Building Trust in an Age of AI with Karen Kranack

Hi, I'm Michael Hartman.

I'm Naomi Liu.

And I'm Mike Rizzo. And this is OpsCast, a podcast for marketing ops pros and rev ops pros created by the MoPros, the number one community for marketing operations professionals.

Tune into each episode as we chat with real professionals to help elevate you and your marketing operations career.

Hello everyone, welcome to another episode of OpsCast brought to you by marketingops.com, powered by the MoPros. I'm your host, Michael Hartman. I'm joined by my co-host, Naomi Liu, today. Naomi.

Hi, it's been a minute, Michael.

Yeah. It has been. Rizzo is, I think the day we're recording, this is day 2 of Inbound for—

I think so, yeah. In San Francisco this year, right?

Yeah, in San Francisco and getting ready for MOPSapalooza, which is now less than 2 months away. So.

Yeah. Well, I see you there.

Fun times. I am still up in the air. I think I said on one of our recent episodes, I need to do it up front, like I am open to doing something crazy and stupid if somebody would pay for me to go. Like, I will dress up in something, whatever. But I think it'd be fun. We could do it. We could do a live episode maybe, right?

That would be great.

Yeah. Yeah. So, all right, well, let's get started. Today we are digging into the intersection of AI, brand strategy, and trust, a topic that continues to evolve rapidly. So our guest today to talk about this is Karen Kranich. She is Director of Applied AI Strategy and Experience at Is it deep, depth, deep, deep, depth? Okay. She's worked with major global brands like PwC and brings a deeply human perspective to AI adoption from internal employee trust to external customer experience. So we're going to explore 6 key considerations for building trust in this new AI-enabled world and what it means for marketing and operations professionals. So, Karen, first of all, welcome. Thanks for joining us today.

Thank you so much for having me, Michael and Naomi.

Well, that's good. Well, so we framed this episode around building trust in the age of AI. And it's interesting, as I have conversations with people, I get really mixed signals about how, like, some people are really like, I'm doing AI do a bunch of my thinking for me, or it's a partner. And others are like, I'm using it, but I don't trust it, right? So why do you think this is such an important and timely topic right now? In the kind of evolution of AI?

I think it's super timely and topical because I think all of us are touched by it on a daily basis. So, for example, like 2 days ago, I was on YouTube, you know, and you always get served ads beforehand, right? And I saw Oprah selling Himalayan pink salt, and I thought, oh, it's Oprah selling Himalayan pink salt. And then I thought, wait, would Oprah sell Himalayan pink salt? And Just in taking that pause, I looked more closely and I could sort of see that the audio track was not totally syncing with her voice track. And I realized that this was AI generated. And so, you know, it's one of those moments where if we aren't really paying attention anymore, we're just lost in what's real and what's reality. And so I think with all the AI slop out there, it was a great example of like AI slop. And so I think, you know, we're being barraged with that now.

Yeah.

And that's the first The first reason why I think, uh, it's important to like sort of learn about these tools and pay attention to them. Um, the second reason why I think it's such a timely and important topic is because, you know, we were recently going on vacation, literally like a week ago, and I used ChatGPT's agent to both find me a gluten-free restaurant in Wellfleet on Cape Cod, uh, and book it for me, book a reservation. And what was so interesting about it, I don't know if either of you used that yet or played around with those. No.

I keep meaning to, but I've, I've been playing with other stuff.

Yeah. I find that a lot of things that I ask ChatGPT to do and the recommendations it gives me from, you know, some of the vacations over the summer that we've taken, half of the places are closed and it doesn't recognize that, or it doesn't exist anymore, or the hours are completely wrong, or they've moved locations and it just doesn't seem to update that. And then if I call it out and be like, hey, this place actually closed, Oh yeah, you're right.

Let me find something else.

Come on.

Right.

Totally. So that's interesting you say that because it did in fact find me some good options, but when it started to actually book it, you can, you can actually see on the screen that it's sort of, you can see what it's doing as it's interacting with like Resy or OpenTable. And it was making tons of errors. Like it was making a lot of mistakes, like it, but it was catching itself for the most part. And like going back and like retyping my name or selecting, you know, it first started out by selecting like one person for the reservation. It should have been 2. Um, so it was really underscoring for me too, like, it didn't make me think badly about Resy or OpenTable, but it did make me question, obviously, you know, LLMs and like their actual abilities and where are we. And then the third thing, and I think makes this so important, is like, I read an article in the Wall Street Journal 2 days ago that said there's recently been a study that proved that the less literate a person is, the more likely they're to be awed and wowed by AI and actually use it. So, it kind of brings to mind that old—

Interesting.

Yeah, that old, um, what was the, uh, Arthur C. Clarke, like, uh, sufficiently advanced technologies are indistinguishable from magic.

Yes.

Yes.

Like, so literally, like, literally people, the more they think it's magical, the more they are to, you know, likely to use it. And that brought up an important marketing question for me, which is, should we be transparent if it reduces marketing efficiency? And I think as human beings, I think one of the ways we can distinguish ourselves from machines is to be honest, right? And actually try to actually provide truthful information for each other. So I think there's like this human element, but I also think the awe kind of passes sort of quickly because like once you start using these things, you start to get increasingly more used to it. It's kind of like television. I mean, the people who first saw television were, you know, you know, blown away. And then now it's sort of like is a box that sits in all our lives and just does its thing whenever we want. Um, so I think technology is very similar in that way. And so I think we absolutely should continue to be transparent because we're just going to continue to be bombarded with— um, brands themselves too are kind of endangered. Like thinking about like the Oprah example, right? Like this is someone who is trusted. Um, brands are trusted. What becomes that? center of truth and how do we support brands in actually promoting themselves as that truthful entity?

Yeah, it's interesting because I think video is quickly becoming harder and harder to detect. But I've started noticing on, I don't remember if it's on Facebook or Instagram, but I get these ones that pop up and it always begins with the same clip of Joe Rogan from his podcast going, have you seen this one? Jamie, go pull this up, right? His co-host. And it's like, Wait a minute, every one of the— I started noticing like every one of these is starting with the exact same clip. I'm like, this is not real, right? There's just like people trying to take advantage of how well-known he is. Um, so it's kind of like that too. I don't even know if that's an AI thing, but it's clearly like— it's e— like it took me, I don't know, 10 to a dozen times of seeing that to kind of like click and go like, wait a minute, I keep seeing the exact same clip. It's weird.

Totally.

So, um, so, um, okay. So all that being said, right, I mean, I'm— I don't know about you, but like the organizations I, I'm kind of interacting with, I keep hearing people like there's actively people trying to adopt AI into their organization, um, and they're rushing, but a lot of them are stumbling. Um, some are more public than others. You know, what are some of the lessons you've seen or learned from all that?

Totally. It's interesting. One of the things I'm really seeing as I work with like large-scale enterprise clients is they have this intense need to keep up with the Joneses, right? And so, AI is top of mind. I mean, literally when I go into meetings, like, you know, one of the first questions I get, you know, it'll be like the entire agency there where we, you know, we do builds from soup to nuts, but like the first questions we get are like, What AI are we including? So, I think every organization, every company feels this intense pressure. But what's interesting is that it still takes time to build those tools, and it's really important to assess whether these organizations actually need these tools. And so, that's part of what my job is, is to help come in and say, you know, is it the appropriate time either for a front-end experience or a back-end experience to improve workflows? So, it's what's interesting about technology and wanting to keep up with the Joneses is that a lot of things appear to be table stakes to clients. So I'm hearing them say things to me like, yes, we must absolutely have conversational search. Yes, we must have next best action, you know, opportunities for when people are looking at content or products on our site. But when I talk to them and try to explain them about technology is that these things really require thinking about like what marketing platforms and what tools and technology are required to actually build those. And it's not a question of just plug and play. Like, you know, like Adobe Experience Manager, for example, like doesn't have a plug and play conversational search at this point in time. So, Sure. Although they all want these things, they're actually not necessarily that many that are really great out there in the marketplace. And although we need to build and work on these things, it's still, it's still a nascent technology. And it, even though, like, you know, all websites have a search on them, migrating over to say conversational search isn't necessarily like the simplest thing to do.

Yeah, it's interesting. I mean, the thing that I've seen in an organization I'm at that's been most useful is implementation of an AI tool that can be added into meetings when it's recorded and it transcribes and it does a really good job of it. I've been truly sort of blown away at how accurate it is. I may get occasional, like, to me, minor things like people's names are misspelled, but that's an easy miss. Humans do that all the time with my names, so I would hold no grudge. But it feels like beyond that, right, I haven't seen— I'm starting to see more and more, but usually it's led by like a handful of people really kind of taking advantage of it and sort of leading the way on what has worked. So, yeah, so at the beginning we talked about sort of like, I forget, was it 6 considerations for building trust in this world? Like, what did you mean by that? Right? Can you talk us through what that is?

Sure. So really the first one is kind of what we've led with, which is leading with transparency and honesty to being really clear and forthcoming. I mean, one of the things we've seen in the marketplace, I mean, everyone, I think the biggest story of the year was probably like Duolingo, um, sort of stating publicly, or actually it was an internal memo that got leaked, but stating that, you know, they were going to start replacing their translators with AI. And there was a public outcry about this, especially around like translation is actually something artificial intelligence can do really well for the most part. But, um, there's the emotional human piece of this and the whole purpose of Duolingo is obviously, um, to, you know, provide, uh, language training for people. So it really like, as once this leaked, like people were very angry and upset. And so there really was a lack of transparency and honesty about what, um, Duolingo was trying to do. on the background. And I think that's what really sort of hampered them in that case. So, thinking about things like that, transparency and honesty, very important. Secondly, clearly articulating and demonstrating computer— sorry, clearly articulating and demonstrating consumer value and benefits. You know, again, like going beyond AI strategy and use and really thinking about like, why would a person care about this? Why would your customer want this? And so, You know, thinking about like a problem example was Air Canada. Do you remember this? I think this was about a little over a year ago. But anyway, Air Canada had a customer who was interacting with their chatbot, and this customer was trying to get a bereavement fare. And the chatbot told them that they could in fact retroactively like get a bereavement fare. They thought it was all good. They bought the full price ticket, and it turned out that when they tried to do this later, they could not. And Air Canada said, said, oh, it's not our fault that the chatbot told you that, even though it's our chatbot. And so, again, like poor customer value, right? Like obviously you can't tell your customer one thing and then undermine it in another way.

So I do have like a follow-up to that, you know, as, as we were talking about adoption Within an organization, right? How do you balance, you know, as someone in marketing ops, right? How do you balance like the speed to adopt, right? So, like, for myself and the folks on my team, like, there's all these tools and like features of our existing toolset that we wanna implement. How do you balance that speed to adopt with building sustainable trust internally, right? Because it's just exploded in the last, you know, 18 months, right? But trust, I feel, You know, as we've been talking about this year, hasn't really kept pace, right? There's like concerns about misinformation, data privacy, job displacement. Like you're, you know, there's all these LinkedIn AI influencers talking about how they, you know, vibe coded a platform in a week and it replaced like, you know, their entire team, you know? So it's just like, how do we, how do you balance that internally? And have you seen examples of where A lack of trust potentially could have, you know, derailed adoption somehow.

Totally. I, I have a couple of thoughts on that. The first is that, you know, I'm talking to a lot of different corporations and what I've heard from most people is that kind of like what you're saying is that a lot of the people working in companies are trying out these tools on their own. So there's that piece of it, but that obviously undermines like confidentiality potentially. And like if people are using this, you know, sort of outside of school, so to speak. Um, so that, that is a problem. And what I'm hearing is that they're not, a lot of people, employees are actually trying these tools and taking them back to their employers and saying like, hey, we should, you know, investigate whether we can adopt this. And they're getting a lot of pushback at a lot of times. And it's so, it really seems to be kind of like the cultural, like top-down where a company or corporation organization really needs their CEO at the top to kind of be open to investigating and trying these things. Within our own organization, we're super into experimentation, but we're also actually codifying and collectively creating, actually our own group, the group I'm in, is actually doing a large portion of our work on internal enablement and how do we actually, you know, pull in these tools to improve project management? How do we improve our creative tools that we're using, our marketing operations tools, our media buying tools? So, you know, it really is about like, I think the organization's maturity cycle. And for those people who, for the most part, organizations know that AI is important and most organizations are shifting. Now, in highly regulated organizations like, you know, banks or healthcare, I'm seeing, I'm still seeing actually enthusiasm for adoption, but the rollouts are kind of slow, you know, where it might be like, oh, sorry, all you get is Microsoft Copilot because it's already embedded in Office, the Office suite, which we already have. I'm not really seeing resistance, which is super interesting. And the people that I've worked with, even the people who don't really know or are a little scared of AI, are still open to learning about it. So, I think there is overall an openness and a willingness to learn these tools and try them. And it's really about the maturity of the organization and the willingness to adopt.

That skepticism about it, right? Totally.

Yeah.

Yeah. Yeah.

That makes sense.

Which is well-founded, you know, it's like, it's very, I mean, obviously we know that like AI tools are often used to like train, you know, like you, it's important. Like I know the enterprise models will actually say, you know, like of Gemini or ChatGPT, like this, this model is not training on your corporate data. Um, so it's very important for, for people to, for corporations to get like, you know, enterprise licenses.

Sure. Yeah. Okay, so you've covered, I think, 2 of the 6, did I? If I— if my math is right. Uh, yeah, totally. What, what, yeah, what are the other, what are the other, other 4?

Okay, so, uh, number 3 is proactively addressing and managing privacy and data concerns and, you know, explicitly communicating to people like, how is your data being used? The big sort of, I think, stinker in the marketplace has really been kind of Meta because, you know, their algorithms are a black box. Nobody really knows what's happening with your Instagram, Facebook, WhatsApp accounts, whatever their claims are, we don't really know. And I think that that has been really underscoring a lack of trust versus say like Apple, which interestingly is falling behind in the AI race because of their emphasis on privacy. So like Apple tools mainly are on device. So like when you're using AI tools on your Apple phone, it's just kind of whatever's personal, personally identifiable information about you is just staying on your phone. Whereas, you know, when you're using Meta, you're obviously helping feed the beast, as it were, and like helping train all their data no matter what you do on it. So definitely proactively addressing privacy is important. Fourthly, committing to ethical AI and actively mitigating algorithmic bias. This is one that, you know, it's, it's, it's been scientifically proven and well received that, you know, there is bias inherent to these systems. For example, I mean, Amazon kind of got themselves into a pickle a couple years ago. They had an AI-based tool, a hiring tool that would look at resumes. And because historically Amazon had gotten more resumes from men than women, it was deprioritizing women. And if the resume said anything about, you know, women, like they had been involved in like women's sports or something, it would actually kick out the resume and not—

Ooh.

Um, so obviously they recognized this bias was happening and was like a problem. Um, you know, so obviously, and interestingly, I just read about, uh, NVIDIA is using synthetic datasets to reduce bias. So, um, because they know that there is a lot of inherent bias on the way that LLMs were trained, um, they're actually trying to come up with, um, synthetic data that is more generic that will actually sort of create a more level playing field.

Very interesting.

Fifth, emphasizing human oversight, empathy, and accountability, you know, really just reassuring users that, and consumers, that human oversight remains, that like, you know, things aren't just going into a black box. And, you know, someone who's doing this well in the marketplace, especially from our ops, is Salesforce. They're really, with Agentforce, they're really positioning their, their AI-based tools as your partner and sort of emphasizing the human element in a way that I think is really, um, kind of refreshing and that they, they want you to be freed up to do more of your tasks. Um, and really emphasizing like that empathy piece, which is pretty, pretty profound.

Very interesting. Yeah. I thought it was, I saw just today a headline on something or, um, a Salesforce thing where they described themselves as an AI AI-based CRM platform or something like that. I was like, oh, they're just going to throw AI into the name of what the description of what they are, right? Totally.

I mean, I think all of these major platforms are a little out over their skis in terms of what the AI can actually like do to the degree that, you know, but sure. But yeah, and the last thing finally is just really educating and empowering consumers, you know, to like demystify AI. And, you know, part of that is, again, giving explanations for when and where it's going to be used. You know, there was a lot of controversy over, you know, Midjourney and other, you know, image generation tools, just sort of like siphoning in artists from all over the world without any permission, right? And so, you know, that's really undermining to people. And so, you know, looking for ways to actually say like, hey, we are using this data to train, we're not Or we're not, you know, or, you know, like Unilever had used AI called Unibot, has a robust, like, training blueprint that, like, sort of actually provides some education for people. So, you know, letting people know, like, this is why we're using this. It's not just some random feature either that we're just bolting on also to just keep up with the Joneses, because that, that's not helpful either.

Yeah. So, I'm just curious, the thought, because my experience with AI being used inside of organizations has been mostly for As one CEO I know called it, time dividend, right? So, sort of leveraging that as a tool to— so, I think it felt like a lot of what you described in those 6 things was mostly in the scenario where the tools are used—

By consumers.

Either consumer-facing or close to consumer-facing, as opposed to, like, do you see, like, are there— do those same principles apply when you think about it from the standpoint of employees are the customer in this case?

Totally. And I think employees have to like really have a great reason for using these tools. We're doing a lot of, and I'm personally doing a lot of implementations of looking at internal workflows, like particularly around CRM and helping optimize those workflows for, for employees. And one of the things we're finding, again, like there was one like major technology company we worked with, you know, there were like basically like 100 steps to kind of basically like, you know, pushing these emails out through this marketing funnel, right? Sure.

Yep.

And one of the things we found is that, you know, there might be, there were literally like 30 ways that it can be enhanced with AI, but the other 70 were really about improving human communication. And so, that's one of the ways that, that we, that we actually support our organizations to help build that trust because we, we come in and we say, you know, this is something that like you can be improving on, but it's also about looking at your actual workflow and your human interactions. And people can sort of then see that like, okay, you might be able to use this LLM to sort of, you know, improve your, um, generative engine optimization so that, you know, you get better, like, sort of SEO traction on LLMs themselves, you know, and like you can use this little tool to do that, but it's not going to like take away your larger job, for example.

Right. Okay. Yeah, it makes sense. Okay. Yeah. So this is, um, it's interesting because I think like it'll be like these 2 sort of scenarios, like customer-facing or client-facing or whatever you consumer-facing versus internal, how that plays out, because it feels like a lot of the things are similar with slightly like the nuances is where you get into differences, I think. So, so—

I'm sorry, I'm sure you saw the— oh, do you mind if I—

No, no, go ahead, go ahead, go ahead.

Oh yeah, I saw— I'm sure you saw the MIT study that came out last week. Oh yeah, like 95% of AI, you know, projects fail. But what they did say that does succeed, to your point, is that internal operations optimization is actually the place to focus more than the customer-facing side. So, I mean, that's something that I am seeing in the work that I'm doing, and that actually, like, sort of, you know, creating those relationships with the internal teams and then helping them level up has definitely probably seen more success than, say, like, some CPG saying, oh, we just need to stick a chatbot on our website.

True. It feels like MIT's got people out there researching why AI is a bad thing, because I think there was a study I thought came out of there, maybe it was Harvard, so it's up there in Boston, right? That was like, I think they had done something where they had students come in and either not use LLM at all, use it as a partner after writing something, and then those that just used it to to basically generate, and they basically said it makes you dumber. I mean, that would— at least that was the shorthand that people used promoting it. And I was like, well, so first off, like, I don't know that it's a general— that is a specific case that's probably not generalizable, but also kind of matches what I would expect to some degree. Like, I don't know.

Yeah.

Like, when you use it as a tool, if all you did— like, I have found with my use of ChatGPT, this is what I mostly use— like, if I don't actually spend time to Do a good job of providing it input context and reviewing what it generates, it's not great.

You know, Michael, I used to be amazing at math and now I'm like, okay, what's 10% of my phone?

Right?

It's true. You don't use it, right? And, you know, we have all these other tools, but at the same time, it's like, why do you need to do back of the napkin math anymore? Longhand math. Right.

Yeah.

So you don't use it, you lose it, right? So it's true.

We just have to hope we don't lose the technology.

Well, I think it's interesting. So I get fascinated, this total, total, total rabbit hole that we could go down to. But I get hooked into these videos of people doing like survival type stuff, right? Or whatever. And I think that's like part of that is this like desire to not forget like how to do things by hand or without all the technology and tools that we have, right? I, so I think, I don't know what it is, but like maybe there's some, I need some sort of psychologist to evaluate me, but like I just, it feels like there's, I do fear that, right? Is that I do think like if all we did was like just let AI stuff do all this heavy thinking, then we will lose some of that. But I don't think that's the way we should use it, to your point, right? I mean, it's like the calculators came around, you know, I mean, abacus was around before that, all these things, right? They're all disruptive technologies. Tractors came around to replace cows pulling plows behind them.

But I think you're making a good point about not losing critical thinking skills, right? I think that's the key. Um, you know, whether you're talking about education or, you know, the workplace, it's like we still need to be thinking big picture. We still need to be able to pull things together. They're recently saying, I just read another article too, saying like, you know, like taste, like human taste is actually something that like, you know, AI can't replicate. Yeah. So, you know, how can we?

Well, I was talking to somebody who's in a, like a journalism role and I, we were talking about something else, but at the end I was like, what do you guys like think about it? Because of the people I know who are Like writers, actual, like that's their, they don't, they're not really fans of it. But she said something that was interesting in that because they are in the business of breaking news, right? New stuff like AI can't really replace them because AI is dependent on other stuff to consume from right now. So right now, yeah, right now. Right. Right now.

Yeah.

Okay, so let's keep going. So we were talking about the customers versus internal. So external customers, let's stay focused there. Like what, I mean, we talked, you touched on a few missteps, but what, like, are there major themes of missteps that companies are having when they're introducing AI into their kind of customer experience or customer-facing interactions?

I think the biggest missteps, honestly, and, you know, first of all, it's expensive to implement these things. And I think it's like that sort of like willy-nilly, not putting a lot of thought into it. Like, you know, we do hear from some clients like, we need AI, you know, and it's sort of like, to do what? You know, and that's part of what trying to figure out, like, how do you really serve that customer value? And so, I think the missteps are really just like putting the time, effort, and money into something that's like not really going to necessarily go anywhere. Which is why I often recommend just doing like a proof of concept or like a pilot study, um, rather than like sinking, you know, like a ton of their money into just overtly doing things that, that may not matter. The other piece of this too is that, again, it takes a lot of effort to, like, you know, you can't, people think that like, sure, we have a search engine now, we can just, you know, plug and play conversational search. But the truth is you need to actually, you know, set up That LLM, train that LLM. It needs to observe the interactive behavior over time of customers. And so I think there is this misunderstanding in the minds of some clients where they think that like, well, search is table stakes, therefore conversational search should be table stakes.

Mm-hmm.

And I'm seeing that quite a lot where it, it's sort of like this sort of misunderstanding of just the technology itself, where we are with the abilities of that technology, and then the time it takes to still make really great technology, which kind of brings me back to what you were saying about, like, you know, you have to actually put a lot of thought into how you're engaging with AI. You can't just say, like, write me a poem and expect to get, you know, as good of a poem as if you, like, had something, you know, that you were giving it as a framework to go to move from. So.

I'm laughing because my first sort of playing around with ChatGPT was to get it to write rap lyrics first.

Nice.

That was a long time ago. It feels like forever ago. So I'm curious, is one of the things, because I know, again, we talked about this before we started going live here, but there's an organization I'm working with where AI is being really embedded into the way that the company works, and it's from the top down, but there's dedicated resources for it too, to try to drive that adoption. So are you finding that Any difference? Are you seeing that much with your clients or people you're talking to? And do you see it making a difference in the success rate?

I see a huge difference. And it's interesting because it kind of depends on— so especially like the highly regulated companies, they, if they really sort of, you know, put forward a dedicated like staff to work on that, I'm seeing them look at very successful use cases of You know, like say, this is not a client, but I've read about like, you know, like a mortgage lender creating a chatbot that can actually, you know, access secure data without actually pulling that secure data like into the chatbot responses, right? Or whatever. So, or it can, it can sort of relay them, but it doesn't train or stay like with the chatbot. So, things like that are very successful.

Mm-hmm.

But, you know, there are cases too where if you don't get, I think it really is about the organizational maturity and organizational buy-in and, you know, are they willing to invest in the time and the people to solve those internal problems. If they're not doing that, if they think it's a bandaid, if they think like, oh, we just need to throw something up on our website, that's not going to work.

Yeah, that makes sense. So I want to go back to the internal folks a little bit because I do think there's, you got a real mix of people. So, when you're rolling things out internally, like, what should be— what's your guidance for people in terms of getting the best chance of success when they roll out and get adoption of AI systems internally for, like I said, the time benefit there? So, and if you've got any examples, that'd be great.

Yeah, success really comes with Again, looking at those workflows, like that sort of marketing CRM workflow I talked about, like, you know, success there was again, like talking to the people. It's funny, you know, I've taught qualitative research methods and one of— it's still one of the most important parts of this job. You know, actually there was— I was listening to another one of your podcasts and it was, you know, about like doing this work is almost kind of like psychology and being like a kind of therapist in a way, you know, like—

Yes.

You've really got to bring in the human element and really talk to people. And that's how I found like the most success is that talking to clients, saying, you know, sitting down with them and literally mapping out their workflows with them. I do a tremendous amount of like that mapping out and they can see what that looks like. They give input to that and they understand like, oh, this is, this is very clear. And I think it's about providing that clarity is what provides the most success for clients. I think no one, you can't just say to a client or anyone really, like, go adopt Adobe Experience Manager for the cloud and then figure it out. Or like, you know, like, or I actually did a keynote talk at Adobe about Adobe Experience Manager or Gen Studio for performance marketing. And, you know, it was hands-on. It was a hands-on learning session. And I think that's the really important kind of work that we need to be doing is that kind of educational piece. And I think that's where, you know, I personally provide the most value because it is about like coming in and working with people, being very one-on-one collaborative, and then you get the organizational buy-in. And then those folks that you're working with can then, you know, obviously cascade that out to the other employees that need to learn.

Yeah, I know Naomi, that probably, she was probably reacting like, to the idea that you just roll out technology for technology's sake. Because I know, do you still measure like adoption on your technologies as part of that? For sure.

Yeah.

Yeah.

Because, well, especially if like there's tools that are like user-based licensing, right? It, you know, you, you kind of need, you kind of need to be able to make sure that that doesn't get overlooked. And I think just, if, if I, I feel like we could have like an entire episode just based on like internal trust, employees and adoption, technology adoption, how do we change, like change management, all of that stuff.

Yeah.

And, you know, I think the thing that I have been finding that I'm coming up against right now internally is there. I think it would be interesting if there was some kind of playbook for building confidence with your colleagues who feel unsure about AI or feel threatened by it, or if there's people who are digital native employees versus those who maybe technology is not the strong suit. And It's kind of this like threading the needle of like, how can, you know, then, and I ask myself and the team about this is like, how can we as marketing ops leaders, how do we champion adoption internally without making it feel like it's some kind of like forced compliance, you know?

Right.

Like, because I, I just never think that that's gonna turn out in the way that you want it to. Right. I almost feel like every time I use the word AI now internally that I just never really know the reaction I'm gonna get. And so something that I've been thinking about as well is putting together an AI bootcamp, um, almost like an internal training session for, um, the people that I work with on a very daily basis to make sure that we're all on the same page, right? Because it's very easily, you know, it's something that I consume and learn about quite extensively, but that knowledge gap needs to be bridged.

Yeah.

Right.

Yeah, absolutely.

And you can maybe even do that with something fun to start, you know, it could be like, you know, learn, learn to do like a mini Vibe Code session with Claude where, I mean, I've done this with people where like, it's literally like, pick something you like, like, or just, you know, like, uh, I had it make a crossword puzzle, interactive crossword puzzle based on the TV show Severance, you know, or something like, um, just getting people to kind of get familiar with something enjoyable.

Totally. Yeah.

Yeah. I mean, I think it would be good to do. I know as an individual, I've tried to do things like, I've gotten like pretty, I feel like I've gotten, I wouldn't say I'm an expert because I don't know what that even means for like using ChatGPT, but I feel like I've gotten pretty darn good at it as an individual. What I've run into when I try to do automation stuff, I haven't tried the ChatGPT agent thing, but it's on my list. I've tried other things and it's just like, Like I did one the other day where I wanted to, I had an audio track for something and I wanted to transcribe it. I was like, oh, there's got to be a free, low-cost option for that. And there is, but it requires coding with Python to use an AI or API. I'm like, how am I going to do that? So I tried something else with a kind of drag-and-drop solution and it worked great, but it I'm on a free level, went right through all my AI credits, like, in no time. And I was like, there's like, that's the part for me. Like, I want to try doing that stuff. And if I had like a company that was willing to support that and spend a little bit of money to try it, I think I can think of lots of things that would be super valuable. Right. And that's, that's the part I like, I'm struggling with. Like, I want to learn, I want to try to do more, try not to spend a lot of money on it as an individual. So, uh, if you've had any suggestions for where people can try to do some of that, or how to convince their organizations to do it, I'm all ears.

Can you use ChatGPT? Sorry, I was—

no, you please.

I was saying, like, can you use ChatGPT to, like, learn about ChatGPT?

You can.

Right?

Yeah, I've done that.

Yep, totally. And I think one of the things that we've done is tried to collect, like, um, we've been, as we have in our internal enablement team, has collected, um, data on, like, asking people, like, just surveys internally, like, what are you using? And then going out and trying to get some licenses for people to try it, maybe like the most That's, yeah, I think that'd be great.

I think that's really what I'm interested in is like, how can we do it? And there's just some good things that can happen from that. So you kind of touched on something I want to get into. And I think we talked about this when we first, when we met before, Karen, is like, what are the implications for all this for the soon-to-be people in the workforce? And I think I shared with you, right? I've got, and it's kind of an updated one, I've got 2 college students and a high school student. College students are basically told don't use Wikipedia. They were all told don't use Wikipedia for research, which I always thought was dumb. Now they're being told like you can't use LLMs, uh, you know, you can't cheat. Um, which is interesting. My son who's in high school, we just got through his curriculum and they're actually specifically calling out like, here's— we're going to tell you when it— when and how you're allowed to use these tools as part of work, which I think is a— to me, that's a much more reasonable one. What I worry about for my 2 who are in college is that when they come out, because they were told not to use it, they won't have that experience, but all these companies are going to be expecting it.

Interesting. Are they— what I tend to see at the university level and having taught at the university level is that it tends to be instructor-based, like whether they can use it or what degree. So like what I've basically seen is that To your point, it can either be banned entirely. There can be a hybrid approach, which is what I personally have used where like, you know, so I teach qualitative research methods, right? And one of the great things about qualitative research is that you can't fake it. Like you actually have to go out and talk to people. So, but I do allow my students to use LLMs to like analyze that data, or at least, you know, get some ideas or thematic concepts like from that data using LLMs. But They have to tell me and prove to me like how they did it. Like, so you can't just like go off and do it and, you know, turn in a paper and say, that's that. Um, and then of course there's those who have unrestricted. I think for, for, for anyone these days, it's funny, I Googled, I was thinking about this question and I Googled yesterday like, Gemini, what, what do you think like should, you know, what is the best path forward for students? And it said a hybrid approach is the only way forward or something.

You know what I'm saying?

I'm like, of course the AI is like, you must use AI. But I think, um, I think that the reality is it is a hybrid approach. And I think that, I think the most important thing is teaching people critical thinking skills. And that has to come both, I think, from like parents and also from the schools themselves. And that's like even thinking about, and this is getting esoteric, but like, you know, logical fallacies, like, you know, what is a red herring argument? Like what, what is a straw man?

You know?

Yeah. Like, what are these? And sort of starting to think about like, when you're using these tools, are you getting back content that actually makes sense? And then in training, you know, for yourself, always saying like, your students saying like, I have to start with like a draft of my paper. I cannot, I can't ask ChatGPT to start the draft of my paper. It might be able to help me like brainstorm ideas, but it can't be the main tool that I'm using. And I think it's unfortunately something that, that kind of has to be, investigated and taught. It's very, I think human nature like pulls us towards a bit of laziness sometimes, you know, like we're kind of like, oh, I'll just Google that.

Sure.

Um, but I think, I think young people today do have to learn to use the tools. And I think, you know, maybe using them in very specific contexts, like teach me how to code, you know, something that will analyze this data or, you know, like take my data and turn it into, um, graphs, you know, graphs and charts that I can then put into my, presentations. I think it's about looking at things in manageable chunks and also still getting— I'm hearing about too instructors doing oral tests or it's like you can turn in a paper, but then you also have to do an oral argument around your paper, for example. So I think it's about, it is probably a hybrid approach.

Yeah.

But there's no easy solution. I think actually going way back to my Oprah example, it's almost like when you're working with any of these tools, it's like Can you take a pause and bring more consciousness to what you're doing and how you're interacting with these tools and really put some thought and care into it? Which I'm not sure how the exact way to teach that is, but other than to model it, right?

I think it's interesting that you've at least twice in this conversation brought up the term critical thinking. And it's— I have a big criticism of the educational system, at least in the US, Naomi, I'm not going to say anything about Canada. But I think our educational system has kind of lost that as a core skill, right? It's more about compliance and testing and things like that. But it's interesting, today I was in the car listening to a totally different podcast, totally different thing, and the guest was someone, and I forget the name of it, she has started a school that is using AI. agents, like truly the way it sounds, like I haven't seen it, right? But students get only 2 hours a day with using some AI tools to do their core curriculum stuff, and it meets them where they are regardless of where they are in the grade. And then they have— the teachers are not like teachers where they're up there lecturing. The teachers are like coaches, more like coaches. And I think it was interesting because she also brought up The idea, like critical thinking, she like repeated that over and over when she was talking about the way they do it. It's like they still want, like they're focused on teaching that and like discerning it. And they, they're not just like opening up ChatGPT for these kids and using it. They're, it sounded like they're using some like, uh, bespoke solutions that kind of picks and chooses the engines behind what they're doing for each student. But I thought it was really interesting. This is like the first. truly like time I've heard about AI being embedded into education where I thought like, that's, that's really out there. That's also a really expensive school, so it's not like scalable. That's part of the challenge.

I mean, it sounds amazing. I have not seen that like in action at like the university level, but I'm sure that there are some professors out there who are doing something similar to that. But I mean, I think that's where we want to get to. And I think that that's, yeah, it's an amazing app. I'll have to look for that.

I mean, you're, you've, you've got one is all the way like way early. Are you seeing anything like that? Yeah.

I mean, she just started, uh, junior kindergarten today. Yesterday was her first day.

Nice.

You're right. I know. Uh, no, not at the age of 4.

So, but, but so this, this woman talked about, they have kindergarten kids who are doing that, 5-year-olds.

So, you know, I'm, I'm just, I'm trying to, you know, as it's interesting as someone who works in tech and uses devices like a little bit too much, I'm very much okay with not giving her a single device for as long as possible.

We, we did the same thing with our youngest, you know.

As long as I possibly can anyways.

Yeah.

Um, yeah, there's, you know, especially after— sorry.

No, I think it's smart. Sorry, I didn't interrupt.

Yeah, no, especially after reading that, that book, um, The Anxious Generation. So.

I just started listening to it. I'm not reading it.

Yeah, it's, it's actually a bit, you know, you, you, I, I'd love to talk to you about it after you're done.

Yeah.

Yeah.

Well, and well, for me, I grew, I grew up at the point where a lot of that stuff he says started, right?

Yeah.

The cusp of that. Yep.

Well, one of the things that to this point is, you know, I'm sure you've read about the, the unfortunate, the suicide that, that occurred, um, the ChatGPT, OpenAI being sued. And I think that this is one of the things that's really important to also help young people and younger people understand is that it's not a person, you know, and it feels like a person. It acts like a person, sort of. It seems like a person, but it is not. And, you know, really helping because obviously I saw that OpenAI is now going to institute some parental controls or whatever. But what we know about LLMs is that they, they really can be jailbroken really easily, like depending on how you talk to them, like they don't always adhere to their guardrails. It's really kind of like not a totally safe space. And I think that's something that we also need to emphasize.

Well, and I think the more they act kind of like human— there's a post today on LinkedIn that I saw where someone said there's research. I have not gone to validate it, so I won't vouch for it. But it basically said that some people did some experimentation where they put like LLM-based engines as virtual people in a system that was like a social media platform that's intended to gravitate people towards controversial stuff. And it actually led to this really divisive thing with a bunch of chatbots, right? So it's kind of like you've got that plus these engines that are designed to keep you engaged for as long as possible. And so, yeah, it's gonna be an interesting time over the next few years. I was gonna say the next 5 to 10 years, but I suspect that the window of time is actually shorter than that.

Yeah, am I allowed to say, like, I actually think no one should be on social media at this point, but—

Yeah, totally. I think you can. Like, I think that's okay. I would love to There was a point in time where I deleted stuff off my phone and then I slowly added it back on and I kind of on a daily basis, I'm like, what am I doing?

Yeah.

So yeah, because the danger is that we've talked to our kids like, don't let it control you, you control it, right? That's the thing we need to do. Karen, lots of fun. I don't think even think we scratched the surface of what we could have covered with you. So thank you so much for joining us.

Thank you.

If folks want to learn more about what you're up to and what you're doing, what's the best way for them to do that?

They can catch me on LinkedIn. So just Karen Kranach, K-R-A-N-A-C-H.

Yeah, got it. Okay, good. Perfect. All right. Well, thank you again, Karen, Naomi. Always great to have you. So thank you for joining. Thanks to our listeners and audience out there. We appreciate your support. If you have ideas for topics or guests or want to be a guest, you can always reach out to Naomi, Mike, or me, and we'd be happy to talk to you about that. Till next time, bye everybody.

Bye everyone.

Thank you.