#319 Building & Managing Human+Agent Hybrid Teams with Karen Ng, Head of Product at HubSpot

Generative AI is transforming industries at an unprecedented pace. But as AI changes how you or your team work, one thing is clear. Your skills also need to evolve. At datacamp, we offer everything you or your team need to adapt and thrive with AI. Whether it's business users looking to get the Most out of ChatGPT and Copilot, or developers and data scientists looking to fine tune models. You can learn the entire AI skills spectrum on Datacamp. Power your AI transformation today. Start learning@datacamp.com I do think a lot of the ways that we work together as a team is going to be really important in our agent world. We believe a lot in hybrid teams and I believe that's the way that things are going to grow. And hybrid teams are both humans supercharged with AIs and agents that need to work together with humans and other agents. Data is probably the currency that AI is going to run on and why it's so important that people start thinking or are thinking about it. I always believe that there will be in a world where kind of humans lead and AI accelerates and kind of the magic of humans and how creative and breakthroughs like that is still going to be there.

Welcome to Data Framed. This is Richie. For all the hype that AI agents get, every expert I've spoken to agrees that humans are still rather useful at work. That means the future is hybrid teams, humans and agents working together. So we have a human in the loop. But what does the loop look like? I want to know how to divide work between humans and agents, how to incorporate agents into my processes, and what success looks like. Our guest is Karen ng, the executive VP of product at HubSpot. Karen's building a modern customer relationship management platform and her big focus is HubSpot's Breeze AI agents that assist with sales, marketing and customer support. So naturally, she spent a lot of time thinking about how to make agents that people will actually use. Fun fact. Karen was an early project manager for Microsoft's C programming language, which means 20 years ago, younger me was using the tools she was building. Back to the present. Let's find out how to welcome our new agent colleagues into the team. Hi, Karen, welcome to the show.

Richie, it's so great to be here. Thanks for having me.

Yeah, I'm looking forward to this conversation. So I guess to begin with, I always want to hear about cool uses of agents. So what's the coolest AI agent you've seen across sales, marketing and support?

Oh, gosh. You know, it's interesting to see how agents have progressed and do more work for us across those. I feel like the most interesting one I've seen in all spaces have been kind of like a customer agent that resolves support tickets. It's the agents that give value immediately and fast and things like customer agent. Being able to expand that across the whole customer journey from like catching leads to resolving support tickets to understanding renewals is a really interesting like way to think about how agents are doing work for us now.

Okay.

Yeah.

I think like the customer support thing is perhaps where everyone starts is such an obvious idea is like answering the same questions that your customers have over and over again. Kind of boring for humans. If agents can do it faster, then that's going to be an easy win for them. We've got chatbots, we've got agents, we've got copilots. There's all these sort of different formats. Do you want to talk me through when you want each of them?

Yeah. Well, chatbots are still kind of in the world we are coming from. So when you put chatbots on your site, a lot of it is still very deterministic. So chatbots can answer questions that it knows about. It can address a whole bunch of ones. I almost see the chatbots that are now AI powered as like being able to resolve those. Those questions. They're smarter, they're sometimes non deterministic. For copilots, I think of those as assistants to humans. So you use a copilot when you're a human and you kind of want to augment or supercharge your work. Like I'm going to be the one kicking off questions. I want to be the one that kind of asks for summaries and insights. And so that copilot is really a human assistant. And then in agents, that's when you start getting to autonomous work. Today, you know, I believe deeply in like agents even still will have humans in the loop. Or you'll have these hybrid teams that are humans that are super powered by copilots and then agents that help you do work. But the agents get into autonomous work. So I mentioned customer agent. That was the first thing everyone started with. We're getting into a couple of those new agents that I'm sure we'll talk about. But one of those agents is a data agent. But now you start getting into things like an operations agent that works 247 and it's making your data clean. It's making sure you have the most accurate data and it goes out and finds external data that makes your, you know, CRM data even more powerful.

Okay, so tons of different use cases. And I like the idea that we've had chatbots like forever, well, at least a decade now. And so, yeah, this I've been fairly deterministic. Now you've got a bit more flexibility and then there's kind of, I guess a progression right through copilots where you've got human and AI working together through to agents, which are at least, at least in theory, mostly autonomous. You mentioned a couple of use cases and I know with your breeze platform you've got tons of different breeze agents for different commercial use cases. So you mentioned like data agents, you mentioned the customer support agency. Do you want to walk me through like how you think about categorizing these things? Like, what are the different use cases there?

We deeply think about, you know, if you have, if you're forming your go to market team, you know, you're running your company, what are the kinds of roles and jobs you'd want to do and what would you want to hire for? And so that kind of gives us the basis of the really key agents that will become essential in every go to market team. We think everyone will have a customer service, you know, kind of a customer agent because you'll have support tickets, you'll have resolutions, they'll very likely need a data agent. So data cleanliness, your AI is really only as good as the data that powers it. So, and then I'll say that data, for instance, since it's a data podcast, it matters so much around context. So, and having complete context. And sometimes that complete context comes from the external world. So for instance, there's a set of companies that I, or customers I care a lot about and they get a brand new funding round. I want to know about that instantly. And so what the data agent does is help you create those custom properties that you might be interested in, like funding rounds or, or major executive team leadership changes. And it helps you build that context layer. You know, another kind of agent that's coming still is like a closing agent. So a lot of times when you look at what happens in a deal life cycle, it might be that deals get stalled. 86% of deals actually stall at a certain phase. And the customer wants to buy, you want to sell, and what's happening that makes it so slow. So the closing agent helps you get from kind of cash to close all the way through by making sure it's really easy to create quotes, custom line items, et cetera. But like all the way we think about agents is these are the essential agents you'll need for your team. And then we think of it also as like, we need to create a platform. We are launching a Breeze Marketplace and Breeze Studio that help you customize and train these agents and also craft your own if you wanted to.

Okay, that's cool. I mean certainly like the decoration is very like important to remind people as everyone gets excited about AI. But actually the AI is only any good if you've got good data to go in there. So having a bit of assistance to improve the data quality, that's going to be incredibly important I think for the success of all your other agents as well. And the closing agent, I mean I feel like every quarter like the entire sales team in basically every company is like sweating because like, okay, we've got like a week to close all these deals, we've got to hit our targets and having a bit of assistance there that will maybe smooth things out. I mean I don't entirely believe it because like it happens at every company. Like they always like the sales team is like trying to push to hit a deadline at the last minute. So maybe technology will finally help.

Yeah, probably every, at the end of every quarter seems very stressful.

Yeah, yeah, for sure. I know because you got all these different use cases. One thing that seems to be very difficult with AI agents is determine like how hard it's going to be to make one of these things. And since you've built like a few of them, do you have a sense of like what, what makes an easy AI agent? Like what the difficult use cases are, like, what sort of proving stubborn to build? Yeah. How do you judge difficulty?

Yeah, I mean part of the reason for creating almost an agent platform is to make it a lot easier. Because the way us as humans would think about creating an agent is it's almost like, like it shouldn't require a lot of code. Right. Nowadays. So the way we think about kind of building agents is give it a set of instructions. But the instructions are like the same way that you would train a team member. You can give that a natural language which is I want you to be a customer health agent and I want you to flag when things are, you know, give it constraints. So I want to know whenever anything drops below like 50% resolution rate or if you see some of these risks and I'll define the risks for you. So that's how easy it is now to kind of build an agent once you have the platform to do so. The things that become really hard with agents is multi agent orchestration. So what happens when agents need to talk to other agents. So I mentioned customer service agent. That one resolves support tickets. But as support tickets get resolved, it's still escalating to humans. For the most complex questions, we have another agent called knowledge base agent that watches which escalations are happening and then helps recommend knowledge bases. So that interaction of two agents interacting together, we want them to be like human people. And that multi agent orchestration is very tricky. The other thing that's tricky is when you have multiple tools in place. So you know, we have agents that work across the HubSpot CRM but the CRM works across like the closing agent needs to generate a quote from my payments or billing system and needs to send that quote back to my invoicing system. When you have multiple tools in play, you know, we're seeing agent platforms still develop, but the ability to choose and rationalize which tool to use and which one to pick right now still has like limitations for the number that you have. You know, with all AI things are going to get better and better. But those are a couple of things that become really tricky. Multi agent orchestration, how to think about which tools to use, what is the reasoning? As you start choosing between different things, those get complicated. Those are the hard problems. But you know, being able to create and express an agent we want to make easier and easier.

I like the idea that customizing agents is relatively simple. So if you start with the standard agent tweaking is easier than building something from scratch. So it sounds very useful because I guess you just want something that's a little bit different. Maybe. We'll come back to that in a moment. The, the multi agent orchestration. I mean I know when you have to work with colleagues, it's like you work with someone in your team, it's fine. You work with someone in a different team. It's like they have stupid ideas and they want to do different things, they have different goals to you. You end up these into office fight. Is that what happens with agents? We're going to see office politics for agents as well.

You know, it's funny we might see office politics, but you know what will be interesting is to see how polite people are to their agents. If they're using the same kind of politeness and niceties or if you're just going to tell your agent that was a bad idea, please try again. So I think we'll have some of that as they work together. The interesting part about inter office dynamics will be which agent takes weights. So what happens when they do conflict or what Happens like, how would you resolve something if you have two pretty equally weighted priorities? How do you describe constraints for your prioritization structure? And so I do think a lot of the ways that we work together as a team is going to be really important in our agent world. We believe a lot in hybrid teams, and I believe that's the way that things are going to grow. And hybrid teams are both humans supercharged with AIs and agents that need to work together with humans and other agents.

Okay, go on. So tell me, tell me what this looks like. What does this hybrid team consist of? Is it going to be like humans in charge and then agents supporting them? Is it like, what do they consist of? What's the structure?

Well, the blueprint for a hybrid team, I think if people are thinking about how to create them. You mentioned this too. Like, AI is only as good as the data that powers it. So I think the first part of the blueprint for any hybrid team is how do you create shared context? So you run to all kinds of shared context, issues of context and data is spread between different tools. So you have this siloed data today. A lot of that data is also trapped in things like emails or calls or contracts. We call that like unstructured data and trapped data. And then it's also data that potentially is really bad. Like, there's a lot of bad data out of this world. So first part of the blueprint for hybrid teams is for sure, you know, create that unified context, connect your data. The second part is enable your human teams to do what they do best. So believe really deeply in. We're going to see taste and creativity still be a really key element in the AI world, where that tastemaker, that, you know, creative marketer, they are the brilliant marketer that might be breaking through. And so we want to enable these human teams to, like, focus on what they do best, which might be building customer relationships, understanding renewals, reading the room. So that's the second part of the blueprint. And then the last part is now you have to build your actual team, which is probably going to be a bit of both, which is the supercharged human with a breeze assistant or a co pilot. And then which agents do you bring into the framework and then how do you get them all context. So you need shared context across humans and agents or humans and AI. You need tools that kind of empower your people to do what they do best, get rid of like the manual stuff. And then the last one is combine the supercharged humans plus agents.

Okay, So I Like, there are different sort of steps. There's almost a workflow for, like, how do you build these teams? So you said start with shared context. Now I was chatting to one of our account executives, Joe. So I was, you know, at Datakin, we're trying to build out all these agents ourselves. And so I was asking him, like, what are the problems you have with like building agents for sales? Things like, well, you know, some of our data is in salesforce, some is in, like all these other tools. Some of it's on BigQuery and just dealing with these data silos. So getting that shared context is really difficult. So, I mean, data silos is a really old problem, but still a challenge. Do you have any advice on how you deal with all these sort of different data sources and pulling the shared context together?

Yeah, I mean, definitely understanding and using tools that help you bring together that context. So, you know, we are launching Data Hub, which, you know, it's both two things. How do you connect data that is across these silo systems like BigQuery or Snowflake or your analytics database Warehouse. But also there's a ton of data that are in spreadsheets that are probably just collected across Google Sheets, CSVs, other kinds of things. And so how can you easily bring those tools together? What Data Hub does is it gives you a home that I wish data. Bringing it together was as easy as adding a column like in a spreadsheet. And that's what Data Hub and Data Studio allows you to do. So today it's fairly complicated to still bring together multiple data sources. You almost need like a CRM or data ops team. But equipping people with tools that like, you know, if I know I want product analytics and it comes from BigQuery, and I know I want these fields, could I add a column that just lets me pull out almost fields like in a spreadsheet, the same way to create a pivot table. So Data Studio is allowing us to do that. It's one way that we're helping to solve silo data.

Okay, so the idea is really just, it's kind of, well, I guess, spreadsheet. So it's point and click sort of interface just to say, these are the bits of data I want. Let's bring things together and hopefully you're solving these sort of connectivity issues.

Right? Because like, once you have silo data or once you're trying to bring together data, you almost always have an outcome. So data itself, you know, while you have the shared context, it's not that useful unless you can use it for something. So I'll give you an example in the sales and marketing world, where oftentimes, if you find a set of customers that are high value customers, they have higher propensity to buy. They often have characteristics like maybe an executive would have entered the conversation at a certain period of time. I almost wish I could find lookalike audiences. And so that's what we're kind of trying to enable with silo data. If you can bring the data together, you can start seeing patterns across it and then you can do things like create lookalike audiences and then run a campaign if you want. It makes it a lot easier to reach people that might value your product the most.

Okay, so it sounds like there might be some sort of data engineering involved just in the, in the set of this, just making sure that everything is ready so that other people can use it. Is that correct?

There will be both. You know, our philosophy is we've always tried to build tools that are easy and intuitive to use and then have fast time to value. So like, if you wanted to create this lookalike audience, it shouldn't be like a number of steps that you have to train up an entire team. Ideally, there's ways that we can help you do it. So I give you that because, yes, we'll have data engineering, you'll see engineer data engineering teams. But like all functions, AI will disrupt a lot of that manual work. And so what we're doing with Data Studio is even if you don't have a full data engineering team, you could start bringing this data together in a very intuitive way, as simple as adding a column. That's, you know, it might look like advanced joins, but if you had four columns across, your BigQuery, your HubSpot, CRN, your QuickBooks invoices, you could start adding these as columns and then mixing them. That is essentially an advanced complex join.

Okay, so not having to learn some SQL for exactly joins in SQL is like anyone could do them. But in general, like salespeople, marketing people don't want to have to do them on a regular basis.

Right. So giving them kind of intuitive tools that like match what you're trying to do.

Yeah, okay, that makes all the sense, is have the right format for your audience. So actually, I guess this leads to a sort of more general point in that often like your, your sales marketing people don't necessarily converse well with your engineers. A lot of the time when you're like, when you're trying to build out these agents, it can feel like you do need technical people and you need people with domain knowledge. And they have to work together, they've got to collaborate, and it's hard work. Do you have any advice for making those collaborations easier?

Finding shared language, to be honest, so a lot of times, if it's really solving for the customer, there might be different ways that we think about solving the customer. But at the end of the day, that is the joint. That's what both are trying to do. Sales, marketing and engineering. So finding some of those joint terms, we use them a lot to describe what we think customer value is and how we measure it. That helps us build an example. So for instance, I talked a little bit about these lookalike audiences. The goal of lookalike audiences or personalized email or campaigns is really to understand and reach customers is to generate more leads. And so more leads, more leads, close leads to higher revenue and growing your business. And so just kind of using shared language in how you're trying to do.

This, that makes a lot of sense. It's like if you're both talking the same bits of jargon, then there's going to be a lot less confusion over that and hopefully you're going to have some shared goals as well. So you were talking about the idea of a hybrid team. So it's going to be humans and agents working together. So do you have message agents being ephemeral things? Because, like, if you think of them as, as a colleague, maybe they ought to have some kind of permanence. Whereas often it's like, well, this agent ran for like five minutes and then it no longer exists. So I'm kind of wondering, does this change in the future? Are we going to have these sort of more permanent agents as part of our team?

I believe that there will be hybrid teams where you, you might have agents that kind of take over certain parts of jobs. What's, what is interesting and I think is happening just across the world now is some agents are agentic. Like there are people will say they're ambient or agentic and they're triggered off of something. So like an ambient agent just kind of just means it's triggered by an event. Like it's not triggered by a human. Like a human might say, please go research this company for me, and I kicked off the job. Or it might be, hey, an email comes in. And every time an email comes in, I want this agent to do certain things. So I do think you're going to see a mix of both, where sometimes they will be ephemeral tasks that will be done. But if you think about it as like, which ones do you bring in into a recurring workflow? It will be a bit of both. Like humans will drive some of these workflows that will have these ephemeral tasks. And then I also think that actually the way I think about agents too is agents themselves can do have many different skills. And so some of those skills will be kicked off as ephemeral skills and some will be more permanent. Like, rather than hiring the five new customer support folks on my team, if I have a customer agent that resolves this kind of these, this amount of tickets, I now know what I'll need is the more complex escalations and I might look for different skill sets there.

Ah, this is fascinating. So suppose you're a manager. How do you change what skill sets you're looking for when you're hiring?

That one depends on what you think are. You know, I look at some of the companies today that have been really successful in, you know, AI native tools like uc, Harvey and Legal. You see a couple of these that are very specific in verticals. What they've done is they've taken jobs that are more mundane and repetitive and those will go first. That's kind of why I had customer. You know, customer agent is probably, it's, it's not the most unique, it's the most valuable right now. And it's because, you know, you used to be able to calculate if I added X amount of people, I would likely get X amount of support tickets resolved. And what's possible now is when you look across that team, I can give almost like an equation of if I have customer support agent running, I reduce my ticket volume by this. But I would expect as humans, you just ask harder questions. So, you know, then, then it's like maybe the skill sets in that will be. Can you. The more complex questions may often rely on. Like we have a big product portfolio, it might go across the product portfolio. So kind of people who can navigate that ambiguity or navigate like a product portfolio, but those might be ways that it could change. Yeah, it'll be, it'll be an interesting world for managers as we think about how to create a hybrid team. And then the same way you would onboard a human or like a new person on your team, you'll have to onboard the agent. Like customer agents are only great if you train it on your company material. Otherwise it's answering generic questions, which is not as useful.

Yeah, I mean, this is interesting. You hire a new human, okay. They kind of read some documentation they're going to have to watch some videos on what you're doing, then maybe shadowing another human. So yeah, go on, talk me through agent onboarding. Is this just a case of like fine tuning an existing agent, like using company data? What does it look like it would.

Be less probably fine tuning but giving it context. So I think that's why the context layer and why actually why data is so critical. Data is probably the currency that AI is going to run on and why it's so important that people start thinking or are thinking about it. Training an agent today, you know, you might have a place to go. So like I'll take customer agent as an example, giving it PDFs or your knowledge base or your company website or you know, the same way that you might have got a getting started guide for a human, you probably have a getting started guide with like a bunch of different links. Today it looks like just giving the agent those links. It's almost like creating a project, you know, in like one of the LLMs where as long as you can feed it different example types and like good content, it starts learning and that's probably what it's going to look like, you know, even as we go into more complex places.

Okay, I feel like we've got, so we've got like bits of components of a plan here. So if you want to get your really good customer service agent, so you, you start off with like generic agent and then you hook it up to all your data sources and then you discover that those are out of date and a mess and then you have to use the data agent to fix your data sources and then once you've done that then you can onboard the customer service agent properly. And, and in theory things work, we.

Should make it easier and easier for people to train their agents. So today we have a smart CRM that is key part in building context. CRMs really used to be this manual system record that people had to keep up to date. But now with AI you don't have to do that. So our smart CRM now self generates data so it monitors things like emails or call transcripts or the support tickets coming in. And I can tell, you know, if Richie got a new job, I'll update my content in my CRM of your new job and your new role. If you're at an office and you sent that to me an email, I can see that. So I would know, hey, I don't email, you know if I'm going to email you, don't email me when I'm on vacation. Email me just kind of right after when I finally get through that. Like the amount of email I have for instance. But is smart CRM that helps you build self generating context. Means I don't have to train you on everything because I already have unified context in my CRM.

Okay, so unified contact, unified context sounds very powerful. But I guess there are going to be some challenges around like data privacy about who's allowed to know what. You can have pretty strong data governance there. Can you talk me through how that works? Like what some. What you might need to worry about?

Yeah, data governance is going to be incredibly important because you want AI that you can trust. And if you can't trust AI and you can't trust your data, that's like a very bad thing. Governance will mean a couple things. So user permissions and even agent permissions is something that we're very thoughtful about today. There are roles that you might have in teams. So someone on a CS team may want default views that look like this. Someone on a sales team. Oftentimes deals are split across regions or across different owners. And, and so you don't really want everybody seeing everyone's deals unless you're like our sales leader across that team. And I want to see what the forecast looks like across the board. And so user permissions are incredibly important. Same thing with like where does that data coming from? How is that data being used? How would this data be trained? But I think of how do you access data as like a core component of solving the data challenges here. Agents will be the same where agents, if they act on behalf of a user, you, you inherit those user permissions if it acts on its own, you know, are we going to enter a world where agents need to have licenses and seats the same way that humans do, which will be really interesting.

Yeah, I guess. I mean there's probably a whole separate discussion on like how you go about pricing your own software if it's going to be used by agents as well as by humans as well. It's maybe a whole separate episode. I don't know whether you've thought about this at all.

Yeah, you know, the concept of hybrid teams kind of gives you a mix of both, which is you have people and agents that may have seats or may have core seats. And so I think the monetization strategy will be hybrid, so it will both be potentially, you know, core seats. But as AI does more work, you will need less seats. So the other half of the hybrid monetization is seats plus credits and that shifts that shifts work more from like paying for software as paying for outcomes and work. So you've seen a lot of different pricing models happening in the market today. But that concept of a hybrid monetization model where you still protect permissions, you have seats and then you monetize on outcomes and credits, I think that combo we'll see more of going forward.

Okay. Yeah, I guess the dumb thing over the last decade or more has been like the SaaS model where you pay a monthly subscription per person. I guess because generative AI stuff especially agents can be more expensive or more variable in terms of the costs. I think. Yeah, the usage pricing seems to be a lot more popular there.

Yeah, I think it'll be a bit of both. Like because permissions is going to be so important, you might have subscription pricing per month, which is access to feature sets or capabilities. And then it's ideally outcomes and value is tied to, you know, how much you want your pricing tied to value and outcomes.

Okay, so outcome pricing, would that be like if you've got a customer service bot, then it's charged by the number of tickets it closes or something like that?

Yeah, that might be it. But I think there's a lot of different pricing models and like thinking right now. And again, for me and for us, it really is like perceived customer value to what people are willing to pay. And so that is the easiest version of customer agent, which is support tickets resolved. But there could be many different reasons why you have support tickets. Like that high, for instance, maybe it's an instant or something else. So those are all factors I think that need to be considered when people start thinking about outcomes or value based pricing.

Okay, all right, so since you mentioned outcomes, that gives you an idea. So we were talking about agents as being team members. Would you have some kind of performance review for your agents? Like, I mean, if you're monitoring, tracking outcomes, then I guess there's some way of saying this is how well the agent's performing. Talk me through, how do you do that? How do you evaluate the performance of your agents?

Yeah, I. So I think so. You know, one of the important tools that are happening around AI right now are evals. And so even on my team we think a lot about evals of like, how do you create an eval system for an agent? And the answer, especially in the non deterministic world, like you're not exactly sure where they're going to ask, there's going to be an answer that's roughly in the same shape. But how do you build an eval system that is like, okay, you've got it, You've nailed it. Like, you know, maybe the first bar is like, there's pure accuracy that, like, gives you answers back that it knows is true and relevant, never makes anything up. So, like, easiest bar and then the harder bar is like, if it's non deterministic. How do you determine that? The range of this answer was, like, roughly in the right spot. I think evals and AI systems are becoming increasingly important, and I think you'll see almost every product you know have some way of evalu evaluating a agent. Did you do your job the way you were meant to do it? And do I need to give you more context? Did you make stuff up? Or did you actually help a customer, like, resolve what they were looking for?

Yeah. Are you seeing humans having to do the evaluation of agents, or is there a way for other agents to evaluate agents? Or what's the current best practice?

Yeah, I'm seeing both. Like, I definitely see humans creating eval systems. And then you can. You can write it, like, even today, internally for us. Like, you can write an eval agent that looks across its other teammates and kind of holds them to a high bar of if they're doing their job. So you'll see a mix of both for sure. Like right now, I see a lot of human in the loop to make sure human in the loop, both in terms of how we interact with AI. So iterating on something that you've asked about and then going deeper. I use AI myself, like, that way. And then you'll see human in the loop with evals as we teach agents how to eval other agents.

Okay, nice. So I guess eventually you want it to be mostly automated with other agents doing it, but you probably need a human in the short term. I guess just thinking about it, like, human messes up their performance review, worse that's going to happen. You get fired. Agent messes up the performance review, that's getting turned off permanently. It's an existential risk for the agent.

Which one's worse? It's better for an agent to have existential risk.

Yeah, but I think so, yeah. I certainly want a situation where humans are getting turned off for playing the performance review. Okay. All right. So I'm curious as to how far you can push this. Like, so do you have a sense of, like, what fraction of work, sort of long term is going to be done by humans? What fraction gets done by agents? Like, how much can you realistically outsource to AI?

I always believe that there will be in a world where kind of humans lead and AI accelerates and the kind of the magic of humans and how creative and breakthroughs like that is still going to be there. So I don't know what the exact percentage will end up being, but I think higher order work will happen. So a lot of times today I think humans do repetitive and manual work and I think all of that will be gone. So almost as agents get better and better and more autonomous, they will drive humans to do even more and more. And like I'm, I'm excited to see what kind of breakthroughs humans will do. So I definitely believe like there's going to be a human accelerated AI world worlds versus like purely autonomous.

Yeah, I guess it's good that there is going to be something left for humans in the end, so there's room for us. Suppose you want to get started with this. You want to really push hard and create some agents, get them doing your work. Where do you begin? Like what's step one?

Step one is still the blueprint. Like you know, you're a data guy, it's unite your data, unite your context. Some of it will start with things like a source of truth, like a CRM. And some of it CRMs, as they get smarter, they're going to do it for you, but you'll still always have siloed tools. And so connecting your data should get easier and easier. Cleaning up your data. If you're coming from legacy systems or like actually any system, right? Like we coined the term inbound marketing, where it was you get form submissions once you put your value online, how many of those forms are half filled? How many of those names are still messed up? You know, how many duplicates do you have? And so getting your data clean, that's step one. Step two is then, okay, deciding which agents you want in your team or what kind of jobs. You know, I've seen both sides, which is are there jobs you want to get done or are there human roles that you want to hire for and that you are looking to have an agent for? Determine where you want that use case and then train your agent. So again it's giving your agent context and then putting it in, you know, putting in kind of in production. But a lot of people will test out an agent first, see how it is. Our data agent that I mentioned to you is really good at making sure your custom data. So like if I care a lot about, for every company in my CRM, if I care about their top product, who their top competitors, are and every funding ground, I can just tell the agent, please watch for those three things. And then every time new information comes in from the web or from earnings, it will populate your CRM. So, you know, making sure that you have like a great system around you and then deploying those agents.

Okay, so your point about if you are hiring for something is you said that might be a thing where you want to incorporate an agent. So is that the idea? You're like, okay, well, we're a bit resource constrained. We think we might need to hire for this particular role. Therefore, we start trying to create an agent to outsource some of the work to AI there before we start hiring. Is that your suggested flow?

That is one suggested flow. But I think what I'm seeing with people adopting agents today is usually there's a, there's a thing eminent in front of you. So, you know, it's, it's just picking a single business process that you might want to augment or change with AI. So for instance, we've really been seeing the marketing funnel change. Essentially the old playbook just doesn't work anymore. You know, a search is coming from different mechanisms, not just kind of search engines. And so that might be an eminent thing that people want. So how do you one, create more personalized or tailored outreach or how do you find search traffic? And so like I would say, my, my advice is probably pick one process that is not working the way that you want and think about how AI can augment it. That might be a place that you'll find an opportunity for an agent or a co pilot.

Okay, so yeah, I'd love to talk about how you go about re engineering these processes, but you just terrified me slightly with the marketing funnel is broken. So talk me through. How do you fix your marketing funnel then?

You know, we coined inbound marketing in the cloud disruption, and it was really about bringing value out. I think we're seeing a new funnel or a new way to market. So what we're introducing, you know, for an AI era is a different kind of, of marketing playbook. And that playbook is called the loop, where you have kind of four phases and it's very dynamic and it needs to think about, like how you would use AI in each of those loops. But the first one is around. Can you express very clearly who you're trying to reach? You know, one of the reasons I mentioned lookalike audiences is it is possible now to find cohorts in similar areas. So can you express what you're looking for in Your core icp, let's say I'm looking for, you know, leaders that are in the education segment, but I also sell to the healthcare segment. I want to be able to find those folks and then personalize their both segments. And so express is the first part of that loop. Then you might want to tailor it to say I want to reach specific people and what would that look like? And then I want to amplify it, which is not only do I want to reach these people, but I want to reach them in wherever they are. So a lot of search results in that funnel is coming from LLMs today. So how can I create content that can be discovered in LLMs? LLMs, like, if you look across, you know, a lot of what we've learned, LLMs prioritize things like community influence content. Like Reddit, it influences, you know, certain types of content. And if I could craft my content the same way I did an Inbound, which was on a website, I could do that for an LLM. And that is, that is something we're calling, you know, that you're seeing across the industry. That's called aeo, which is, you know, AI engine optimization, a little bit different than SEO. So while the old SEO funnel is broken, I believe there's a new funnel, and that new funnel probably looks more like aeo.

Okay, that's fascinating. So, yeah, SEO is sort of very well understood at this point. Also, again, a little bit scary that all the dumb comments I've made on Reddit now being associated with my name, going straight into ChatGPT and Claude and all the rest. Yeah, that's terrifying. But good to know, good to be aware of. I think for a lot of people. We were talking before about process re engineering. So how do you go about changing your processes to incorporate these AI agents? I'm sure it's not as simple as just like, oh, let's switch out a human for AI.

Yeah, no, it's not. I think, I think people have to try it. And so it's nice to pick targeted problems. And then, you know, we do a lot of what we call proof of concepts. And the proof of concepts might be okay, we know the funnel's changed. We know the source of that traffic is coming from LLMs. Like, we know that content is changing and like, there isn't an SEO playbook the same way there has been established writing cloud yet. But I expect it to come. What are things that you would try? And, you know, once you try a set of things and AI works, then I would replace you Know, I would start replacing it. So for instance, or. Or I would say the other thing I've seen people do is give people the impossible task, which is, could you do this? You know, like, if there's a new idea for a project, can the project be done at a faster speed with less resources? If you used AI and simply kind of giving that as a constraint, which is almost like an impossible task. It's been amazing to see what teams can do and in what timeframe. That's the other route I would like suggest, which is almost the inspire others with the impossible task and people break the box with it, which is super fascinating.

Yeah, I mean, I do like the idea of just trying stuff and maybe it is a case of do something completely different and solve that impossible task and then share. Share what you've done with everyone else. I think everyone's kind of struggling with this problem at the moment. So I guess communication is an incredibly important aspect of this. So, yeah. Do you have any advice on like how you share your wins across your organization?

We do it in a structured way. So we might say, hey, there's like four hypotheses we're kicking off. All of them deserve a proof of concept. We're going to have that proof of concept team. It might be a virtual team across different disciplines. And then they get six weeks and then in six weeks we look across all the proof of concepts and like, what did you set out to do and did it work? So for instance, we use a version of like, we use our customer agent to resolve support tickets. And it is dramatic how many support tickets it was able to resolve. But that uncovered the next level which is, oh, wow, we're getting, we're getting more complex requests now. And so how do we think about the team that way? So I think of it as like, have I set a hypothesis of what you want to achieve? Give them an impossible task. Do a proof of concept, have a six week, like give it a time frame because people can do amazing things in limited amount of times and then see which ones worked and which ones didn't. Once ones work, scale it out. And so that's how we're doing. We're doing that on a repeat.

Okay, I like that. And yeah, I think the timeboxing is incredibly important. Otherwise, you know, this thing can go on forever.

It takes forever. Right. And there's no way to like, there's no way to check in on the learnings. And that's what really you're looking for in the proof of concepts. You're looking for the learnings.

Okay, is, yeah, have, have your employees learned how to do something new and maybe that's even more important than just like the thing they built.

Yeah. We also leave time, so sometimes it can feel super overwhelming with like the number of AI tools coming out. And so we leave employees time. Like we have something called we call Grow day where we might be like, okay, we're training everybody and then like try it in your own craft. Share different wins. We have Slack channels that are called AI Fluency or Grow Fridays. Those are fun ways to share things and then give people time to like try and learn. Once you get the aha moment in something that you're doing yourself, it's dramatic. You know how powerful it could be.

I do love the idea of just like give people time to learn and experiment and try new things. Suppose it all goes well, you push hard on AI agents. It works. What does success look like to you?

Success looks like the outcomes. So it kind of depends. Like some of the agents are as easy to understand as like the support ticket one. You know, some of the agents, like I even run in my day to day job is I run agents over my inbox. And so, you know, for me it's like sla, like how many, how many emails from my executive team am I taking a really long time to reply to? Those are like small things across the board. You know, another agent would be like the prospecting agent. For instance, we have about a 17,000 person wait list on the prospecting agent. But the prospecting agent is trying to find cohorts of customers that are very interesting. So that might mean we have something we call research intent. So because we have had inbound marketing, we have all these websites that people are all over the Internet are like doing things. And so when you shop for a product, you're starting to shop for a problem before you decided which product to buy. And so that's a research intent that you are in the market for, you know, new plants. For instance, I see your plants behind you and you're in the market for like new plants that don't take a lot of water. And so as you start looking for that in search engines, the website and LLMs like, to us, that's research intent. So that research intent is built into the prospecting agent. It's built into the CRM by default. And you can start finding these signals now. And so prospecting agents, measure of success, number of meeting it books, number of customers it reaches, high value customers that it reaches. And so like what kind of outcome are you looking for when you start bringing an agent into play?

No, that's very interesting. The idea that, like, as soon as someone starts searching for something that's intent to research, like they might want to buy something in the future and therefore they're already feeding into all of your systems. Yeah.

And prospecting agent can monitor. Monitor the Internet better than you can. So, you know, it kind of gives you that, like these set of people are starting to look for, you know, new plans.

Wonderful. All right, so as we sort of close off, what are you most excited about in the world of data and AI at the moment?

Oh, I'm. I am really excited about hybrid teams because, you know, I don't believe in a world that humans are just obliterated. You know, it really will be the kinds of things that make us more creative. I love the craft of, like, creating things. You know, I'm a product person, so I think I'm most excited about that, which is what AI enables us to do. That wasn't possible today. Like, even in that research intent example, that would be really, really hard to build. But it makes me a better. It would be make me a better salesperson, it would make me a better product person. Just to understand that, you know, the majority of people now are starting to look for this type of thing. So I'm excited about what other opportunities AI unlocks.

Absolutely. Yeah. So hopefully we'll see a few more human agent hybrid teams in the future and yeah, some great products to support that. And finally, I always want new people to follow. So whose research are you most interested in at the moment?

I love Leonardi Brachicki's stuff. So, you know, there's a series that he's just started called How I AI and I think that's a really interesting kind of podcast and others to learn about how you can practically apply AI in your daily life.

Okay. Yeah, certainly practically applying stuff. Very, very useful. Even more so than the theory oftentimes. Wonderful. Thank you so much for your time, Karen.

Awesome. Thank you so much. It's great meeting R.

SA.