LIVE from SaaStr Annual 2024- Workshop Stage - Wednesday

Sa. It. Ram. Ram. Sam. Sa. Sam sa. Sa. Ra. Sam. Ram. Sa. Sam. It. Ra. Ram. Sa. Sam. Ra. Sam. Sa. It. Sa. It. Sa. Dance. Ladies and gentlemen, please take your seats. Our show is about to begin. It. Please welcome to the stage alex plugarou, cto at gorgeous.

Thank you, thank you, thank you. I think some people are still kind of pouring in. Let's see if this thing works. All right. Hi everyone. Welcome, welcome. All right, I'll be talking about AI agents and customer support and what works. What we found works. But first a couple of things about myself. Been co founder Institute of gorgeous for past 10 years. Started in 2015. You can follow me on LinkedIn if you want or don't. But first was Gorgeous. We are an e commerce CRM. We are serving around 16,000 brands from very big ones like Fashion, Nova, Glossier, Steve Madden and so forth. We are a usage based SaaS software and we have two sets of products. One is a support suite which is our main one and we started going into the marketing as well. The support suite has a help desk which is where human agents go in and write tickets and respond customer requests. We have an automation product called Automate. It's all about automating customer service and we charge on full automation. And then a phone product with voice and SMS which is a paid channel. And finally we have Convert, which is a conversion optimization tool. It observes the customer behavior on the website and then pops up chat campaigns to close a deal to offer some discounts. Start first 201560 million ARR 300 people. So yeah, that's Gorgeous. What is this talk about? It's funny because when I sent the first iteration at Saster, it's like it's way too technical. You gotta make it less technical so it needs to be processed. So I tried to have a decent amount of high level stuff, business stuff and a little bit going into the weeds. And if you want to ask more questions, if you're curious about a specific subject, we're going to have some Q and A, so feel free. The microphone is going to be over here, so feel free to remember your question and come ask. So I'll start with a very high level about how the agent works. What are the different data sources that we have? Basically what is the data that we fit into it? What are some major challenges that we encountered while developing it? Why are actions or external actions very important to this goal? And then finally some rollout results. So basically when we start Alpha, when beta, how did we do the ga and some of the results and Closing thoughts afterwards. All right, moving on. Demo time. So you can see here how VA agent. So there's someone who accidentally placed an order to wrong address. The AI agent uses guidance and help center articles and you know, order information to correct the incorrect address and send back an email. What's important to understand here is that there is no human in the loop. It's all fully automated, end to end, which makes things a bit more challenging. If we take another scenario, you can imagine an email provider like Gmail and Outlook. You send that email to a help desk, it creates a ticket, the AI agent picks it up, assigns to itself, uses some knowledge from a help center article about. In this case, customer asks for a refund. So it's like what are we doing for refunds? Right? Also uses the order information in this case from Shopify and then decides that okay, I need to issue a refund. However, when it tries to do that, there are some, let's say strict filters. For example says if the customer does not have a higher LTV than $1,000, then do not perform this action. And that's not part of the LLM. So it's something that prevents accidental actions and because it fails, it escalates to a person. So that's another scenario. Obviously this is quite simplistic. There's a lot of data that goes into any large language model pipeline like ours. For example, we found. I was actually very surprised that the number one thing the brands ask is like we want tone of voice, we want to control how VA agents speaks, we want to control what type of words it's using, how it addresses customers and so forth. So there is a apologize, there's that level of control which comes first. Second is that you don't want to just dump like a giant JSON file into your LLM. It's not going to work really well. So you want to make sure that you're filtering some of metadata and rearrange it in such a way that it's processed so that the LLM can basically need to make your life easier for an LLM to work function properly. And third, like very very important is exclusion topics. A customer can send a message saying I'm gonna sue you and you don't want the AI agent to respond to that threat, so you want to escalate it directly. So some customers want to or like allergy claims or any sort of things that are medical related or something like that. That is something which we found very important to work where the customers can control which topics are absolutely a no Go for the agent. All right, Rag. Basically it's a fancy way of saying search what it does. It finds relevant articles to feed into the LLM prompts, guidance snippets. We call it guidance. What it is is like private internal help center with different processes that you don't necessarily want to expose to your customers. Things like discount policy, right? You don't necessarily want to say hey, the customer spent with us $10,000. If they ask for a refund, no questions asked. So that is that type of guidance that goes in there. Macros are things that human agents use a lot and actually are quite good source of information to feed into the AI agent. Then you also have the actions which I mentioned. So I'm going to talk about actions a couple of times because I feel like they are key and I'll explain why at the end each action. For example, third party API such as Shopify or Recharge or something like that. Like it needs to be integrated with AI for it to be able to do a full automation. However, as I mentioned earlier, it needs to be. It needs to kind of have a very strict set of filters, especially for dangerous actions such as refunds where money is involved, pretty much. And then help center articles on public website data. I think these are usually questions serve questions like is this product made of, you know, is it vegan or do you ship to Canada? Or that type of stuff. So usually you find it there. Moving on. Because it's an end to end AI agent, there is essentially no opportunity to correct an error before it goes out. And so some people are okay with it, early adopters are okay with enterprises or not. So what they want is they want more control, they want to be able to roll it out in a safe way. And so because of that we developed a couple of features which we find quite useful. I already mentioned exclusion topics. Really, really important. There is preview mode or draft mode if you will. It just creates the draft. A human agent can validate, even change if they want to. We also use the that as a feedback mechanism to see if we actually respond correctly. So that creates some safety sampling what it is. It's like you only send 10% of the traffic, you don't send more and then you roll it out progressively. So that's another technique which works quite well. And Playground, definitely my favorite, where you can test different scenarios without actually going into production. So it's like an internal chatgpt where you talk with your own agent hallucinations and factual errors. So basically the nature of LLMs is that they are just gonna generate stuff, and it's not necessarily gonna be based in reality. I know some humans also do that from time to time, and there is no way to avoid it. But you can mitigate it. You can mitigate it by specializing LLM prompts. You can invest in a test pipeline, in evaluation data sets. There are ways to reduce this, and we found that we can reduce them significantly if we were very careful about tuning them. Obviously, because it's an AI agent, it means that you pretty much cover 100% of your traffic, which means that pretty much all of the. All of your traffic is gonna go through it. So you should expect that if you roll it out. Well, I mean, compared to a copilot, where a copilot is more on demand, AI agent is like, on demand of your customers, right? So it's like it just goes 100%. So what we found compared to, like, our copilot implementation is that, like, the cost increases a lot more. So that's something to expect as well. But really what it comes down to is this last piece, and it's between precision and recall, really. If the AI agent is trying to be too safe, you're escalating too much to people, you're creating extra work for your customers. If it's too eager or too aggressive and responds to everyone, potentially creates a lot more errors. And then, you know, your customers churn you, create a lot more work for. So it's really. It's work between these two pieces. It's the art of balancing the errors versus. And by the way, another thing that I think is important is the bar. The expectation for the agent compared to people is higher. So that's another thing. It's just like, well, my agent needs to be way, way better than my bpo, so. And cheaper, obviously. All right, More challenges. So you will notice a trend. I'm a cto. I'm only talking about problems. Context sizes got a lot bigger with bigger models, but that doesn't necessarily mean that they perform better. In fact, we noticed if you just have, like, just a giant, you know, text and you dump everything into one LLM call, doesn't really work. So what you do is split it up in multiple specialized pieces. First piece could be a router, which decides what is this message about and then which path I should take to treat it. So that's like, number one. Then you can have a summarization step, then you can have a generation step, and finally we have what we call, like an arbiter step. It's like the last step before it goes goes out to the customer which simply asks, does this look like something like a support agent would respond? So it doesn't respond anything like. So that actually last step is quite important, the last validation step. And you gotta decide as a business, like what do you value more? Do you value being extremely accurate or do you value basically like how forgivable is the business that you work in? For example, if you're determining, I don't know, legal documents or something like that, you're working with that, like that can be less forgivable than a support response in B2C brand. So I think as a business important to decide this aspect and that allows you to build the agent in such a way that is very useful. Then there's the retrieval like I guess this is for the tech people. Rag versus fine tuning. If you don't know what fine tuning is, you don't need it. If you do know what it is, you probably still don't need it. But if you do think you need can be good for very well defined steps and you can use it to achieve quite good results. But I would say it's like there's new models coming out all the time and what. And they are cheaper and cheaper. So we found that like it's not that useful actually. Not necessarily something to look into, but it can do very well for improving F1 scores.

All right.

I mentioned a couple of times external actions and why they are important and the reason they are important is because today the default expectation is full automation. That's what people expect. That's what businesses expect. They don't expect you to just do some of work. They expect you to do all of the work, at least in customer service. And the ability to perform external actions is really key because if you only respond questions but you're not able to change someone's shipping address or perform a refund or do a return or do all of that. That's just half of the work. And a lot of the brands just don't find that compelling enough either to switch from a different vendor or upgrade to your product. So that's why like actions are very important. However, there are challenges with them. As I mentioned earlier, you absolutely have to have some strict filters outside of the LLM to create this safety. What I mentioned earlier, we have the example where the LTV of a customer is higher than certain threshold. Then you perform the action and that's processed outside the large language model to make sure that it never happens, no matter what. The biggest let's say fear is that like customers try it, it makes one error and then they never try it again and they go to a competitor. So you want to avoid that at all cost. Right? Because the cost of switching is high. And also a lot of people are very skeptical these days, and rightly so. Everyone promises 80% automation like anywhere you look. I'm sure your inbox is full, including from us. So another challenge, which I think is really important, especially for more advanced actions, is that sometimes you need to call multiple different APIs to retrieve the information. Then based on that information, you need to know, do another API call, or sometimes you need to do them in parallel, then get the response and then decide what to do next. So it can get quite complex. It's essentially becoming a workflow with multiple steps. I mentioned the tip of the iceberg. The truth is the fact that you're probably not seeing results that you expected from your AI agents is because the iceberg is very, very deep. And I know it sounds cliche for the CTO to say, yeah, it takes longer than expected, give me more money, more people and I'll do a better job. But that's the truth. It just takes a lot longer to build a useful product. There is many, many different components in the process. However, what I will say, at least from our experience, is definitely worth keeping, like pushing forward. It's just good products takes time and that's just, and you should just keep going. So if you're, if you're looking for some encouragement, here it is. Keep going. All right, let's talk about rollout a little bit and how it went for us. So we started with an alpha, took us three months. We kind of started coding in sometime November, December, then we start coding kind of like a more production ready version of it. We selected 10 brands who are very, very eager to get started with an AI agent. And we were very clear about the fact that we need very, very tight feedback because the timeline for us was six months until we get to the ga. So it was very tight loop. One technique that worked is that if they didn't respond really quickly, we would threaten them. Okay, like, we're not gonna work with you anymore, so it's like we're gonna get someone else. And actually that helped quite a bit to get the feedback that was necessary for us to work on this. The Results, we got 10% full automation, 30% on the kind of on average, and 30% on the best customers. We didn't have actions, we just had help center articles. And that's it. That was pretty much it. We at the end we added another 20 and because some people dropped out and we needed more information and so. And then moving on to the beta, we had 50 brands. We started writing the playbook for activation and that I think was one of the good decisions that we made. Very early on we involved the product marketing team, success team, we involved them together to write a playbook together on activation, on how to actually sell it, how to position ourselves. So that was very useful and I think that played a huge role for the success of rga. And I would highly recommend, if you're building a product, to involve the people who actually will have to onboard customers and sell it really early on. And then we continued by adding more data sources I mentioned before, like macros, public pages and so forth. We introduced some very basic HTTP actions, just generic HTTP actions. And then some of our ecosystem integrations, Shopify, recharge, loop and we went from like sampling to 100% rollout for the email as well. So depending on your stage of the company, you can think six months is a lot of time. Why are these people slow? Or if you're a big company, you'll say this is incredibly fast. I think it was fast for us given that we have quite a few brands. But the way we went on the rollout is that we coincided the actual release with VGA with our first ever user conference. So we went on stage, we presented, we did a live demo which worked. So that was good. That was a relief. We made a lot of buzz on the social media. We changed our website to be AI first. Like everyone.

Yeah.

And the results are that we have right now we have around 500 brands using our AI agent. Around 10% automation, maybe a bit more. I didn't look at the numbers. I think this is probably a bit higher and 4th percent top line. Basically it's like P95. I think my main learnings is that usually when there is such a big company initiative, if you think you spent enough time aligning with everyone in the business, you probably didn't spend enough time. So you should expect that there's gonna be some pushback because there's a lot of drift in any company's development. And so my learning is that like you need to spend an inordinate amount of time making sure that everyone is pushing right direction. So definitely worth doing so. Closing thoughts. I think like the key takeaway is that full automation is the new expectation. It's no longer something that you can kind of not do copilots are nice to have even table stakes these days. And so yeah, it's just, it's just the truth. And it takes time to build. Like the POC that didn't get product market fit in nine months. Sometimes you gotta wait longer. You can just keep going. It's gonna cost a lot more, but I think the cost will go down. There's a lot of GPUs that are gonna be available, I'm hoping, and better models, so. And the infrastructure tooling is not very good yet because it's pretty much like 2 years old. And so a lot of the things that make the development of successful large language models is not there yet. Like testing, evaluation, orchestration, observability. It's just a little bit different, a different way to develop software. And so the tooling behind it, the infrastructure is not yet super mature, which makes things a bit slower. So don't be surprised that like there's not not much on the market for this. And finally like GPT4 level, at least for us, the type of models are good enough. I will take a faster, cheaper model at the same level of accuracy. So if you waiting for GPT 6 or 7 come out, probably don't wait. That's it. Thank you. Thank you. So if anyone has a question. So we have, I'm not sure exactly how much time out the time, so we have 20 something minutes. If you have a question, the microphone is right here in the center, so feel free to take it and I'll happy to dive in any question.

All right.

Thank you so much for this in depth presentation.

I really liked it. I have a question about innovation is great.

I really believe in it.

But how about tech depth?

Tech depth versus these new innovations.

How do you handle that as an organization, as a cto to align the troops internally.

Yeah. So you also have to do enterprise, you also have to do the existing product that you have. Yes, it's very difficult. I think what's really important is to align at the top to make sure it's like okay, how many resources we actually allocate for this. Is it like 30% make sure that everyone in the business understand that that is the allocation and we are carving out the space for this. It cannot be. Yeah, just do some AI and figure it out. Like no, like tell me exactly how much do you want to invest in this and then we can have a discussion about like what I can deliver. Right. Because just starting and not really aligning on this, it's not gonna work out really well. It's gonna Create a lot of friction internally because of the existing. As I said, there's a lot of inertia to any business. Right? Yeah, I would say that is where you need to start. And then on the engineering side, I think it really, like, you gotta paint the picture of like, this is not just hype. We actually have a use case. It's a real use case. Here's how it's gonna work and these are the results that they're looking for. It's definitely worth investing. And we're going to pause, we're going to sacrifice some of these other things in order for us to deliver this. For example, for us, what worked is that we have. It's kind of seasonal the business. So we have a Black Friday Cyber Monday in E commerce is really, really important. And we were like, if we don't deliver by July 1st, we're missing the window. So we need to wait another year. That creates a lot of urgency. Like three months before Black Friday, we need to be ready. Otherwise the customer. Customers don't have the opportunity to adopt it. And so that created a lot of pressure and I think good pressure in the team to deliver on time. And yes, you have to sacrifice some things, but you need to be open about it.

Yeah, got it.

Thank you.

Of course.

Hi there.

You mentioned earlier in your presentation about.

An agent that would pop up to help convert a sale perhaps at a key moment in the sales funnel.

My question is about how end users reacted to that. And I asked because I had an.

Opportunity on our E commerce platform to literally watch live sessions and pop in.

And do that live.

And I tried it two or three.

Times and every time I popped in.

And asked a question, the user just aborted.

But I was very early on and I was going to on my gut.

My question is, after iterating that have.

You found it to be helpful or.

Does it kind of freak customers out?

So some customers decide to disclose that it's a bot. Some customers decide to impersonate them. We leave that decision to the customer. We recommend that people disclose that it's an AI and not a human. But people decide what they want to do. What we found is people don't really care. Care as long as you solve the issue, actually solve it. So for example, if they ask you, hey, change my address, they don't care if it's a bot that does it, they just should do it. Right.

So I was thinking more in an.

E commerce sales funnel, if someone's about to buy. Yeah.

And then all of a sudden a bot Pops up.

Yeah.

Asking as if they've been watched.

Right, right, right, right. Oh, yeah, the convert side. Yes, sure. So we found that, I mean, on average we're able to influence around like 5% GMV uplift as tested by an A B test where you have zero, don't pop up anything and then you have 50% which pops up something. As you know, in the E commerce space, there's like these pop ups on the models and a bunch of stuff. We found that the chat kind of bubble actually is a bit more friendly and doesn't occupy all of the space and it's, it's less intrusive.

So it did increase sales.

Yes, it did. It did, yes. Yeah, yeah. The next iteration of this is that when the bubble comes up, you actually have a conversation with an AI that is, you know, trained to do sales. So that's the, that's the next step that we want to do. Yeah.

Thank you.

Sure.

Hey, Alex, Vlad here.

Great to see a fellow Romanian on stage. First question, there's been many viral examples of chatbots going haywire on social media.

Right.

If you ask, you know, things they shouldn't care about, like, what's your opinion on the Harris Trump? Have you encountered this at Gorgeous and.

How do you protect for that?

Yeah, so that, that is the last validation step that I mentioned. We literally asked at the last step before it goes out, like the arbiter or whatever you want to call it. Does this response looks like a typical customer service response given the guidelines that the customer has. So for example, if it's a cosmetics brand or something like that, we kind of feed in that. And does this look like something someone in customer service would answer? So if you ask a question about the nature of universe or, or the Kamala, Harris and Trump debate, then it will not. It will just block that. It will block the response and it will escalate to an agent. Makes sense. That's a separate LLM response. Yeah. And actually we try to keep it as isolated as we can so it's not biased from the rest of the pipeline. The challenge with this is if it's too restrictive, then a lot of answers do not go out. So then you don't provide as much value that you should. And by the way, like, we don't charge customers on partial automation. We only charge them full if actually the ticket is closed and resolved. So it's very tricky because, like, we need to be. If we don't, if we're too restrictive, then we don't make money and our customers don't get value. If we are too permissive, then you know the issues that you mentioned. Yeah, makes sense.

So it's success based pricing and a follow up.

If I may.

How often do you change models? I know you just launched in June or something. Do you change models often?

You mean like LLM models? Yeah, yeah. So different steps now have different models. To be honest, like my team every week tests something and the reason we're able to do it is because we invested in an evaluation pipeline and we have data sets for each step. So Sometimes we use GPT4 or Mini or Sonet, it depends and we just test the different things. We also have some like hugging face things as well in the pipeline. So we changed it a lot. But I think the only reason we are able to is because we have some safety in when we deploy it before it goes out in production.

Thank you.

Hi, could you tell us a little.

Bit about your resources like engineering, CEO.

CTO quality, Whoever checked the edge cases.

And tested to get to your 3 months, 6 months data and then to general. So what were the success criteria to go to the next step? No, the resources, like how many engineers needed. Was there a product person, the CEO or CTO had to jump in to.

Test it before it got out there.

Like the execution. Right, right, right. Great question. So early on we like the first three months and like it was mostly the team that was building it and I would say that was more like 20 or something engineers that were working on it in some capacity because some engineers were working more on the data sources. For the, for the AI, there was like a team of five that was actually working on the orchestration on the AI itself. So it's not like if I'm saying 20 engineers, it's a lot of engineers. Yes. And actually doing that like so many right away, it's probably not a good idea. But if you split the work into different categories like I mentioned, then it can actually get better. We had our first like prompt engineer as well, who came from a background of customer service, actually someone from our team, technical person that was very willing to kind of test this thing out and actually configure the prompts and create the data set. So we essentially. Okay, you work in sport now you work in engineering. Good job, you're promoted. Now you're a prompt engineer. And so yeah, we did have to shift some roles. And then on the exec team we decided every one of us in the exec team will have one customer that's gonna get onboarded on this and we reported that every week we reported on what was our success with that customer and what are the kind of the pushback that they had and so forth. So we were kind of essentially telling the story in the exec team and that propagated downwards to everyone else. So a lot of these sort of tactical things. And then As I mentioned, 20 engineers, but some of them were in the platform as well. For example, the external actions is really about integrations and they live in the platform, not in the automate. And so like there was a lot of collaboration inside the team, which was one of the reasons why I was mentioning, like you gotta be, you gotta invest a huge amount of effort in making sure that. On the alignment of all of these teams. Yeah. So even the three months alpha, you had 20 engineers all kind of focus on that goal together, plus, minus. Yeah, I think, I mean, look, it started with a few and then it grew to 20. Yeah, yeah, yeah. All right, thank you. Sure.

Hey Tom from Spinach AI, thank you for the talk.

Super helpful.

I'm curious about pricing. So we have an AI product in market and we're also doing usage pricing.

Which made a lot of sense to us because, you know, it aligns with the cost structure.

It resonated well with small customers because.

You know, like they use it more.

They pay more, they use it less, they pay less.

Like it makes sense what we're seeing with bigger customers that, you know, they like predictability and control and you know.

Being on top of spend.

So I'm curious, what are you seeing.

Since you are working with some, some big brands, are they, you know, resistant.

To it or does it work?

Like what's, what are you seeing?

Yeah, so we have, we have, let's say packages and we're like, okay, we can. So actually maybe there's the CRO and maybe he can answer better this question. But what the, the way we do it is that we have multiple products. So we have a help desk, we have the automation product and they have separate pricings and then we have a voice and SMS product and the convert one. So usually what we do is we have some packages, we have a suite that we sell as a package instead of just the automation part. For example, you buy 10,000 automations and that is a specific cost and if you go beyond that, the agent just does not work, for example. So that way they can control the cost and we discuss on that. So it's like, it's usage based but it just stops after a while.

Got it.

And they can predict how much they'll.

Need based on historical data.

Usually we look at what traffic they already seeing in the support size and we say okay, we can guarantee you that we're going to automate 30% of that and that usually already has a number and we work with them to actually onboard it. Remember that it's not like flipping a switch. You're actually going to have to go and create these guidances and the help center, you got to have the data because if you just put the AI there and we've known nothing behind it, it doesn't work.

Yeah.

All right, awesome, thanks.

Hi Alex, thank you very much much for the talk.

Was pretty helpful as well.

I have just a question. How do you balance basically the level of personalization you manage to give each.

Customer versus kind of the performance or.

The, the things they are able to tweak into their own platform?

Right.

I think that's something very interesting. We've seen our customers as well like we're having to deliver super personalized solution as you said, like the tone and all that stuff. Like how are you balancing these two things? I think we are cheating a little bit because we are in the E commerce space and a lot of the brands have similar use cases. So I guess like one of the reasons we found that it's successful is because everyone has to deal with returns. Everyone has to deal with where is my order? Type of questions. So we were able to, even without the customer being involved a lot in kind of providing all of that guidance, we were able to kind of hide it behind scenes. Such the experience is very good and there's good results. So that's if your customer base is very different. Right. Like healthcare and SaaS and everything. I would say it's a little bit harder because you have to think about all of the use cases. One technique that we, I mean we actually use it internally is that you don't have like a single AI agent. You have multiple ones and every one of them is specialized. So for example you develop like okay, I have a healthcare question and then you have a router. You have a router. So it decides it's healthcare question. It goes into the healthcare AI agent pipeline and then you, you only work on that. So yeah, every vertical will have some like nuance I would say. And that's perfect.

Perfect.

Thank you very much.

Sure.

Thank you very much Alex. All your talk here is imminently useful questions surrounding sms. Firstly, what is the rough percentage of completed automations from your platforms are through that channel?

We don't support sms. Oh just email and we're releasing chat.

We might be able to partner on that.

The. Sorry, we support SMS as a channel but not for automation. We don't do AI agent with SMS yet. Understood. Yeah.

Second question then just now that you clarify that is have you found through different channels you need to do a lot to change either the prompt or the orchestration or in order to give the same level of performance?

Yes. So actually while we are working on the chat, the latency for email, like for example, if someone sends an email to customer service, they don't expect like a 10 second response. Right. Or even quicker on chat is totally different. So we have different, let's say sophistication, sophisticated models. We use cheaper models. Basically instead of using Turbo, we are going to use Mini and vice versa. Right. So it's. And depending on the different steps because as I said, we're splitting things up into different pieces. So the fact that you're splitting actually increases latency as well. So latency is really important for the chat and so we made a lot of effort to kind of use smaller models. And actually by doing this work we also reduce the cost on the email side as well because we found there is no difference in some steps. There was no difference between like using Turbo versus Mini and we're like, okay, let's just use Mini for both. Right. And so that's our experience. And also we also tell the customers that hey, like we're gonna use cheaper models for the chat because the latency needs to be lower with time. I think, I just think it's just a temporary thing to do. Be honest with you, like we're gonna get better, we're gonna get more GPUs and faster models. So you just gotta wait a little while and pretty much all the models are gonna be fast enough. That's my prediction. I believe in Moore's law. So yeah. Thank you so much. All right.

Hey, thanks so much for the time. Wanted to get a little more insight into your journey. Kind of standing up in eval to.

Evaluate those different steps, especially early on.

When you might not have all that data. How do you make sure you're not overfitting to yourself?

Yeah.

Tell me a little more how you.

Structure your approach to that.

This is really the guts of it all. That's the tricky part that usually you never see outside. We use the tool prompt layer first of all to do the template registry, the prompt template registry. Then we did the evaluation and now we're doing a B test with them as well. The reason we chose Them as a vendor is because they are very eager to develop. Like we were like, maybe we should develop this internally. It's gonna be a core thing. So like why would we go to a vendor? And as you remember I mentioned, it's like it's two years old, so there's. Everything is like YC level, you know, development. So it's. To actually have something in production needs something, something very mature. But to your question, how do you, how do you make sure you don't overfit? So as I said, keep your data fresh so that. Yeah, yeah, yeah. So they have, they have some functionality where you can see new data and you can add it automatically to the data set. Some samples which you find like they're representative, especially when there are bugs. So what we do is we include like a trace ID in every response and then the prompt engineer can, after someone complains on the customer, they can take that and they can go in the tool and then they can add new data set and remove some things from the data set. It's a lot of manual labor, to be honest. It's. I don't know. I mean, if you know better ways, let me know.

But I'm doing it manually too.

So that's, that's, that's what takes. So we're actually increasing the team of prompt engineers to scale that. But it is tricky. I mean there are like techniques where you use LLMs to automatically improve your prompt DSPY. There's a bunch of them. For now, we're doing it manually because we just want to make sure that like we can afford it, so to speak, at this moment.

Cool, thank you.

Sure.

Alex, thank you very much. Like, very helpful, very insightful. Just want to ask you mention about the Playbook you created. Like do you actually, apart from the Playbook, you actually help people to build or help them to understand what actually how do they measure their success?

Sorry.

So I just say you actually, you know, it's a DIY people can build themselves.

You give them a Playbook.

But do you also offer some kind of more like a professional service type thing to actually, to help people actually go through the whole building part?

Yeah. So the Playbook is both for the customers and internally. So when we did the Playbook, it has essentially two versions of. And that's why I think it's also important to work with the product marketing as well. So we have a product marketing team and they are, they are the ones that actually built the Playbook, both for the marketing use case, but also for the success team and sales team. So they did both. And there's different versions, there's different versions of detail on this. Right. So there's going to be a lot less detail on the public, public side. I say then what's kind of key.

Kind of KPIs to measure success for.

Your customers for this specific product is automation rate. So we look at like how much we automated and automation rate is measured by, by this criteria. If you, if, let's say the AI agent responds and 72 hours pass and it closes the ticket. Sometimes the agent does not close the ticket because it feels like the response is not done and it waits for the customer to respond. Human agent takes over before when we don't charge. If 72 hours passed and the customer said thank you, for example, and there's no more messages, then we consider that closed and that is the automation rate that we measure.

I see.

Okay, so thank you very much.

Hey Alex, what's up?

I want to quit.

I want to ask about how deep.

Knowledge tech should a CTO be as the team grows? Like as you do your team scales.

How should a have actually really knowledge of the technology or be more like.

A product person and balance with like.

A people person and management skills?

This is probably going to be controversial, but I definitely think basically it's like if in your title there is a T technology, you need to know the tech really deeply. You need to know how things work in like a lot of detail. Like that's how you're going to be helpful for your team. And, but also if you have a big team, for example, we have around like 100 people engineering team. I have directors reporting to me. I have like senior engineering, engineering managers, staff people. You got to be a good manager too. You got to do both. So yeah, you got to be everything. Yeah, let's put it in scale front 0 to 10. Like how, how do they be for.

Tech, product and management skills?

Yeah, I would say maybe, maybe think about, about it a little bit differently. Like as, as a, as a technology leader, you gotta understand really well what the business needs. What are the top three things that the business needs? And make sure that like you understand them clearly for yourself. And then the team understand what are the top technology needs. And then you say okay, from these three things, I have a list of 10 things that I really care about a lot that needs to happen. For example, for us it was analytics needs to be really good. It needs like the plot, the stability needs to be really good and some specific things you communicate that with the team. And you tell the team, hey, whenever you have a subject around this or we need to make a decision around this, I want to be involved. These other things, you guys need to, to figure it out. So you're gonna. Depending on the stage of a business, that list of like 10 items that are really important for you as a, as a tech leader are probably gonna evolve with a business needs. So you just. I would say it's like the way to say, okay, where do I need to be in the weeds and where I should be a bit more hands off and let the team cook. That is for you to communicate to your team. You need to tell the team what's important for you, you to be involved in and what it's not that important to be involved in. And it doesn't mean that you don't care. It just means that, like right now, the priority of business is these 10 things. And you guys need to tell me whenever there's something there, when to involve me. You need to give a playbook on how to use the cto, how to leverage the cto. So, yeah, I would say that that would be my advice.

Thank you very much. Hi.

Enjoyed the talk.

Three small questions, they're probably interrelated. How long is a typical implementation with the customer?

Another question is how much of the customer data do you need to train it on?

And when you do implement it, do you have some measure like ROI that.

Kind of like, you know, work in.

Collaboration with the customer to determine? So for I'm gonna ask, I'm gonna answer maybe the questions, which are quicker first. So for the roi, we have like a calculator that says, okay, we saved this much time. This is how much you're paying per hour, and this is what the result will be, like, how much cash we're saving you. So there is a calculator in product that you can use and you can plug your own numbers as a customer. The other question was like, how long does it take to onboard the customer? We have actually this program which is called 30 in 30. So 30% automation in 30 days. And when we go and pitch to the customer, AI agent or automation in general, we say, hey, we commit that we're gonna get you 30% and we're gonna work with you, but you have to commit as well that you want to get, get to that 30%. Because, for example, if you're a merchant, I cannot determine for you what's your return policy. You gotta tell me. And we can build it together and do it. So depending on the plan, of the customer. If it's an enterprise customer, there's gonna be more, let's say enterprise CSMs that are gonna go there and work with them and gonna have multiple sessions. I onboarded a couple of customers myself and what I found it to be very, very useful. I was like, I'm coming to your office, we're gonna do a three hour session and we're gonna get you to 40% automation by the end of that. And just like make sure you reserve like basically in person instead of doing it like multiple times. Obviously you can't always do it and for SMBs it's not worth it. But if you have very big customers, high acv, usually it's better if it's like a very, very long session where you go after every piece, every different use case. And I forgot the second question. What was it?

Yeah, how much data, how much of.

The customer data do you need to train it on?

Right. So what I found useful is actually looking at from the use case point of view, you say, okay, we're gonna solve the returns first and we're gonna only focus on that and everything else is gonna be an exclusion topic and we're leaving that for later. So we're gonna improve that with later. In terms of data it on depends. Depends on the use case. Right. So if it's like returns, it could be as simple as a help center article. Typically a lot of customers already have a lot of information on their website that shoppers can already answer. So what you do is say AI agent, go and take that information from website and use that to answer so that already can give you 10% like this, like very quickly. Every 10% afterwards is a little bit harder and incrementally harder. Right. So you don't need a lot like that's the thing. I think it's mostly just like you just take one simple use case. People also need to develop an intuition about how it works. So you don't want to go ahead and like, okay, we're going to do 80% right off the bat. We're going to get to 80. It's going to take time. We're going to start at 10. I guarantee that we're going to get 10% automation by this date, you know, and so forth. And we're gonna have milestones. So I wouldn't say it's like usually a lot of the questions are easily answerable by whatever public information you can find about the company. Yeah, awesome, thank you.

Hey, Alex Sapreet here. Yeah, so basically my question is all around, let's say there is a lot of dependency on the existing open LLMs, right?

Yeah.

Let's say down the line if things go south, LLMs are not available. Even though if we build our own.

LLMs it's not so powerful enough just.

Like the public LLMs.

So what are the workarounds are you thinking? Let's say.

So you mean like OpenAI shuts down and anthropic and like this multi billion dollar companies. Okay, just giving a thought, that's it. Yeah, yeah. I mean Llama, Llama three and I think llama four when it's coming out I think you can still probably achieve good results with fine tuning which we did some tests and there are some good tests, good results. Obviously there's nothing as good as the top proprietary models yet, but that's what I mean. I think it's like my prediction is that eventually it's going to be more like a commodity. I mean and, and we are very much vendor agnostic. We don't really care whatever is the like let's say the best performance. Accuracy really specifically accuracy. Not even recall that's what we're going to choose. So my, I mean I'm. The way I look at it it's like you get a similar level model which is 10 times like 10 times cheaper and it's just the trend is just going down in terms of cost.

So.

So and I think open source is catching up which I think is good for us. It is very good news and I think for everyone. Perfect, thank you. I think we're at time but let's do one more.

Yeah.

Hey Alex, this is Sunil. Sorry I had to step out for a few minutes Dr. Rome because I.

Was heading out so apologies if this.

Question was already asked. No worries. But you did mention the copilot versus going with full automation is hard. Yeah. How do you make it a science? Are there some leading indicators from a data perspective that can help you make the decision? And the part two is on the error rates, right? Because as you said one or two errors can really bring down confidence for customers. How do you help track. Right.

So.

People said don't talk about the future of a product offering but what we are doing is we are developing what we called internally auto QA metrics. What it does is like we're using large language models to look at how the person that got the response reacted to the response and look at like what is the sentiment? Is it like are they satisfied satisfaction? Like are we satisfied? Not satisfied. Resolution was the Ticket, like when you look at it, was it resolved or not resolved? Were all the questions answered? Was it done in a proper manner, like any grammar mistake mistakes, like that type of stuff. And when we use these metrics to indicate where the manager who's managing the AI should look at, because let's say you have 10,000 tickets, which one should you look at? And that's why we have these magical tools today. Large language models that are good at generating text, but they're also good at classifying. So that's what we are doing as well. And then you can create all sorts of graphs, you can create all sorts of interfaces for people to actually go and debug. So I would say it's like first, like you have a metric that says, okay, look at these 10 tickets and then you go inside each ticket and then you look at like what happened to understand like kind of the source of the problem and then correct it. So that's how the job is changing of the support agent. It's no longer responding to every ticket, you know, using macros anymore. It's more like I'm looking at the stats, I'm seeing where the problem is, then I go drill down into the specific thing. Oh, it used this help center article that says we're shipping to Canada and actually we stopped shipping to Canada a month ago, but I didn't update the this thing and the customer is pissed. So it's like, okay, now I know where I need to go and correct it in the public information. So it's more that type of work rather than there is, I think that's all. Thank you. Thank you all for coming.

Appreciate it. Sa. It. Please welcome to the stage jeff perry.

CRO@carta and brendan cassidy, founder@high5.

A little balancing act.

Let's stand, stand.

Here we are, Here we are. Okay, go.

Hi everyone, welcome.

Thanks for joining us. My name is Jeff Perry, I'm the CRO at Carta. I've been at Carta for just about six years now and previous to that I spent about 15 years at Oracle in all different businesses, everything from enterprise to SMB software to hardware. And then made a transition to DocuSign and spent four years there helping build the SMB organization and some of the verticals like legal and not for profit and then had a chance. Carta found me, I found Carta and it's been exciting to help build this company from at that time, roughly about 20 million in ARR to 400 million plus now and continuing to go. So excited to be here. And answer any questions that you might have about the journey along the way.

Yeah, my name is Brendan Cassidy. I'm the co founder of High5. Prior to that, I've been in the startup world for about 20 years on the sales and go to market side. So I was the first 15 employees at LinkedIn, Echo Sign and Talk Desk as the first VP of Sales. Great experiences and interesting experiences all and yeah, super excited to be here. I've been at Saster really from its inception. So I was actually Jason Lemkin's VP of Sales at EchoSign years ago. So excited to be here. Yeah.

Any questions?

Yeah.

So any questions? Step to the front.

I think you have to use the.

Mic just so everyone can hear.

Okay.

Hey.

Oh, it's working.

Awesome.

Can you understand me from my accent?

Yeah, we can.

Everyone's been struggling the last few days, so. Hi guys. My name's Nick Armstrong. I'm co founder of a startup called Chirp.

We're a co pilot for sales teams.

So my question is really if we think about outbound automation at the moment seems to be the hot space, there's a lot of people operating there.

How do you guys sort of see.

The, the role of the SDR in.

This world at the moment? Can you just explain to me how.

You see an SDR operating in this.

Current world with all these outbound automation tools? Yeah, well, we were actually just talking about this about a half hour ago. Obviously, like just not with copiloting and with AI and all the different tools that are available now. I, we believe that like that doesn't really replace the SDR organization. I don't think it ever will, but I think it will help us get more efficient in how we use some of these tools in the future. At Carta, we have sort of two plays. We have an outbound SDR function that I think will continue to exist and we segment it by type of product that you're selling or who the ICP is for the different business units that we have. And then we also have an inbound SDR function that I think we will continue to obviously need as we qualify. Inbound leads to that come to us through our website and through all of our marketing channels.

Yeah, I think, I think there's a misconception that the largest companies and the fastest growing companies are looking to replace all of their SDRs with sort of AI bots. And I think they're actually a lot more wary of this than people realize. And I, I know because I know them, Jeff knows them. It's actually a pretty small circle of VPs of sales and CROs. And so I think for me, for me AI is about how do you help salespeople? That includes SDRs, but AES as well. And I think it would behoove anybody that's selling sort of that play to that market that you talk about. How are we going to make your, how are we going to make your team better and more efficient and you can cover more ground. But I think as a replacement concept I don't think certainly hasn't succeeded so far and I have a lot of doubts about whether people want to be sold to by AI bots. But I guess we'll see that conversation play out over time.

Hi there.

Thank you.

My question is a lot of times when you're selling an enterprise SaaS product, it's not necessarily replacing an existing system that's already been put into like a line item budget.

It's taking little bits that might be.

Happening in a conglomeration of like four.

Free tools and other things.

So can you talk a little bit.

About how you pitch to carve out.

Something that isn't necessarily budgeted for so that we can get into like a.

Real project based selling cycle?

Yeah, well, so for me, you know, it comes down to value selling because you know we deal with a lot of what you're talking about in each of the business units. Oftentimes in our cap table business for corporations you're, you're rip and replacing from Excel or from a lawyer managing a cap table. So for us it's about the value in the software and making that hopefully more efficient and cheaper and eliminating any potential human errors that occur in the cap table. So it becomes about the value that the software provides to make that more cost effective. And that's how we approach it. And then on, on our, on our fund admin business it's a bit different because there's a software play to it and there's also services as a big component to that as well. So you have to be able to sell the value of what the software provides in making that more efficient, but also understand that there's a service component to how we deliver and service our customer base.

Yeah, I think every successful software company in history probably started out not selling a line item budget right where they were trying to carve budget out of budget that already existed. And so, you know, obviously you have to find pain, real true pain, pain that doesn't just exist for the organization but for the buyer themselves. So I've always believed when I'M selling, I'm selling, you know, pain risk management to a person and opportunity to a person and not necessarily to the business. And I think if you sell to the person, regardless if you're a startup or a larger company and you find something that's pretty personal to them, real pain for them, you're going to be able to sell to them irregardless if it's a line item budget. So I know this guy.

What's up guys?

John with Maestro QA Question for Jeff.

I've read somewhere on LinkedIn recently that one of your big bets is event.

Strategies for pipeline development.

Thank you for reading that.

Can you talk to me a little.

Bit about how does Carter approach events.

More broadly and what's the tactical game.

Plan to engage people at conferences?

It's a madhouse.

Yeah, thank you for the question and thanks for reading the post about about it. So I have to answer each question and sort of in three parts. It's kind of how CARTO is broken up in our business unit. So on the corporation side how we think about events, it's events like this and to be honest this isn't maybe the best top of funnel event for us because many of you are already customers, but it's a chance for us to come extend the brand and thank our customers and talk to them about other products that we have to cross sell. For example our total compensation product. That's very different than how we think about these event strategy which is more what I posted about in the fund admin space for venture capital and private equity. And so we've taken a different approach there. We've decided that we're going to do very high touch expensive events for higher pipeline value, for higher ACVs. And so we've started doing what we call CFO symposiums and bringing existing customer CFOs along with prospect CFOs together at exclusive private events because it's such a network driven sale that we have in that fund admin space pulling those people together and have our own customers sell on our behalf and talk about their experience with Carta we found to be most effective. So that's how we're using our budget as we move forward in those two businesses which is different than what we do in our cap table business like events like this. Yeah.

Thank you. Curious your perspectives on balancing sales and revenue and collaboration with is legal the things that work to drive the sales process in a faster way or slow it down and the things that help drive the revenue and like what doesn't work and kind of walking that line of risk because, you know, you obviously could go really fast, take a bunch of risk on and find out later versus too slow and the sale doesn't close. Just curious your perspective on the business side.

Well, I mean, that's best consulted with a lawyer, probably of which I'm assuming you weren't. And we aren't for sure. Although we probably have served as amateur counsel before in a deal, although we won't disclose where. But I think, yeah, I mean, obviously if you're a VP of sales in a fast growing company, you want Legal to be along for that ride, right? Where, you know, I mean, obviously there are parameters in place that you can't go over as far as like what type of risk or liability you can put into it. But I think legal can be an asset or a liability for a growth company. I've seen it. So when I was the VP of sales at Echo Sign, we got acquired by Adobe post acquisition and competing against Docusign, by the way, which is where he was. And post acquisition, the process and bureaucracy of Adobe on the legal side was. It became a massive liability for us where we could move super fast. And our CEO is Jason Lemkin, who everybody knows here and he was a lawyer. And so when we were Echo Sign, we had Jason like in this sort of office above us. If we needed a contract reviewed or redlined, I'd just be like, hey, Jason, review that. You know, and he could, and he'd do it in five minutes. And so as we lost that type of speed and agility as a company, it was materially damaging to us when DocuSign could out hustle us in the sales process and the contracting process. One of three or four things that I would say shifted post acquisition. The tide of where we were really beating DocuSign. We got acquired and ultimately Adobe slowed us down. I don't know if I answered your question, by the way, but I'll let him talk about it.

Yeah. So from my perspective, like it's the balance between like the legal part protecting the company, but also not blocking sales or slowing sales. And so at Carta, like our mentality from CEO down has been like, legal doesn't get in the way of sales. Sales. But we also need to be smart in how we use the legal team. So my.

So you're saying you're not negotiating your own contracts?

Okay, yeah, correct.

Good.

Exactly. But my recommendation is for us to like partner with your legal team and bring them into the deal earlier so there's not these Contractual surprises or MSA negotiations. At the last point of trying to get the deal across the goal line, you put that work in ahead of time to where it doesn't become the block of in the end. And I found the earlier we partner with our legal team and create that relationship as they're part of the sales team getting the deal done with us. You create that partnership and it allows us to all be more successful together.

Yeah.

I think dealdesk. I don't know if you guys have your own deal desk. We do, yeah. That's wildly varies in companies. But like I personally would love to see DealDesk be a pretty broad sort of partner with SIV and not the guy in the shop that says it's going to take us 72 hours to fix your car.

Yeah. And I think that's the important part of the partnership. And early.

Yeah. Hey guys.

Way louder than I thought. I feel like a trend in the.

Space right now is the increasing overlap between sales and marketing and the evolution of the GTM term in general.

I'd be curious what tactics you guys have seen for real efficiency on sales.

Collaboration with the operations and the marketing team, either for lead handoff, closing deals or anything across the pipeline.

Yeah. Another thing we talked about earlier this morning. So to me, I feel very fortunate. We have a new CMO that joined Carta roughly six months ago, Nicole Bayer. She's speaking at 2:00'.

Clock.

She is awesome. And frankly, she's a partner in this with me. And you know, it's our number together. It's not the same sales number to where we're competing and like they need to feel a little of the heartburn of the sales number and us all getting there together. And I really look at a couple metrics that are the most important things. Like there's all the noise out there and too much data. In my mind it's how many s zeros become true leads. So the conversion there and then how many of the true leads become opportunities to close. And you measure those two things and if your teams are working toward those same methods metrics, then you're in to win together. But to me it's like that. Again, similar to the legal conversation. It's a partnership between sales leadership, marketing leadership, and we use a theory called a trifecta model. And for us it's the. In each of the three business units that I mentioned, it's the sales leader, the business development leader and the demand gen leader. And those are the three most important people in inside of those business units. Making sure they're all driving to the same metrics. And we have that in corporations, venture capital and private equity. So there's basically nine of them, three pods of the trifectas that are all working together in conjunction with marketing and business development for us.

Yeah, I would say in 15 years there's been just a quantum shift from ownership of demand gen from what used to be 100% marketing owned sort of function to increase over a sales zone function, right. As it shifted from declining effectiveness of marketing sort of marketing demand. And then more and more of that onus was put on sales to own, right. From like an outbound sales development standpoint. And so, you know, I think the optimal scenario is where marketing has a true commit. Like for me, if marketing owned a lead commit, right, that was like sort of tied to growth into sales numbers. Other people want more. They want like pipeline commits or other stuff. But for me, if marketing would just own a lead commit, I'm fine with that. As long as there's sort of a definition of what is a sort of marketing generated lead. There's a lot of companies that are getting zero marketing level traction around demand gen 0. And it creates almost an impossible situation for a sales leader to sort of do that on their own, right? To own all demand gen and own, you know, sort of the total revenue number and all the rest. That's a. You're gonna have a hard time getting a great VP of sales to take that job. And by hard time, like a 0.0 scenario. John Parisi, right here is one of the best startup VP of sales I know. You know, he knows this as well. So he was a, he was an early leader sales leader for this company called GuideSpark, which had to me one of the best sales and best sort of outbound sales organizations I'd ever seen. No offense to any anyone there, but marketing never really contributed, right? So they were essentially on their own. And it's incredibly hard, almost impossible. But one of the things you see from that team is they're all good, right? Because they all had to, they all know, they all understand the challenge of owning that burden solely to where not only do we have to own the number, close all the deals, but we have to generate all the demand. That's almost an impossible task. So.

Yeah.

Thank you.

Yeah.

Hi.

I'm really short, so we'll see how this goes. But just want to know, do you have a favorite book or blog that you both read and like that you'd like to share?

I don't have a favorite Book, not a sales book, honestly. And like, you know, there's obviously all these like sales methodologies, medic value, selling, spin, selling whatever it is, in my opinion, like in a startup, certainly anybody that's going to fall, follow one methodology to the letter is likely going to fail, in my opinion. Now some people disagree with that. That's fine. I don't have any issue with it. But if you're a VP of sales in a startup, you're probably going to need to have your own philosophy and methodology. That's probably cobbling a lot of different disciplines together. That's my opinion. But no, I just follow whoever the latest hot LinkedIn content generator is that's 23 years old, never had a job, but man, they know it all.

So have all the answers. A couple things I would add we had. It was actually required reading several years ago for folks that joined carta. The essays of Warren Buffet, I think they're a really good example. Examples of like foundation of building businesses and like the methodology of how Warren thought about that. So there's maybe a recommendation. And then most recently, I've just been listening to a lot of podcasts. Bill Bench is a guy who's at Battery Ventures. I've known Bill forever from the Oracle days. Like, he interviews folks like myself and roles like CMOs and CROs. For me, it just makes me feel.

A little better saying you watch yourself on podcasts.

Weird.

Bill hasn't invited me yet.

Okay, yeah, we'll make sure, we'll make sure we get it. Get you on.

Maybe there's a little like comfort in feeling like, okay, some of the folks that are in similar roles are experiencing the same things and have the same challenges. So it feels good to hear them kind of say that out loud as well and makes you think about like the things that they're doing and how you can kind of optimize your org. So I do a lot of like Spotify GTM podcast listening now.

Yeah, I would say just capping it. I'd say any sales leader that's taken a startup from inception through some level of scale and success when it wasn't obvious when they joined is somebody that I'll listen to and I respect. And there aren't an infinite number of people that have done that successfully. But those are people that, like I said, it is a small circle. We all know each other and so yeah. Yep, thanks.

Hey guys, thanks for being here.

My question is around, I think account management in that function, expansions, renewals, mid cycle expansions, how has that changed in.

The last couple of years with a push towards efficiency.

And how do you guys think about.

It within your own organizations?

Yeah, it's a good question. It's a tough question and I don't know that anyone has the right answer. So I think it's about what is best for your business and that model. We have changed our model probably six times in the six years I've been.

At Carta and I think that's okay.

Like part of what we do is being fluid and adjust the organization based on the needs that we have. We currently have a model that is a new business, a growth and a csm. The CSM is responsible for the renewals and customer success. The growth AE is responsible for upsell and cross sell. And new business is obviously new business. That's the model that's worked for us most recently in the corporations business. I wouldn't be surprised if over the next couple months we do all the data analysis and figure out like do we need to be more efficient in some ways? Do we combine three roles into two? And that's what we do. We do it every year and it's a matter of looking at what's working and try to optimize on that and then figure out where you can best use resources in other places that maybe have more TAM or there are bigger opportunities for us to help grow faster. So I don't know that anyone got the silver bullet on that one, but to me it's like constantly tinkering with the organization to optimize for where you're at in that moment.

Yeah.

I'd say as account manager, it's a question of is it a revenue generating role or is it a customer success role? Exactly.

Because you have different commission rates, might.

Be easier to convert existing customers to upsell. Is that if you're going to.

Yeah, if you're going to do a true hunter farmer model that's extremely expensive. So LinkedIn. LinkedIn did that through their scale and I think they did it effectively. But they had unlimited resources to throw at it and were willing to spend in an efficiency era. I think I see a lot of people saying cutting short on customer success and just doing sales and support. Support sort of being a combination of success and support. I'm not saying I agree with it, but I see a lot of people doing it.

So thank you. Yeah.

Hi guys.

So I'm a new sales leader for a B2C company that's going B2B and.

Obviously comes with a ton of product challenges.

So I'm Wondering if you have any advice for sales leaders that are trying.

To quickly align with product and engineering.

And those kind of things.

Well, you said you're new to it so you can still get out of it if you want.

So what is the question exactly?

Sorry. Yeah.

How can sales leaders quickly align with.

Their product team on new go to.

Market, give them proof. Right. So you know, like if you have actionable feedback from the market or from who you view the new Persona or ICP to be, then that's super valuable and product will respect it, Engineering will respect it. If it's, you know, if you have a abundance of proof, then they really can't shut you out if you have one sort of sample size of hey, I know this 30 person startup and the head of product says I'm right, you know, that might not be enough but yeah, I would say go out and have those conversations if they haven't already. Yeah, that would be the best way. Proof. Proof always wins.

I would say obviously I'm big on sort of the team sale, whether it's legal marketing. I'd say the same with product.

Right.

And I think there's a couple ways you sort of force product to join calls with you and it helps them learn the feedback directly from the customer. But you also should be having product help you sell as well. And the more you get them involved and the earlier the better off and then I think it also helps for the future roadmap of product.

Are you the sales leader?

Yeah.

Yeah. So I could tell you everywhere I've been, I've had some pretty epic conversations with product and engineering around what I believed, you know, and so you better be willing to have have some uncomfortable conversations. Right. So I think at Echosign, Jason even made myself and the VP of product sign a e sign a document agreeing on features. Right. So, so that nobody could say later that you know, something was promised or over promised or whatever.

So.

Yeah. And by the way, that's one of my best friends in the world so it can work out in the end. Thank you. Yeah.

Hi there. I'm going to ask a follow on question to what someone had asked a little bit earlier and this is related to customer success support and sales.

And I know you both said something.

Slightly different so I wanted some clarification.

You mentioned that customer success teams eventually might go away and just sales and.

Support because that's new generation.

Yeah.

And I kind of aligned with that. So in any case.

But I also know that you mentioned.

That they are customer success managers and your organization are revenue drivers because they.

Take care of upsells and renewals. So how do you see that eventually converging and would you think that going forward sales tools will eventually transition to customer success and support? And if, sorry add on, if customer success actually goes away, who's going to manage the wholesale to renewal life cycle?

I'm building a product for that.

Which is why, which is why the question.

Well they have the resources to do it. So it's Carta. I don't know what your valuation is. What 8 billion?

Somewhere in that range.

8 billion, you know and so most startups don't have the resources to be able to sort of fund a true hunter farmer model. And so I think that's where you know, customer success people are going to look at it and say can I get something 80% as good for my support team?

Correct.

Which again I'm not saying I agree with that at all but I'm saying that is a trend in a sort of era of efficiency where everybody's trying to be capital efficient. Yeah, I'll let you go.

Yeah.

So one just quick point of clarification is our growth. Our CSMs do not do upsell, they just do the renewal. So upsell is done by a revenue growth team but neither here nor there. It's still like what is the function of the csm? And they do the renewal because they're part of mitigating churn and so if you create a great, great experience then you don't lose the customer. And so that's why they, they're attached to the renewal component. I do think there will be a point where it's like you have to decide if you have X amount of resources or money to spend to support an organization where, where are you okay making trade offs and so then it goes back to like how good is the damn product to make sure that like you don't have issues to where you, you need less TSM hopefully because the product solves itself and you don't have those issues.

Right.

And so that's where the is problem. Partnership with the product team I think comes into play and hopefully over time you would need less CSMs because there's less sort of support work to be done there.

Thank you, I appreciate it.

Thank you.

And the last one, my name is.

Lala, I'm a co founder of EdTech B2B software business.

My question is around hiring of a VP of sales.

Although I have read all the blog.

Posts by Jason throughout the years but.

We still failed two times miserably So.

I hope to get some advice, you.

Know, what are the tips on how.

To spot whether there's a fake during.

The interview or how to test it.

Or even if we hired in the first 30 days or somehow to spot.

That things might not be on track.

Yeah.

So, yeah, we talked about this. We talked about this. I know Jason, who you're talking about. And funny story, when I was looking to leave ORACLE Back about 10 years ago, I watched a, either a YouTube or a podcast, something Jason had written or a video he'd taken where his guidance was to not hire a VP of sales from Oracle specifically. And I was like, I'm that guy that he's saying don't hire, you know, And I like to think that, I.

Think he reached out to me is like, what the hell, dude? I'm like, hey, you know, like, hey, man, I can.

So for me, I ended up using it as a little bit of a chip on my shoulder to sort of prove that theory wrong that like, I wouldn't be the right guy. But I think like, you really have to get in. And we talked about this like, you know, do your diligence on who the hire is. And I think the most important qualifier of that is the reference checks. And you know, we do this all the time. He's asked me about candidates, I ask him about candidates.

Candidates.

So I think you really got to find people close to your circles and do your homework on the reference checks of people that you really trust and don't believe. Like maybe it's the VP of sales from Salesforce or Oracle or pick your big company, are they the right person that has the right intangibles and the grit and perseverance to be able to work in your fast paced, startup, nimble environment? Which is everyone's going to say, right? Can you be the big company guy and the fast paced person? It's not for everyone, but. But you gotta use your reference checks to be able to vet that out and do that very quickly.

Yeah.

Somebody that's vouched for by somebody that you trust sort of in network.

Right.

I don't know what stage of company you are. It could be an investor advisor. If you don't have those things, seek them out.

Right.

Like I get asked all the time, hey, Brendan, who should I hire as VP of sales? You know, like, who do you think is good? And I think who, you know, for us, who we think is good usually has more capital than what does their resume say?

Right.

Who is vouching for this person who is Saying that this is somebody, somebody that they would go to bat for means everything to me, like who's vouching for them. And that's informed most of my hiring in startups.

Right.

Where like you look at the first 10 people I hire, most of them are vouched for by people I know. And so it's going to be a long process. You're very unlikely to get strategic. Top candidate Jeff's not coming.

Right.

Most of the best VPs of sales are very risk averse and they're going to spend a long time vetting you. There'll be more vetting of you as a founder than you of them. And they're just looking for any excuse not to do it, by the way. So, yeah, that's how it's going to go. So I would try and create a council of people that can help open some doors and make some intros.

I'd also be okay with knowing again, you got to vet the person. But it's likely going to be a stretch hire. Maybe it's someone that was a director or senior director, but really damn good. And this is their moment to step into that role, to really prove that they can can expand and do more. And you got to be okay with that part.

Yeah. My number two at EchoSign became the CRO at Brex. My number three became the CRO at Rippling.

Right.

And so look for teams that did it successfully and go to that VP of sales and say, who reported to you?

Right.

On your org. Maybe that wasn't vp. Maybe they were director level, head of level that is eager for that kind of shot or opportunity and that's a pretty good way to do it as well. So.

Yeah.

Thank you so much.

Thank you.

All right.

Good job, I think.

Yeah.

Thank you all.

Appreciate it.

Sam. Ladies and gentlemen, please take your seats. Our show is about to begin. Sam, Please welcome to the stage Bradley.

Lawrence, senior director of customer success at Coursera.

How we doing? I'm Brad Lawrence and I'm officially overdressed for this conversation. But it is so good to be with you all today. I love the CS conversation and account management question we had before. I'm excited for the conversation today. Today I'm going to talk about driving grr. Renewal rate, turnaround. And I come with tips, but what I don't come with is all the answers. And I'm someone who is in it with you in what we do. I'm going to talk about some tips. What you don't see here is tips around prayer, long hours Anxiety, missing a vacation that's not here. But something is the reality of this. So I lead CS at Coursera, based in Nashville, Tennessee, originally from Kentucky. So you hear the accent come out a little bit. What are we going to talk about? You don't care about this, but I think this is really important for setting the stage of who we are as leaders and as people. Whether you're on the front lines or leading a team, who you are matters in today's world because renewal rates are tough, SaaS is tough, B2B is tough. And I think as individuals and as leaders, we need to think about who we are. So for me, while I had some sugary cereal this morning is one of my favorite things, Frosted Flakes, the hotel. As an individual, being caring about our people and our teammates, caring about our customers, creating operational process for yourself and thinking differently about account management, sales and customer success is going to be critical. So who is Coursera? I always hesitate to do this, but who has heard of Coursera? All right, keep your hand up if you're a user, okay.

All right.

And maybe an enterprise customer. Okay. If not, if you are you have any feedback, please, we can talk later. But I set the stage here to understand what Coursera is. So in the bottom left you can see what I do. My team is a B2B customer success team. We work with sales, we work with organizations, corporations, universities and governments. So if you think about it, we create opportunity for individuals in different markets. Our goal is to transform lives through learning. That can happen through our enterprise sales motion can also happen through consumer. So anyone you know who goes and signs up for Coursera and we have a ton of industry content and university content. I share this only to understand our go to market and who we are. So how do we turn around? Grr. From 47% I'm surprised sometimes that I have a job to 86% from Q2 of last year to Q2 of this year. I'll never forget I had a really great chill week planned in my in laws house, as chill as that can be. And I tell my wife as we're on annual fiscal we're approaching July 4th, like I think I need to work a little bit after the Q2 that just happened. So I say that just understand the reality and the journey that we're all on and what we can do together to learn from each other. So I'm going to lay out some problems. What were we facing in our North America corporate market? So this may be a market very similar to many of you and I recognize Coursera may be bigger than the organization you're working for. My hope is that all of these principles will apply to you in what you're doing now and in the future. We had weak deal management. Has anyone gotten to the end of a deal and you feel like there's a churn or the new deal lost? We start to ask questions. Oh, we didn't know about this step in procurement. They told us they had budget. We didn't get to the ultimate decision maker. I just felt good about it. We call that happier sometimes. And it's really easy to blame the team here. It is not the team's fault. It is our fault as leaders if we don't have the right process in place to have strong deal management fundamentals. We'll talk about what we did in a second, but I think it's critical for you to think about what is the operational rhythm that you want to have to manage your pipeline, new business or renewals that we were talking about today. The next is our roles and responsibilities. If we have any CSMs in the room or you've hired CSMs, they're a fun crew and one reason they love their job is they get to be the intersection of sales, customers, product marketing support and drive a lot of value that way. But especially in today's efficiency market, we can increase the responsibilities on CSMs and there are consequences of that if we're not clear about roles and responsibilities. We had that. If you work for a sticky business critical solution, count me jealous of you. I spend the last part of my career in a non sticky solution where it's easy to turn us off. And what we found last year was that as organizations got smaller, as online learning became less prevalent, maybe we had more return to office and we just hadn't focused on driving health and utilization of our product. What happens? Organizations say we just need as much. How many of you have an octa tile that you just don't go to? Maybe HR offers it for too many organizations because we weren't multi threaded and driving utilization. We saw CHURN because of that. Fourth, we had a lack of scale. Scale can be a dirty word and I think it is something that we really need to bring to a proactive leadership mindset as it relates to account management and customer success. We can call it digital touch, we can call it tech touch, we can call it throw right experience with the right customers. We had incredibly senior talent working on small customers. We had less senior talent working on more senior customers, more high dollar customers. This is a bad cycle. People are in the wrong roles, they're dissatisfied for different reasons. The senior talent doesn't be working with a small customer. The more junior talent might feel a little overwhelmed with that massive renewal, whatever that looks like for you. And we are overserving some customers and under serving some others. So I think really thinking about segmentation is critical. And last, we had some big deal fundamentals and I recognize these dollars might be different for you, but we had 18% of our churn driven by two deals. And again, you might have that and you know which customers you lose. And we need to be thinking about what are the fundamental accounts in deals that you sold that are likely going to turn next year or in two years. Because what does the product need to do to evolve and how are you going to get there? What is your process to make that happen? So I'm going to pause for a second. We're not going to go to the MIC apparently, but anybody have other challenges that they're facing? Yeah. Value realization. Absolutely. Others? Yeah. Long implementation. Yeah. Particularly the more technical things get. You can get to one year and the implementation's still going. Yeah.

Upskilling CSM on our advanced product.

Upskilling CSM is an advanced product.

Absolutely.

We can have some relatively generalists so good at so many things, but maybe they haven't had the depth of technical expertise. We haven't screened for that important talk about value realization. How are those CSMs driving value and driving enrollments? One more. Yeah. State of migration. Data migration. Absolutely. Learning content migration. Yes. When I get pulled into a technical situation for integrations, I am nervous because as a leader I'm not good enough there and I know we might have some. Some tech debt and some personnel debt that we need to fill. Those are all huge. And let's talk more about that and I will hang around to have more conversations after this. So what did we do? And again, there is no secret formula here. Brad's five tips. I am very worried about next quarter and the quarter after that because there is no such thing as smooth sailing in today's market. And what we do. First, we require joint execution plans for all of our renewals. If you've heard of maybe a customer action plan from Medic. We want to go every single renewal by renewal and talk about is there a JEP? We call them the JEPs, but it's basically a shared document with the customer that works back from the renewal date. When is the contract going to be coming in? Ideally well before the renewal date. When are we going to be having that final conversation. Procurement, when is the proposal due? What is everything that we're going to do as a team and who on the customer side is responsible for that idea? We have a champion decision maker who are also committed to getting this deal done. We also want to set targets for early renewals. Ideally you enter a quarter with over 40% of your revenue closed for the next. This is much easier in certain markets. This is much easier in I think at the end of the calendar year. But if we can set a dollar around it we can get customers working ahead and avoid any slip ups because we know how killer it is to have a deal slip over the course of the quarter. The next is roles and responsibilities. The ever present question in CS is who own renewals. It's very emotional. People get very upset about it and it is different based on the product. Set the stage of your company, what the customers need. For us right now it was right to remove the renewal responsibilities from the CSM and create an account management function. We had piloted this in this business segment and I think we saw the best results as a result. But we kept a revenue and comp incentive for the CSMs. I have worked in organizations I imagine some of you all have with a CST team is not comped on revenue and that is a recipe for disaster in my opinion because we turn in to operational CS teams that have different goals. Could you imagine comping your CS team on just utilization? The sales rep closes an amazing 3x deal and the CSM gets no part of that but has to deal with 3x the licenses to implement. I think it's critical with their shared goals but we don't comp them the same. I think it's about finding the right needs because again we ask the CSM to do everything. You're hiring people with a range of experiences who may not be the true revenue killers that you want in an account management role. Next is utilization and health. We have to focus on health. We have to get our customers using the product. We increase the accountability for CSMs there. I'll be very honest, we had gainside dashboards last year. We had health metrics but we didn't use them enough. We are doing that again. We also have comp tied to it and we're also having very honest conversations. I think we have honest conversation about performance management and are we operationally efficient. Lastly, I would say there is a CS and I think this is a SaaS tendency, particularly as you get outside of traditional AM and AE roles is to Cram everything in to comp. Well, there are some operational metrics that matter in non maybe overlay roles, supporting roles that don't necessarily need to be in comp. For us, that's how often are you talking to your executives and how often are you talking to your admins? Called that exec engagement and CSM engagement. That is not a part of comp. I don't want it to be. But it's a part of our weekly conversation of how are we performing. So surfacing KPIs making sure we're having the right conversations increases that likelihood of success. Next, as we talk about the lack of scale, we have so much power in how we organize our teams, organizing the ratios, we increase the customer we called scaled having a more scaled touch with some great talent that might be more junior in their career. And we increase the coverage ratios for our more strategic accounts. We also organized our books differently where we decreased the number of accounts covered by our most strategic CSMs but kept the revenue high so they could really focus. And we had slightly mid career talent with slightly larger books. We have scaled high volume, lower acv, but very high volume. We have growth or mid volume, mid acv, potentially high acv. Then last is strategic low account volume, high acv, more senior talent. Again, that's different for every company at a different stage. If I was at Corsair five years ago, I probably wouldn't have done it this way. But I think there's trends that you could probably take to each part of your organizations. And lastly, these big deal fundamentals, again, they might be different for you, but we have to have deal acceleration conversations. And again, at smaller companies you might be like, dude, we are always talking about every single deal and I'm the CEO. Well, as you scale an organization, you may not have that. And we need to have accountability for the deals we sell. We need to have strong executive engagement because the bigger you get, the easier it is to be in internal meetings and pointing fingers. Why didn't CS do that? Well, the SCRs didn't have enough leads for expansion. The support team dropped the ball. The first product doesn't work. The sales rep was this and that. We have to have alignment. We also have product alignment velocity. What does that mean? That means that we as an organization need to be responsive to our customers. I've been dealing with escalation this week from a large customer and it's different if they're large versus their small. But we have to make a change. And in a matrix organization, it's really Easy no to do that. So establishing a strong relationship with your head of product, a strong responsiveness but also a self awareness on the go to market side it's not all products fault. I don't slack them. Every time that things are wrong that we create a process around we set a roadmap we commit to. We also have some flexibility. So we have 10 minutes left. Would love to make this a conversation. What our challenges are you facing, what are you working on on and how can we support each other together. Those are the tips that I have. But excited to learn from you. Yeah, please. Yeah. Come on to the mic. Excited to talk more.

Thank you for this.

You mentioned earlier it's when your product.

Isn'T sticky it's easy to rip you out. At the end of your cycle we experience that too.

Like we'll have a three year deal.

And at the end of the three year deal it's actually pretty hard to keep them engaged because it's easy to rip out. So there are like certain features that are a must have in order to make your vitamin A pain pill so to speak. I mean ultimately we're struggling with that.

We think the system brings value but.

It'S still ultimately too hard to rip out.

Right? Absolutely. The first is integrations. You know can you integrate and what kind of non sticky solution is it?

Let me just strategic planning.

Solid.

Okay. Yeah. What kind of system can you integrate into them technically so we can get single sign on if we can be a part of their HRAs that is helpful. Now that to the point around long implementations can take a while and it can be frustrating to all of us. But if we can be a part of that solution that will both increase adoption and make that rip out process harder. The second is business process alignment. We are just an L and D solution of go take Coursera. If you want learning as a benefit it's fine. But if we are a part of an organization's strategy, if we are upskilling the data science team, if we are part of the leadership development training, we have pre work on Coursera that is more impactful and I'd say the last is that we have a team who obsessed about health and it's not just the CS team responsibility support cares about it, the sales team cares about it product. So those would be the three that come to mind.

Thank you.

Yeah. Yeah. Hi there.

Sorry.

Yeah, my company sells a lot to.

Mid market and a handful to the higher end of SMB like high eight figures. One of the things that we struggle with whenever you're adopting a new software to run different parts of your business process. Something critical is really getting that institutional buy in.

Right. It's not just am I paying money.

For it, but do I have enough.

Customer risk on the line where we're.

Going to have resources allocated to make sure that this project is important and a success.

We're having difficulty appropriately scoping sizes of like proof of concepts.

Let's say for a customer that's a.

$75 million a year business, if their entire deployment looks like a hundred thousand dollar, hundred thousand dollar investment, we would typically look at a proof of concept.

Say, you know, maybe it's 20% of.

The normal size, but a lot of.

Times that's difficult for that to be enough of an investment that it prioritizes.

Their, their investment in the success of the project. Could you talk a little bit about how you structure that to make sure.

That you give a look at maybe.

A proof of concept de risk, the.

Sale a little bit, but also set.

Yourself up for success?

Yeah, absolutely. I think establishing those deal reviews and having a kind of a joint execution plan is key. It's not just about how you operationalize but how you scope it. We also have a pre sales solutions consulting team and they can both map the content that an organization might use and look at the technical piece and really understanding how you get set up for success. And that flows into implementation. And so I think having a very strong, strict pre sales and implementation process, we actually won't let customers out of implementation until they reach a certain threshold. We'll keep the implementation manager on it. And I think that is key because when we don't do it well, you're at six months, nine months, they want their money back and then we don't, you know, go well from there.

Great, thank you.

Absolutely.

Hey, great turnaround. I had a question on cost, hard cost. You talked about more software dashboard board segmentation. Right.

Did you bring in more CS folks, account management?

What does that look like from the resources side?

Yeah, we, it is a tough market right now. We kept it neutral. Yeah, we kept everything neutral. We organized. You know, is this imperfect? I think we're also starting to feel the challenges of resourcing debt. To be honest with you. Our team is stretched. We have higher ratios than others. But I'm also not going to business asking for 2x of headcount. I know where we are and so I'm trying to do kind of more with less. I think that's just the reality that we're in based on our growth Rates. Every organization is different. One reality that we face is I think we as finance teams can often do like undervalued CS as a back office function. Once we face this we're like oh, we need more CSMs. You know like I think that can be this tension. I think it's really important to have that alignment to the CRO sales ops, finance head of sales. Ultimately if the head of sales wants more of you, it's a good place to be in. But I'm also. I want a small kingdom. I don't want a kingdom, but I don't want to be too big. Yeah, thank you. Great question.

Hi.

Yeah, thinking more on how you moved.

Renewals over to account managers. How are you comping CS DSM's revenue when they're not working on that renewal?

Is it based on implementation?

How does that work?

Yeah, absolutely. So we have a 7030 comp plan. I think that is a little bit higher variable than most CS folks. Half of that is a utilization, a health mbo. The other half is an NRR goal. So hypothetically it is the same as the account management teams. So if you spend X at the beginning of the year we, we want to get to 1.3x that will be your NRR quota. So any renewal and or upsell will count towards that goal.

Thank you.

So based on the book of business, you know it depends on mapping but that's the high level.

Hello.

Yeah. Hey.

Talking about enterprise deals, the kind of.

Deals that you lose that can really affect your growth for the year.

Yeah.

What's your take and your experience on churn accountability between sales and post sales.

Yourself being focused on cs?

Yeah, absolutely. It's everybody's responsibility and it's unique. I think that's it goes back to the R and R because it's really easy when sales just owns new business and expansion to be like they didn't do it right. Their team's not good, they didn't drive health and ultimately sometimes and you know I kind of mentioned earlier there's things outside of anyone's control, right. Funding goes away, companies goes away. It's a bad deal. The fundamentals there was as can happen with everybody more conflict last year because things were harder. And it's also though easy to say it's everybody's responsibility. I think we have to build it in to the process and sometimes it takes struggling there to do it. I think part of it is like making it more part of the forecast calls talking about the next renewals that are coming up. We're trying to talk about Q1 renewals.

Right now.

We're in Q3. We should make that a conversation. And I love when we have small group forecast calls that is everybody involved. We have the general manager, we have the head of sales, we have me, we have sales managers to kind of co solve. And I think as we are more in person, this is like a very small thing spending time together. So we're a fully remote company, which I really think is an amazing thing if you also are that way with love to learn from you, but having an off site investing a little bit of capital just to break bread because there are some people that I've been able to like, honestly hug it out with after maybe riding the waves of the reality of last year together. Thank you. Yeah, thank you. All right, I think we're all done. Any more questions? One more. All right, going once, going twice. Well, I'll be around. Let's hang out and hope you all have a great rest of day. There's an amazing session coming up after this with Glean and cs. I hope you stay around for it.

Thanks.

Sa. Sam. Sa. It. Sam. Sa. Sa. Sa.

Gentlemen.

What's your name again? My wife is very early. Please welcome to the stage AJ Tennant, VP of Sales and success at Glean.

And Lauren Kennedy, Head of Customer success at Glean.

All right.

Hi everyone.

Hello. How we doing?

I thought we said we were going to do a high five.

Oh, sorry.

Yeah. There we go.

All right.

Hi, everyone. My name is AJ Tenant. I'm the VP of Sales and customer success here at Glean. Really excited to be here today to.

Talk to you about how Lauren and.

I work together to help to 10x. And actually Lauren announced yesterday she's 100x the end user adoption of Glean. Just some context and intro. So prior to Glean, I spent eight years at Slack, helping to take that company from 6 million in ARR to over a billion. And then prior to that, I was at Facebook and Meta is what they call it today. But super thrilled to be here. But I wanted to hand it over to Lauren to have her introduce herself. But I wanted to share that Lauren is just one of the best, if not the best, customer success person I've worked with. She's a great human being and she's also just a badass at what she does. So over to you, Lauren.

Thank you, aj. I keep telling him that if he keeps calling me a badass, we're gonna have to have an official title change at our next career conversation. I will. Yeah, it Looks good. A little bit about me. I've mainly worked at very small startups, typically 100 if not first 50 employees building out the customer success teams. Spent three and a half years at Gainsight where I led different teams of CSMs and really kind of honed the CS craft. But this isn't just a journey about us, it's a journey about Glean too. So we'll give you a little bit of context to Glean. I do want to start with a little interactive poll, so raise your hand if you heard of Glean before coming to this session. Pretty good number. Keep your hand up if you've used Glean before in the workplace. Perfect. I'm coming for video testimonials from all of you after this. So thank you for making my job easier.

I'm going to go for anyone whose hand was down. Yeah, get you on Glean.

Perfect. But yeah. So a little bit about Glean. Just some context setting before we jump in. Glean was founded in 2019 by Arvind Jain. He was one of the first and a distinguished search engineer at Google and then went on to co found and was the CTO at Rubrik where he kept hearing from his engineers, I can't find the information I need to do my job. And so he got together with our co founders and founded Glean first as an AI powered enterprise search platform. We're going to fast forward a whole bunch of to 2022. When AJ and I joined there were, as he mentioned, dozens of customers, thousand, couple thousand users, couple thousand paid for seats. We're going to fast Forward again through 2023 with the advent of the AI Workplace Assistance as we all know them now. And we're going to then fast forward again to today and this week and really excited to say that at this point Glean has hundreds of of customers. We have, just like AJ said, 100x our end user adoption and we're approaching a million paid seats. So yeah, thank you for whoever did that. So with that we did want to share a little bit of a secret with you all. We have to be honest, Glean did help us write this talk and what's funny is I think I did this like a week and a half ago as we were pulling this content together and it already looks totally different. We've had multiple product with releases since then. We have had a full brand refresh and we announced a round of funding yesterday. So while things look very different in the world of Glean from a week and a half ago, it really did help us Help us put this together. But AJ I'll let you give a little bit more context.

Yeah. And I think we want to set the context before we share some of the learnings on how we scaled Glean and worked across gtm. I think number one is we were starting over two and a half years ago, three years ago, helping to define a very nascent category and really create the category of work AI. So that's number one. I think number two is we had no plg. So I spent almost a decade at Slack and I know there's some Slack people in the audience that I worked with. We had this amazing inbound motion where we had tons of freemium, tons of users and people that we could reach out to. We had none of that at Glean. When we were getting going, it was all outbound and it was all really true enterprise selling. And then the problem was very complex. When you think about what Glean does, we index all of your corporate data, we build a knowledge graph. And so that requires us to really understand permissions to navigate a lot on the security side and especially when you add in LLMs and all the change it creates for a really complex problem to solve. And on the last point I'll say is we were also selling and going to market in a situation where we were telling customers, hey, enterprise search is a need to have, it's a must have. But a lot of customers and prospects didn't see it that way. This is, you know, two and a half, three years ago, the tech Armageddon people were saying, no, this is nice to have. And the world's obviously changed a lot where work AI and enterprise search is now viewed as a must have. But that was the challenge we had. So what does this really mean and how did Lauren and I and the entire Glean team partner together to drive success? What we did was we took a very hands on approach as we evolved the motion. So early days when we were getting going, we had engineers, we had CSMs, we had AES in this pre sales phase as much as possible because we wanted to have a high touch motion that allowed customers and prospects to feel like they were getting that White Glove service. Now as we've evolved the motion, that White Glove service and that high focus on customer success hasn't changed, but the way we we do it has. And it's mainly because this category has been created. We now have case studies, we have market awareness and we have great things like yesterday where we announced a lot of exciting things. So Lauren, I'll hand it over to you to kind of touch on a little bit more.

Yeah.

What I really want to touch on is the bottom part here, which is the relationship aspect of the journey we went through with our customers over the years. So as AJ spoke to, we were incredibly high touch and we'll talk a little bit about that in a second. But the reason we were doing that was because we truly were learning alongside our customers. We needed to learn and grow together. And as we evolved and as we developed, we were able to start building that technical trust again. Because we were so hands on, the technical trust with our customers allowed the CSMS to start focusing on true success criteria, not so much the in the leads. And once you get there, you've earned the right to ask for that executive alignment. Once we have that the CSMs, we're really able to focus on driving adoption with our existing customer base. And this is where you're going to see the most kind of boundaries start to be set between the pre sales and post sales motion at Glean. And now we're in this world and going forward where because we've earned that trust, we've earned those relationships, we've built that adoption, we're now in more of a case study world. We've had that market awareness. And I know something that the sales team approach appreciates is that also helps. Every single prospect actually has to go through this journey as they go through the sales process. But having those customers having the case study is helping to shorten that deal time as well. And there's a great story I do love to tell. AJ knows this is one of my favorites. It's a little bit different. It was, it was a couple years ago at Glean. It was an existing customer and they got a new VP of it. And he walked in day one and said, I don't believe you that you fixed Enterprise Search. I do not trust the technology. I do not believe in the permissions. We're getting rid of Glean. Obviously there was a bit of a scramble. It helped that the employees at this company all threw a riot. But more importantly, the GLEAM team really rallied around this and went back to that high touch. We started the motion over with this customer, went incredibly high touch, built that technical trust, got that alignment with the executive and now he'll know who he is. Now he actually tells this story very happily on the multiple webinars and case studies that he's done with us. So it's not totally over that we're doing this, but it's much more a case by Case basis.

Yeah.

And I think the we'll segue into just emphasize again that as we were getting going and Glean as a product did not have any plg. So a lot of this was that high touch. So talk a little bit about what you did there.

Yeah. When we say high touch, we mean very high touch. We had a Slack channel with every single customer. Did not matter size or spend or anything like that. Slack channel with every single customer. And there were channels where it felt like, and it probably was true that like every single team member at Glean was in the Slack channel all the way up to Arvin, our co founders, our engineers, our sales team, marketing, everybody was in there and that was amazing. That delivered a ton of really extreme customer. Love it built that technical trust, especially because we had the engineers, the founding team, the product managers in that channel too. But as you guessed, not super scalable. It was breaking. It was breaking for us, it was breaking for the customers. And so we eventually had a moment. AJ, I think would agree with this. We should have had the moment earlier. Earlier, yeah. Than we did. But we eventually had a moment where we said we got to fit this and we got to change and we got to get creative about how we're.

Going to solve it.

And we As a team, CS&AJ and Sales said, look, if we can learn so much from our customers, they can also learn from each other. So we leaned into this idea of community led customer success where we deprecated 50 plus customer Slack channels. We threw them all into a Slack channel together and we said, help each other. And now they're answering questions for each other. They're addressing those technical concerns. The example I want to give, I know the font's a little small, but Matt Mangia is one of our amazing AI builder champions. @ a customer, he asked a question about, hey, how can we get public information into our Glean instance? You can see there was a nice thread going and it resulted in him and multiple other customers implementing this new use case. So they're really taking some of the lift from the CSMS and doing it in an incredibly collaborative way.

Amazing. Well, I do want to go back for a second to the security piece. Piece of this. Security is everything to us. How we in the pre sales cycle focus on delivering really powerful messaging around how we secure your data, how we ensure that the permissions align is a huge part of what we do. It's why customers, some of the largest banks in the world have deployed us. But one thing we learned was that it's not Just a pre sales message. It is a constant conversation in the post sales side with customer success around how we're evolving and how we're meeting the needs of customer success when it comes to security. So one example to share is about two and a half years ago, this is before my time, one of the largest security companies in the world was a customer of Glean and they said to us, hey, we bought your product but we don't want to deploy it. We don't want to deploy it because data governance is a nightmare. Our permissions inside our company, we have Google Docs that are shared with everyone. We have Salesforce records where people are putting in, you know, putting in credit card information. We have open text passwords that are sitting in JIRA and Confluence. And what we turned that into was let us learn more, let us understand. And over the course of that collaboration we actually turned that into a positive where now our product has features that enable customers to actually do that scanning to address the data governance issues they have, which has become a huge asset in our go to market motion. Every company that's deploying AI needs to be thinking about data governance. And that's been something that, you know, was an issue in our deployment. Now it's something that's an asset that makes us really valuable to companies. Speaking of just trust and building that trust security in our products, number one. But I think to take a step back and to really hit on what Lauren and I and the majority of the early folks focused on was culture. And so defining our culture. When I joined what I did with the TENSO people we had in the GTM organization, I brought everyone together. I said, hey, we're going to do an off site. We're going to align on not only the value that we want to bring to customers, we also want to align on our personal values. And it's not surprising that, you know, you can see empathy, humility, grit as some of the personal values that we wanted to align on as a team that would bring us together. And so that was one of the really core things we did, was focus on building trust in internally the other piece to it. And this is more of a kind of evolution of the culture, collaborative tension. This is, I think everyone knows that culture is not something you just defined once three years ago. It's something you evolve. Two of the values we have at Glean is make it better and make it together. I started to notice over the last like two, three, four months that people were being more just make it together, collaborative, but they weren't challenging and they weren't creating this pushing people to do better. And I have this great story where two weeks ago, I called my CMO and I created some collaborative tension. I said to him, hey, we have an opportunity to get better. I think we need to improve our video content, our marketing, and if anyone has seen the marketing that's come out in the last week, should be impressed with work AI for all. And that's an example of how collaborative tension is something that we're talking about internally. The other part to this is hiring people that are comfortable with change. And I know that's trite to say, but I think it's important that you and we did in the interview process really look at, okay, what are the drivers of this individual? Where did they have challenges in their life and in their career? And how did they navigate those changes? Because having people like that inside our company that can navigate change is really important. And last but not least, it's the one that my team will laugh about. Creative problem solving is number one for me when I look at candidates and people to bring into these early stage companies. How you can think on the spot, how you can think creatively, and problem solve is something that all of us assess for in our hiring. That helps to drive what we do. One of the things in the culture we have is I like to take selfies a lot and send it out in slack channels. So I'm gonna take some time now and do a selfie with everyone here. So come on, get in here.

Bear with us.

All right, raise your hand.

Say hi.

Come on, say hi.

Hi. All right.

All right.

Over to you.

Yeah, Now I have to follow up on that. But look, this one could not be more timely for Glean. This has been a sprint of a week. We've had major announcements yesterday. It is exciting. I was telling AG earlier, it actually feels a little bit like an alternate universe where everybody's very excited for us. And that's amazing and wonderful. And inside, we're still dealing with support tickets and making sure that customers get onboarded on time and making sure that deals are getting closed. So it's been a crazy week, but it's been a sprint, right? This is really exciting. It's a wonderful milestone moment for us. But as we build the team and as we grow, we need to be thinking about sustainable growth. And so really viewing this as a marathon is just a critical piece to the GTM culture. And then the last one is knowing the why that everybody's joining and making sure they're Achieving it. I also think, really importantly, and this is something that I learned from aj is assessing during the interaction interview process not just why they want to join your company, but the why behind every decision they made that got them to the conversation with you. It's been a wonderful way to just one, get to know the people that you're potentially bringing in, but two, really understand their why. And then as leaders, it is our responsibility to constantly revisit with our team members and for them to revisit themselves and with their peers and back to us as well. Like, why are we all here? It's not easy. So you've got to have a why and really make sure that people achieve it. That's why you join a startup is to achieve your why.

Amazing.

Well, we. We're going to create some space for Q and A here in a second, but hopefully we can round up with just sharing some key takeaways. So I'll hand it over to Lauren to hit on the first, the first and second, and I'll close with a third.

Yeah.

So this was our journey. Right? These were the pieces of context, the challenges, the opportunities that were specific to us and Glean. But I would say taking the time to figure out what that is for your company is critical. We're not plg. Your company may be, it may not be. You really just need to take that time to get to know your business and what those unique contexts and items are. And this one I obviously personally love. But making customer success and everyone priority. We talked about the high touch that came from our early days with everyone involved in in Slack channels. But that needs to continue. That doesn't change because we got rid of 50 slack channels and got everybody together. We are still everybody's focused on the customer success up to Arvind to our co founders. Everybody's talking about the customer success. In fact, we have a Slack channel. That's my personal favorite favorite Slack channel that's just customer win moments. And it's just a channel where people send in screenshots. Engineers will send in screenshots of text messages that they got from friends as CSMs launch new customers, they'll do screenshots of the celebrations in internal Slack channels. It's just a space for us to all see the value that comes out of Glean and the success of our customers. So if you don't have one of those channels, I recommend getting one too and making one. It's super fun, but I'll let you take it away.

Amazing. Yeah. And just to close, I Think when I think about Saster and all of you that are attending or like listening, you're all in your journey with an amazing product that you're trying to build and launch. Create your own new category. And I think you know that journey is one that takes a lot of hard work and I hope some of you will come up and do a 10x or 100x presentation next saster. But I think when you're building these products and when you're driving new categories, it's important that you don't lose sight or discount what it means to have a really good culture and the trust inside your organization that has definitely helped glean be successful. And I would urge you all to don't discount culture and trust. And hopefully folks enjoyed this talk and we'll open it up for Q and A. So I do want to just do a round of applause to sastr, especially the event speaking team, Caitlin and Taylor who helped to coordinate with us. And so just round of applause to the Sasser team. All right, we're going to open it up to Q and A. Folks can just jump to the mics and ask questions.

Come on up.

First of all, thanks a lot for the talk.

It's very inspiring for JNai Power. Like a enterprise search and QA systems.

We sometimes find that there are unique obstacles for customers to use them well. For example, certain users don't know how to ask good questions. The data qualities and availability really varies among different companies, organizations and even departments.

So my questions are what kind of.

Like major gaps do you guys find between companies or customers are able to.

Promote and utilize the products well, comparing.

To those who are not?

And what kind of data and signals.

Do you guys use help with the observation and analysis?

And lastly, what kind of actions do.

You guys take when you find that.

Certain customers are really struggling using your products, especially in early stages?

Yeah, I think there's two answers I'll take on this one. I mean number one, yesterday we, we just launched a prompt library. So a big part of how to get value out of work AI and genai is how you are prompting and asking questions and so enabling more of the builders and folks to sort of give some templates to people to ask the right questions and to customize the responses is one way we're doing it. But from a kind of hands on kind of CS approach. I think it is about. We're, we're looking at this as helping people with AI transformation. So we do need to be there to share best practices, connect customers with other customers and do it in a way that is truly like kind of blocking and tackling, like getting on the phone, understanding what's working, what's not working, and taking that feedback back to product so that we improve the experience for customers.

I'll add one more thing on there really quick to. To your question of what are the companies that are being successful at it doing compared to others is. And I wish I could name this customer, but we have a customer that has a. They took a team member and he's basically his entire new job is defining their AI strategy as a business. He's reporting directly to the CEO. He now has a team of a couple of folks below him. And that's entirely his focus is just how are they implementing AI throughout their organization compared to some other companies where they don't have have quite that much focus on it. It's a bit of this like we know that we need to be using it mentality. But if you can have somebody that has just really specific focus on how are we using it within the business, that's when you start to like just go off the charts.

Thank you.

Yeah, welcome.

Thanks for being here. What rules of engagement would you say.

The two of your organization have put.

Together that are critical for your success to better the relationship relationship of CSMs.

And AES or account managers?

Yeah, I think one thing that helps is so I oversee both sales and customer success. And so I think having that sort of a centralized leader does help, but that's not the answer to every organization. I think it comes down to, you know, evolving to the needs of the business. Meaning early days, the rules of engagement we need to be loose on because we were still defining the category. But as you expand and as you get more product market fit, it's about what does the customer journey look at, look like? What are the needs across that customer journey. And we realized that hey, we wanted to have more specialization on the pre sales with AES and SEs. When it came to post sales. We have a services organization, an implementation and then customer success and support and creating clarity for everyone on what, what they need to do in that journey is something that we don't have the perfect playbook right now, but it, but we've evolved it and now have that. Those roles that speak to the journey that we're on, which is much more mature than we were two, three years ago. But I don't know if you want.

To add three months ago.

Yeah.

If we're being honest, I have a really cheeky answer to this and Part of it is we tell everybody glean it first. So the team members are actually using using Glean to collaborate a lot. But AG is exactly spot on. It's that every single process and rule of engagement that you make is going to be iterated upon over time. And so it's just about the ability to move it and move it forward.

Thank you. Yeah. Hey guys, thanks a lot for the talk. It was really enlightening, very helpful. My question is that for a product like Gleam, you would have to have.

Buy in from everyone want in the.

Customer'S organization because entire leadership team, executive team, everyone needs to be bought in.

On the fact that, okay, our entire organization is going to be using a.

Product like Lean for enterprise search. I'm the founder of a similar kind.

Of a product where we need buy in from everyone.

So my question is that one of the key problems that tend to happen.

Is that during the onboarding process you might have your champion who dropped off for whatever reason left the company, whatever else.

Or you like that it person's example.

Right?

So my question is, as you went.

From fewer number of companies to now.

Hundreds of thousands of people who are.

Now using Lean, how did you scale that up? What were some toolkits or some things that work for you guys? Wherein.

And did you involve sales to once.

Again do the selling when something like that happens, how did that work for you guys?

Yeah, maybe I'll start with that one and then jump in. Yeah, so definitely it's a mutual effort between the sales and the CS teams. I think ultimately what you're asking about is a little bit of multi threading and that's a big focus that we've put on in the last six months where it's something that we track, we track in Salesforce. So I can track all the different multi threading meetings that CSMs are having. We actually do a spiff off of it every quarter as well. So the folks who have the most multi threading meetings get a nice little prize at the end of the quarter. And we have, I mean there's so many ways now that I'm thinking about it, we also, we have a channel called Everybody does BD where it's Everybody does business development. It's this Slack channel where anybody can go in, in the company and say, hey, I have an issue with XYZ Customer or XYZ Prospect. Lauren, I see you're connected to this person on LinkedIn. Can I draft up an email for you to send to them? And it's something that's Just really, again, built this very collaborative motion around doing that multi threading and that business development if it's on the sales side of the house. But I think that's it. I think it's having a focus on multithreading, making sure at the handoff from sales to cs, you're getting the contacts, they're getting introduced. I think really importantly too is that they're getting introduced to whoever's on your post sales side from an executive level and then putting a focus on multi threading from day one.

Yeah, I think the highest leverage advice on this is executive alignment. And because if you have a champion, usually champions can vary, right? There can be a champion that's not an executive, but even if you have a champion, you need to have an executive that is that person, that's that point of contact. And we have this concept of there should be three points of executive engagement that enable you to de risk in scenarios like that where a champion leaves. But at least you've got connections into multiple executives. Executives. And I know we're at time, so we're asked to wrap up. I don't know if we have time for one more question.

One more question.

We'll be standing over to the side and we'll answer some questions. If anybody has any additional ones, we'll.

Wrap there and we'll step over here and answer your question. But I just wanted to say again, thank you to everyone here and appreciate the time and attention.

Take care.

Thank you.

Sam. Sa. Sam sa. Please welcome to the stage Steven Hsu.

CPO at Calendly, and Jess Clark, Head of User Experience at Calendly.

Hello.

What's up, y'? All? My name is Steven Hsu. I am the Chief Product officer of the beloved scheduling platform that is Calendly.

And I'm Jess Clark, head of User Experience at Calendly. We're building products to help it you be easier to meet people, achieve your outcomes and close deals. Before this, I led design teams at LinkedIn. One of the first projects I had in AI was building recommendation products for product onboarding.

So all that recommendation AI work that you're all using LinkedIn on LinkedIn today connected with each other. A lot of that was Jess's work. Myself, I've been in AI working on AI for about eight years now. I've built AI chatbots before the current wave revenue intelligence products and even AI developer tools and accountly. We are experimenting a lot with AI, not just within the product, but across the entire customer experience. And that's what we're here to talk to you about today, how we're leveraging it. What are the best practices that we have observed out there, as well as things that you probably should not be doing. All right, so let's get started. Actually, one thing, by a show of hands, may I ask if you've ever clicked on a Calendly link before? Can you raise your hands? That is everyone all right? Thank you very much for using our products. So you all know that we removed the difficult back and forth of email scheduling, right? That was the pain point that we solved with scheduling links. If you asked our founder, Tope, what he's the most proud of when it comes to calendly, is that he changed the habits of people, people all over the world, 10 years ago. Now, there's a lot of. There were a lot of companies doing it, but Calendly made the mark. We changed those habits. And that's what we're going to talk about in terms of AI, how to change habits of our customers across the entire experience.

And beyond that, we're also just product builders, really passionate about building products that help people be more successful and productive every day. AI truly has the power to transform the customer experience within an app, across apps, and in every way that products overlap with our human experience.

But first, let's talk about the current climate, because I'm sure all of us have been saying, oh, the pace of innovation of AI is so fast, so fast changing. But let's for today just be on the same page and I'll just describe what we are seeing in the current climate. We can all nod our heads and say that's where we are today and tomorrow will change. There have been many waves of AI over the past few decades, and many of us have seen it. But what's different about this wave? Well, I would say for one, consumer expectations have shot through the roof. Consumers all over the world are adopting AI ever faster than ever before. As a result, their expectations, when it comes to simple, intuitive experiences with your company, are up there, higher than ever before. Number two, with these macroeconomic conditions, those budgets are tightened, which means they have to make sure that any AI tool they adopt has to have high roi. Shelfware is not going to cut it anymore. I used to work at a big company before shelfware just sat there, right? And that was basically what SaaS was for a long time of the last decade with AI tools. If it's not working, if it's not being adopted, it's being thrown out the door. And so there's a big challenge and transition we're doing right now in our industry, which is beyond the delight moment with AI. How do you get that retentive experience not just within the app, but everywhere else? Now, remember several years ago, all this talk about the consumerization of the enterprise, we don't hear those words as much anymore, but it's still relevant. It's even more relevant because AI now is a part of that.

And now user experience, which was considered a nice to have before, is really critical to the buyer journey. So when buyers are thinking about what AI products they're going to bring into their organization, they're really thinking about that user experience and making sure there's a clear return on investment for bringing that product in and making sure that their organization is productive. So AI acceleration is fanning the flame of product LED growth and changing the way that we all think about SaaS. Now, as you navigate that narrow path between AI boom and disillusionment, how can you make sure that you're building products for your customer in a smart and effective way? So we're gonna give you three takeaways today. First, prioritization. Prioritize the problems that actually matter to your customer and the experience. Make sure that you're instilling confidence through the entire journey in and outside your app and guardrails. Be aware of the things that matter to your business and the things that impact your business specifically.

Let's dive into the first one. Now, I've been in a lot of AI experiments over the several years and one of the things I've observed across many of those experiments is it's fun to do those things, but very quickly you lose sight of the customer problem that you're solving. Because I mean, with generative AI LLMs everywhere, you can create some amazing experiences, but very often the risk is that it's not aligned to the problem that your business decided to solve. So it's important to focus always with first principles on what the problems you're solving for your customer align it to your business. One of those key areas, generally speaking, is around improving the productivity of your users. What is the deepest problem that your customer has that you're solving for? Calendly, of course, we're solving the problem of scheduling a meeting. We want to get meetings on the calendar so you can close deals with your customers, so you can close candidates, you can connect with partners, VCs, etc. Right. And once you get that in on the books, there's a higher chance of closing business, which is ultimately what you're trying to do, you've got to focus on the productivity of your users.

The key here is to deliver real value. Make sure that when you're building your product, it's actually hitting that bottom line of productivity for the team and for the organization.

There's actually, there's a recent Menlo VC report that states that enterprises have failed to deliver on the promise of AI so far. And that's the transition state we're in so far. And the reason is that they have failed to create the workflows and habits in our daily lives. There are some tools out there are obviously very successful right the product, previous speakers up here. There's plenty of tools that are actually cracking it, but there's a long tail of companies out there that are still experimenting. And it's so important though that beyond that delightful experience, what is the thing that's gonna stick with your users, not just in the app, but everywhere? Number two, fast tracking Personalization Calendly is a very well known PLG product led growth company. Personalization matters so so much in PLG companies and it's going to matter more in general in SaaS as the consumerization of enterprise continues even in the SLG world. But with AI, the way we're thinking about it is transformational beyond just the initial onboarding phase in terms of personalizing the experience to our users. What can we do beyond the app in the website, on support channels and in such a way that we can help them discover advanced capabilities that they didn't know about before. We are about to launch a couple of new capabilities in the next few months that are going to be transformational for our users. That's going to provide accretive value to their entire meeting automation experience. To do that, we're going to need that personalized experience as they discover that new value and we will become a multi product portfolio that they will be able to extract value from.

So as you think about personalization, consider how AI isn't just a new product but can actually understand your customers context. So use AI to understand who they are, what they're trying to get done and what they need to do on your product and make sure you're using AI to surface those recommendations and help them streamline those workflows.

Yep.

Third, solving for a frequent use case. We're saying this here just to describe low hanging fruit. Now all of us here have been scheduling back and forth for a while. Even at Saster, we're probably scheduling meetings with partners, investors and the like and that's really tedious. If you're not doing it with some kind of automation or AI assisted tool, right? That's the low hanging fruit that you should be thinking about when it comes to AI or even just automation. How do you focus your customer on. Well, let me put it, put it this way. How do you get your customers to focus on higher order, higher value problems they can solve so they don't have to deal with the mundane repetitive tasks. None of us like that.

Right?

Identify those use cases that happen on a frequent basis so you can throw AI at it and take that burden off of them so they can realize that they don't have to work on lower I work anymore. They can focus on the higher value work. And actually let me just describe this, this internal use case that we have here and I'm sure there's many companies here that are doing the same. But I would add this extra dimension, which is calendly is basically a B2B company with a ton of B2C dynamics. We have 20 million users all around the world. We have mom and pop shops on one side like masseuses and therapists and agile coach consultants on the other side. We have financial services companies, Fortune 500 companies and top universities around the world. How do you think we make sense of all this data? How do we know which customer to focus on and how do we map that to our product strategy? Well, I'll tell you. Historically it takes weeks and weeks of data gathering, segmentation interviews to figure out what these companies are doing with their automations and their use cases. But with AI and a strong data architecture underneath it, we're able to automatically gather these insights, synthesize it into our in a mapping to our product strategy and our roadmaping process becomes much, much simpler.

So as you think about building an experience, the best experience for the problem that you're trying to solve, you need to make sure to instill confidence in that customer journey. End to end in the app and outside the app. So we're talking about four ways that you can do that today. First, make it visible when it matters and invisible when it doesn't. Research is already showing that people are getting exhausted with the term AI. When you think about going to market and your first instinct is to put an AI tag on it to make it purple or maybe add the shine emoji. Really consider what your customers are looking for from you and what they expect and if you can deliver on that expectation. Even if you think about your own experience over the last couple of years with AI, most of the time you did not know that it was AI powering it in the background. There wasn't a tag that told you these recommendations were coming from AI. It just simply was easy to use and helped streamline, streamline your workflow. Next, make it integrated seamlessly, not as a parallel experience. So when you think about building an AI product, think about the product that you already have, the experience that your customer already loves, that you're giving to them, and don't build something completely separate. Now, we can talk about separate in a separate product, or we can talk about separate as in a parallel experience within the app. So having a separate experience in the UI where they're engaging in some kind of AI chat and then they're also trying to use your product, this kind of parallel experience can really cost you, and it can cost you in a couple of different ways. First, it can cost you in just tech debt. Now you're building for two things at once. It can also cause you with customer load. So they're really trying to have that cognitive load of understanding your. Your product already. And now they're trying to relearn a different part of your product in a different way to get things done. And also it can lengthen your time to learn. So as you're trying to learn, is this AI product or this AI feature even successful if you're building it separately or in a parallel experience? Now you're trying to get product market fit, and it takes you longer to really know if it's actually making a difference with their workflow.

Actually, can I jump in here? Actually, I'm going to do a second question here. Who here has built a chatbot experience in their product or their company has? All right, that's still a lot of people. And I'll bet many of us who built those experiences failed to get them to stick the way that the original, the core product was able to do.

Right.

I'm going to be vulnerable and share a little bit about our experiment through this journey. About a year ago, we built our own conversational scheduling chatbot. What ended up happening, or the way it was built was a user comes into our app and says, hey, I just want to open up a few times next week and share a link out to a customer. It worked flawlessly. It picked up the right times over the next couple weeks, produced a link, they could share it out. But then after a few weeks, that user never came back to that experience.

Why?

Because the original way was way more efficient. It was really cool to be able to tell an assistant how to do it. But the original experience was so much better. That's why our users are still retaining our original features, but not this conversational experience. So we went back to the drawing board, working on some new AI capabilities. But it's important, like Jess said, that you're not building this parallel experience just to replace an existing one. You really have to be thoughtful about the entire user experience in the app.

Yeah, it was a great learning for us and that's what we're sharing with you today. I think a really key learning in that was don't fix what's not broken. So if you have something that's already working with your app and people find it as delightful and it's working for them, don't try to add AI on it to try to fix something that is already working for them. So next, be clear on what can be done and what can't be done. So when you're delivering your AI product, make sure that the inputs and the outputs are really clear to your customer. Don't make it really fuzzy for them to understand how to engage with this product. Put yourself in your customer's shoes and start using the product and make sure that you're getting success out of it as well. When you input something, do you get the output that you expect? So make sure that the first thing that you bring to market is predictable and consistent as much as possible. Now, when in doubt, there's always something that's going to go wrong. Make sure that you're building in those mechanisms to give them and communicate those error states or when there's misinformation so that you can continue to build that confidence and trust throughout your experience. And lastly, make it easy for users to achieve what they were already trying to achieve, similar to what Steven mentioned. If something's already working for them and if they already get the job done, don't add friction to that experience. Make sure that what you're adding isn't just adding in the name of AI innovation, it's actually making it easier to use. It's actually making it smarter, safer. There is some value add that's happening so they can make sure that they they're getting their job done and you're not losing sight of their job to be done just for the sake of innovation for your business.

Yeah, just to tease a little bit here, we have been looking at the jobs to be done outside of the core scheduling use case. What are our users doing in addition to scheduling? We all know the answers. We can probably just offer them up in this room. But those are opportunities for us, but also for you all to think about adjacent use cases that you can take an AI first principle approach too. And kind of on a related note, there is a report out there that says that 86% of customers remain loyal to your brand retentive when the products provide onboarding and continuous educational experiences. Most products out there today have onboarding initial onboarding flows. Some of them are personalized, but there's a drop off because it's been really difficult to provide the additional the continuous education experiences to discover all the other advanced capabilities of your platform. Whether it's an upgraded experience or a new product, AI has the opportunity to gather your customer profile data, their behavioral data, what features they've used, what they have not used. So you can provide a contextualized experience for continuous education. It's just a human experience too, right? If you had a consultant with you or a salesperson helping you discover other products as well services, you would be much more likely to adopt those solutions because you become familiar with them. If you provide an AI solution that does the same, you have a great self service motion that you built. This is an important point because there have been many failures that we've seen out there. When this does not happen, it's an organization wide effort. By organization, I don't mean a department, I mean your entire company. I mean your leadership team has to be in lockstep on a holistic strategy and approach for your customer experience. You do not want to be shipping your org chart. Whether or not it's AI. You should not be shipping your org chart. But with AI it becomes much more risky. And the reason is think about the times when you've interacted with an airline. You talk to a support rep, someone at the gate, someone like the chat experience. They're all saying different pieces of data or saying different things to you, pulling the wrong information from you about you to create a disjointed experience and the NPS for that airline goes down. The same thing will happen with AI if you, your agents, your AI agents are not speaking the same language with the same data. So start with the great data architecture so that your AI agents, no matter where they are, speak the same language to your customers.

At Calendly, we're doing this by bringing together a lot of our help content for our customers. So we take everything from our help center and different AI chatbots and we bring them together. We're also doing it to streamline all of our content creation by building custom chatgpts to make sure that we Ensure that things are consistent, have quality at scale, and have personalization for the things that we're bringing to our customers. As an example, we have a chatbot for ICPS to make sure that a Persona is getting what's useful for them. We also have a GPT trained on our style guide to make sure that we can have a consistent voice and tone. And an executive style bot that can not only talk about the product features, but can talk about the business use case of using our products.

All right, we're all technologists here. We all love what we're building and shipping for our customers. But at the end of the day, everyone here represents represents a business. We're all running businesses. And so it behooves us to call out a few key tenets and examples that should always matter even when we're thinking about the cool things AI can do. Number one is value. We always have to make sure that we're shipping the right customer value to our users. And we're going to do that by measuring it. We're going to measure the success or failure of our applications. I'll give you an example of how we do this. One of our top KPIs is what we call meeting intensity, which is the frequency with which meetings get done.

Right.

The more meetings booked by our users, the more users pay us money or upgrade. But there is a world where we create AI that actually reduces the number of meetings scheduled, but helps them get to an objective faster. Right, that actually hurts our meeting intensity KPI. But it's a conscientious decision we have to make. And that's the kind of conversation we leadership should be having everywhere. Cost. Generative AI is expensive. Costs are going down. But even as we transition our users to new habits, those costs will stay up. And we have to have those conscientious discussions about are we going to pass through those costs to the customer? Are we going to absorb it in our margins? Data, performance and quality matter more than ever. Have the architectural discussion on how to support the business. It should not just reside in product and engineering. It has to be be supporting the entire company. And lastly, your users, your customers, they have to have a single unified engagement experience no matter what happens behind the scenes. I used to work at an integration platform company. There are big companies out there who spend enormous amounts of money just integrating the data in the right way so the customer's experience is consistent across any surface area that they interact with the business.

And I want to talk about cost real quick. Make sure that when you're looking at this back to being a return on investment that your investment is not outpacing your customers need. When that happens, you see examples where they might have sticker shock when you're moving up in pricing upgrades and they do not want to pay for the thing that you found valuable to build that they did not find valuable to build. So we talked about three things today. We talked about the prioritization, the experience, guardrails. So what's next? We want to make sure that we measure success and adapt along the way. So a couple things I want to leave with you today. Define clear metrics. Make sure that whatever you're shipping, like everything else in AI, you are defining those clear metrics. And make sure that you are measuring that consistently. You can look at something like user engagement. Are people using the products that you're shipping? You can look at something like task completion. If you're adding in different AI features as actually helping them complete that task, or is it harming it? And you can look at customer churn and engagement. When they're using your AI products, is it actually helping them be a stickier product or is it actually driving them away? Next, make changes often. Make sure you're retooling your prompts, you're looking at the back end, you're looking at the experience. You're doing user testing, you're doing AB testing, and you're making sure that your product really does truly have a product market fit with your customer. And then of course, feedback, feedback, feedback. AI is new to some of you, and most importantly, it is new to some of your customers. So make sure that you're putting mechanisms in place to gather that feedback really early and often. Make it easy for them to give feedback on the things that you're shipping so you can be honest with yourself to say, is this actually have an investment that has a return for my customer or does it not?

So let's wrap this up back to the three big things. Prioritize the problems that matter to your customer. Customer first, customer first, customer first. Number two, when you're describing building this experience, we cannot impress the importance more than just saying that it's important to make sure that when your customers are interacting with your AI generated experiences, that confidence has a whole level or a higher threshold of quality or else they're gonna fall off, they're gonna move somewhere else. And last, let's just make sure that we think holistically about the customer experience everywhere they touch your brand. Because you're basically many of us are describing ourselves as AI first companies. Now if that's true, you better make it true across every touch point with your customer and with that. Just wanted to say thank you very much. Jess and I will be off the side if you have any questions. Thank you for your time.

Thank you so much. Thank you.

Sam. Ra. Ram. Sa. Please welcome to the stage andrew steele, partner at activent capital.

Adam guild, CEO and co founder of.

Owner.Com sarah du, CEO and co founder of alloy automation and josh browder, CEO.

Of do not pay. It is tall.

Hard jump here. Make sure I don't fall off this chair.

Hello everyone.

Thanks for joining us. Incredibly thrilled to be on the stage today with some of some good friends and some of my favorite people and also some leaders in vertical software and software more broadly. We're here to talk to you a little bit about how to leverage AI in your toolkit as a vertical software company. Vertical software can mean a lot of things and we're going to hear actually a little bit from both consumer and the business side as well as from the more infrastructure side with Sarah today.

But this is a theme that we.

Think is really, really important. As investors, my chair is moving here as as investors I'm a partner at a growth stage venture firm and we're focused pretty heavily on enterprise software more broadly. And this theme is one that we think is going to be one of the defining categories in AI going into the future. So let's today we're going to try to keep this panel as practical as possible. Really the big focus is on how these folks right here are using AI in their day to day and how it really is changing the way they.

Think about their product roadmaps, how they.

Think about the customer experience and how they think about everything else. So with that I want to introduce my amazing panelists and friends and we'll start with you Adam, if you don't mind.

Absolutely. My name is Adam and I'm the co founder and CEO of onr. You can think of owner as being like Shopify meets HubSpot, but specifically for restaurant owners and eventually all local service based businesses. They use our platform to do everything they need to succeed online. We're their website builder, their CRM, email and text message marketing and a variety of other functions that help them succeed in this shift that's happening in their business digitally.

Hey everyone, I'm Sarah. I co founded a company called called Alloy. We're an API integration platform. So what that means is we help you as a SaaS company launch integrations faster and we do Both that and we also help you automate business processes. And so been doing this for almost five years. We're series A backed by Andreessen Horowitz. And I've known these two gentlemen since I think all of us were like 18, 19, so grateful to do a.

Panel with them today. Hi everyone, my name is Josh. I'm the founder and CEO of DoNotPay.com and I'm very lucky to be one of the few consumer founders at sasta. Serving consumers. You guys can think of do not pay as sort of a power to the people AI that helps fight for legal rights. So the first use case when I started eight years ago was parking tickets because I was a terrible driver and got a lot of tickets. And finally, fast forward to today and we have AI agents log into people's utility bills to start negotiating with the big companies. So if you see any AI agents in your business, you can blame us. And I'm really excited to be here with my three good friends.

Josh really leans into the do not pay lifestyle. So we were texting before this panel and I was asking why he was late to the speaker lines and it's because he picked up more free T shirts than I knew was physical possible to carry. So thanks for that, Josh. Okay, so why are we here? We are in the midst of a software replacement cycle. At least that's what we believe at Activant. Now these types of phrases get overused all the time. It's part of the hype cycles of all of all moments in technology. But we are entering a world where outcome based software is becoming more and more of a reality. And those who have been built on previous versions of software are really feeling the pain of this new future, this new world, this new competition that's coming from a whole host of different angles really. We think about it as a shift toward taking down the next generation of systems of record. Something we might talk a bit more about here. But this is a very important moment in software and it's why you see the title of AI all over the outside of this building. And really software as a service is to provide a service. And that's something we'll talk more about. We see it as a real shift towards outcome based software and to more self driving software. In other words, instead of just providing observability for another human being to do something, the actual outcome itself can be generated and executed by a piece of software. We think of it a little similar to how you can look at some of the steps of self driving Cars. It's not a perfect analogy, but we are moving quite quickly through some of these steps and in vertical software we think this is probably one of the categories that's going to benefit most and frankly benefit first in all of, in all of software and all of SaaS. The reason for it is a few things and again we'll touch on this in more detail but number one, the workflows are a little bit more contained which means that execution based agents, all the things that we've been hearing about today are much more likely to be, be fruitful and a reality because at the end of the day customers pay for a great outcome, a great service, they don't pay for half baked things. And number two, and I think more importantly, we are now able to tackle some of the largest parts of GDP. Software is roughly 1% of GDP. Services I think is like 15, what does it say, 13%. And we're entering this really interesting moment in time where we can think a little bit about vertical software powered by AI tools and AI as not just a replacement for software, but an actually replacement for a lot of services. And that's what we're seeing, that's what we're hearing, that's what we're investing behind, is tools that can actually drive outcomes in places where human beings, whether they were in your own team or external or previously used and today is in particular a great time to be building these companies. We are sitting here in a world where a lot of the first really successful batch of vertical software was built 15 years ago, maybe more in some cases. And while there are great companies that are publicly listed and otherwise their stacks are just not quite as modern. And we're going to hear from Sarah a little bit about how integrations plays a role. We'll hear a lot about data and we'll hear a lot about other business models including payments, financial services and everything else. But LLMs here, the pardon pinker, whatever color that is, is obviously one of the big step function changes we've seen and why we're here today. So how do we unlock this? How do we practically unlock this is kind of the biggest question. And one of the first things we've been thinking a lot about is really is this going to be something that drives more sort of cost gains and efficiency gains internally or is it going to be more of a net new revenue driver for your products? And so why don't we maybe start with you, Adam and then Josh and then Sarah. So Adam, walk us through a little bit of what it's been like to think through your product roadmap and actually speaking to customers owner as it relates to the AI features that you can today build.

The biggest impact that AI has had on our roadmap is making possible all of these fantasies in our customers minds that they've dreamed of for years. And it is both cost saving and revenue driving. So we serve restaurant owners and something we've been hearing since starting the company seven years ago is how magical it would be to have a member of their team that was trained on all of the best practices for creating amazing online experiences for their restaurant and executing all of the marketing best practices. Whether that's optimizing a website to show up at the top of Google or optimizing an email campaign to drive maximum click through and revenue. And it's just had to be this fantasy for so many years because economically speaking, the average restaurant owner makes less than $50,000 a year in profit. So this idea that they could have a full time member of their team trained in these things has just been inconceivable. But in thinking about how we can begin to apply generative AI to make that fantasy a reality, we've developed everything from an AI website generator that gives restaurant owners all of the best practices in a matter of seconds that help them drive these outcomes, as though a member of their team were going through the thousands of hours of research required to develop the expertise to make that outcome a reality. Another thing we've piloted is the ability to have an agent that is entirely powered by AI as an LLM that responds both reactively to commands as though they were talking to a member of their team, saying things like we ran out of avocado. Can you disable all of the dishes on my menu in my online ordering system? And just having that automatically happen, as well as making proactive suggestions, scanning that restaurant online in every place that they appear, their website, their directory listings and a variety of others, and pulling out all of the ways in which they deviate from the best practices and giving them personalized advice as well as being able to take action on their behalf to correct those things. The way we conceptualize it internally is we think that these small business owners have the ability to add unique outsized value in infusing their unique personal touch to their online presence. That means developing relationships with customers at scale, asking for feedback and being able to respond, or sharing their story with their customers in a format that resonates and helps them build brand differentiation. But the way they've been caught without all of These fundamental breakthroughs is in having to focus on all of the different thousands of little micro optimizations to implement best practices, whether that's conversion rate, optimizing their website, or optimizing that email campaign. And we've now been able to automate a lot of that. That work, which hasn't just led to revenue driving opportunities and differentiating our product, but is also leading to massive cost saving because it's compressed our average onboarding cost, for example, and being able to automate a lot of what would otherwise require humans on our end and similarly free up our team to infuse their unique personal touch and develop relationships with the business owners, rather than having to go through all these rote tasks to optimize customers.

Josh, what are you saying?

Yeah, I agree with that. There's huge opportunities both to save money and also build new products. With Do Not Pay, our average customer value is maybe $200 and our customers have incredibly high expectations. So because we are helping people fight these big companies, if they want a customer service issue from Do Not Pay relating to their subscription, they expect to get a result almost immediately. Immediately. And so we've actually automated our customer support using AI. So if someone goes to Do Not Pay and says, hey, I lost my parking ticket case, I want a refund, the AI will surface their account details and refund them. And that's allowed us to have 11 employees. But we have 250,000 subscribers and millions of people using our website every year. And so it really allows us to deliver a really good customer experience with a very small team. And I think the Klarna CEO said yesterday they're firing all of that. Well, he said a few months ago they've automated 60% of their customer service workload. And then yesterday he said he's replacing all of his SAs tools with AI. So on the internal side, there's huge opportunities in terms of building new products. It feels like AI is making years worth of progress every few weeks, weeks. To give you a concrete example, everything really changed when GPT4 came out. And that meant that it was finally coherent enough to have a negotiation on someone's behalf. So we deployed an army of GPT4 robots to lower people's Comcast bells. And they would open up Comcast Chat and start chatting with Comcast. And for any of you who've a Comcast customer, you know that Comcast actually use AI themselves. So the two AIs are negotiating with each other.

And that's better, Josh, or is there better?

And our AI, wow, is more Motivated. So we all right, good, shift it on the edge. But what's really exciting is AI wasn't quick enough, GPT4 wasn't quick enough to hold a phone conversation. And a lot of customer service stuff that we automate gets done over the phone. And GPT4 took about 3 seconds of latency just for the model to respond. But then when GPT4 came out a few months ago, they reduced that latency by 90% and it now responds quickly enough to hold a phone conversation. And so we've had amazing examples of AI phoning up these big companies and actually doing calls on people's behalf. I guess later on when I'm talking about building with AI, I'll give you some positive examples and some, some crazy things that have happened that might lead to some caution.

My examples aren't as funny, but as an integration platform we do see a lot of different types of software. And so like Josh was pointing out.

AI has accelerated progress significantly.

So in all these different verticals there are hundreds of new apps being built much faster with AI assisted co generation. And so what that means is we need to keep up with the pace of application development, build out more integrations. So we've seen huge efficiency gains in the back end by building out a way to essentially translate API docs and web pages into pre built integrations for our customers. And so it's not a product that we explicitly sell, but it's essentially allowing us to integrate the long tail of applications. So in each vertical, for example, in E commerce there are hundreds of shipping applications or warehousing applications, same in healthcare for practice management. And so it's been a huge boon for us to use LLMs because essentially APIs are like a language that we invented. And so being able to translate web pages and APIs into these no code visual integration blocks has been huge for our customers.

Yeah, thanks Sarah. You point a really interesting point out there, which is cost saving versus revenue generating can be very muddled at times.

Right.

You're saving costs internally and building more integrations, but you're building more integrations which means a lot more revenue, a lot more potential use cases for your platform. So it is modeled, but I think generally what we're seeing is and makes sense, people start with it internally. A little less stressful than putting something out in the wild that touches your customer. But the customer touchpoints are starting to really scale up.

And speaking of customer touchpoints, one shocking revelation that we've had in experimenting on the go to market side in using AI based Positioning is we saw that with headlines that highlight the AI capabilities that the click through rates and cost per leads substantially dropped among our buyers, which was particularly shocking because when you think of a mom and pop restaurant owner, they tend to be be significantly older demographically. They tend to be less of the early adopter as we conceive of it in new technologies. And where this initially was so surprising to me is we were at the International Pizza Expo at a booth that we'd set up there about a year and a half ago, just as the ChatGPT craze was starting to find its way to the mainstream. And nothing about our booth mentioned anything related to AI. We were highlighting our features related to branded mobile app generation and the new marketing automations that we'd created. And yet in manning that booth as a salesperson for the two days of that conference, we had dozens of pizzeria owners come up to me and say, how are we going to be able to use this ChatGPT thing in our business? Like, what are your ideas there? And the big learning here was that as people that work in the tech industry, we have this assumption that while it's captured the imaginations and zeitgeist of our industry, that among the significantly less tech savvy industries as you traditionally conceive of them, that this wouldn't be as exciting or applicable. But nothing could be further than that in reality. Like the pizzeria owners were ecstatically excited when we told them about our AI roadmap. And then as we began to show them demos, we would see literally jaws drop and pupils dilate on what AI could do and mean for their business, because they begin to connect the dots of how this technology is going to enable this fantasy of having somebody in their business that knows their business really well, that can operate on their behalf both reactively and proactively. And one of my hot takes here is that I believe AI agents are going to completely obsolete dashboards within probably three to five years for vertical industries, because it takes the learning curve of a new product from one that takes hours at minimum, even if that dashboard is greatly simplified and opinionated, to one that takes just a few seconds. Because all you have to do is communicate with that agent as though you were speaking to a member of the team. And it not only understands you, but understands all of the best practices of your industry and your specific business to be able to drive that outcome that you want without you having to learn how to actually pull the levers within the software.

Last point on this, because we Got to move. But we actually seen this in our portfolio as well. We have a company that does, it's an early stage business that does kinds receivable work so they collect payments on behalf of other companies and software businesses. And the MPS improvement of people calling on behalf of the company using an AI based caller versus an actual human based collar is like 15%. And that is really, really critical because like this actually unlocks a lot of areas and industries in the economy that vertical software serves that are typically unstructured in nature. And we're now moving to this world where voice you can interface with your dashboard and you don't even need a dashboard. So anyway, let's talk about how you're building it though. One of the most common things we get asked is like, all right, any perspectives on like all of these hard questions from a technical perspective?

Right.

Which model do we do we use? Do we go open source?

Do we not?

Do we have multiple models? Do we fine tune? Do we use rag, do we use prompt engineering? Do we hire for prompt engineers? How are you doing it? Josh, let's start with you.

So I think what Sarah touched on is really important. LLMs and AI is a super powerful technology, but if it doesn't connect to the real world, it's useless. And so in my business, our business is connecting this really powerful technology to of the stone ages. Sometimes we have AI generate faxes to send or letters to send. And in these examples you have to build a lot of guardrails to make sure that the AI doesn't hallucinate because there's huge liability issues. So I'll give you one quick concrete example that both shows how AI is intelligent, but also how it has to be restrained. And so this was a few weeks ago and and our AI was helping a customer lower their cell phone bill. And so it was phoning up this company, Boost Mobile, which is a low income cell carrier, to try and cancel and lower their plan. And for legal reasons we have the AI introduce itself as the person's assistant. It doesn't say AI assistant, just says the person's assistant. And the agent on the other end who is a human said, okay, that's great, but what's your name? And the assistant said, no, no, no, I'm the assistant. And then the human representative said, no, I understand you're the assistant, but I need to know your name. And the problem is the AI didn't have a name. And so it took a while and this went back and forth five times. And finally the AI said Okay, you can call me Alex. And that's really helpful because the AI overcame a problem and but at the same time it's lying. And you really have to worry about the AI to lie. And going back to Comcast, it would lie to achieve the discount. It would say, I had four Internet outages in the past week. And that might be a successful negotiating strategy. But it's terrible for us because we don't want people suing us, saying the AI is lying on people's behalf. So what we've done technically is we built another machine learning model to watch the AI. So it kind of like, and on the prompt we say to the AI, stick to the truth. And I think everyone here who's building an AI, I would encourage you to say stick to the truth in your prompt. And so we have one machine learning model, AI watch another AI to talk to the big company which is using AI. And that's really the future of building these systems together with an integration layer. Yeah, I think what we've learned from.

Experimenting with a lot of different models is they have different flavors and personalities. And what's scary is it just like advancing so quickly. And so what's been important for us is like finding the right tools to continue to evaluate how the same prompts produce different outputs for different models. And so I think like across the stack, like there's no other area like using LLMs where you need to continually test out different tools and tools from different companies, keep up with the latest updates. And so thankfully we've been able to find some great open source monitoring tools, evaluation tools, and I think that part of the stack is going to continue to evolve, but I've been watching that very closely.

So Sarah, maybe just moving to one of the next parts of this, like layers in the stock is, is data.

Right.

And inherently your business is a data business. Right. You do integrations, but really it's a data business. We repeatedly talk to large enterprises on behalf of some of our portfolio companies and every time, number one question is, how do I secure my data and how do I ensure I have the right data? Entering models and, and it becomes very clear that this is a very big data problem. And some of the research we're working, working on at Activant right now is around data Ops and we're about a decade behind is best guess and compared to DevOps. And so there is a big gap here. Just walk us through a little bit of what you're seeing, how you're helping companies leverage their own data or data in other systems and how that applies specifically to some vertical software companies you're working with.

Yeah, I mean everyone's saying data is the new oil because that's how you're able to fine tune and train models. And so even in like Josh's case, like training an agent on how to negotiate on behalf of say like a, I don't know, a pet food company, you need to plug into all their vertical SaaS systems, get all the context. And to do that you actually need to integrate and like pull in data via APIs or RPA. So kind of like scraping methods. And so for us like we do see more of a need for integrations because there's really no other way to pull out data to train models. And so that applies across like different verticals and across different types of tools. So whether it's accounting, whether it's CRM support desk tools, there's so much context that's stuck in a lot of these systems. And a lot of these businesses have used the same say like Zendesk support instance for 10 years. And so being able to pull out that context and data to then train a model is basically the key to leveraging LLMs in the future.

Yeah, I'm actually curious about your view on this because part of owner's value proposition is that you've built an actual platform where there were previously a lot of point solutions. And on top of that your, your user doesn't really care for just adding another thing that doesn't connect to their day to day workflow. Right. If, if it doesn't work in the actual operations of the business and the kitchen and everything else, everything falls apart. And so we saw that when some of the first batches of more in store tools came about. But like walk us through a little bit of how you think about integrations and data as it relates to your business and that theme of we're going to move away from observability and dashboards to like intelligent software.

Absolutely. And this actually connects to the previous slide, if you could go back there on how we're navigating the model stack. Because there's something that's very different about the way we build product and philosophize product as a compound startup. Meaning as Andrew described, we are replacing a dozen different point solutions. When an owner adopts our platform, we're replacing their website builder typically wix, their online ordering system often chow now mailchimp for their email, Easy text for text message marketing, five stars for loyalty. We've got all of these different products built in that are replacing these point solutions that people are using. And as for how this relates to navigating the model stack, we've had to conceptualize how we build product very different, differently as a compound startup, meaning that we don't navigate the model stack top down or a variety of other product decisions. It's not our VP of Engineering, Venkat making the call, or our CTO Dean making the call on exactly what model the team should be building on and using. Instead, the way we've built our engineering, product and design teams is having an extremely high density of founder DNA across those teams. Both previous ventures backed founders, we've got more than seven of them that immediately come to mind from that team. People that have raised millions of dollars in venture capital as co founders and often CTOs. Or we've got people that don't quite have that founder experience, but that we evaluate very carefully in the interview process for having the founder DNA of wanting to take outright ownership over a part of a product and actually be a part of making these big strategic technical decisions versus just following the traditional software development playbook, which is nailing the tickets that the PMs assigned to them with very specific technical specifications. So as for how this relates to navigating the model stack or making other large technical decisions, we ultimately recruit people that want that level of ownership, that are willing to and excited by operating as founders, and then empower them with all of the different business context on why, what the specific product they're building is important to customers and the different constraints that they have to keep in mind in selecting a model. And then from that point there's all these lively debates that happen among our engineering team about what model is specifically best for this AI photo enhancement challenge that they've come across. And they'll try testing 4o against anthropic or a variety of other models for the actual first part of that challenge, technically speaking, which is getting an image described in very specific detail. And they'll try something else for image generation. But it's the ability to navigate the traditional procurement process as founders and owners in the business, the fact that we don't get involved at the leadership level and instead empower engineers to do the research and make the decision based on what they believe to be best, that ends up not only empowering those engineers to do more impactful work on behalf of of our business, because these things are changing so rapidly that that on the ground testing and ownership plays a huge role in their ability to achieve quality, but also enables us to attract the types of engineers that would otherwise be starting their own companies because they're excited by the challenge of starting a company within owner, which is effectively how we've structured things in order to replace that point solution and add value to the restaurant owner in all of the ways that we've outlined. And to expand on the very controversial approach we've taken. We've had 1pm up until like a month ago for about 16 engineers because each of our engineers are empowered product engineers that are in the process of writing the product requirement docs and doing the research and selecting the models and testing them against each other and know that we trust them to make those types of decisions. The other interesting thing about navigating the model is landscape is the fact that these are not super high stakes procurement decisions. They're not having to sign a multi year contract as we would in buying other software, but instead tend to be self serve platforms where we're able to very rapidly test the different models against one another for that specific use case and empower the engineer to make what they believe to be the best decision for their part of the product. Which is how with such a lean team of engineers we've been able to achieve such a broad surface area. Each of our engineers are operating almost like many founders within the company.

That's a great point which we'll come back to about around outcomes based business models and gross margin as you test meter it. It's also by the way full disclosure, I'm an investor in Adam's company and Dean who's sitting with us and for to make this tangible, this idea of the compound compounding effect. It became very clear to me when we saw my eyes light up when we looked at the website website creation portion with AI because that, that I'm sure that's a whole industry and there are companies that are doing that really well. But that value is hard to see in the long term if it's not part of a broader ecosystem. And when instead of having just a, a tool that like generates a site that's a pretty thing, it actually drives outcomes and real business like for your business, like real money through the door immediately. Like that is, that is the time that it takes someone to, to get off of that feeling of excitement and magic and energy and actually saying oh you know what, this is too much work, I'm not going to go down that path. Instead they're going to actually be able to go dine it, actually see value and keep going forward. And so I think from a go to market perspective. It has a really big impact too. All right, so we lost our screen. Going to be looking behind me for a bit. Okay. You touched a lot on this, Adam, which is people, maybe Sarah and Josh, you could touch on this as well because it is such an interesting theme. Like how do you decide design and build your teams and also how do you keep track of like, what tools your engineers are using? I thought this stat was crazy. Stack Overflow has done 3x in the last 18 months. In terms of page views, I think it tells you a lot about kind of obvious things. But like, how do you navigate building, hiring, retaining, empowering? Like Adam's talked about your. Your team.

Yeah, I mean there's a lot of distractions for sure in terms of AI tools coming out. But I think, for example, using like Cursor with Sonnet is like a very powerful combination. So we generally have sort of one or two early adopter type engineers on our team who are testing out various tools. And then once we feel it's worth rolling out to the rest of the team, we'll have everyone start using them. So that's kind of internal, like process and tooling revamps we've done and we'll basically continue to do. And then the other part is, I've seen this with other companies, companies too, but there are just people out there, like prompt engineers or, you know, folks who are more well versed in certain parts of the LLM world that are better to just bring in as consultants.

When you need them.

So we have done that for some, like prompt optimization around translating HTML into API related JSON.

We could also have a long debate on what, what the future of prompt engineering looks like because there's papers now that say that, that the models do better themselves.

So anyway, Josh, I agree with that. I think over the long run prompt engineering will die out, but right now it's needed. I think what you said earlier, Andrew, about outcome based pricing is really important. I think there are two types of businesses. The first type is they're resting on their laurels and they're making the most of an inefficiency in the world. Perhaps it's difficult to cancel their service or they're the system of records, so you can't cancel it so easily. And these are much older incumbents and they're just riding on their laurels and making revenue that way. And then the second is businesses that are driving big outcomes every day and continuing to innovate. So I really tell my team we need to focus on the outcomes and the way we do that is we're very mission driven. I recruit people who are like on Reddit at 4am looking for ways to save people money, maybe for themselves firstly. And they want to scale themselves and so making sure people are in it for the right reasons so that those outcomes get done. Because in a world with efficiency, with AI, you can't afford to kind of be inefficient and rely on people finding it hard to cancel or move because it's a system of record. Yeah.

I mean so great transition to business model and changes in business model. I mean we saw Zendesk come out and talk about the shift towards outcome. I mean that is a huge deal. Huge deal. Salesforce, the creator of, literally the creator of seats. I don't even know if these chairs would exist by Salesforce. Like they openly are now talking about outcomes. BS pricing. It's definitely, definitely going to change the way we think about software as a service more, more broad, broadly. But it also matters not just on revenue on the cog side. Right. If we're not using these components like models for different customers in different ways, if you don't have some way of metering it, you can end up with a power user costing you an unbelievable amount of money from a compute standpoint and then add agents to the equation and all of a sudden that gets pretty scary. So how do you guys think about it really? Like how do we ensure that we provide, we keep our customers sticky when we use icons based pricing and how do you think about it as part of your broader suite? And Adam, like obviously at owner you have lots of ways to charge, so maybe walk us through some of the different tools you have in your business model, how that's evolving, what the evolution will look like five years from now.

Totally what we've realized is regardless of whether a customer verbalizes it during the sales call, there is an outcome in their mind that they're buying the product to achieve. Because we've done significant analysis on our top retaining customers versus customers that don't retain and figured out that there's a number of metrics internally related to sales growth that they've got to hit by certain points in order to justify the cost of the product and in extracting what those numbers are for us that we need to get customers to X amount in the first 30 days and why amount within the first 90 days in order to make them top retaining customers. We've been able to orient the entire team around driving those specific outcomes. Where AI feeds into this is that we can actually get more predictive and personalized in terms of the outcomes that we can drive for specific customers because we can ingest on the front end a lot of data about how that restaurant is already performing online through looking at their website traffic and whether they're following the best practices for search engine optimization and conversion rate optimization. And then using our database of thousands of customers and the sales outcomes they were able to drive get very specific about what we can expect that specific restaurant to drive.

And is this like expectation setting at the beginning, is that what you're referring to?

Yes, partially expectation setting in the sales process, but partially also qualification trying to figure out whether this specific customer is going to see success with the the platform because we are able to drive a lot more precision on all of these different ways that our customers quantify value through analyzing all of the data that's available to us.

That makes sense. Sarah, what are you seeing your business's outcomes based?

Yeah, so when we started out we would sometimes price based on integrations that we're helping our customers launch and then we'd also tie in number of API calls going through our system. So throughput, both of those are kind of value based forms of pricing. We've actually never priced based off of like engineering hours saved or anything like that. But we've increasingly moved towards throughput related pricing.

So I think especially when it comes.

To integrations, one integration say with NetSuite could include 200 endpoints and integration with Twilio might be one. So it's highly variable and ultimately the value is the API calls you're driving through the system. And so for us we feel much more aligned now that we have more.

Of the API call and usage based pricing. Yeah.

Josh, any thoughts?

Yeah, just. And I would say what Adam said about consumer and business purchasing expectations is through the roof. When the foundation model companies are releasing something like ChatGPT, that's what the consumer expectation has, which is free. And so we have to constantly innovate to make sure the value that we're providing is increasing. And we're actually switching business models. So right now we're a subscription business. We have hundreds of thousands of subscribers, but for certain new products we're launching where we're taking a percentage of what we save. So AI is really good at analyzing people's whole hospital bills and instead of helping someone with a $50 parking ticket which do not pay does today, we are increasingly working on helping people get out of a $10,000 medical bill or at least reduce it. And that's much more aligned with a value kind of taking pricing model. So using AI to increase the value and then changing the business model I would say is our core focus.

Yeah, I think I'll take. Big area we're spending some time on to in our work is thinking about how this all plays into ultimately this is a data like in order to charge effectively here, you need to have clean data and it needs to flow on time every time. But if it does, what you can do is you can start to really test and iterate on your business model in much more granular ways. Right. A seat doesn't have to look the same for both people. To your point, expectation setting item can mean different prices for different people in a way that actually is beneficial to everyone. And so we're keeping a close eye on that. We've got a few. Do you have one last thought?

Yeah. One other thing we've realized beyond expectation setting and how this can be helpful in the go to market machine that we've built is we can get way more consultative and personalized in how we actually sell that product when we arm our sales reps with the analysis that our AI has done on how that restaurant is set up online. So we prepared what's called the growth grade of the restaurant where we're analyzing how that restaurant is appearing in all these different places, the areas that they're currently falling short, the best practices that they're missing on their website in terms of how they've written their headline or the body text on the website. And then we are giving that restaurant owner that report. But the salesperson is also able to use it to infuse significantly more personalization into that sales process because we've all realized how powerful consultative selling is. But the problem with selling into vertical software SMBs is that the economics are really tight. So each AE needs to be closing a lot of deals to make it all work. Which means that they're not able to go as deep in that research process to be consultative in their sale until you've got an AI that's actually doing significant research on how that restaurant is currently performing and giving them all of these actionable recommendations that they can implement regardless of using our software that help both build credibility as well as make the recommendations of the salesperson that much more trustworthy, authoritative. Incredible. Because they're coming with this incredible packet of research to those conversations.

Yeah, I think we just level up across the board, whether it's the Amex black card experience for the consumer. Josh, you can provide. So as if they have a full time assistant always in their pocket at any moment on the behalf or whether it's the enterprise experience, the like elite enterprise experience that someone can get as a really small business in your case.

Exactly.

So I do agree there completely. All right, we are pretty much. If anyone has one question that they want to hit, please jump on the mic in the middle of the room. Otherwise. All right, perfect.

Let's go.

So what really great panel.

Loved it.

Would love to hear examples of how you are all use AI like in.

Your day to day.

I would think with this conversation, anytime you get an ad hoc request or you're doing something, the first thing you're doing is going to an LL lab. I wouldn't be surprised if I heard that.

So it'd be great to hear like.

A couple examples of how you guys.

Are using it day in and day out.

This is the best one.

So you should start.

I highly recommend PIM Eyes. It's actually a bit scary. You can take a picture of anyone in the room and it will tell you who it is. There's privacy, his dad, but it could be a useful tool if you're trying to track down someone. Might only work for you since you're very Google able. No, it works for me.

I like when I travel somewhere new, I like to ask the LLM to equate neighborhoods to neighborhoods in my city. So for example, what's the pack heights of Paris? What's the haze of, I don't know, Casablanca. So that's been very helpful.

I find myself using AI most for writing purposes as a proofreader and simplifier of thoughts because I'm constantly writing internal memos or documentation for our team in order to help people be aligned with the way that we as a leadership team are thinking about things. And in the past I would then have to wait on a editor to share a series of comments about how to simplify that thinking or distill the essence of what I'm trying to say. Whereas now I can have this quick back and forth with GPT4O using ChatGPT to more quickly get the essence of what I'm trying to say and propose alternatives in terms of how I communicate both to the team and to customers.

Yeah, amazing question. Very similar to Adam. That's probably one of the biggest use cases. We do a lot of writing. Writing is a very important thing as an investor to make sure you have clarity of thought and that you go through the journey rather than just feel emotion and and don't actually take the time to consolidate. But I will say it has been very useful in just quickly getting a pulse check on certain markets or trends and then enabling us to go deep, because that's a big part of our approaches. We're pretty research led, so it never can be fully part of it, but it gets you going, and it. I think it breaks the inertia, which is the biggest thing for me. It just gets me over that hump, and then I'm moving and feels like.

I procrastinate a little bit less. Yeah.

So thanks, everybody. It's been fun. Thank you to the panelists.