#281 Developing AI Products That Impact Your Business with Venky Veeraraghavan, Chief Product Officer at DataRobot

Hi, I'm Maggie, VP of Curriculum at datacamp. And I know firsthand how overwhelming it can be to keep up with the rapid changes in data and AI. But staying ahead doesn't have to be hard. That's why our curriculum team has curated the best learning content to help you and your team stay ahead, covering everything from data and AI literacy for busy executives to cutting edge generative AI for developers, and everything in between. Whether you're upskilling yourself or your team, we make learning practical, engaging and flexible. It just works. Power your data and AI transformation journey on datacamp.com.

Are you picking the right problem that is really part of your business workflow or your business process? Like, are you going to change something? The way you sort of drive the change is like you start showing small results and then you start saying, hey, look, I can scale it by doing more changes. The idea is that you are collaborating in real time, so to speak, on the problem in a common space. That's what I mean by intentional experimentation, which is like, you pick something, you understand how it works and you gain confidence. You're like, oh, now I know what to do. Let's pick the next problem.

Welcome to Data Framed. This is Richie. There's a new management buzzword trending, namely AI readiness. Like many management buzzwords, it sounds like a good idea, but I'm not really sure what it means. Today I hope to find out. I want to discover what you need to do to make AI work, both from a technical point of view and from a business point of view. And since things are moving quickly, I'm curious as to how you stay AI ready in the face of technological changes. Our guest is an AI product legend, Venki. Veer Raghavan is the Chief Product Officer at Datarobot. He drives the definition and delivery of the DataRobot Enterprise AI Suite. So Venki's got 25 years of experience in big data and AI as a product leader and a technical consultant. So previously, Venky spent two decades at Microsoft, rising to VP of Product Management for Azure Cognitive Services. I'm keen to pick his brains and find out what AI readiness really means. Hi, Venki. Welcome to the show.

Hi, Richie. Good to be here. Brilliant.

So we're going to be talking about AI readiness. So first of all, what does that mean for an organization to be AI ready?

It's being prepared on all parts. There's the technical part and then there's the business part and then the cultural part. So I think being ready on all of them is what it means to get AI ready. I think people generally start thinking about, do I have the right tech? Should I deploy a software, buy a product? But it's much more than that. It's really sort of retooling your company to use AI and depend on AI to get business results.

I think a lot of people think, okay, I just want to buy something and then all my problems would be solved. But if only life were that simple.

Cool.

So maybe do you want to talk me through what are these problems then? So why aren't organizations AI ready? What's this gap they've got across?

I think across the board, AI is having yet another hype cycle in AI and people are really focused on understanding the tooling, seeing how it works, and experimenting and learning about it. There's so much change in the last two years in terms of new technology. I think people are really curious and they're trying to put it in, but they're not finding a lot of business results because there's more to it than just technology, as I mentioned. And so really one is, are you picking the right problem that is really part of your business workflow or your business process? Like, are you going to change something or are you going to invent something brand new? I think everyone, especially with gen AI, people just said, hey, look, we'll just start afresh and there's a whole new paradigm and we'll go fix it. But there's very little guidance in terms of what works, what doesn't work. And so I have a view that, you know, you really should focus on something you already know how to do workflows, and then sort of experiment and intentionally experiment on putting new, the new technology in and measuring how it works. And so that's, I think, to me, one of the big issues here, which is like, for us to fully capitalize on AI, you really have to sort of figure out where it gets input from, where's the data, and then where are you going to use it? Where is the AI going to be used? And is that a problem worth solving? Because it is going to be possibly expensive. It's certainly going to be risky because new technology. And so you really want to sort of find the problem that is interesting to the company that you can measure before you put this in, then measure after, and then you gain a sense of confidence. And I think that's what people are lacking, a sense of confidence that there's a pat answer to go solve. And so they have to sort of do it all by themselves. And it's A lot of work.

Okay, yeah, so that sounds like tricky. If you're using a completely new technology, you're not really sure how it works and it involves a lot of different aspects. So you got to change processes, you've got to get your data right, you've got to then figure out how the AI fits in many things to change at once. So I like this idea of starting with something, you know, so suppose your CEO says okay, we have to become AI ready, we've got to go and do this. Where do you start? What's your first step in this?

I think most people very quickly go into a technology conversation which is like should I use ChatGPT, should I use anthropic, should I use whatever? And really the question is really thinking through a little bit on where are the places where you can use it, where it might helpful. And there's a lot of knowledge out there and I think some amount of research in terms of like what are the business processes that can most stand to get improved, whether it's top line, bottom line. I like to use an example of like, you know, you might be doing marketing mails, you might be sending mails already to your customers for saying hey look, upgrade to the next service. And so you have this propensity model and you might say like okay look, I know what I'm doing today. I'm sending out a number of mails and I like four different templates and I know what the click through rate is, I know what the conversion rate is, I know a lot of things about my process. Now you say great, this gen I think is going to make me to hyper personalize email, right? It's not just like three or four templates that I can use to drive. I can do an email per person because I know them so well because the technology can do that. So then you get saying hey look, who are the people you sent the mail to? You might use a model to say, you know, what is the churn propensity or upgrade propensity that I have for this customer, how do I score it? And then you can say look, everything where I feel very confident they're going to upgrade modelists I say 90% accurate. They can say great, I want to pick those customers. And then I'm going to use the prediction features. Why is the model predicting this person would upgrade? Use those as prompts in your model to then say look, I'm going to create really, really hyper personalized email that says look, I'm going to upgrade to a better telephone Plan because I do a lot more international calls or I do a lot more roaming. You can start using these inputs to build your email and then you test them, you send them out and see whether they have a better click through rate, they have a better upgrade rate. And so now what you're doing is you're putting the technology in the context of a business process and measuring it and then gaining confidence that you can make it work. You might say, oh look, it works at 90%. Can I move it down to 85%? What point do I need to actually have a human in the loop to actually double check what males are being sent? And so you sort of do this, what I call intentional experimentation on a known business problem. And that gets you both the skill and also like really starts giving you the ability to, to start stitching everything together to actually have a business solution as opposed to just a cool technology demo that you can show a CTO and be like, oh, I did this cool thing.

I really like that idea. If you start with a really concrete business problem, just like, okay, well how do I include increase click through rate for emails? And then you're going through like there's stuff that you can measure, like it's got a real, a real metric. You can do experiments to see whether it works or not. And you're going to be able to have some like concrete knowledge of whether or not this is going to work or whether or not it worked at the end. So with this example, something like messing about with email, this is going to affect like a single team, like your email team, maybe a little bit larger, like a couple of teams. But is that where you should go for when you're deciding what to do? Should you be choosing your projects at the team level rather than it being some sort of top down approach driven by your C suite?

You know, I think the C suite can definitely help in giving guidance on what they think is the most interesting problems. And so if you want to sort of solve for like, you know, email, strict upgrades constitute like 0.1%, then that may not be a good one. It might be a good, interesting learning experience. But if you're saying, look, this is my main driver of saying getting more, more ltv, then you're suddenly starting to say, look, this is what we want to work on. So I think you can clearly say what are the priorities of the company and where is it? And I think the technical teams again take that as input to then drive saying, look, this is what we want to experiment with. And I don't think the top, the tops really know exactly which are the right priorities. Right. Because the technology works differently in different conditions. And so you really want to be able to say look, try in this area. These are things that are most important. And then you make some experiments and you then find out. I think it's good to have a bottoms up ideas that actually check what is the return of the technology.

All right, so C Suite is going to be all about just inspiring people to get going, looking at stuff and then the load low down the organization. It's going to be. This is where you pick exactly what's going to be relevant for your own team.

Yeah, like broad guidance, like directionally, sort of like we think we should solve these problems. Then the technical team do have to say like, hey look, integrating for the last mile in this area is super expensive versus the other one. It's really easy. So you also want to sort of be able to look at ability to execute as one of the things and how much risk do you have in getting that out there?

So yeah, ability to execute is incredibly important. I guess for that you're going to have to have some AI skills in house. I'm curious as to what sort of what you need, like particularly what roles do you need to hire for what skills those people in those roles need. What do you need to know?

This is an industry and a function in rapid change. You know, I think like I've been in this industry for now 10 years and it used to be about data scientists and then it became more deep learning, but now there's a whole sort of new set of skills that involve. But you know, I'd say, broadly speaking, I would say that three types of skills you want. One is you still want data science. It is AI being able to do good evaluations, something they're doing outside of anecdotal sort of validation. You still need data science skills. You need to be able to put metrics into place and you're going to be able to test and experiment. Even though you may not be building the large model, you might be fine tuning it. You might be able to compare different model types and everything else. So data science is still an important functional expertise that you want. I'd say increasingly you'd want to also want software developers because I think we are now realizing that the AI doesn't hang out by itself. It is integrated the workflow. So you might have your IT devs sort of really being able to say how do I integrate into Marketo in my example? Right. And so really be able to say what is the end to end flow? Where do I get the data from? How do I push the results back into a business critical workflow? And I think especially now with Genai sort of the business users matter a lot. So again, if I'll use the email example, you might want a product marketing manager to say, what is the reading level that I want of this email? Right. How many minutes do I want it to be? How many minutes do I need this email to be read by? What is the complexity? And so suddenly you're having all these, what is the voice of the email which is aligned with corporate brand? And so suddenly you're having these things that are not developer or data science. They're really business specific things. And so you really want this cohort of these three broad functions sort of working together to solve the problem. And I think collaborating over them. Otherwise you will, you know, you might have a true but not useful solution, so to speak.

So it seems like the most important things there you've got experimentation. So I guess a lot of AB testing and variations on a theme from the data science side, it's all about connecting all your different bits of software together from your software engineering side. And then it's got is this actually going to have an impact? Does it make sense from a business perspective from I guess your commercial teams there? So I guess beyond this is like how do you structure all those different people? Are you going to have one team with all these sort of different skill sets brought together? Do you want different teams communicating with each other? How do you organize it so you're going to get the most out of your AI?

I think it'll be a no. 1, one sort of answer fits all, but it'll be a variation depending on how the company's organized already and sort of where they want to go. But I would say what you want to do is you want to get the teams together on a platform or on a thing in which you can pick and work together. Often there's huge amounts of issues that sort of around just they might be in different departments. You know, one might be in it, one might be in the IS AI center of Excellence and the other one is in the business. How do you sort of pull them together in some common place instead of sending stuff by emails or a file system? So how do you sort of get them in a single place? So having them in a collaborative environment is super important. Being able to iterate quickly over that is important. And I think once you do that then you can Start thinking about, hey, look, these are what works for our company. Then how do we sort of harden it, so to speak? You might create a separate team, a tiger team. Or you might say, look, it's a real team. You might say that data science should be really close to the business or, or the app dev should be close to the business. Or you might say, look, actually we have a more platform approach in the company and so I want a central team that has projects and then we bring in the subject matter experts into the team and we sort of do this thing and then we sort of move on to the next project. I think, I know there's a fixed answer for that. I think every company has its own, but the idea is that you are collaborating in real time, so to speak, on the problem in a common space. And I think like. And sort of being able to do that is, I think, super important.

Okay, so I guess the actual team structure matters a little bit less than just, have you got the right people who need to talk to each other, able to communicate with each other in an efficient way?

Yeah, I mean, team structure eventually does matter, but I don't know, that's the first thing. And I think, and also I wouldn't be so bold to say, like, here's a structure that works for every customer. So, you know, I think I want to be a little bit more, I guess, modest in my guidance over there. But the thing I would say is that's most important is that you start by getting them together and identifying them, giving them time and space to sort of work on the problem because you don't know what you're going to find. And so being able to sort of iterate and work on it multiple times is sort of important in general.

Do you need all these AI skills in house or is it something you can buy in? Do you need to trim existing stuff up? Can you hire new people? Can you have them as contractors or whatever? What works best here?

Cities are going to be part of your true business process. And that's my pitch. You have to have some of that skills in house. Unless you're sort of doing the entire thing as an outsource, but that's a totally different situation. But if you're running these processes in house, I think you need to have some of those skills in house. And the subject matter experts, the business users, they're probably going to be your employees already. You might have sort of app devs, you know, if you want to do augmentation, you want to get someone to write a Little bit of code that's for the project and you can certainly bring those contractors. But the idea is that the heart of sort of understanding what the business process is understanding how the AI helps. I think those things are worth having in house. But you might start with getting as augmentation to sort of get you help to get going and then you sort of build from there. But if you're going to make AI a critical part of your business, some of it should be in house.

All right. So yeah, it seems like if it's going to be part of your core business, you want those skills to be not outsourced to a contractor.

Then okay, there are companies who do that, but that's a different model. Yeah, I'm assuming that sort of they run the business themselves. All right.

And I'm curious as to whether there's a difference between you being a technical person and a non technical person in terms of like what AI skills you need, because it seems like, well, everyone needs some kind of level of AI understanding at the moment. Suppose you are one of the commercial teams, one of the commercial employees. What do you need to know about AI?

The familiarity with the technology is important. I think there's so many ways, I think one of the nice things about Genai is there's so much technology available for non technical users, whether it's ChatGPT or you know, perplexity. Like there's so many copilots, there's so many different technologies available to you that being able to use it and run it gives you a sense of what works and what doesn't work and also gets you a sense of like what is reasonable that you can do. And so I think like the commercial or sort of the field and the business side of the house, they're the vaults of information about the business, about what's happening. And so I think like if you take that plus a sense of what the technology can do, then you can become a really good thought partner with your technical team to say, oh, I was thinking about this, I was thinking about that. And I've seen a couple of times when conference is really, really good ideas that you as a technical person, all you know is how the technology works. You don't know where you can help. So this idea of like this marketing mail, it was brought up with a customer actually. And because we were sort of pitching Genai, we're talking about this and they were like, hey look, how do I know who to send it to? And we were like, oh, how do I know what to write and and suddenly those questions become the way you sort of get this really full fledged understanding of the problem. And so I think like that is super important for non technical folks, the business users, to have a good sense of what the technology can do.

Yeah, it just seemed like you can need these business people to be able to describe what their business problem is and do it in a way that's going to make sense for the technology you're going to use. So I guess it's just got to be enough to have an intelligent conversation with the person who's building the thing.

That's right. And then you can riff on something that you have some broad understanding of. If you have no understanding of the technology, then you can sort of expect magic. And then we went through that phase. If you think about two years ago, we just thought that these gen AI things will solve all problems. They're like, why can't you do that? And we were like, well that's not how it works. Right. And so having a better understanding of what the constraints, what the strengths are can really help sort of channel the situation and sort of the collaboration very quickly to the useful part of the thing versus like exploring the non useful parts of the space.

Yeah, it's amazing like even just a couple of years ago, like the crazy things people were trying to do because it was new technology and no one knew what was possible or not. One of the fun examples is people trying to do data visualization using like Dall E and stable diffusion and all these kind of tools and it will just make something up that's completely wacky. It looks like a plot but it's absolutely useless. So yeah, understanding what the technology can do and then making sure that your projects are aligned with that. So this has maybe got a bit abstract. Let's talk about some like concrete stories. Do you have any examples of success stories for organizations where they've really tried to improve their AI capabilities and then they've had some sort of benefit?

One of the ones I always start with is one of our customers, it's called Kings Hawaiian. They've been super focused on working with us on really concrete business problems. Whether it's office of the Chief Financial Officer or on the supply chain side. They've been sort of really focused on concrete business problems and so they have been able to see understanding sort of how to price. They've been doing some pricing studies and being able to understand sort of how to price the goods. And they were able to get 50% extra money margin by using AI in their models and they were able to figure out what's the market conditions, what is the demand and everything else. And being able to really understand how to predict the price. And this is sort of more of an old school sort of forecasting problem. But they were able to use it directly because they were working extremely closely with the business. Say, look, we want to optimize this thing, right? We're able to forecast today and we want to forecast better. And so how do I get gain on that? So that is a concrete example of a customer who's gaining a lot of benefit and a great reference for us. We are working with them on the next set of problems around cash management and other things. And so they've gotten a taste of solving problems with AI and they're like, oh, this one works, what can we do next? And to me, that's what I mean by intentional experimentation, which is like, you pick something, you understand how it works and you gain confidence. You're like, oh, now I know what to do, let's pick the next problem. Then you sort of need more of a platform approach as opposed to just a app approach. So you're sort of like being able to take what you learned and reapply it in a new use case and very quickly lean from there.

So I like the idea that just messing about with your pricing, you can just dramatically increase your margin and make your company a lot more profitable. Just like, I guess just this is just in one area of the business as well.

That's right. Okay. All right.

And then so I guess you mentioned the next step to that is taking what you learned from the first project and then trying to reproduce it. So how do you go about with that scaling idea of just we can do machine learning in one area of the business. How do we get it everywhere?

That's. That's right. And this is where now the technology part really helps because when you. The way I think about the first one is you're trying to run water through the pipes, you're trying to get the right data in, you're trying to figure out how to get the security worked out, how different teams can collaborate. And then once you build a model, you're able to figure out how to push the results back into your business system. And that's the first one's hardest one. That's why it's important to have a problem that is worth solving. And that when you solve it can sort of give you lots of bottom line or top line benefits. So that's a good example of that. But Once you have it now you have a system that you can do the next one. You can say, okay, now I have related data, how do I do the next set of things? And you have a better understanding of the technology. You can say, if I can forecast, let's say pricing, can I forecast cash? And so the customer really is like sort of thinking through, hey, look, what happens if I want to forecast receivables and who's going to pay money on time? How much do I have? And what they're finding out is that they can predict it quite close closely and they're able to save money on the credit line. So suddenly you're able to sort of get these much, much closer, sort of related problems. But they're now solving a different function. Now they're solving the CFO function. And so that is the transition that you want. And you want to sort of have a platform that actually supports those multiple use cases that are related, but they're different.

That sounds very cool, but the examples you've given there, this is kind of, I don't want to say old school, he's not really that old. But like it's all predictive analytics. This is machine learning rather than the fancy generative AI stuff. Should that be where organizations focus their attention first? If they're just wanting to start out.

With AI, I would say maybe I've been in this thing for too long. When I started it was called classical ML and deep learning became interesting and with NLP and CNNs and RNNs and then became generative and transformer. But now all of that stuff is now considered predictive AI. And that's the gen AI. But I think in a year or two it'll all become all AI. So, you know, that's why I like this idea of like not having the technology drive. The first starting point is to say what is the right problem to solve if you want to get better, Click through rates. Again, I go back to simply such a simple example. You're using gen AI to create the email. You're using predictive AI to predict the cohort and the drivers. And so you have to let the problem tell you which technology to use, not the other way around. And frankly, this marketing thing came from generative AI. We said, you know, it can generate mail, can it do this? But in the end it happened to have a big predictive AI component as well as a big generative AI component. And I think that's the more. I guess maybe I'm too practical a guy. But like that is how I would sort of solve the problem. Not just saying, hey look, this is cool tech and what problem can it solve? But really say, what problems do I have that I think are worth upgrading? And then obviously you can take a look at the available technology, but you sort of make the technology work for the problem.

Yeah. And I guess most business problems are tricky enough or nuanced enough that you're probably going to end up using a mix of different technologies in order to solve it.

The really doesn't care whether it's generative AI, predictive AI. If you want to say I want to understand how to make my call center better and I want to make them get better, let's say nps. At the end of the call, you might have to use a combination of generative AI and predictive AI and deep learning. And you want to use speech to text, you want to do sentiment analysis, you want to find anomaly detection, you got to find these parts. And then you can say, oh great, now I'm going to maybe help customers with a script that says when you run into these problems, say this instead. So I think the idea is that you really focused on how do I use the different parts of technology. And I talked about three different types of AI, but it's grounded on the business problem.

All right, so we're agreed. Now we start with the business problem, then we've picked the technology afterwards. So we've talked a bit about like, how do you choose what projects to work on, what skills you need. The other stuff we've not really talked about yet is the processes. So it seems like as soon as you start automating things with AI, whatever form it is, all your processes need to change. So can you talk me through, like what's likely to change and how you go about doing that in a reasonable manner?

I think the biggest concern people have with thing is the original process generally is super clear. Right. Humans do it or the computer does it, but it's clear what you're supposed to do. It's deterministic. You sort of know when you do this, something happens. With AI, it's more stochastic. You don't quite know what it's going to do. There's always some amount of like, here's the prediction, percentage, accuracy, percentage. So you have to build your whole process around what happens when it doesn't do the right thing. So you certainly want to monitor it to understand is it behaving the right way. And that is a new thing. Because if the old process is running with One set of people and one set of technology. If you're going to put AI in there, you've got to have a new set of runtime which is like it's not just you build a model and go away. You have to have the infrastructure to monitor the model, monitor its behavior, get alerted when things are going not the way you expect, be able to intervene. So you got to be able to fund it. So I think like suddenly you sort of having a different set of activities you're doing to sort of maintain this AI process. And so I think like what I'd say is building that funding and capacity to run the AI at scale is an important part. The second one is like, how do you know when to bring human in the loop? Like if you have a confidence problem in general about these things, then making sure that you know, should you check every piece of email that gets sent out, that probably is not very useful, but should you check none? That's probably too little. So how do you know when to get people involved? I think those are all new things that you didn't have to do before. And that is the biggest problem. Change is like I'd say designing process for like more of a stochastic outcome as opposed to a deterministic outcome. It can be quite different for companies.

I suppose as a data scientist you sort of used to deal with probabilistic things, but if you're a software engineer, not so much. And I guess for, well actually most of the rest of the business, probably not so much. So that's going to be entirely new.

That's right. And you can see it in a lot of the new generative AI. Like we look at these ChatGPT or copilots, they sort of in the UI tell you that they're going to be approximately right, but they're not 100% right. Right. And so we didn't used to do that before. We used to just say ask a question, get the answer back. But now it says like, hey look, you know, this is indicative. It's I am an AI, I may not get it right. And these are all little things that you're trying to get set up. The promise of like it's not 100% right. And so that is one particular example. But the idea is that you have to think about it as a real thing. You just can't assume it is the old style deterministic outcomes. You have to really sort of say what happens when it's wrong or if the values are not correct.

So I guess the Main thing here is you just have to have human check steps within your processes just to make sure that the output from any AI is correct. Is that about right?

Yes. I think you certainly have to sort of figure out both during the build phase, like, how are you going to get the right input from the right stakeholders to make sure that it's. You think it's behaving the way you want it to behave and that many people almost get right because you fall into those problems. But many people don't get the operational element right, which is like, once you put it into production, it just doesn't run like I installed a piece of software, a web server, and it starts working. It just. It doesn't change behavior in AI. It does change behavior because it's very dependent on the data and the distribution of the data. So you have to still manage making sure that you have time and effort spent to manage that part of the process as well.

So you've got to monitor what the output looks like and I guess continually test how much of the time it's given the answer.

Right.

Actually, do you have any more details on how that might look like? What does monitoring some sort of AI product look like?

You know, I'll use generative AI at this time. You know, when you're chatting with a chatbot, you're chatting with OpenAI and you're trying to make it your own, you're trying to say, look, I'm grounded on my data, my company's data. You might have certain policies like say, I'm never going to talk about my competitor. And so in which case you have to say, you know, you can do a prompt that says never mentioned your competitor. That works kind of. But what happens? How are you going to be sure? Because these things do hallucinate. And so what you want to do is to say, you want to have a guard model that looks at. On the way into the LLM, you might say, look, I want to make sure that I test for any mention of any of my competitors. And it could be a heuristic to just code. It could be another LLM that is checking for it. If you find it, you can then say, I want to intervene. I want to tell them, look, I don't wanna talk about my competitors. You could just return back. So you sort of. You're training your user. Now. The second part is you might do the response, a guard on the response side, which says, the user asked for a good question, but the LLM returned something I don't want to see. Then you can say, great, I wanna have a guard here, I wanna again detect it in real time. And I want to modify and mitigate that outcome by changing the output. So now you need a pretty sophisticated framework to actually host all of these guards and run them in real time and then be able to detect and then be able to mitigate the problem in real time. So that now becomes a little bit more of an interesting infrastructure problem. Obviously our product, for instance, supports that kind of framework that you can sort of just put the guards in and really focus on the business problem of what words do you not want to talk about or what topics do you not want to talk about versus figuring out how to run eight different models and run them reliably and then sort of hook them all up. That's done by a box.

Okay. Yeah. So unless you're an infrastructure person, you don't really want to have to care about infrastructure. You actually want to care about the high level business problem stuff. That's interesting that you mentioned you got multiple levels of checked in there. Just because there's probabilistic nature. You've got to check user input, is that okay? And then you got to run through the model and you've got to check model output as well just to make sure that that works as well.

That's right. And it gets more complicated with these more of the sort of new style agents where the LLMs talking to each other and they're making plans. You're like, they're all talking to each other like, how do you know what happened? How do you debug it? How do you make sure you're, you know, you have control over what it is. And that's the heart of the confidence problem. If you don't know what it's doing, it's hard to trust. And so being able to see it and then being able to say, ah, I know it works. And this is the kind of failure modes again, very much like software development. Like you sort of learn that part and then you mitigate for those problems.

So I guess the other part of changing process is how do you persuade all your colleagues that they need to go along with the new processes? There's a sort of change management aspect to this. Do you have any advice on that?

Change management is a huge part. You know, I think the way you sort of drive the change is like you start showing small results and then you start saying, hey, look, I can scale it by doing more changes. It's almost like you have this sort of pay for play thing which Is like the more we do, the more benefits we can get. And that gets people really aligned on the benefit and why the change is interesting. I think too many times we end up with like we're doing the implementing this software to do this sort of hypothetical gain. They don't fully buy it and the art is skeptical. You know, a lot of people are skeptical of the sort of change in the status quo. And so if you don't motivate it with the benefits and smaller scale experiments and say like look, now we can go back to market mail. I sent 50, it was great and I increased it by 300%. Click through it by 300%. It'd be better if you can do it more. But that means I need to actually integrate with your system. That's changed for them and more risk for the system. But now you're saying, no, I'm doing it because there's a possible upside. Maybe we can then sort of work out the next. So I think to me this just overall change management as anything else. I think most people have to figure out like how do you motivate the change and why that's super important to sort of align it with the business benefit.

Okay, so this is going back to the idea of experimentation and just saying, well, if you can demonstrate that there is some benefit to some business metric, then that's at least going to encourage people to try the new process.

That's right. And I would say that is the more sort of radical revolution style, which is like, look, I've completely upset the business process. I have a brand new one. That one's obviously going to have a lot more higher bar for change. And so I think the lower bar, the more of the evolution style is much easier to get done. And there's some cases where you might want to completely rethink the business process. But many of these problems to date at least you know, technology hasn't like sort of said throw the old thing away and start it all afresh. I think like, you know, they're all sort of how to assist and aid while we learn and then we can sort of find more disruptive ways to solve the problem.

So I guess the tricky part is when you're using AI for automation and then people get asked to automate parts of their job. I think a lot of people get nervous like am I being replaced by machines here? So do you have any sense of a good way to deal with that aspect of using AI?

I don't think I've written the final chapter on that. We're all learning based on what we have seen so far. I think like, it probably allows a person to do a lot more. You get more capacity than not. But what you do with extra capacity is still tbd. And you know, some companies could argue, hey, I don't need as much capacity. Or you could say, I can do more. I think like that. I think it depends on a case by case basis and we've seen examples of both. So I don't know if there's a single answer to what do you do when people get a lot more productive with the new tools?

Okay, yeah, so once a year more productive, you got more capacity. And then there's like good ways of doing things, I guess, like reskilling people, bad ways of doing things, like just getting rid of everyone.

That's right. You know, like as we get more productive with coding, for instance, one of the places where I think it's best understood I have a giant backlog, I can do more of my backlog. So I think like, depending on how you are, you might really love it. Saying like, oh, I love the 30% gain that GitHub says we get. I'm like, I'd like 30% more deliverables. That's on the tech debt side or on the customer facing feature side, I would like more. And so I think, like, to me it's a great benefit. It's not a challenge to sort of keep people employed, but really it's like, I can get more done.

Yeah, I suppose it's going to depend a lot on. Do you have more problems that you need to solve than you have like time and money to solve at the moment? Or are half your employees just kicking their heels at the moment?

In which case, and that's the part I think that is very, very case specific. So I think it's very hard to like make general statements about that.

So we talked about some success stories. I also like to hear stories of things that have gone wrong. So I guess, are there any mistakes you've seen happen when organizations are trying to improve their AI capabilities?

To me, the biggest thing is sort of like getting caught in the hype cycle, trying to get the latest and greatest and be like, okay, this is going to fix all my problems? They generally tend not to fix all their problems, but they tend to change. And so it's sort of like the question is like, how do you not get FOMO or panic? And we've seen examples of that. People are like, oh my God, I'm going to throw this thing out there and then everyone's like, this technology doesn't work. And I think you'll see a lot of like the early Gen AI things were like riddled with errors and hallucinations and it was like almost too fast. Doesn't mean you shouldn't try, but I think it's just sort of like don't panic. So I think that'll be one thing. But having said that, I think the most common error condition is having a plan to start, not having a plan to finish. Right. Which is like, because it's new, exciting technology, you want to learn a lot. So you want to say I want to build it all by myself. But then you're taking on these costs, which are again as I mentioned, like not just build time costs but also operational costs. Suddenly you're like, oh look, am I building a serving platform for LLMs and guards and hooking them all up and building, building a software infrastructure for scaling up and scaling down. Most companies didn't sign up for that thing. So really what you want to think through a little bit more, and I think I would say a lot of not necessarily failed projects, but products have had challenges, sometimes failed, is that you haven't thought through like, hey look, what should you build yourself and what should you buy? And sort of that mix I think is changing and I think people have to figure out like how to make sure that you are in the right spot on that spectrum because you want the company is authoritative on its business process, its data, but it's not necessarily authoritative on like how to run LLMs at scale or how to orchestrate a bunch of guards. And so knowing when to sort of buy a platform versus like put everything up from scratch with open source, I think that's a place where I think a lot of companies end up picking the build side because it looks like it looks very, I would say appetizer. Appetizer is not the right word but like they're very easy to say, oh, I'm going to get started, I have all these open source tools. But then owning and running them over time and maintaining over time becomes a much bigger cost that they have not planned for.

Yeah. So I guess the idea of starting or knowing how to start something but not knowing how to finish this is very common with Agile methodology. It's like, well, I can think two weeks into the future, maybe six weeks at a push. But yeah, getting to the end of the project is the problem for another sprint sometimes. So yeah, that's interesting.

And running an operational for forever. Right. If you're going to do this stuff, you got to marketing device is going to count on your mails. Like you got to have a system for doing that and you have all that workout. You're like oh, I hadn't thought about those five things. I thought I was just going to do this project and move on to the next project. You're like, no, now you have a mission critical system that's running. Someone's got to fund it. Like oh, we haven't thought about the funding. And so that's where a lot of these products end up sort of wrapping themselves up around the axle because they haven't thought through what happens if it is successful.

That's kind of a bad sign when being successful is also a problem as well.

So generally people do solve problems of success, but if you have a middling project that doesn't quite break out, then you're like, oh, am I in? Am I out? So then that's why so many projects today people say don't go to production because there's all this learning and you haven't sort of thought through all the downstream implications.

So you mentioned build versus buy is kind of the crux of the issue here. How do you decide whether to do one or the other?

I think it depends. We've seen customers of all kinds, like highly mature customers who are, who think that owning the technology is a competitive advantage and building it from scratch. They have funded their team, their, their IT orgs and engineering orgs are very, very deep. Then they have to buy less because they have the capacity. They're paying for it because they think it's a highly valuable technical, competitive advantage. But there are other businesses who would say look, owning technology and running technology is not my differentiation. What I'm really good at is the core of the business for this selling fruit or whatever else it is. Like, you know, pick your problem. And so what I'm really good at is my understanding of the industry or my understanding of the problem. And if you're that then you're sort of a technically less mature in the sense that you don't want to run all this stuff infrastructure, but you want to focus on that. So then that gives you a sense of like, oh look, maybe the infrastructure work I buy because that is someone else will warrant and run that run that system while I focus really on what is unique to my business and so sort of knowing where it is. And so my view is like with the exception of a few point apps, a lot of the AI will actually end up having These customized solutions for the company, where the customization is about what's unique to the company, not general purpose, like building the thing up. So you want to sort of buy the bottom, so to speak, the infrastructure and the platform type stuff and having you focus just on what's unique to your company. And that's where you sort of like get your differentiation really to sort of show up.

Oh, okay. So this sounds a little bit like the same thing we talked about with the skills earlier. So if it's kind of close to your core business, you're going to want to be like, dealing with that at quite a low level, like quite in house, and then you can just buy everything else that's a bit further away from what that's what you're doing is super. All right, it sounds like we kind of have a plan now then. So I guess maybe not. So, yeah. What are you most excited about in.

The world of AI? Two things. One is I'm excited that we are sort of coming off the peak of the hype cycle. I think the conversations are much more grounded, which actually makes us have rational decisions, because for the hype cycle, people just have to guess and hope they're right because there's nothing to solve hype. Right. And so now that we are sort of a lot more on, like, it works for this, it doesn't work for this. I want business value, which forces all the vendors and all the technologists to be like, really kind of focused on how to solve the problem. That becomes a much more interesting. And it's a game where we understand the rules, because in a pure hype cycle, no one understands the rules. And so it's sort of both the implementer and the vendor, all of us sort of have the sort of know what to solve for. So that's sort of my view, which is like, I'm most excited about that because I think we are going to see the golden age because there's so much technology. But now that we're going to say, hey, look, it's not just. It's a magic panacea, but it actually is going to be something that we can apply and test and do, and results are expected. Suddenly, I think we'll have this amazing sort of blossoming of like actual use cases that work and people trying to fix that. So that, to me is the number one thing. The number two thing is as a technologist, I just love how quickly things are moving. It's just like every day it's hard. It's hard for me and my team to, like, keep up. And every day there's a new competitor, every day there's a new startup. And sort of being able to keep up is hard. So it's an exciting time. But. But really, in the end, like, if it has to be sustainable, it has to generate value. And that, to me, is that. Focus back on that is super exciting for me.

All right, less hype, more value, and lots of stuff going on. Yeah, two very exciting things. Wonderful. Thank you so much for your time, Venky.

Of course. Thank you very much. Thank you for the opportunity.