EP 524: Agentic AI Done Right - How to avoid missing out or messing up.

This is the Everyday AI Show, the everyday podcast where we simplify AI and bring its power to your fingertips. Listen daily for practical advice to boost your career, business, and everyday life.

AI agents are all the rage. I literally just left one of the sessions and it was standing room only. But I think one thing that business leaders are constantly thinking about when it comes to agentic AI is getting it right. And, you know, you can either mess up or miss out, or you can do it correctly and really see a new level of productivity for your enterprise that you maybe haven't experienced in a very long time. So that's what we're going to be talking about today and a lot more on Everyday AI. What's going on, y'all? My name is Jordan Wilson. I'm the host of Everyday AI. This is your daily livestream podcast and free daily newsletter, helping us all not just learn what's happening in the world of AI, but how we can leverage it to grow our companies and our careers. And if you're joining us on the livestream, you've probably This is quite a different setup. I'm here at the IBM Think conference. Very excited to partner with IBM to be able to tell some of the stories. And the story has definitely been so far agentic AI. So that's what we're going to be talking about today, how you cannot miss out on it. So I'm very excited for our guest, Dr. Maryam Ashouri, who is the senior director of product management at Watsonx. Maryam, thank you so much for joining the Everyday AI Show.

Thanks for having me.

Yeah, the special edition here at IBM Think. But, you know, before we get into all the new announcements, Agentic AI, all that, can you tell us a little bit about what your role is at watsonx and IBM?

Absolutely. I'm the head of product for watsonx.ai, and the past 24 months have been super exciting. Like every day a new piece of technology is coming to the market. And mid-year last year, we saw the excitement around LLMs taking actions as agents. It's been revolutionizing every corner of businesses, and we are excited. Get with the new features and capabilities that we are announcing to roll out as part of watsonx.ai.

Yeah, so we're going to talk a little bit more about all of the agentic AI and all the new announcements, but a little bit on your day-to-day, you know, and maybe for some of our audience that isn't super familiar with everything that watsonx has to offer, can you tell us a little bit about the different, you know, products and services that IBM has just for those that aren't aware?

It's about AI. Like, we are building AI, but also we are consuming AI. So we have the platform that is helping enterprises customize their AI solutions, but every solution that we are designing also we use them to enrich a series of our software products. We have a series of products like Intelligent Environmental Intelligence Suite that are powered up and enriched with the foundation models that we are delivering. We have some new products like Watsonx Code Assistant that are powered up by the Granite code models, and we are also having a series of services that are helping together with the customers to look into their problems and see how AI can benefit from them. So we are looking at the very wide spectrum of how AI can help businesses through the platform, through the services, and through the products.

So speaking of solving problems, right, that's ultimately what this is all about, you know, new products, new services, new techniques to solve customer problems. What would you say with everything that, that was announced, and there's a ton that was announced here at IBM Think, what would you say is the biggest solution for those enterprise customers that are already maybe on the watsonx platform, you know, what are they able to maybe accomplish now that maybe last year at this time they weren't able to accomplish?

So the market is still experimenting with agents. They are still looking for a wow factor, an aha moment. But what we are designing is for production and scale. As the enterprises go through the journey to production, they soon realize the path to success is not as straightforward. There are major challenges there that are amplified with agents. And I tell you why. Let's start with ensuring a responsible implementation of AI. All the limitations that the LLMs historically had, now they are carried forward through agents because agents are powered up by LLMs. But at the same time, these agents are taking actions. They can access data, they interpret code, they connect to external services, right? They can leak data potentially if not designed well. So the transparency and the traceability of actions is essential for agents. Observability is challenge number one. Challenge number two, optimization. When you're looking for a wow factor, the larger the model, the more capable the model is. But we all know that the larger the model, it also requires larger compute. That translates to an increased cost, that translates to an increased latency, that's your response time, that translates to an increased carbon footprint and energy consumption. So the pattern that we are seeing in the market is moving toward getting, grabbing a much smaller, LLMs, even for powering up agents, fine-tune it on proprietary data of the enterprise. That's the data about your users. That's their domain-specific data to create something differentiated that delivers the performance they need for a fraction of the cost for their target use case, right? At the same time, why is it amplified by agents? Because this was the story of LLMs. You know agents, they have advanced planning capabilities. They have chain-of-thought reasoning. Inference time scaling, that translates to additional compute. So think about the scale of enterprise, the cost adds up, and that brings it back to optimization, cost-performance optimization, and why enterprises should pay attention to this. These two have been the guiding principles for basically everything that we announced, I think. Thinking about agent lifecycle, managing the lifecycle all the way from building it to deploying it and monitoring the performance of the agents is what we've been targeting.

So, you know, building, deploying, monitoring, it seems like even those three steps have improved a lot, right?

Oh yeah.

On the front end building, you know, now you have the agent catalog, you have the build your own agent, you know, you can use them as templates on the back end, you know, being able to trace and monitor a little bit better. And I love seeing like the chain of thought reasoning in an agent that you build for traceability. It's huge. What would you say from everything that was announced here, you know, whether you want to pick one of those three areas areas, but which one do you think is the area where enterprise leaders should first focus on? You know, are they— should they try to rebuild a different way? Should they monitor what's already, you know, working, going wrong, and adjust? Like, what is the best next step to make sure agents actually work?

Yeah, in order to deliver these agents in production, they need all of them. They need to build the agent, they need to deploy the agent, and they need to monitor the performance of the agent, right? If you are in highly regulated environments like finance or insurance, they have serious guidelines in terms of monitoring the agents. So for example, making sure the agent behavior is adhering to the policy of the company as an example, or they are monitoring the tracing of what happened, the agent behavior, not just for the purpose of logging, but auditability, right? So they have to pay more attention on that. But you said if you pick one, I'm gonna pick two. I'm gonna pick the one in the middle.

Okay, there we go.

The deploy one, right? Enterprises in average, the developers in enterprises are spending 18 hours in deploying and scaling a GenAI application. 18 hours. We don't want the developers to spend 18 hours. We want them to deploy their agents as a matter of seconds and scale it as a matter of minutes, right? That has been one of the examples that we've been focusing on. The deployment service that we just announced and released in the market gives developers a single-click deployment from the UI or single-command deployment from the command line. Ease of use is just deploy, and it's designed for the scalability of enterprise. Let's say that you are an enterprise, you want highly available agents. If one of the instances fail, you don't want to fail your workloads, right? The other one automatically through load balancing comes up so easily as a matter of, let's say, 2 minutes. The one that I tried yesterday, in 3, you can increase scale and instances of your agents. The last factor that I like to highlight here is access control. Enterprises are very concerned about security. Even for some of them, like some of the telecommunication companies that we work with, they have very unique security requirements. We have designed these deployment services in a way that the access control is managed by projects and spaces. So you have full control over who can access this agent on their for what circumstances to do what, which is essential for enterprises.

You know, one big, I guess, mindset shift that we're seeing a lot with enterprise leaders is, you know, they've been looking at the past maybe, you know, 2 years since large language models became popularized and they're like, okay, we probably made some mistakes along the way and that's with our smartest humans in control, right? But when we talk about now multi-agent orchestration and, you know, these agents that are actually so easy to get out, right? Less than 5 minutes, but then they're so powerful, you know, there is this fear of maybe messing up. So how can companies not miss out and also not mess up and kind of get it right when these, you know, models are, and these agents are so powerful and so capable?

Yeah, I would say that they should focus on the problem they are solving versus, hey, there is an agent, how can I use that agent, right? Because when you have your problem, you know exactly what are the expectations from this agent. And then if the technology delivers or not, if the technology delivers perfect, if the technology doesn't deliver, mitigate with everything in-house or the existing workflows that you have mix and match. Then look into the sensitivity of the workflows. For some of the workloads, the risk is just too high that you need to make sure a human is in the loop. But for some of the low stakes, like the example that I'm using is like if I'm using agents to provide recommendations for dinner, I probably don't care if there's a human in the loop or explainability of why I arrived at that decision. So the stake matters, right? And the third one is like just the industry. Industry, like what are the regulations, and, and think about the future, not the regulations for today. So just bringing that together, human in the loop, understanding the problem and the stakes, like what is the use case, what are the requirements for that. And then the last one was the designing for a responsible implementation of these agents.

You know, all of these, you know, capabilities that I even look at now, that with everything that's been announced here at IBM Think, I'm like, wow, this really not not only changes what's possible, but it also changes maybe how work gets done, right? Because if you would've asked me, you know, 2.5 years ago when I started the show and said, hey, today you can connect your enterprise data with an agent that can reason and it's a non-technical person that can put it together, I would've been like, okay, what does that mean for both technical people who would generally be building these things and then non-technical people that maybe wouldn't usually be taking advantage of all these capabilities? So how do all of these, you know, new capabilities just change the way that, you know, developers work and non-technical people taking advantage of it all?

It has already started changing every single one of us' life. Like, we ran a study with 1,000 developers across the States, the developers that are building AI applications, and we asked them, are you using AI-assisted coding for development? The majority of them, the answer was yes. We said, how much time saving are you getting? Most of them, they said 1 to 2 hours a day. Just think about it, the additional value that you can create by that 2 extra hours per day, that translates to acceleration in the speed of creation, that translates into freeing up the time of developers, or it's not just developers, every single one of us to do higher value work. And I feel like that's really where the opportunity lies and where I'm personally excited about because I feel like collectively as humans now we have way more time in our hands to do more higher-value work.

Yeah, you know, one piece of advice that you gave, which I think is great, is don't go out there and try to use agents. Go out there and find a problem to solve and find the right agent that aligns with it. You know, one thing we've talked about and we've heard is, you know, IBM had this massive, right, $3.5 billion in savings because of AI and automation. So, you know, when business leaders are seeing all of these new announcements from Watsonx everything else that IBM has going on and they're like, okay, where do I go? Where do I go to save time, right? Where should businesses be looking? Because it's almost like there's so many different agents, there's so many different places you can apply. Where should they be looking?

Two things. The first one is looking to LLMs itself and how they can help businesses. The most common use cases for LLMs are content-grounded question and answering. Customer care is a very good example of that. Code generation or content generation, classification, information extraction, summarization. So basically anywhere in your business that you have these workloads, they can be accelerated by GenAI. But then the opportunity that agents represent is bring that all into every single corner of your enterprise, blend them them these two words together through function calling and tool calling. So literally all of that acceleration can be mapped to even your legacy systems in enterprise. And I think that's where the opportunity lies. So I would start with LLM application itself and then look into, one, how can I bring that acceleration to every single corner of my business? And two, focus on problems, workflows. Can I use agents to automate some of If the answer is yes, go for it. If the answer is like explore and build your own and watch and see how the market evolves to solve your problem, then that's the path forward.

So I'll even ask you, so, you know, how might your work change in your department, your team's work change with everything that you've just announced? I know I'm sure your team has already been, you know, testing it out for some time. But, you know, I think maybe our audience can learn a little bit about how even your work might change with all of the new tools and features that we have available now.

That's actually fascinating. I run a team of product managers and my product managers are white coding. When we think about a new feature, an idea, they are showing me the fully functional prototype that they had coded and they are like, Maryam, this is it. And I'm like, is it real or what am I looking at? So I feel like This is literally changing everything. Like the way that we are thinking about technology, the way that we are thinking about solving problems, our problem-solving processes is already changed.

Yeah, that's amazing. And, you know, I always think, okay, our internal presentations and, you know, internal, you know, long rollouts are those things of the past when you can just like, you know, go in. I know there's the new, you know, Code Assist that you all updated, like, like, is that just going to be a thing of the past where it's just like, no, I'm just going to go solve the problem first and then talk about it and see how we can use it?

Is that it?

Like, is that going to happen?

Well, back to start with your problem.

That's—

don't get distracted because it keeps changing.

All right, so we've, we've talked about a lot here. I wish we could talk for hours, but, you know, as we wrap up today's conversation and, and hopefully advising, uh, you know, business leaders on the right way to take advantage of agentic AI and do it the right way, uh, what What is your one most important piece of advice or the one step that business leaders need to take in order to not mess up on agentic AI?

Know your limits and lines. It's like, what are the risks associated with your use cases that can't be jeopardized? Understanding the risks gives them a true and good lens to assess the technology. And along these lines is don't limit your people. Like closing your eyes doesn't erase the problem. It just lets you not be able to solve it and sit on it. So I would say that understand the risk, provide guidelines, establish the guidelines, go talk to the experts in the field to understand how can you mitigate those risks and be open to that and make it accessible to your staff and trust your workforce to find them the right way and help them and empower them to move forward as the AI moves forward.

I think that's, that's great advice and some great practical next steps for business leaders that are looking at all of these new agentic AI capabilities and they're like, I don't want to miss out, I don't want to mess up. Now you have the blueprint. So if you missed anything, don't worry, we're going to be recapping today's conversation and sharing a ton more, both at what, what was at the IBM Think concert, convention and a lot more. So if you haven't already, please go to our website at youreverydayai.com. Sign up for the free daily newsletter. Thanks for tuning in. We'll see you back tomorrow and every day for more Everyday AI. Thanks y'all.

And that's a wrap for today's edition of Everyday AI. Thanks for joining us. If you enjoyed this episode, please subscribe and leave us a rating. It helps keep us going. For a little more AI magic, visit youreverydayai.com and sign up to our daily newsletter so you don't get left behind. Go break some barriers, and we'll see you next time.