Agentic AI Architecture: RAG, Tools, and Multi-Agent Orchestration

All right, thanks everyone, thanks. How was the lunch? Good. No, good? Good? Okay, great. Of course, we can talk about it later. All right, and here we are. I want to quick poll before I go into the talk. Anyone from the New York local? Great. Anyone visiting from other cities? All right, so you probably can imagine like from our social media, everyone imagine like New York's like a great place. You have a lot of things, the food, the culture. But however, in reality, you might probably spot something, for example, the train's always late. There's also food trucks everywhere. So imagine you are dealing in the business, your company are facing the digital transformation using agent, agentic AI. This is the world we are facing now. It's kind of chaotic, so many dynamics. How do we learn from that about how to conquer that? So that's what I'll talk about today. Okay? Cool. I want to share a personal story. This is probably one of the biggest projects I've been involved in. Anyone heard about the Wizard of Oz on Sphere projects? Yes? So what we tried to do back in the day is my team got involved into the project called Wizard of Oz. And we're trying to use AI to transform the video, become like in the Sphere Theater in Vegas, 180 degrees. So a lot of things was not in the original movie, so we're using AI to do that. But that's a challenge. Think about using AI is not the way to solve it. How do we use AI in different ways? We need to have the agentic, maybe design a way. That's how we start thinking, how do we solve the problem? Right. So I want to bring you to this topic. This is me, Jonathan Chen. Nice meeting you all. And I'm the manager for the Generative AI Solution Architect at Google Cloud. And my team's main mission— here's what we've been really privileged to working with. My team building the cutting-edge frontier AI prototype and solution. And our mission is to showcase what is possible for different scenarios, industry, and maybe also different use cases. So we got very lucky and privileged to working very closely with the high leadership at Google. We've been working with Sergey Brin. If you know him, he's the founder of Google. So we've been working with a prototype for building for X Games for Sergey Brin. But we also have a privilege working with Sundar for Google AI, but also with Thomas Kurian, our CEO of Google Cloud, that working on very many of the high visibility events. So my team's job is to taking sure what's the scenario, what's the use cases, what's the challenge for different industry, Then we build prototypes to showcase this is the way to do it. That's why a lot of time we have so many of the opportunity to think through, hey, saying using AI is not a solution. How do we use the AI in the right way? How do we use agentic AI in the right way? That's what kind of challenge that we're facing, my team facing every day. Okay. All right. So I want to frame today's talk. It's not about, hey, here's a solution. Solution you can take away and start executing. But I want to frame this talk about here's some ideas, here's some trends that we've been observing in the market, and here's we've been seeing other companies, other customers, they've been using AI or agentic AI in different ways, what kind of challenges they are facing, and what kind of pitfalls, what kind of things we need to be aware. I want to frame today's talk about this way. Of course, there'll be a lot of talk about, hey, here's agentic AI that become the future, that can become wonderful things. But I want to throw a few ideas for you, and that will be your takeaway today. All right, so take a pause, think about how we got to this stage today, right? And maybe 100, couple hundred years ago, there's an industrial revolution. You can imagine that was a time when the steam engine invented. There will be other cool things that come coming out from the revolution, right? And how that become the full automation today? You can imagine there is a time when there comes an engine and people put together become assembly line and making sure there's all efficiency about how to make the product more faster and cheaper. And about coming out today, maybe past a decade, a lot of enterprises starting using the what they call rule-based engine or rule-based automation, right? As you can imagine, here's how you think about the logic. When A, then does B, or when B, then does C. Yeah. This kind of linear logic that become like a full fundamental for the automation for the last past 3, maybe 2 or 3 decades, right? But of course, you can imagine there could be a lot of limitation coming out. If there's anomaly, there's exception, there's always some scenario you've never seen before, then things can break pretty quickly, right? That's why a lot of companies spending a lot of time, even including like consulting firm, they're trying to come in, understand the whole flow, and try to manage this. So, Right. So a lot of costs are coming into this. That's why we kind of getting into the AI era. So on the left-hand side, we all know that there will be a neural network. So doing prediction, predictive AI, you have a model that can predict, hey, what's your customer churn rate? What's the weather for tomorrow? Or what's the Bitcoin price for tomorrow? Sure, I want to know. But now we're going into the next phase. Of a generative AI. This will be powerful for, by for example, Gemini, for example, the OpenAI, also Anthropic, many, many companies coming out with the LLM models, right? So a lot of times being very repetitive, but this is where we're seeing what's the next chapter when we talk about the automation, right? The industrial automation is the first wave, maybe a couple, a couple years back. Then here's the next evolution of the automation. Now, take a pause here. What does that mean to us? What do we try to automate? A lot of time we think, hey, automation could be just taking one task and making automatic done by a human being. Here's one scenario. But however, we are facing a lot of challenges. We may see some unpredictable things, complexities. There can be other scenarios you've never seen before. Can we use this technology to design a way that solve your problem for you and more productive, right? This is a number that you've probably seen before. However, Gartner, they project that by 2027, more than 40% of the enterprise, they will have embedded with agentic AI. So think about that. What does that mean? Does that mean like this is a solution, this is a way to go, everyone has to adopt agentic AI? Maybe. Put that thought. But the way that people starting to master agentic AI today will be the thought leader in the market tomorrow. Think about that. Yeah. So that's why some ideas I want to throw out here for you to take away. It's not about, hey, you have to use agentic AI, but it's more about can you do that something for today? Okay. Everyone knows about LLM, right? Yes? Yes. Okay, if you don't, we can have a talk later on. Yes. Here's a prompt, you throw it into the model, and the model gives you something as output. Cool, easy. This is called zero-shot approach. You can have multiple-shot, you can have some examples, you can give it some output examples, so you give it something out. But this is still more like reactive model. You have to tell what the model to do exactly what you want. So, we are seeing the trend, which basically is my— our observation across the whole industry. We are seeing the evolution from LLM, starting from the single prompt, the single LLM, you get some output that you want. Yes, that's great. As people starting building maybe last year, 2 years ago, people starting building RAG, right, the retrieval augmentation model. Basically, you got some kind of infrastructure getting information, you have a context from your databases, all from that become augmentation about your information, all from LLM. What's the next step? People thought about, hey, maybe I can get multiple tools. I can get to databases. I can connect to maybe your ticket system. I can connect to other things that become the LM, RAG, and the tools. That's why there are lots of frameworks coming out. For example, LangChain, the ADK, and many things out. But people are not stopping there. We're seeing many, many more coming up become wrapping this together, make that become agents. That's why the concept agent comes out. Right, it's not because just one model, you have other things supporting this agent, this model, to support you to get more context, to get more responsive. So that's why people kind of grouping them become agent. But the next wave is about, can you put multiple agents together to achieve the one thing autonomously? That's the trend we're thinking about, and this is happening today. So bringing back to the agent, what's the agentic cycle? What is the agent actually doing, right? Imagine I'm a person, I'm a human, maybe. So, I may be thinking about, hey, when I learn a new thing, for example, yesterday I was learning about Japanese. I try to perceive, I try to read those characters and see what I understand. That's the perceive stage. Then the next one is reasoning. Do I understand those characters? How do I put them together to become sentences? Then the act, meaning I put those sentences, I state it out to my Japanese friends. Then they don't really even understand what I'm saying. Then I learn. I get the feedback. This is the same way when you design an agent. Become the full loop of learning. So it's not that different from how human beings learn about. So imagine this is a full learning process. You observe, perceive, then you also reasoning what's going on, then also act. That becomes the full cycle of learning. Then Put that into a more complex example. Imagine you may have one master agent that may be taking care of your master, the goal, the main mission. For example, I want to master Japanese. Then the master agent will say, hey, okay, cool, let's break it down to subagents for you, subtasks for you. The first, you want to learn some characters, you want to learn some sentences, you want to learn other things. Those subagents will take care of breaking down into sub-missions. Then will be attached with the different tools. For example, the tool can be going to your microservices, your API endpoint, or maybe your database or internal document. Those will be the whole architecture look like. Hey, if you want to put one mission, just like in you're running a company, you may have a CEO. That CEO say, hey, that's breaking down the mission maybe to your marketing department, maybe to your engineering department. Then sub-agent will taking care other subtasks for you. you, right? That's one approach of it. We'll cover many, many more architectures there. Do you recognize the organization chart before on the social media? I don't want to name the company, I maybe insult them, but you may recognize this chapter, this organization diagram. Imagine when you're designing an agentic system, agentic AI, sometimes it may feel like you're designing organization. How you put those master or maybe the agent together, how you put who taking care of the charge, how you frame your architecture in a way largely influences how this outcome becomes, right? All right, then I want to make sure this is very important. Think about automation, but however, when we say automation, it's not just making sure that tasks can be done automatically. Most of the time in the business, especially when we come, when we're doing a lot of like organization charge or maybe structure, more about the workflow transformation, right? How you put the place, how you're starting the task, how you're breaking down the subtasks into different departments. That basically the whole end-to-end workflow transformation. So, This is the way we think about maybe traditionally, say, hey, let's do automation. Let's do step 1, 2, 3, then 4, then 5. Right. That's how we design, for example, the rule-based engine. That's how we design the task flow. That's how we design all the traditional automation. But going into the agentic AI era, agentic AI world, we are not doing this type of step by step. We're basically framing AI agent in the center And putting together what the context, what is all the things to support, what could be the edge cases, what the examples, and putting in that AI to learn by itself. As you can see, the cycle again: perceive, reason, act, and then learn. The very important part is learn. And we put the center agent in the center, making sure all the things supporting that learning. Of course, there'll be some time you need to make sure the AI actually learn and pick up, but once it learn, You don't need to deal with maybe redesigning all the steps. Imagine if there's any like exception in your workflow, you have to redesign the whole workflow. This is what in the past, right? That's why we think about it's not about automation, it's about the digital— it's not, it's about workflow transformation. A few quick examples for you is we are already seeing agentic AI everywhere. Right, AI been transforming your shopping experience already, but it's not only the shopping on the website. You can think about it's actually a master agent taking your request, breaking down to other subagents and doing for you. For example, there will be search agent, there'll be other, for example, your recommendation engine, there'll be also other things that doing all the subtasks together for you. So you're already seeing that every day, and this actually not just happening at Google but also for And any other company as well. They're thinking ahead. How can you bring this agentic AI framework to the world? Also happening, same thing happening for the travel, right? Imagine you're asking, say, hey, I want to go to, for example, next month I want to go to Norway. Hey, I want to get some, maybe a really nice trip to see the Northern Lights. Then I'll be the sub-agent to taking care of, hey, what should I prove? What should I book? Where should I go? What's my itinerary look like? Subagents will be taking care of the rest and collecting together become the result, right? We also been working many of the travel company. They're already thinking ahead how can they design in a way that can be future-proof. But that's also the moment for you to think about what's your business, how can you bring the agentic AI for your business? Would that be just for efficiency, adaptive, How does that transform your workflow, right? This agent can be broken down many different ways. We'll cover that later. But how would that become your goal? What's the problem you are solving? I want to make sure this is also a very important key point, right? But now let's take a look at, okay, cool, we understand what's agentic AI. It sounds great. It sounds like a lot of opportunity you can do, but how do you do that? Now we're going to the how part. Of course, you don't want to build from scratch. You want to leverage existing frameworks, the tools, the library, the things, external technologies. You've probably heard of LangChain, LangGraph. There's Google's ADK, Agent Engine. There's also AutoGen, many, many things, MCP, and many things. But even though those names differ, maybe they have a slightly different in their design, the tool they provided, However, the concept is still the same. What we are providing is a framework, a way that people can choose and pick what kind of architecture you want to implement for your agentic AI. This is probably the same thing why Google offers not just basically a tool, we are providing an ecosystem. For example, we have Agent Engine. You can host multiple different things. You have Agent Framework. You can use whatever you want, LangChain, LangGraphs. Google's ADK, or you can choose different models from the model garden, right? So what we've seen as a future is not just saying, hey, we as Google, we're taking care of everything. It's more about the community, what we're building as a whole ecosystem. But it's the same. It doesn't really matter, but the core concept, the core probabilities, properties for agentic AI is the same. You want to make sure your agents are atomic. You also have a goal-driven, very important. Everybody think about, hey, can I just throw AI agent for this problem that's solved automatically? But you have to take a step back. What the problem are you solving, right? And also important thing is adaptability, right? How does agent being also quickly adapt to whatever they never seen before, the new scenarios, anomaly, maybe exception? Right. So those will be how agentic systems being really powerful. Now let's go into really details how we actually implement or build an agentic AI system. Right. From the left, you can imagine you could just have one agent taking care of everything for you, maybe one small task. That can be what we saw before, which is you have the agentic cycle. You have perceive, learn— sorry, perceive, reason, act, and learn. That's a whole cycle for the agent to take care of it. Of course, in a single agent architecture, you can imagine this agent can be taking care of, being responsible for too many things. You've probably heard there'll be hallucination, there can be not enough example, maybe agent not sure which one to go. That's why the people introducing, a lot of companies introducing the multi-agent architecture. The central part is what we saw is one way down, you have a top-down decision, top-down, one agent Passing down to subagents and going through different tasks, right? And that can be talking to different tools, the resources like databases, API endpoints, microservices. But the new wave that we are observing now is the last one, which is called multi-agent collaboration. So imagine you are not— you are creating a company. The structure is not like, hey, the one top-down decision. It's more like people collaborate. Maybe more like academic. You have a sub-agent, many agents, they can take— taking care of their own territories, but at the same time, they can even debate. They can say, hey, I don't agree with the other agent's output. You can have a multi-agent become like a committee. So, that's a new wave that we think we are also observing. How you decide your agentic structure will largely influence its outputs. So I want to pause here to throw the idea. It's not we saying, hey, you have to use this one or that one. It's more about think about what kind of problems that you are solving. Okay, so on top of that, of course, you have a lot of agents in this place, and how do you orchestrate that? How do you manage that together is also an art. For example, you can have a sequence organization orchestration. For example, one agent Pass down the task to the second one and pass on the third one. Once they have the third one has the result, they're coming back to the first one. That's called the sequence orchestration. You can also have a concurrent orchestration, like the first one passing down to the second tier. They're all working together in parallel. Or maybe you can have the hands-off pattern, say, hey, this agent, I'm done my— agent A, I've done my job, I pass over to agent B. And B, just working independently, isolatedly, and passing down to the third one. So, there are multiple different patterns that we've been experimenting about, but more so, like I said, it's not about which pattern you're choosing, it's more about what problem you try to solve, right? Okay, I want to quickly go to this type of the risk. So, imagine you have so many agents, That being taking care of your task. You have the credential from you, the agent able to get the token, access to the database you have. But think about that become more complicated. You have multiple layer agent, right? You may have the agent you trust able to access databases. You have other tier of agent try to get talking to the agent that you trusted but able to pass it down to your databases. That's a huge security potential risk that we are seeing today. So I want to make sure this is really good takeaway for you. The first one, think about the days when there's a JavaScript Ajax been coming out, maybe 20 years ago. There'll be a lot of people trying to do what? Trying to insert the code or insert some JavaScript code to trick the API to execute for you. So now we are seeing the same thing for agentic AI cycle. People may be trying to think about, hey, how can I make the AI, hey, make an LM to Do the things I want, maybe getting your username, your password, your credential. That is the remote code execution. And second part is the— we call it memory poisoning, right? The LLM has a long-term and short-term memory, and most of the time people keeping that short-term memory become something you can extract from, right? You can say, hey, LLM, tell me about what's the last conversation. Tell me about what's the last database table you executed. Something about that. So, that would be the memory poisoning, trying to influence— also memory extraction as well— trying to influence the agent able to get something out from it. So, I want to pause here a little bit quick. It's now, right now, we are providing an ecosystem, but we are also thinking really ahead. It's not about this is the way to go, but this is about, hey, we've been observing it different ways. But I want to pause here. It's not, hey, here's the AI, then let's do it. People might think about, hey, that's throwing AI, genetic AI, everywhere, every problem I've seen. But this is probably basically a traditional problem, hammer and nail. Everything looks like a nail right now, right? So what is really a takeaway for you? I think we're along the time, but thinking about AI's genetic AI cycle, genetic AI structure, It's a workflow transformation. It's not purely just automating your task. It's redesigning, for example, your org, your workflow end to end. Think about that. But also the second part is making sure you have really good governance. Like we talked about security risk, your token, your credential, your privileges. Is that good enough? Is that covered? Is contained? Making sure you have that governance in place from day one. But those are very important. Start from small. Don't try to reinvent the wheel for the whole company for everything you've been observing. We don't want to replace people. We want to elevate people, but we also want to start from small. Something you can test, you can have the really good experiment, or something you can start from working and just doing tomorrow. Okay? All right, I think we have about on time. Saves 2 minutes for questions. Questions for Jonathan. That was, by the way— Thank you, everyone.