#316 Enterprise AI Agents with Jun Qian, VP of Generative AI Services at Oracle

Generative AI is transforming industries at an unprecedented pace. But as AI changes how you or your team work, one thing is clear, your skills also need to evolve. At datacamp, we offer everything you or your team need to adapt and thrive with AI. Whether it's business users looking to get the Most out of ChatGPT and Copilot, or developers and data scientists looking to fine tune models. You can learn the entire AI skills spectrum on Datacamp. Power your AI transformation. Today start learning@datacamp.com from the foundation model perspective. And we have a lot of good models, Answerpic models and Gemini models, Openlab models. Oh yeah, what make the ChatGPT difference is agentic features, right? Think about it, right? And I'm pretty sure you use ChatGPT all the time. But what makes Chat GPT different? I think that's a secret sauce. And also my prediction is like GPT5 is not a model, it's an agent. I think mostly maybe you know, five years later everyone going to have this R2D2, right? And you know, we all like R2D2. I think that's clearly trend, right? And we all have our digital companion physically. If you already have open search base knowledge base, you can easily add another layer of the reg on top of it, right? Build a decent enough reg system and however your data refreshing data management system can be as the same as they have today. You don't have to redo everything. So this is one possible way, right, to move direct system forward quickly.

Welcome to Data Framed. This is Richie. AI agents are the hottest of hot topics. But there's a big leap from your CEO declaring your company to be all in on AI agents to actually getting some value from them, especially in an enterprise setting. So today we're going to try and figure out how to cross that chasm. I want to know how to make your organization AI agent ready, how to manage your corporate data for use with AI agents, and how to manage and govern those AI agents once they're in production. Our guest is Jun Kian, the VP of Generative AI Services at Oracle. Jun oversees the development of Oracle's Generative AI Services, Generative AI Agents and Code assist platform. In his career, Jun's been responsible for creating some pretty high profile AI products at Microsoft. He founded Microsoft Power Virtual Agent, now part of Copilot Studio. And Jun also ran the AI and machine learning department at fraud detection platform Sift. So Jun certainly knows a thing or two about getting AI into the enterprise. Let's hear his advice on adopting AI agents. Hi Jim. Welcome to the show.

Hi, me too. Nice to meet you.

At the moment, I think AI agents are the hottest topic around. So first of all, can you just tell me what's the coolest AI agent you've seen so far?

Ah, that's a very tricky question. I think you know it. You use it every single day.

Go on, what's that?

Yeah, which is ChatGPT.

Okay, so we're counting ChatGPT as an agent now then.

Yes, I think you think about it, right? And everyone have the model access for OpenAI models. But why? And the ChatGPT experience is quite different from a lot of agent you saw, right? And then think about, right, and go back to 12 months and even more than 12 months, right, when ChatGPT first started, it's missing a lot of features. For example, web search features and memory features, code generation features. Right. Some now actually it generated code, the wrong code. So think about it and from the foundation model perspective. And we have a lot of good models, enthropic models and Gemini models, OpenAI models. What makes the ChatGPT difference is Agentic features. Think about it, right? And I'm pretty Sure you use ChatGPT all the time, but what makes ChatGPT different? I think that's a secret sauce. And also my prediction is like GPT5 is not a model, it's an agent.

Interesting. Yeah, that'd be very cool. And I certainly right there with you that just having these simple sort of tool use features, being able to search the web, being able to just interact with like Python or whatever code tools, it just makes such a difference to like the quality and the range of things that you can do well with generative AI.

Yeah. And also key thing is memory. Memory is also very important because you probably noticed and openhead launched called memory features and not long ago, three, six months ago, however they're actually using these memory features for more than six months ago. If you go to your ChatGPT settings, you can actually see what they memorized all the conversations you had with ChatGPT. And also there are another secret command you can type in ChatGPT called bio. We just had a bio into ChatGPT. You can see all your informations they saved for you. So I think that's one of the secrets, right, and why it makes ChatGPT so relevant to a lot of questions you have. But I think that's agent, right? So memory to calling, that's the key component of the agents.

Yeah, yeah. So the ChatGPT memory feature is really interesting because I went into my settings. I had a look at like what it was remembering about me. It thought it was really into ultra running, which is very odd. I'm very sure like I will go jogging once a year. So I don't know where it's got that idea for me from. So yeah, you just need to be careful that it is remembering the right things about you. I know at Oracle you've just launched a RAG agent. Talk me through what is that and what's it for.

Correct, correct. I think everybody knows rag, right? And we don't need to explain what is rag and however I think why it makes rag so popular. If we ask these questions, for example Rishi, I think you know the rag concept from your perspective. What do you think why is rack suddenly get too popular? Right? And remember about eight years back, right, we had chatbot and everyone's building chatbot, but now everyone is building rugby's chatbot. So what is the difference between the chatbot we built 8 years ago now the rack? So I'll give you my answer because eight years ago I was at Microsoft. We are building this chatbot for enterprise customers. At that time we built chatbot for Macie for hp. And what is the key challenge at that moment when we build a chatbot in the high level? There's two technology when you build a chatbot. First one is we call it implant understanding. When you ask questions and then the model need to understand what kind of questions you're asking. We call the implant classifier. This is a very generic NLP technology. Just like Alexa, Ashuri, they all have similar technology. You ask questions, they try to identify what is your intent. In the handle you can think about is a classification problems. So you can have like 100 yin time and 200 yin time. So that's the first step. When you ask the questions the model try to understand what's your intent. And the second part is once you understand the intent and will try to give you answer. So essentially you will hard code all these answers related to this intent. Again, I'm trying to simplify the problem. Now you have this intent, classification and the knowledge retrieval kind of like two part view, this chamfer. But what's the problem? The problem is most of the chatbot cannot follow up your questions because they don't understand the context. That's one challenge. And the second challenge is because your knowledge base has to keep updated all the time, right? Because once you have a new product launch then you have updated your answers. So this ongoing knowledge based maintenance become a challenge. And also when you ask questions, for example, you ask okay, what's your open hours in Christmas time for Macy? And then you have to understand all these details. Daytime location, we call it slow filings. And you need to understand all the details. We can use classic NLP technology for sizing all the data. This is a classic NLP technology like eight years ago, Internet understanding and knowledge building and that's one of the parts. However, I think the challenge is this kind of chatbot or you can think about SRI and Alex as well. It's kind of silly. You can only answer specific question you have. You cannot really have a dialogue. So that's why the chatbot I kind of got popular solve a certain problem but didn't really go too far. But now in this large language model world, you see how this large language model help to solve the problem. You don't need internal classification anymore essentially, right? Because the language model understands your language really really well. And secondly, all these large language model can very easily understand your follow up questions. You don't need another guess. So basically the built in natural language understanding of the large language model is so advanced. So we just solved some of the critical problem. So you don't have any language understanding challenge anymore. And however the second part, you should still exist because the reg system, right? You know the reg system, right? And so we still need to keep updating other knowledge base or the database you have keep all the knowledge. I think that part is still existing today. No matter what kind of rank implementation you use. How do you keep updating your knowledge base or your database to reflect the question you're asking? I think this is still one of the challenges we're dealing with today. That's from the high level.

Absolutely. And it sounds very simple. But just the idea of having chatbots that your customers don't hate, that's just absolutely huge for so many organizations I think.

Exactly, exactly. Yeah. I think one of the success use case we have, right? Because once we have the rag system, right? And even we only have the generation model, you can see a lot of teams started building their own racks because on the high level, right? And very simple knowledge, right? And if you have already have a database, for example, if you already have your data, you open search, you can simply do the query to the OpenSearch database and then you get the search result back. You do rerun, you do generation. Everybody understand, right? Even very simple few steps, you can already generate very useful system. Of course we need to do other issues and knowledge updates hallucination. However Just introduce a very simple concept and already solving a lot of issues. For example, inside Oracle we have the, we have this Slack channel, right. Essentially you can ask questions when you run into issue. Oh, I cannot log in my laptop. I have the data string issues before. We have the human support team to open ticket for each question you ask. Now it's just like the automatic chatbot RAG bot. Just answer the question. Do I give you the answers and the instructions. But I think it's just like a very simple implementation of RAG in the support system can save a lot of the effort, right. And support effort at any enterprise, right. I think that's kind of a low hanging food, right? To build a rank system for support. And I think, right. That is a very basic starting point and you can achieve dramatic results immediately.

Yeah. So I like that with these sort of large language models, okay, you solve the sort of natural language understanding. You've solved a lot of the issues around context with rag. It's like you've got accuracy from your knowledge base. But you mentioned the big challenge before that keeping knowledge up to date, making sure you're retrieving exactly the right information in a, in a scalable way. I guess that seems to be the big problem. So it sounds like knowledge management, data management, these are the big challenges. Do you have any advice for organizations on how they can manage all this data for their, for their agents, for their chatbots?

This actually go back to the traditional enterprise search problem. So let's say you forget RAC and you just want to build the enterprise search because search the same thing, right? You want the latest information. So essentially you can think about, okay, yes, we're not born in the ocean here. If you already have very good enterprise search system, for example, like a lot of teams using OpenSearch, if you already have your knowledge base and data in the OpenSearch, which you already keep refreshing regularly. So essentially you already have a good knowledge base to start with and then you can build the RAG layer on top of it, right? So basically you have the query rewriting and the rerank and the generation. So you already build a very decent reg system to start with. But again that's not the end. But so essentially you already have a real time RAG system with your existing knowledge base management. So I think that's actually one of the interesting challenges in the enterprise is because when you saw a lot of the open source POC implementations, it sounds very easy, right? You can easily build a poc, right? And you can easily show the result, however, right. Once you get enterprise customers, you will find the challenge because a lot of enterprise systems they already have the knowledge base, right? Clearly it's a lot of effort if you want to rebuild the knowledge base, for example using vector dbs, that'd be huge effort and a lot of challenge. However, you can leverage the existing knowledge based systems, right? You don't have to rebuild everything from scratch. So for example, in this case if a customer already have, let's say they already have open search base knowledge base, you can easily add another layer of RAG on top of it, right? Build a decent enough reg system and however your data refreshing data management system can be as the same as they have today. You don't have to redo everything. So this is one possible way to move RAG system forward quickly to have the team or enterprise systems to adopt the regular concept to start with.

Okay, so it sounds like if you've got really good enterprise like knowledge management, then creating the that RAG layer like it's just embed everything thrown to a vector database, that's probably going to be give you a decent start. There was a big if though, that's like if you've got the good enterprise management system.

Exactly, exactly, exactly. So that's actually get the second part. But if you're just not newly saying oh okay, plug into this open search, right? I send a query, get a result back, I do a rerun generation, I'm done. Clearly it's not. I keep it simple. Right when we started I said wow, yeah, that sounds very straightforward. And then we run the issues. Why? Because the traditional search results, right? And if you don't do specifically chunking, it could return the whole document. So you run a query and the search system give a return the whole document which is not useful for the further processing because once we get the results back and we want to rewrite, for example to measure which result is bad, however you just refund the whole documents back. It was helpful. So that way then you have to think about oh okay, our data ingestion pipeline actually needs a change. But in this case when you ingest data into the open search then you had to change it, right? Instead of sending the whole document as a single chime, then you want to chime into smaller pieces and then for each smaller pieces and for example OpenService, they already have the built in embedding mechanism, right? So basically now you get into the second part and how can you modify your existing knowledge management system? How do you modify your existing data ingestion process to make it better for the New reg system. So I think that's exactly getting to the second phase of the react implementation. Right. So the new phase is oh, I just poke into the existing system and do re ranking generation and reflection. You know, that's probably okay, start and then you get oh, that doesn't work but some use cases why? And then you get into oh actually we need a more smarter trunking algorithms. Right. So you get a second part of the problem and then eventually you will get into the more concrete problem. For example, when you ingest the data, if you have some graph the image, the existing data ingestion pipeline may not work because you don't really processing all the image data. Then you get the next piece of the rep system. So oh, actually we need to do this multimodality ingestion and the generation. Right. So I think. And that's why I like your podcast. Right. With that guy for the rec 2.0. He's very practical. I don't remember his name. Yeah, because once you start building the system layer by layer problem, problem, then you get the deep and deep of all these issues. However, if you just read a paper, read some blog, you don't build it, you won't be able to fully understand what are the challenges here. Clearly opportunity is huge, but if you don't build it in real, you won't understand all these challenging features we are facing in the real enterprise environment.

Yeah, okay. Yeah, so that was Dawah Keela, one of the adventures of Rag who was on the show recently talking about the place innovations in retrieval, augmented generation. But yeah, so that's interesting. It seems like then you've got a very iterative process. So you're going to start off, you're going to have like some kind of chatbot and then there's gonna be a lot of, a lot of tuning goes on just like figuring out all the algorithmic stuff to make it work properly. So you kind of covered the technical side. I'm curious as to who's involved like on the talent side, like which teams or roles tend to be involved in this.

Oh yeah, that's a great question. Yeah, I think generally speaking, right. And you need a data scientist, right? Engineering. Yeah. For example in the RAC system. Right. It's of course you need the engineering team to do all the integration work. However, I think the data science is super critical because once you build the RAC system how good it is, you need a measure. So essentially you need to measure not the single system, you want to measure the end to end accuracies and recalls. This is typical data scientist work. We prepare the sample questions, we know the ground truth. You want to build the end to end evaluation data set to start with to systematically measure the improvement of your RAC system. So I think that's the one key part is not only build a RAC system, don't measure, of course you have a human and testing to measure yourself. However, you should build your evaluation data set. You should have a data scientist and measure this right systematically and continue to improve your evaluation data set as well. Right. You should add more and more questions, more and more answers continuously to measure and improve your system.

Okay, so measuring does seem incredibly important. I agree that's very much data scientist work. But what sort of metrics will you use to see does the system work on it? I mean especially if it's, if it's non deterministic. So you're going to get different, slightly different results each time. Talk me through it.

Yeah, definitely. I think in the support scenario we have one key metric called deflection rate. So this is very standard across industrial even eight years ago when I was in Microsoft. Right. And this deflection rate is the question the bot can answer and result. Need a human involvement. Right. For example, when we start a deflection rate of 20% is pretty good, right. Essentially 20% question you can answer by bot accurately without talking to human. That's great. 20% is pretty higher years ago. Right. When we started this. So wow, you can solve 20% of the question. That's awesome. But nowadays and for example the RAC system support system we built, we could even reach 80% defection rate. So that dramatically improved. And for example, the experiment had eight years ago, right now we come from like 24, 15, 20% to 70. 80%. That's the dramatic difference. Again, that's just one of the metric you could measure like a deflection rate and we can use as one as a proxy of the and accuracy of the system.

Okay, that seems like a pretty useful measure. And I guess do you need like is it going to be largely around like end user measures? Like this person found the answer to the question they wanted or like they clicked on the first results or it feels like you need like a whole suite of different measures and sort of different stages.

Oh yeah, yeah. This is one simple. Right. For example, for any chatbot system. Right. And once you provide answers and if you don't have further questions. Right. And they just quit a session and however, if they continue to have questions. Right. And there are always Buttons there. Right. You can click talk to human for example. Right. I think every single chatbot system have that function. So I think that's a good proxy. So if the customer don't click button and talk to the human, I think that's one very strong signal. And also because when we generate answers that you also have links and if a customer clicks the links, that also very strong signal. That means oh, you provide the useful information once they click link and can find their answers they need. So yeah, but I think all the chat bot system will be with Microsoft and Oracle and also on this market. And this signal is very clear. Talk to Fuma as automatic. I was thinking though.

Oh yeah, that's true. It's like oh, this chatbot's annoying. I want to speak to a human. That's a sign that your project is not working. That's yeah, very, very clear, very straightforward.

But very, very useful.

Okay, actually that's off my default behavior right straight to you. So I'd like to talk about some of the issues about adopting chatbots and agents within the enterprise. So I guess it seems like data privacy, data security are sort of top of mind in terms of the challenges. Can you talk me through what people should be worried about with these things?

Yeah, I think this is one of actually the advantage for the existing enterprise systems, right. Including Oracle, Microsoft traditional. They're very strong on data privacy and the securities. Right, yeah, same thing. I remember you talked with the entitlement in the previous podcast. I think that's actually it's very true. Right, because when you build a new RAC system and I think one of the key challenge is how to integrate with existing enterprise system including entitlement and authorizations. Because certain information is only available for certain customers. You cannot dumbly index all your knowledge base and then just throw it out. Right. I think you're probably saw some of the use, right? And some company build some react system. Now every single employee you get the same information as a CEO. I think the key thing is and security and the data privacy compliance is super, super important. So when you design your system you had have this in your mind how do you integrate with your existing security and authentication authorization system? Which is actually is very tricky because if you don't design your system carefully, including if you just stumbling index everything, I think it'll be very tricky. But the good news is a lot of the enterprise providers, including Oracle, like Microsoft, they already have very strong security and compliance policies. I think for us if we start building the new agents or the Gen2AI solutions. We work very closely with all the security, legal and compliance teams right. Within the organization and leverage their existing experience. I think that is the key things. I collaborate with existing security, compliance and data privacy group. Every single solution we build, we always go through the security compliance reviews with the internal team. So make sure the solution we build is fully compliant and secure with your existing policies. You don't have to invent new things, but make sure the thing you build is secure and compliant with your existing policy. I think that's the case.

Okay. So yeah, in theory, the existing sort of enterprise software using hopefully those policies that you've already developed over the years, they're just going to sort of naturally apply in the, in the AI use cases.

All right.

One thing that's I was having a conversation with one Datacomp's customers. They were struggling with. Okay, the technology works, but in terms of the organizational side of things, there's teams wanting to like get started with AI and like build things as quickly as they can. And then you've got a compliance team that is like, nope, nope, nope, nope, nope. Because they move a lot slower.

Yes.

Do you have any sense of like how you can resolve this sort of tension between governance and speed?

Yes, exactly. I think that is our ongoing challenge. However, I think challenges also opportunities. I think the first question is why is compliance? Say no. Then you have to go back to I get a no every day. You have to delve back. So I think the question is you need to educate the team. Right. For example, two years ago when we started a lot of our compliance legal don't really understand how the large and the model chain AI works. I think. And we spend a lot of time working together with our legal team. Compliance team is more like education as well as learning together. So I think the build a partnership is very important. Of course. Right. And from the compliance legal perspective, it's so much risk. From their perspective, they seem, oh, everything is so risk. Right. If you model general anything, how do we control this behavior? Right. I think it's more about education. Right. And you so service, for example. Right. And we need to understand. Oh, okay. Large model itself is also software. It's a piece of software. It's not like magic things and it's a piece of software. And then how do we control this behavior? How do we monitor the behaviors? And once they understand the concept and once understand how do you control the behaviors of the large language model and then they will have better understandings. Actually sometimes we will provide even more insightful suggestions for you to solve another problem. Right. And now our legal team is now get very knowledgeable. Right. When we start discussing about compliance, data privacy, sometimes they are actually on our side. Try to ask the people, okay, this is not a legal issue. And as you do the zero data retention, you don't keep any data. Yeah, it should be okay. Right. Because we start some discussion with the partners and then we understand the concept, then they can help you sometimes even accelerate some of the process. I think the key thing is really build a partnership, right. And educate and your team, your legal compliance team and also work together. It's not easy, right. I spend a lot of times working very closely with it. But now once they have the knowledge, right. Once they have the understanding and then it's become much, much easier. Of course certain compliance you still have to do for example. Right. And we, you know, like llama when we launch Llama 3 models. So it's multimodality capability. We cannot launch a new right because that's just a law. We have to comply with the EU law. You cannot use the multimodality model. So for language model then in certain, certain law you have to comply. They just had no way around. But some of other things are more about I think the understanding, collaboration and make sure. Right. And we all on the same page.

Okay. Yeah. So I'm definitely an agreement that just educating different teams on like what. What's possible, what should be allowed, what's a, what's a good idea or what's a bad idea. That's really important. And yeah, it sounds like that. The trick then is that you've just got to have some sort of agreement with these compliance teams on like general principles for what should be allowed in general and what shouldn't be allowed and that that should resolve most of the sort of use cases just to enable you to get on building stuff.

Yeah, yeah, exactly. I think I will put this in the beginning. You know, you probably want to put extra effort because from legal compliance teams they also want to learn the new technology. For example, I build a few PoCs demo to them and have some sessions work session together with them to tell them what exactly is the knowledge behind the scenes. I think that could help them understand the technology and in the long term to benefit you to move the project forward more quickly. You don't have to repeat some of the discussions again and again. Once we truly understand and how the large language model get trained and how the large language model get called in the inferencing time once and they understand all these concepts and it'll be much easier.

I just want to return to something you said a moment ago about not being able to use LLAMA for the large language model from Meta in the Euro. This is something I've not heard about. Do you want to talk me through what are the issues there?

I think they have a policy, right. If you process image and there are certain restrictions in certain region where you cannot using the image processing features in the EU regions. I remember right. When the Llama three start. Right. I think Llama three on three starts all the image and sending features. However, I think that certain EU rules, right. You cannot process image. I think that's why we had to turn off the image processing features. Right. In the EU regions. Right. We only allow that in the US regions, for example. Yeah, that's one of the very specific rules or had to complex. Okay.

Yeah, that definitely seems worth taking note of at the point where you're choosing your models and also deciding sort of where things are being deployed.

And yeah, I think again that's actually go back to the data privacy and the sovereignty things. Each region and state may have their own policies. For example, data privacy policies in Singapore and Australia and it could be different from the US you had to really compliance and with your local policies.

Absolutely. Yeah. Always a good idea to check all the different local policies around the world. Okay. So for organizations who are sort of wanting to get started with these things, we talked about having to put governance in place beforehand. Do you have any advice on like what are sort of good first use cases? Like I don't know whether you can jump like straight to where creating some fancy agent or whether like you want to start with something simpler.

Oh yeah, I think I mentioned. I think RUT actually is a really good use case because I think it solves a lot of problems. Right. To service. Right. And if you already have the knowledge base built today, right. For example, you already have some internal search systems and build a rag system on top of it. I think that is very tangible, right. You can generate very tangible results. And for example, the use case I gave. Right. And support the chatbot click the system. Right. I think that's a very tangible things. Right. You could build and show the immediate results and of course. Right. And a lot of other use cases you can think about. But I think to start from support channel, I think that should be a very decent starting point and you can see some tangible results. If you already have some knowledge base system and inside your company, I think that should be a very straightforward Starting point. Okay.

So yeah, start with chatbots and then kind of work forward from there I guess at the other extreme of things. So there are a few startups that are trying to have complete like AI employees. So you've got Cognition Labs has this like Devin software engineer. There's like Julius AI has like an AI data data scientist. EMA's trying to create like a universal AI employee. Like some really, really ambitious projects that I'm not quite sure work entirely yet. Is there like a sweet spot for like where you should be aiming with like with agents?

Yeah, I think that's actually if you think back right. In the past two years, what are the two prominent use case about large anchor model? I think one probably we already talk about react and then the second one is really code agent. I don't know, I'm pretty sure. And you already tried some of the code tools like Windsurf, cursor. Yeah, I use cursor as well in my spell time. So I think if you're using cursor for long enough or any other code agent they evolved quickly. Right. And so start with is just complete your code and then you can generate the whole code for you and then you can read your repository, generate more config code and now they can generate the multi steps and they say you give a client they can execute. Right. So you can just from for example for any of this code generation tool you can see they're involving from very simple task completion to now kind of like a fully semi auto coding task. Right. You can run for example 20 minutes. Right. All the tasks. Right. And you can plan in the task and run all the tasks. I think we call it Async and Code agent. Right. And the latest I think codecs and cloudy code we already went to intranet direction. So clearly I think a code generation automation is one trend very clearly and is very productive already. I'm pretty sure a lot of the developers if you're not using co generation tools, you're probably out. Yeah. But again go back to your question. I think and this is, I think it's still just the beginning and we saw the adoptions internally. We also have the code generation tools where we developed and our team we called Oracle Code Assist and we beta launched this service and last year. Right. And internally and we have a lot of teams using it as well. So I think code clearly is another trend you probably we all know, right. The wind SERP news. Right. And all this news, right. Code I think the code is clearly one winner. Right. In this Generative AI world, right? Rag code clearest winner, right. And we'll discover more use cases. I think this just started, right? And we clearly see some patterns. But I would assume. And once we get into, deep into the application, for example, I'm always thinking, because I've been doing Software more than 30 years, so what are we thinking, right? And is there any fundamental, like a computer system operating system and even compilers and should we rethink some of the fundamental building block we are doing today, like OPERATING systems, right? And for example, all the command line today we use still a little bit of dynamic, right? And you need to find the. Grab whatever, right? And come out some of the information you need from your laptop, for example. However, think about, right? If you have a natural language interface, you should just. Oh, okay, can you find me the resume I saved in my disk a month ago and with the name blah blah. And then this smart command line should find my data from here. I don't need to run this crazy, you know, UNIX command, right. Python information, right. And this could be built in very soon. I would assume this should be available very soon as a part of building your operating system, right. For example, when you get a max, right. You should already have this very smart command line, right? You don't need to remember all this UNIX command anymore, right? It's just there.

That's funny. Yeah. So on the coding side, I'm right there with you. Like if you don't know how to use these sort of AI assisted coding tools, then you're not going to make it as a software developer these days. I mean, I think the point is though, they're quite easy to use. So it's something that everyone should look into if you're writing code. But yeah, I do like that idea of just better software search. Like there's so many times where, yeah, I cannot find a file that I'm pretty sure I downloaded or I created like a while ago. Even like, like we use the Google Office suite and somehow, even though Google is a search company, like Google Drive search is absolutely terrible. Like they're supposed to be, they're supposed to be good at this stuff.

Yeah, exactly. Same thing, Dropbox, all these things. So think about the software you're using every day. Clearly there's a lot of things you can do better. So that's why if you think about and always get very excited and basically I'm thinking the software we built in the past 30 years can be rewritten in a dramatic way. I'll give you another example. So we all know the web scrapper five years ago when we built this web scrapper. You get an HML page, then you try to do regular expressions. I do all the things, I try to match a certain data field for example. However every website is different and then the website change all the time and then you have to change your, you know, web swiping tools for example. However this is, it's very simple. You just send this to to large language model. You dump the HTML, send to the LLM said oh I want this field. Then they will give you all information back.

Oh yeah, I'm with you on that. Like web scraping is such a tedious task to do as a human. It's. It's so annoying. And yeah, I do like that actually.

Yeah, so actually I like to do all the experimental things in the vc. I do two things right. One thing is oh, I just download the whole raw HTML I sent to the area that gave me all the answers back really decent. Another way is actually you take a screenshot sent to iron back too. So either way works perfect.

Nice. So yeah, lots of opportunities then I think for just making software better. So you mentioned the idea of like enterprise search and also just like operating system search. Since you've been in this game 30 years, do you have any more predictions on like things that can be improved using AI over the next few years?

Yeah, I think you know like for example, right. And if we this industry long enough, right. For example 30 years back, right. You know I still remember when I was high school, right. I heard also AI and the expert system. I was so excited. Wow, expertise, that sounds great. And then studying. I remember it's a perlog, right? That's one of the language at that time, Perlog, right? They call this AI language. So I started using Perlog that built my own expert system. And then I found out oh actually the so called expert system just if then else I got quite disappointed. I said wow. Then you had to build all this knowledge based knowledge graph, handcraft, all these things. It's just like another practical, right? And then 30 years later we had this large language model. Essentially any of the large language today is a super super smart expert system. We cannot even imagine to build 30 years ago. If you're 30 years ago you can build an expert system like any of these larger today it's like an impossible task. So that's why you know I was so excited when this thing happens. I said wow. Finally after 30 years we finally built some decent expert system. So and then I think thinking Forward, I think, I think again, I think there's like a plenty of things even go back to 30, 50 years ago when we started thinking about AI. And I think a lot of the vision and The Pioneer had 50 years ago, now we can finally implement them. Like the third tooling test. Right. Oh, we finally encountered. And now the expert system all we already built it. I think I probably could just go back to read all these old yearbooks to say what kind of dream they have. Fifteen years maybe now it's become a reality.

Yeah, I mean definitely like there's just been so much progress over the last few decades. Hopefully it's going to continue in this area because like stuff that we've been trying to get working for decades, it now actually does work.

So yeah, robotics. I think robotics probably next line. I was just joking with my case. I think probably maybe five years later everyone going to have this R2D2, right. And we all like R2D2. I think that's clearly trend. And we all have our digital companion physically. And now today we already have this digital companion like ChatGPT. However, I think after five years I would assume, right, we will all have our little cute robotics robots with us and we will all have the autonomous driving. I would assume. Right. I don't particularly like drive myself, I think. Right. And you know autonomous driving should be zero. Right. In 10 years. Right. Why we need to drive. Right. And we have all this Waymo and the Tesla we're building autonomous driving. I think this reality is already here and just on the heating side. Right. A lot of things already happening. We just didn't realize. Yeah.

Jensen Swan, earlier this year, the CEO of Nvidia, he was talking about physical AI being the next big wave, which I think is just a rebranding of robotics. But yeah, it seems like this is going to be an important big thing.

That'S exactly the same. For example. Right. The robotic things we have been envisioning for more than 50 years now just become a reality. So I think we are just living in our own fantasy, right. We imagine all these things 50 years ago. Now it's happening one by one.

So what's the impact then of physical AI? I presume it's going to hit manufacturing and industrial use cases first, is it?

Well, I think industry already happening. I am more talking about how this impact our daily life actually. And this is the only one area actually I do have concerns. If you have a physical robot in your home, how do you control it? If it's misbehavior I think this actually once you get into your daily life, I think there's a lot of things we have to thinking carefully. If you build a small cute robot dog, probably it's okay. However, if you have like a really human bot can lifting things. Right. And can do all the heavy labor work. But what happened to you out of control. Right. And actually I do have concerns. I put such a huge human body in my house. Right. Not only privacy, it's just like what happened to the malfunction.

Yeah, well, I mean Isaac Asimov, he wrote hundreds of books on this topic. It's like robots going wrong all over the place.

Yeah, it's all over the place. Yeah. I think that's kind of like. Yes, I'm always optimistic about technology. However I think and once we get into the physical space I think we definitely need a more careful how do we design these robots can co live with us peacefully. Right. I think that's gonna be one of the things we had thinking right now.

Yeah, yeah, definitely some potentially great things or potentially very scary things going on. Like Terminator was not necessarily a happy movie. So yeah, before we wrap up, I'd love to talk a bit about Stargate because Oracle's obviously involved in this. It's a ridiculously large infrastructure project. Can you just tell us a bit about it?

Yeah, yeah. I think you already saw all the news and I won't repeat. Right. And but however I think I would more think about from computing perspective and for example when we start building all the GPU clusters and to service 1,000 GPUs back to three years ago, for example when the GPT2, GPT3 OpenAI when it started, I think about probably thousands of GPUs to build the large language models. You probably heard from Elon, right. He built like a 100k H100 in his Memphis data center and now the Stargate is already close to equivalent about 500k h100 scales. So besides think about this computing infrastructure is getting more and more powerful and it's very soon we're going to have 1 million GPUs in a single GPU cluster. So I think that's kind of the target. Right. And like Stargate and also other provider too, right. I think Microsoft and Amazon Red Meta, they're all trying to build large scale TPU clusters. I think what's unique about the Oracle OpenAI collaboration is this is huge GPU clusters in many regions. The textiles one is just the first one. Right. You probably heard right, we're going to build One in the Middle East. And we're also working with a lot of state and government. We're building the large CPU cluster for them. Again I think this is just another starting point and we just dramatically change our GPU infrastructures to start with. And a more generic is more the computing infrastructures back to. Again back to. That was 10 years ago, right. When I lead a team launched the first Amazon machine learning service. We're only talking about a CPU, so hundreds of CPUs, but now we're talking about thousands, even thousand GPUs. It's only 10 years, right. And we dramatically already advanced the whole computing industry. So given another five years and that's why I think I'm super excited and because from the computing infrastructure computer science perspective, it's just like a so dramatic exchange in the past few years. And this kind of lead us to many things, right? We it's very hard for data. Put this way, having a million GPUs.

In a single cluster, that's just a ridiculous. I mean it's a crazy ambitious engineering challenge. Can you talk us through like what are the benefits going to be? Like what do you expect to come out of having such an massive compute infrastructure?

Yeah, I think one perspective is just iteration for example, right. And let's just use the NTP industry as the sample, right. And you know before it could again it could take you 12 months, right? Do one iterations. But now if you 100x tower then you could do iteration super quickly, right? Everything else you could probably have one iteration, right? So the way you do the. I think Jeff have very good blog post about this, right. I think a couple years back. So essentially you increase your iteration speed. That makes a huge difference. Right. Before for example, if you take your three months to experimental one algorithm now probably going to take you three weeks and once you have more convenient powers you could take you to three days. So that means your innovations speed is accelerating. That's the difference. It's not necessarily your model get larger and larger. It's more about the iteration speed. And then think about it then you could have more teams doing all the experimentation in parallel once you have the computing powers and then I think the integration could accelerate. I think that makes a huge difference, right? It's not only about the size, it's really about the speed you innovate.

Okay, that's interesting because I had assumed it was just okay, we're going to create some like giant new model. But actually just being able to create similar sized models but faster and being Able to keep tweaking them. That does make exactly.

Yeah. Before probably going to you know, take you know, one team three months, do one iterations. Now you could have 100 team doing the same thing. So basically you do 100x your iteration. Right. I think that's kind of dramatically and accelerate. Right. And no matter what you build agent or foundation models, I think that can accelerate our AI revolutions.

Okay.

Yeah.

Certainly a bright future for being able to develop stuff once all that infrastructure is in place. Okay, cool. So just to finish up, what are you most excited about in the world of AI?

Yeah, I think for me because I've been doing the computer science AI for a really long time. My thinking is still. I think it's still a lot of fundamental computing problem. Right. I think one thing really excites me is I'm thinking is right how the way we build a computing system and how the way we build a computing software could dramatically be changed in next few years. I think that's one dimensions. Right. What you've been thinking, should we just do we still need a compeller for example building other things. Right. Is our current MacBook or Windows system. Do we rebuild all this operating system as well? And again I think that clearly a lot of things we could do. We could make our software computer system much smarter and much better. However, I think the second thing is I still believe there scientific wise. Right. We haven't made many breakthroughs. Of course large does great. We still have a lot of potential say build super powerful large language models agents. But again I think some of the fundamental technology like reinforced learning is still 50 years ago. Right. It's not many dramatic I would say scientific advancement yet. And I think right. Zero have a lot of opportunities developing new algorithms and have a new. I think another thing is maybe the current algorithm large language model, for example. Right. Make at plateau. Right. In next few years. Then we still need to keep thinking. Put this way, I didn't think we done right. We just started. And as a humanity I think lifetime is perfect great. However it can't solve all the problems the very beginning. I think from a research science perspective it's a lot of things and I think we should continue to evolve and continue to do more. Not stop here. Large is great. We can build a lot of applications. But I still think there's a lot of potentials for science community continue to involve, continue to reinvent. Not stop here.

Absolutely. I'm right there with you. A lot of the basic scientific research questions. These are like very, very cool things. To work on. There's a lot of ways that both data and AI can help out here. Hopefully if we can speed up scientific research, that's definitely going to be a big win for humanity. Wonderful. And just finally, I always want recommendations for people to follow. So whose work are you most excited about?

I think not particularly, of course, like Andrew Capacity, Andrew N, Jeff Dean and all this Eurosus. Follow them and I started to follow, for example like Open Anthropic and because here are the top tiers, Frontier and DeepMind and all this and Enterprise, we follow them very closely and yeah, I think be open minded. Put this way, podcasts, including your podcast. Right. Things.

Absolutely. Got to follow Data Frame.

Exactly. You got to follow Data Frame. Right. The one you had with React 2.0. I think that's fantastic. There's a lot of practice, practical. I think my, my recommendation is just like listen and read as much as you can. Don't be bounced by any one of them. Right. The more you learn. Right. And the longer you learn, then you can build your own judgment. Right. And so, but don't bounce by single person. For example, oh, I will only follow this person's blog or suggestion, you know, and I think I'd be very open minded and try to learn. Right. And from different companies or different personnel. Right. Broadcast, you know, podcast, YouTube. Right. And anything. Right. You could learn and yeah, that's my suggestion.

Yeah, definitely. I like the idea of just speaking to different people, getting different points of view.

Yeah.

I mean I speak to different people on the podcast every week. I do like it when guests disagree with whatsoever that was last week. Said brilliant. Yeah. But good advice just to get all those different points of view. Super. Thank you so much for your time, John.

Yeah, thanks for having me here. I'm super excited. You can see I'm super, super excited. I think we just started. Yeah, I think a long. Again, it's very exciting work ahead of us, but still I think a long way to go to fundamentally change how we do computing. And I think we still have a lot of opportunities and we can make a difference. Absolutely.

It's all about making the difference. All right, thanks.

Okay, thanks so much for having me here and very nice talk to you, Richie.