Open Source, Agents, and Specialization: What's Next in AI?

Hello. Hello. Thank you for joining me today.

Thanks, Carrie.

I am Carrie Brisky, Vice President of Generative AI Software at Nvidia, and I am joined with a man who needs no introduction, but we will. A man who has the coolest name and an even cooler job. So, Konstantin Bueller, thank you for joining me today. You are an AI engineer turned venture capitalist and a partner at Sequoia. Tell me a little bit about what you do.

Well, thank you, Carrie, for that very kind introduction. And I did bring my mandatory leather jacket we get to wear here. Yeah. So I'm an engineer by training. I learned about it when I went to school. I went here out west and studied at Stanford. Got pretty deep into the deep learning revolution in the early 2010s, fell in love with this concept of massively parallelized and sequential compute, being able predict almost anything, and worked at a couple companies in that area. Past six years, have been at Sequoia.

So you are working with founders who are building the future of AI. You are sort of known for being able to predict big technologies over the next decade. I think that's kind of interesting because when you're in the AI space, it tends to move so fast and sometimes even having a roadmap that is one year out, let alone a decade out, what is your protocol or way of thinking to come up with like these big visions?

So for me it was, it was very simple. I knew AI would be a really big important trend just from the technology. When I was in undergrad and grad school, it felt like this is something that's going to change the world. I wasn't sure exactly how. You know, a lot of the smartest academics at the time thought that it might be more elegant models they'd talk about than a raw, unbridled neural network. And it turns out that wasn't exactly true. But I, I did know that this idea of prediction was incredibly powerful because at its core I felt like intelligence itself is prediction. There was a previous AI hype cycle. This was 2017, 18, where every company or many, many companies claimed to be doing some AI and it wasn't really core to what they were doing. So I felt like my role for my friends would be for the technical friends, hey, is this a real business? And for the non technical friends, for my friends from business, hey, is this real technology? And that's kind of how I created my little niche in the AI venture world was companies that are real technology and real businesses.

Wow. So today we're here to talk about open source. We're here to talk about specialization. We're here to talk about the challenges for both of those with adopting agentic AI.

And.

And then we're also here to talk about innovation and investments. And so if I had a timeline of this is years and this is adoption of AI or let's talk about just open models, right? People really weren't adopting them. And then I go to the era of 2024 is about rag like retrieval, augmented generative systems. You start to see the pickup of open models and frameworks and then you go into 2025 with reasoning and just agents. And so you see the adoption of open technologies go. Do you think that open technology is driving a lot of the transparency and adoption of agents today?

Yeah, so I do generally agree with that timeline. I think that there's always the future is already here. It's not evenly distributed. It's old freight saying I feel like there are companies that are now getting there and then there are companies that are ready on the next thing. So I do think that the timeline you outlined is what core enterprises are now thinking about. I think that the startups are already thinking about what's next next. So most of the startups that I work with are thinking about swarms of agents. So agents working together, agents actually making decisions with each other or running background agents or having that kind of communication. Yes, definitely. Open source played a key role in that because it's a lot easier to be able to take an AI into your enterprise when you actually control the weights. And we're really excited about the prospect of open source because it gives more power to the enterprise and ultimately to the consumer. We actually have a shared investment reflection where we just raised $2 billion and Nvidia was a participant and a wonderful partner in this and they're building, trying to really carry the banner of the U.S. open source development ecosystem. That's a really good thing for everyone because enterprises can then build it into their solution and enterprises can now compete and win with their own workflows, not just the startups that use closed source APIs.

Two things that you mentioned. One was about the communication of agents to agents. And I think that you and I actually have used this analogy. I don't even know that we've known that we've done this independently. But let's talk about it together. Is that the stage of agentic adoption? And I always explain it as comparing it to the Internet because you had arpanet and then Internet and then Internet and so sort of the stages of adoption standards that needed to be put in place. Do you, do you kind of see the evolution? Like 20, 25 isn't the year of agents. We're gonna have agents to come because there still need to be standards. There's still definitely. Yeah. Could you talk a little bit about that?

Totally. So I think there's at least three things that are in the way of us having really great agents. You know, the first is agent memory. So that is an unsolved problem. Some of the great researchers right now are trying to figure out agent memory. If you think about interacting with a person like Carrie, you and I have had great interactions. If every time you met me I was a different personality, it would be pretty weird, right? With agents, how do you actually carry that memory along? Because if you're going to be working with an entity, an agent, you want it to have a persistent memory of you and of itself.

See, I think most people are talking about agents having memory of me, not of itself, because I think people are working a lot about memory in the moment, or memory procession, or working across key value pairs of kvcache, across similar queries, across different people, but not necessarily that the agent is aware of itself and keeping that memory. And so I think that's really important.

Totally, totally. Because it's like, of course you want someone to understand you. Like when you interface with someone, remember their name and their family and all those good things, that feels really good. But consistency of the individual themselves is also really important. Think about someone you work with and they're a really good partner. And if all of a sudden they were dramatically different every day, you probably wouldn't want to work with them very long. It would be pretty weird. Same thing with AI agents. So this memory problem is open. But the good news is some of the greatest researchers are now focused on that.

Do you think that enterprises are going to have to fine tune or build their own AI to stay competitive? Because I think of agents having memory about themselves as being fine tuned or gone through reinforcement, learning about themselves. So then they become who they are rather than having. Maybe you're saying memory isn't. It's been trained in. Is that what you're talking about? Or a different type of memory?

I think it's both. We both are computer enthusiasts. So I think some of it's going to be having the code that's been loaded into the processor itself, some of it's going to be ram, some of it's going to be large storage and you access that in different ways. So in this case, yes, some of the Models will be changed over time. That's one of the values of open source, where you can take those models and basically decide which neural pathways to change in which direction based on what type of new information. That's a technique some people are using and it's, it's really smart. At the same time, rag and vector databases and all these other techniques, they're not going to be forgotten, they're just going to be used in different ways. And there will be other ways of us thinking about memory that haven't even been brought mainstream yet. That's going to be one of the three most important things, at least in the next year, for getting AI agents to work. The second is communication protocols. We've talked a lot about MCP and this idea that in order to have agents work together, they have to be able to talk together. We both speak the same language, not only the same technical language of English, but a similar business language and a similar technical jargon that helps us communicate really fluently. AI agents are going to have to have something similar. And that actually reminds me a lot like the Internet, where in the early days you had tcp, ip. That wasn't the finish line. That was the starting gun. And then the third is AI security. So now that you have agents that have memories and are communicating with each other, we're going to have to figure out ways that they communicate very securely. And it's actually Jensen who said this in a previous conversation. I thought it was pretty awesome. He said, think about agents as not needing to have the same requirements of the physical world. So in the physical world, you might have one person as a security guard and a thousand people that they're protecting. And that's because of physical space and also because of cost. But in the future, he pointed out, maybe you're going to have a thousand security agents around one cognitive intelligence agent. And so we have so much to solve in the security side of things. Those three areas, memory, communication, security, are all being solved real time and are going to yield the agent economy.

You said the agent economy. I love that term. We call it digital workers. But there's going to be some sort of niche market that's built up around new frameworks, new training types, just to support this economy. Right. Just like we have an economy for humans, we're going to have an economy for agents. You also. So you talked about those three problems. Another problem that I see is not necessarily for agents, but maybe in the people adopting the agents is evaluation. So you and I were talking about stochastic mindset and determinism. Talk to me a little bit about, because I see when we've all heard the study, about 95% of pilots don't make it to production. I actually kind of disagree with that because I see where the pilots are focusing on targeted use cases or specialization. They're actually succeeding and getting into production. And, you know, this definition of AI, of what's done or what we're used to and what goes into production, people tend to forget about. So we can talk a little bit about that, but talk about this stochastic mindset and evaluations.

Sure. So I'll answer a little on the stochastic mindset and then I'll share how I'm seeing evals evolve over time. So on stochastic mindset, the idea is that we're changing how compute's happening. And frankly, you guys have been at the forefront of changing how compute's happening for decades now. And this type of compute is no longer fully deterministic. It is stochastic. So there's some randomness in the system. And for those EEs out there, I know nothing is fully deterministic, even historical types of compute, but pretty close to deterministic. If you and I put the number 37 in a spreadsheet and we check that spreadsheet tomorrow, it would almost certainly be the number 37. But the way AI works is a lot more like humans. So if I said, hey, Kerry, can you remember the number 37 for me? And I checked back in in six months or a week, you're really, really smart, so you remember 37. But maybe the average person might also remember 73 or 38 or 36. The point is, there's a distribution of how that memory is stored. And over time it probably gets worse and worse. And that kind of change is a difference in how you interface with a computer. So it's going to be a lot more like interfacing with people, which is mostly a good thing. I love people. I think you do, too. Like, we're people people. But there's challenges where people forget things in certain ways or there's randomness in certain ways that aren't true of computers.

And so therefore, when we expect or evaluate or retrain to keep agents up to date, we have to think about this.

Definitely. Absolutely. And I'll share one way we're seeing this happen in the industry, but I'd love to hear how you are seeing it because you talk to the biggest enterprises that are doing this. One thing I've noticed in the startups is we've moved from academic evals to real world evals. So we work with a company called Expo. They set out to build the most effective AI hacker in the world. Basically penetration testing. Can you find vulnerabilities in a software platform? And they said, hey, how do we show that AI can be so good at finding vulnerabilities? So they said instead of academic benchmarks, which they did, and they succeeded in, we're going to go out into the real world. And they registered on HackerOne, which is the leaderboard for all things hacking globally, and they started competing to find vulnerabilities. And within a few months they ranked number one in the world. So the number one hacker on HackerOne in the world is an AI, which is simultaneously scary and exciting.

I mean, all technology for all of history is people have used technology for good and bad. And so if you're using AI to do bad things, you need to combat bad things with AI. It's the only way to go.

Yeah, exactly. Which is why with this company, I get to work with them. It feels national security level important, because without question, you're right, the bad guys have also seen this and they're taking action on it for sure. And so the response isn't to put our head in the sand, it's to say, hey, we have to be even better. We have to find these vulnerabilities before the bad guys do. We have to patch them. Our job is even harder. So for a company like that, I think it's amazing what they've done, not just because of the impact, but because they said we're going to evaluate this not just on an academic benchmark, but, but in the real world for specialized.

AI, I foresee, especially for agents, is this reinforcement learning and the amount of gyms. And so when you talk about this evaluation benchmark, you're going off and letting the AI train in an environment or learn through an environment, or learn through doing and then taking all that and reinforcing it back in. And so the more gyms, just like the more we work out, the more we exercise or the more variations on scenarios that we're put into, the. The better we are at engaging in new scenarios, new systems, new input.

Totally.

And so I foresee new studios, new gyms, new RL environments, specifically for specialized niche domains. And why do I say that? So I think when we look at what has been successful, has been coding agents right now. Right. I think that's because the verifiers are very simple, because the code compiles or not. And you're able to reinforce that, learn. Right. You have a lot of people who code and they can say, yes, that was right, that's good code. That's not. Now when you get into, I'm calling it like sort of this last mile of expertise when you get into an enterprise or a specific domain like you were just talking about, where you have experts and expert knowledge, the verifiers become very hard because they're either expensive or there's people that can verify, need to be doing their job rather than verifying an AI. And so I think that synthetic data generation is going to become really important there because you have high quality seed data to create specialized AI. You also need again to be able to create the verifier. And again, code is yes or no or math. The answer is right or wrong. It's not so easy when the verifier is like, it depends or it's close, but not that. Right. So you obviously work with all these startups. Is there a field where you see that people are getting more into kind of moving up the stack, moving away from models, building models more around the agents and these verifiers or gyms or environments for which models can train better for specialized AI?

I think what you said is really, really a great point and I've actually been wrestling with this concept for a while. What, what is the gatekeeper for progress in AI? And I think what you said, which is being able to verify it quickly, is a brilliant point. And it's so true of code. It's so true of code because you can see compiles, but then you can just build it and look at it and then run unit tests and you can iterate fast enough that the stochastic nature of AI doesn't prevent you from making progress. In fact, it's fine, you know, you can just go quickly enough. It's harder in other areas. So I'll give an example. You know, my, my wife is a surgeon, so she does, you know, it's a high consequence field and you know, she, she does a specific kind of surgery that like you got to get right. So the importance of verification is extremely high there, there are other parts in the process that the verification is much quicker. So she might use AI for scribing when she's talking to a patient, taking all the notes, allowing her to speak to the patient, allowing her to focus on that. She might use a company like a bridge for that. She might use a company like Open Evidence for doing research and answering questions that that a patient has, that's an area of where it's going into an application that is actually really hard to verify. But she has so much expertise, she can then read it, accept the notes, accept the conclusion, and then move on. And by the way, that makes the models better too.

That's right. And so we might think of scribing as not as needing expertise, but you actually do because it's the language, it's the jargon, it's the technical expertise of the words that you're saying are accurate about a patient or what might be happening. Right.

And so definitely, what do you see enterprises mostly wanting, like when they, when they come to you and they say we need generative AI, which they definitely do, you know, to not fall behind. How do they think about the use cases that are reasonable and verifiable and how, how do you guide them through that?

I think for whatever use case they're coming to, they want the most accurate model on the tiniest cost of ownership. And it's a really tough trade off. Right. They want high accuracy, but they want it on a small footprint. But they want it on their own data, they want it private.

Makes sense.

And they don't want to send their IP or intellectual property and private information out of their data.

I sent this input to my partnership a couple of years ago when we had these big, big models. I said, at some point it's going to become about distillation, which is this concept of taking those big models and then making them much more efficient. So usually much smaller and then focused on a domain. And I feel like we're still in the bigger, bigger, bigger model part of the cycle. But I do think the smart enterprises say, well, I need this to be cost effective, I need this to be efficient, I need it to be fast. And the best way to do that is that kind of distillation.

There's distillation, there's looking at model architecture, there's all kinds of ways of making models more efficient. I think honestly, if you look at model development, just like any regular software development platform, you kind of demystify it for a second. You're always, you're writing code, you're compiling it, you're putting it out to run totally. And so when you're building models, you're getting the data, you're retraining, you're putting it out there. So no software application is just ever out there and done. You have bugs, you have feature requests, you have to improve it and then write the code, recompile, re release, I think that with every release, it's not just about the features, it's also about optimization. The most performance that you're getting out of the software at runtime. So if you think about model development just as a new software development platform, it takes the scariness out of it and understands that, hey, you want to get your mvp, you want to get your mvp, right? And then you can optimize over time.

And that's well said. I couldn't agree more. That makes a lot of sense.

Okay, so we talked about challenges. You talked about persistent identity, seamless communication, security. How can open source contribute or accelerate solving those challenges?

So I think something like communication protocols have to be open source because definitionally many applications have to participate. I could see a world where memory and security are closed source memory. There are benefits to open source, but we store memory differently in different contexts. Software memory and security is also pretty proprietary. But I think the area that open source will be most critical to is having these base models to go build on, of course, which we get to work on together, which is amazing. And then separately, communication protocols, how are models gonna talk to each other?

Okay, so you talked about working together, so I'm gonna talk about investment, innovation, and we talked a lot about agents, we talked about model training. And so I think a lot of the capital investments are moving from this model space to like up the stack to more of the agentix space. So the agent layer, what is driving that shift? What do you see? Why do you think that's happening?

I think it's because at the end of the day, it's all about people. No matter what, it's always all about people. And you as enterprise leader here probably know that better than anyone. It's always a people problem, it's always a people opportunity. And that's why we're shifting from the model layer more and more to the application layer. So I'll give a concrete example, and this is an example of going from model layer to automated model to human in the loop model. I work with a company or portfolio called rox rox and they are building an agent operator operating system. I think Nvidia uses them and they have basically built a technology that helps instead of a base layer, you know, right out of undergrad automation of some salesperson, they want to have the best sellers sell even more effectively. And so they connect with models and they might do a lot of research ahead of a meeting or draft some material for follow up to a meeting. They introduced a feature that allows for emailing. So if you're an executive, you had a meeting, you're sending a follow up email. And like a lot of teams, they started by offering that as an auto send. So we've created the model and we're able to send it. But they realized that they actually could get much higher conversion if they brought the seller back in the loop. So the seller now is able to edit every email before it gets sent. Kind of like that idea of the human supervising or verifying and return emails went up 3x so materially higher here. And that story gives the essence of what's going on. The essence of what's going on is at the end of the day, this is all about people. And especially in something like selling, it's about people. And so the idea of cutting humans out of the loop is just naive. And the idea of if you really want to help the best people do better, integrate them in quickly, have AI help them build into their workflow and you'll have the best possible product.

And I think that that's building in trust, right? It's a trust layer. And I've heard you talk about before, like the definition of AI because at one point computer vision. Oh, that's not AI anymore because it's already happened. I trust it. I get that you're pointing out whether it's a Chihuahua or a blueberry muffin.

I love that era. I love when you were talking about computer vision, I was like, people still underestimate how awesome computer vision is.

It is. But putting it all together, it's still really hard, right? So the visual understanding and the language and speech and audio and using it all together. And so maybe that we always keep moving the bar for what is AI, right? And so totally. But it's always about people.

Always, always. I feel like you and I were talking about this beforehand where AI is such a tortured name for the industry where you're naming it intelligence. And by definition people will consider intelligence whatever the machine can't do. So as soon as the machine does it, they'll start to forget it as being intelligent. And I already see this in tasks. I mean, when you and I were getting into AI 10 plus years ago and the idea that a computer could understand the difference between a Chihuahua and a blueberry muffin was awesome.

Mind blowing.

It was awesome. It was a huge deal. And now that artificial intelligence is relegated to the field of computer vision. Even self driving cars, I'm now seeing self driving cars are on the streets of San Francisco and I don't really hear a lot of people Talking about it as mind blowing AI. They're just calling it autonomy or autonomous vehicles. So with every era of AI, it eventually gets pushed back. And that's my job too, to be, hey, what can be that next form of AI that's going to make the world feel like it's magic?

You just had the best lead into. My next question was, where will we see the next big wave of innovation?

Ah, I feel like you guys are all over this, which is, I think world models are going to be extremely exciting. And I'll go a very base layer, kind of like you were talking about, of the progression over time. AI generally moves from less data intensive to more data intensive over time. And we had eras of AI that were numbers in and numbers out and the generative AI text in, text out era. And then we've gotten pretty good at images and we're getting pretty darn good at videos. And in between those two, we got really good at Voice. So basically richer and more intensive data types. Logically, what comes next? Well, of course we're going to have voice get really good in the next year or two, and video and all of that. But then what starts to ramp is world models. These are extremely data intensive. You know, it's the whole space around us and streaming in and streaming in.

All the data streaming into us at all times, whether totally we accept it.

Or not, 100% absolutely. And then it gets really philosophical, like, how do you perceive the world and do you perceive it as we all have mental models that we're testing against, or do you perceive it as, you know, our eyes and senses are constantly streaming in new data? I'm probably the former, actually, of how I think we perceive the world. In any case, it becomes about that. And then from there, once we have those good world models, we can do amazing things. I think that's actually gonna be the base layer for all this robotic opportunity that will be truly massive.

That's amazing. We always try and work on the hardest problem, the biggest workload. We always say that if you want to go to the moon, Nvidia will take you there. If you want to walk across the street, you're on your own.

That's awesome. That's funny.

Okay, so companies who are moving fast and you're working with those companies, they seem to iterate often. And it sort of reminds me of the design thinking approach where you want to fail fast or iterate often, pivot if you need to. And if you were to talk to a leader today or a new founder, that you're just talking to and you're going to give them advice, how would you tell them to adopt that mindset? Or would you tell them to adopt that mindset at all?

Okay, so I feel like this is a very safe place for a very nerdy technical analogy here.

Absolutely.

Okay, excellent. Which is I do believe that you should iterate incredibly quickly. And I get inspiration from gradient descent algorithms or just invert it and call it gradient ascent algorithms. And whether it's a truly stochastic gradient descent algorithm or batches where you get a bunch of information and then update it, I think that's the best way to build a company. So, you know, you don't know all the information. Nobody knows all the information in the market. Because the beauty is the vast majority of that information lives in other people's brains. You know, they know what their preferences are. And the whole point of the market is to serve other people. So we'll never know all the information. Even if we have a perfect world model, we'll still only have some estimates. So it's the job of the founder to gather as much information as possible, a batch of new information, and then run a stochastic gradient ascent and try to find something new. So then that begs the question, in this super hyperdimensional universe, with all of these hills and convex hills in this dimension, in this space, how do you find the highest one to climb? And Nvidia did. I think that ultimately happened. You guys said, we're going to think about all of compute and a new way of computing and acceleration at light speed. That is a huge mountain to climb. So you picked a great starting place, and it took many, many, many years to climb that I'd encourage founders to pick a starting place that's worthy of them. Pick something that is sufficiently exciting, big challenging, that if you do succeed, it was worth it.

Big, big reward. Yeah.

What do you see from startup founders selling in enterprises? Do you feel like they're iterating too fast, breaking things or going the right speed?

I think enterprises are a completely different world. Not that they're second movers per se, but I definitely think that they have more risk. They have legacy systems. They have data locked in legacy systems. So that curve of adoption is taking slower for enterprises because they were figuring out how to integrate something as fast moving as AI. Right. I think that enterprises aren't moving as fast as, say, the startup world, but they definitely get their influence from the community, from open source, from the startup world. They look to that to see where, again, what is everybody working on and what do I need to adopt? That's why I think Enterprise hasn't adopted generative AI like the way that we wanted them to within the last year. But I think now that it's become easier to fine tune and specialize, easier to generate data through synthetic data generation, and easier to evaluate or put through these reinforcement learning environments, that they're going to be able to start to really specialize per domain. I think some enterprises started thinking like this one model is going to solve the world and there's not one model to rule them all. There's many smaller models or different sizes of models. It's systems of models to actually achieve a use case.

I feel like we speak the same language here. I think a lot of it comes just from being from having been in other areas of AI where you've noticed. Yes, it seems at some point that one model will be able to do everything. And always it turns out that a specialized model ends up doing the thing better. There might be one, like you said, to start to get things going, to start training to kick things off. But turns out specialization is a superpower and I feel like that's going to continue to happen in the Gen AI realm.

Me too. Well, that's all the questions I had today.

All right.

Thank you for coming. Thank you very much for headquarters and having this glass of water with me.

Amazing. Yes, it was. Strong glass of water. Cheers.