Product-Led AI: Mustafa Suleyman on Defining Intelligence

Hi and welcome to Gray Matter, the podcast from Greylock. Today we're featuring another episode of Product Led AI, the series hosted by Greylock partner Seth Rosenberg, where he discusses opportunities in the application layer of AI. His guest this week is Mustafa Suleiman, who leads Microsoft's AI efforts. You can find more information about this episode and the entire series@productledaipod.com now here's Seth with Product LED AI.

Hi, I'm Seth Rosenberg. I'm a partner at Greylock and the host of Product Led AI, a series exploring the opportunities at the application layer of AI. Mustafa, thanks so much for joining.

Hey Seth, good to see you again. Thanks for having me.

So very excited to have Mustafa here. As many people in the audience will know, Mustafa is one of the leading pioneers in AI, currently the CEO of AI at Microsoft Venture, partner at Greylock, former co founder of Inflection AI, as well as co founder of DeepMind, which was of course acquired by Google, where he became VP of AI at Google. So Mustafa, I feel very privileged to be able to work with you through Greylock, but also to have you here to kind of share your take on the current space which is evolving so quickly. So maybe to kick it off, I'm curious to hear more about how you decided to focus your career on AI before it was obvious.

It's kind of strange looking back on it. I sort of had to write my TED talk recently and I reflected on how crazy things are 15 years later when we first started talking about founding DeepMind in 2010. It's hard to overstate how weird we were. Often people say being an entrepreneur is choosing to obsessively work on something that is really contrarian and, you know, attempting something that everybody else thinks is impossible. I think in our case people didn't just think it was impossible, they thought it was completely absurd. I honestly am not quite sure how we came to have so much faith in our ability to try to do something so out of distribution and exceptional.

What was the initial insight around deepmade?

I mean, we didn't just start working on AI or machine learning. We were fully committed to working on artificial general intelligence, producing a system that could exceed human capabilities and knowledge at all levels. And the reason we were motivated to do that is because we genuinely wanted to use AI to solve other problems and to make the world a better place. I think there was no academic lab or setting that could accommodate the scale of investment that we thought was necessary at that time. In academic labs, there wasn't a focus on large scale engineering. There certainly wasn't a focus on products. And even in the kind of big national project investments that you get from working in a government, there was just nothing that was like a technical effort to try to understand what intelligence was at scale and deploy it at important problems. And so it was really only like startups that were as brave and courageous, I think, as was necessary to be successful on this mission. Like, by that point, that was my third effort at a company and it just felt like this is the only vehicle I kind of. Because you learn so much along the way by screwing it up and figuring out how to do it right. I had come off the back of a bunch of years working in the nonprofit sector in governments in conflict resolution and facilitation, had started two small companies, actually, one that was focused on selling networking equipment and electronic point of sale systems to restaurants. But this was like, way before that was possible. That was a failure. And the thing I sort of realized is that we need more knowledge and insight in our world to help us address the overwhelming complexity of our systems. It's so difficult to make an intervention into a complex social system today, like an economy or, you know, food production system or the financial system, and be confident that that intervention is going to make the impact that you think it is. And that's really, I think, one of the reasons why we need amazing AI. We need to be able to make good predictions about complexity in our world in order to create value and to change that world to help people live healthier and better lives. It sounds super cheesy, but like, that is really what was motivating us back then and still does.

I'm curious. So you had this mission that unlocking abundant intelligence for the world would actually solve problems that matter. How do you actually define intelligence in this context?

The thing that gave me confidence that we might actually be able to make progress towards inventing Intelligence was our third co founder, Shane Legg, had spent his entire PhD researching various definitions of intelligence and trying to aggregate them into a single metric that we could use to turn the science of intelligence and the neuroscience of understanding biological intelligence into an engineering effort and really make that a measurable kind of quantifiable exercise. And the definition that he came up with was the ability to perform well across a wide range of environments. So again, emphasizing generality. And that was like a major. Now everyone takes the AGI part for granted as though GE is the central part of intelligence, but that's an assumption. You know, generality happens to be one of the characteristics of intelligence. But it's not the only important characteristic. And it also turns out that it's also very hard to measure and to scale it down to something that you can really grasp. Whereas another sort of definition was the Turing Test, of course, is that a system would be intelligent if it could deceive a human into thinking that it was itself a human during natural conversation. And in some ways, we've sort of crossed the threshold of that intelligence. Right. We have systems now that are really very good at conversation, and at least for a few turns, could. You know, it's certainly better than humans in many respects. Right. You can still tell that it's an AI or a chatbot and not a human, but in a few years time, you really won't be able to tell. And yet that doesn't really tell us anything about whether these systems are actually intelligent. Every time we cross a threshold in terms of a benchmark or a milestone in AI, you then sort of turn around and go, okay, well, here are all the problems with that mechanism of measurement, and here's the next sort of thing that we need to measure.

I've heard Reid say that AGI is the AI that we don't yet have. It's kind of always pushed forward in the future.

Exactly. And it's. It's like the constant carrot that we dangle to chase ahead, you know, to sort of move ahead. So another measure that I had proposed is that we should focus more on the capabilities of the system, the actions, the things that it can do, the things that we can observe, that it can have an impact in some environment, rather than this kind of abstract idea of, is it general or is it good at having conversation? That would basically be like, can it produce human quality labor in a practical environment and actually earn money for that? Or could it write software, for example? It's a very measurable thing. I sort of called that a modern Turing Test and said that in the next five years, a system would be able to take a very abstract goal like go create a new product, get that sort of designed, manufactured, drop shipped, and then distributed and marketed and try and earn a profit from it. And then you could measure that profit in terms of making a million dollars or something like that. I think there's going to be a system that could actually do that, certainly before 2030. Yeah.

That's amazing. And it's possible. Do you expect that that type of system would actually trade off on the G, on the generalizability and it's actually built for a specific use case?

Yeah, I think that it's more likely that we'll have really powerful systems that are specialized for specific use cases that have real deep domain expertise than we're going to have this sort of very general purpose system that can switch from being a marketer to a, you know, clinician, to being a doctor sort of to being a lawyer or whatever. I mean, clearly the general case is going to come afterwards.

So I want to spend a moment just getting your take of just the state of large models today. And maybe you wanted to start with just level setting for the audience of what was the inflection point that led us to the current state of the GPT3, GPT4 style models, the Inflection 2.5 style models, this combination of the transformer architecture and scaled compute.

I think that this revolution has been driven by deep learning. We are still building deep learning models, albeit with a slightly different flavor now, the transformer architecture from 2017 and we're now turning those into composable units that are essentially going to act like parts of our software development ecosystem. You're just going to turn to your AI and it's going to actually generate code for you. I'm already sort of seeing that G copilot and you know, we're seeing it like as a member of your team able to take a natural language instruction and just sort of act in unison with you. And I think that what people sort of don't quite realize is that these models are not going to be large forever. In the history of all technologies that have been valuable, anything that is significant gets cheaper and easier to use over time. And that curve is sort of double exponential over the last like couple years. It's incredible. I mean Microsoft AI just released Phi 3 fully open source. It is close to but not quite at GPT4 level. It's fully open source. It's 3, 8 billion parameters, right? So that's like more than 100x smaller in terms of inference compute than basically the absolute frontier of models today. Like I said, it's not quite as good, but it's certainly, you know, as good as or better than GPT 3.5. That's mind blowing. I mean that's something that can fit on your laptop or on a phone like I mean in the future. And so we should expect that trajectory to continue. I think open source models are going to be very close behind the closed source proprietary API models. Months or maybe even just a year or a year and a half or something. And that's just going to change the whole creation landscape.

Basically what enabled this model to be kind of almost equal in performance while also being much smaller.

For the last couple of years, everybody has been focused on reinforcement learning from human feedback, where in the final stages of training, at the stage of fine tuning or post training, as people say, you have a bunch of trained raters or judges that compare two possible responses or completions from a model and that pairwise comparison provides large scale feedback for the kinds of behaviors that you want your model to exhibit. Everyone's familiar with that now, but what we've been focused on as soon as that was showing promising signs, so for the last 18, 24 months is reinforcement learning from AI feedback, where we really want very smart and capable models to do that pairwise comparison because obviously we can automate that process and we can produce an even larger number of supervised fine tuned labels to give more feedback to the pre trained model across a wider range of experiences and moments that might be in tension with one another if you only have a small number of samples that come from expensive, highly trained humans. So that was one method is reinforcement learning from AI feedback and then the second is generating training data from these models. So sometimes people refer to that as distillation, where you're sort of trying to absorb as much of the best bits of a big powerful model as possible and then you're using that to post train or sort of align your smaller model. And parameter count is no longer the primary proxy for capability. High quality data is the real valuable asset here in addition to the architectures. So for the Last sort of 6 to 12 months, everyone's been focused on compute, compute, compute, can I get compute? And obviously that's kind of important or like, you know, large models. Large models, but really it's investing in high quality data. And so from a startup perspective, I think the real trick is finding existing data sources or more importantly, creating a UI that allows you to collect high quality data from an interaction with a product domain that you think is valuable, that is producing a moat of highly valuable data that you can then use to post, train and fine tune your model again and get into that feedback loop. That is a path to creating an enormous amount of value. And it does not require you to depend on the large scale model providers, which I think is a great, you know, that's why it's such a creative time in Entrepreneur Lab.

As a startup, you're competing with incumbents that have access to large sets of data. So I'm curious if you could share anything more about the nuances of opportunities that might exist for startups to get specific types of data that's maybe more valuable than others.

Okay, so how do you go about collecting high quality data? Because obviously in pre training it's about volume of tokens and there the hyperscalers will have a long standing advantage because they already own search engines or YouTube or whatever it is. Right. Whereas in post training you need a small number of very high quality tokens to align the model to the behavior that you want for your product. And you can collect that from scratch. Like when we built Pie, we created and to this day the most high quality human like conversational AI with the best EQ even today in the market. Right? Yeah. And we didn't use any data from big providers. We collected all of it ourselves from scratch by training paid teachers. We called them AI teachers, some people call them raters. But the crucial thing for a startup is you have to really, really, really pay attention to training those teachers. You have to pay them a lot of money. Like I'll just tell you, from our perspective, we selected people who had an undergraduate education, nothing less, that largely spoke English as a first language, with some exceptions that had a domain expertise that we thought was valuable. Like maybe they said they were very passionate about history or they like, you know, had a good kind of cultural knowledge or they were movie buffs or whatever it is. They had to pass 20 hours of training and testing by us. So we would give them reading and comprehension exams, we would give them multiple choice questions. They would have to do sentence completions that have to do spot the differences. They would have to do like really quite hard analytical tasks. And in order to just keep my team super humble about how valuable this task was, I would obviously also have all my team go through the same training. Right. And go through the same tests. And I can tell you, not even a majority of people pass.

Yeah, I was about to say I'm nervous.

I thought you were going to send over the test. Yeah, you know, it's actually not easy. It's a pretty tough thing to do because if you think about the task, you're asking a human to read through two 10 turn conversations, look at the proposed answer by one model and by another model, and then absorb a huge behavior policy, like very detailed line by line. The AI should do X, shouldn't do Y in this situation should do this. And then remember the training from the AI teacher that says all kinds of like subtle exceptions and stylistic tones and brand things and backstory things and capability, awareness. And then you have to sort of find the correct intersection of all of those to decide. Is this paragraph more in line with the behavior policy or is this one more in mind? Yeah, it's a painful task.

That's super interesting. I'm curious how you view this evolving. Right, because you mentioned we're kind of moving from reinforcement learning just from humans but also from AI. So how do you view kind of application layer startups between being vertically integrated versus which parts of the stack they really need to be experts in?

Yeah, that's a good question. I mean, I think you have to be very principled in answering that question. And that's where the bet is of your startup. You have to decide which part of it am I going to bet on? Obviously a bunch of people are building tooling and infrastructure and that's fine. We all are familiar with that kind of strategy. I'm a great believer in building and owning your own product and as much as possible controlling the key bit of the value there, which in my opinion is the LLM and everything around that is secondary. The words that come out of the LLM are what you have to focus on and that means that I think it's reasonable to break off the pre trained model and get that from somebody else. Yeah, that's a good approach, but I think you need to own your fine tuning stack and I would not give the fine tuning stuff to somebody else. Yeah, you have to train your teachers because that's not going to go away anytime soon. We're not going to have GPT5 tomorrow and suddenly GPT5 replaces all humans as the ultimate judge or teacher. I think that's quite unlikely. It's going to be a lot better than GPT4. But even the people that have tried to do RL AIF with GPT4, the quality is okay and it is impressive, is very cool, but it isn't on the verge of replacing humans entirely. You can get an 80% prototype going and it can look good, but a real consumer experience requires you to nail the 99 percentile experience. Yeah, you have. It has to be really high quality consistently and as soon as the AI breaks out of character, gets something wrong, has a hallucination, whatever you want to call it, that sort of destroys the illusion and it breaks trust so then you lose your consumer. And so I think that, you know, the key thing for startups, at least for the next year, is to get really good at data collection and data filtering and data quality.

That makes sense. I'm curious, what are the different UIs that you think kind of AI first companies are going to be building, whether it's, you know, a chatbot, an agent, just regular SaaS that's kind of enabled by AI.

In my opinion, the UI needs to get out of the way, especially for the consumer. Obviously for SaaS you can have all the bells and whistles and all the developer features, right, blah, blah, blah. But for a consumer, the goal is to get the UI out of the way. So we create a very paired back, quiet, soft, calming, I think quite distinct looking UI with very few buttons. But we also have one of the best voices in the world we've got. You know, in the end, I think we have nine or ten voices. They were really high quality and very, very human. Like, I mean, PIE is still alive and people should try it out, but it's like, I think Voice first is, is a big part of the future ui.

Yeah, I like the. Where you have to choose the voice that you like as part of the onboarding experience with pie.

Yeah, because it's sort of like connecting with your AI. It's a porn personalization moment. And 30% of all our conversations took place on voice and they were by far the longest, most engaged, most retained users. People should keep that in mind. I think that's a very important insight.

I've heard you talk about AI has iq, they have eq, and then you've talked about aq. Right, an action quotient.

Yeah.

So I feel like this is a very interesting topical thing at the moment where people are talking about these kind of autonomous agents for certain use cases, which includes reasoning and planning. How far away are we from the kind of chatbots we've experienced, that most people have experienced today, to a fully autonomous agent that you described at the beginning that's capable of executing an end to end task? What's missing between where we are today, that vision in, let's say, the next six months to three years?

First of all, I don't think we're on a path towards fully autonomous and I think that's actually quite undesirable. I think fully autonomous is quite dangerous, I think. And you know, I got a lot of stick after my TED talk because I said that the autonomous capability was dangerous and so on, and that's one that should be regulated and I don't really care. I mean, I still think that. I think that if you have an agent that can formulate its own plans, come up with its own goals, acquire its own resources, just objectively speak, act completely independently of human, objectively speaking, that is going to be more potentially risky than not So I think about it as these narrow veins of autonomy where you give it a specific goal and it has limited degrees of freedom to go off and act in some specific environment. So like making automatically calling some API to check some registry to collect some information to observe some state, maybe writing something into a third party API that is not yours, but is again restricted with some specific degrees of freedom because I think the security risks here are significant. So I think we should tread carefully on the autonomous piece. But in terms of on the actions piece, it's still pretty hard to get these models to follow instructions with subtlety and nuance over extended periods of time. I think that they can do it, you know, and there's a lot of cherry picked examples that are impressive, you know, on Twitter and stuff like that. But to really get it, to consistently do it in novel environments is pretty hard. And I think that it's going to be not one but two orders of magnitude more computation of training the models. So not GPT5 but more like GPT6 scale models. So I think we're talking about two years before we have systems that can really take actions that makes sense.

And if you were to break down, I guess some of the unsolved research or technical problems to get there.

Well, an action is no different really to predicting a sequence of words. So when you ask a model to complete a sequence of action, let's say it's like three things in to basically let's say to book a restaurant that you and I can go to on a certain day. Yeah, the first action would be check the availability in both of our calendars. So that's a correct function call, reconcile the correct moment. So that's the second action. Make sure that it's a restaurant that has availability so that check is another one. And then go and sign in so that you can basically use the correct tool to book the right restaurant at the right time. Put your credit card details down, having obviously also checked that it's a restaurant that we both like, et cetera. So it's like four or five or six different steps just to produce that one action. Subcomponents. Right. So in order to get that right, you're basically saying that the model has to produce perfect function calling for each element and do so in sequence. So it can't just be arbitrary, it has to be in sequence. And that's like saying it has to write a four page document in response to one question that is exactly that document. And it can't be something that is approximate or similar to that document. So we all think that obviously these models are magic at the moment and they write beautiful poetry and creative copy and text and give you good answers and sometimes they're grounded and blah, blah, blah. But for each one of those answers, there's a wide range of correct answers that it could have picked. Right. Tens, Hundreds, thousands, maybe. So it isn't producing a specific perfect. Every single piece of every single token that is outputted is correct answer for each one. It's not there yet. Right. So to get that level of precision, we have to scale up these two orders of magnitude. And that's what's happened so far. The last five orders of magnitude of transformers. With every 10x of compute and data, we get more precision. It's not just emergent capabilities that's wrong. People say, oh, it's surprising we had these emergent capabilities. That's an anthropomorphic projection. They're not surprising emergent capabilities. It is just more precise attention to the correct mapping between a prompt and an output. Right. So you're just honing in on something more specific now.

Do you think we'll be able to get narrow forms of actions in specific domains before we get to GPT6?

Yeah, definitely. I mean, there, there are some good actions today. Right. You can definitely. You see these orchestrators making good API calls at the right time. The question is, can it do it with 99% accuracy? Because if it does it 80%, then you know, one in five times it getting it wrong, it's not usable for a consumer. Right. So you either have to constrain the action space so that every time you're asking your model to take an action, it's only got five options to pick from and the consequences of it getting it wrong are low. Or, you know, you have to find a problem domain where 4 out of 5 accuracy is acceptable. Right, right.

And if you think about the architecture of building one of these agents, are there any differences to what we just talked about in terms of focusing on post training, getting the UI out of the way, in terms of the architecture of building one of these kind of narrow autonomous action agents?

I mean, one thing for people to keep in mind, I think, is that there's a whole range of tools in the toolbox now. Right. And so the art is in designing a router or a classifier that can take some given input. It's either contextual information, metadata, or of course, the incoming query from the user and redirect that to a model that is appropriate to that context. Right. And that's important for inference management, inference budget management, because obviously you can serve to you can redirect the query to smaller, cheaper models or higher quality models or models that are specialized in particular domain that have been fine tuned for a particular area of expertise or indeed have certain capabilities. Like they might be good at retrieval, they can retrieve from some knowledge base or even from the open web, or they might trigger a model that has been fine tuned for a voice response because obviously the length and style of a voice response would be quite different to one that was producing paragraphs and paragraphs in a traditional sense. So the router is a critical part of the architecture as well.

So Mustafa, I'm curious, if you were starting your first company today, given everything that exists and you know, the crazy rate of change, where would you focus your attention?

I would look for problem domains that make a virtue out of imprecision. So what is a problem that where if you solve that problem, the valuable contribution that you've made is that ambiguity, imprecision, multiple possible answers are the key thing. If you pick a problem domain where the consequences are really high if you get it wrong, and where there's really only like one or two correct answers, then your model is going to struggle. Right? That's the first thing I would say. Look for more of those things.

And just on that point, there is a ton of activity in domains that do require more precision, like law or accounting or tax. Do you think that that is a futile effort at the moment or actually.

Law doesn't require as much precision as taking actions. Right? So even in law most of the applications are retrieving similar cases or giving you summaries of cases. Right. Or where there are five possible summaries, all of which could be correct. Right?

Right.

Or where you're retrieving one case over another, you don't have to. So the law thing is actually a high stakes domain because the consequences if it gets it wrong are like really wrong, are bad.

Right.

But like unlike, you know, generating marketing copy. But actually there's a lot of right answers in that domain. Medical is much harder. Like clearly there are fewer right answers in medical and, and the consequences are really high. So that's a pretty difficult domain. You know, a bunch of my really good friends from DeepMind Health who have now been at Google just released a paper last week. Unbelievable work that they have done showing that, you know, they can basically provide an amazing reasoning engine for clinicians and I think in the future for patients too. It's like, that's coming soon.

What other kind of factors would you think about?

I would say where you can design an interface which naturally collects valuable label data for fine tuning by virtue of the interface. That's really important because if you're successful, you want to be able to compound that success because the more users you get, the more data you get, the more quality, more the higher quality model you can produce and then you get that virtuous cycle. So that's a really important part of it. And then it sounds obvious, but I think a domain where you can monetize much faster than you might think because you need to get people to pay for it pretty fast because GPUs are wildly expensive, as everyone knows.

What's an example of that that comes.

To mind for you? I think the companies that are doing specialist services for not quite 10,000 fans, you know, but like, you know, 10,000 true fans, but like, I mean like people that really do need that kind of niche, highly adapted expert system in your pocket. Yeah, you know, I don't know whether it's a mechanic or a dentist or a, you know, a sort of, you know, someone who's passionate about certain hobby or a chunk of ip. I think there's value in those kind of, those sorts of specialist use cases that people be prepared to pay for.

Yeah, that makes sense. I'm curious to hear just a little bit about the products, the AI products you're working on at Microsoft, what the portfolio looks like.

I'm responsible for Bing Edge, the browser and all of Copilot, which is deployed in basically every Microsoft Surface now. And it's been an impressive arrival actually. The quality of products and their scale and reach is sort of much greater than you might think as, as a kind of default, you know, Silicon Valley person who had grown up in Google.

You don't become a $3 trillion company for nothing.

Yeah, but the rep that we give it in Silicon Valley relative to what it has is like, I think is needs a, you know, rethink and also just like huge scale and distribution. My main goal is to uplevel the quality of Copilot. Yeah, so we're rapidly building some of the best models in the world, partnering very closely with OpenAI, building on top of all of OpenAI models and infrastructure, fine tuning their models. And the next phase is that we're really going to start focusing on memory and personalization. I mean, your AI should remember everything about you, all your context, all your personal data, everything that you've said, and be there to support you and be your aid and your sidekick. You know, throughout your life, you know, so that's what we're going to be focused on next.

That's fascinating. I'm curious, how do you think about the constraint of the existing applications? Microsoft Office versus the ideal version of Copilot?

Yeah, that's a good question. I mean, you know, people have often said that like an AI subsumes all other interfaces and surfaces, and I think that probably overstates it, but it's the right direction. I think there will come a time in a few years when the first thing you think is you just say, hey, Copilot, can you take care of this for me? What's the answer to that? Where do I find this? Can you book this? Remember that, buy this, do this like you're just going to have this ever present aid in your life that is going to change what it means to use a keyboard. It's going to change what it feels like to have apps. It'll move us way beyond the search engine and, you know, the browser. And you're certainly not going to think, I need to go write a document or send a message in a traditional way. You'll still have those things, right? But your AI will just manage a canvas of activity across your entire life and largely be coordinating with other AIs and other services and collecting information for you.

Okay, well, let's leave it at that. Mustafa, really appreciate you spending some time today and always enjoy catching up with you.

Yeah, great to see you, Seth. Thanks a lot. It was fun.

Thanks.

Ciao.

Thanks for listening to ProductLED AI. You can find more information about today's interview and the entire series on the website productledaipod.com you can subscribe to the show on all major podcast platforms and watch the video version of this interview on YouTube. And if you want to get all the links and details delivered right to you, sign up for my LinkedIn newsletter. I'm Seth Rosenberg and this is Product LED AI.