Microsoft CTO Kevin Scott on How Far Scaling Laws Will Extend | Training Data

The things that are brittle right now where you're like, oh, my God, like, this is a little too expensive or it's a little too fragile for me to use. Like, all of that gets better, like, it'll get cheaper and like, you know, things will become less fragile and then, like, more complicated things will become possible. Like that. That is the story of each generation and these models as we've scaled up.

On any given day, Microsoft may be the most valuable company in the world. And arguably no one has been more ambitious, more strategic, or more effective in its AI strategy than Microsoft. The key architect behind that strategy, Kevin Scott, CTO of Microsoft. We've had the pleasure of knowing Kevin for a couple of decades now, dating back to his time at Google, when he overlapped with our partner, Bill Korn. Bill will join us today for a very special episode of Training Data. We hope you enjoy. Kevin, thank you for being here on Training Data.

Yeah, glad to be here.

So, just to start, I know you've talked about this before, but for our listeners who might not be familiar with your story, how does a kid from rural Virginia end up becoming the CTO of Microsoft?

Who knows? Certainly not a repeatable plan, I don't think. I don't know. It is. The thing when I reflect back on my story is it's just a lot of being at the right place at the right time. 52 years old. So I was, you know, 10, 11, 12 years old when the personal computing revolution started to hit full steam. And so, like, right at that moment, when you're a kid trying to figure out what you're about and, like, what you're going to latch on to, like, I had this, like, really convenient thing that captured my interest and was a good place for me to ground my curiosity on. And, you know, and I think that's, you know, one of the object lessons in general is if you happen to be interested and like, really motivated to learn more and do more with something that is at the same time growing really, really quickly, like, you probably are going to end up in a reasonable place. And so, you know, I was interested in computers. I, yeah, I was the first kid in my, or the first person in my family. Neither my mom nor my dad went to university, so I was the first one to graduate with a bachelor's degree. Like, I majored in computer science and minored in English literature. Like, I had this moment when I was at this moment when I was trying to decide what I was gonna go do after I got my undergraduate degree where my two advisors were arguing about whether it should be PhD in computer science or PhD in literature. And I was very seriously considering both. But I was so broke and just so tired of being busted all the time that I picked the pragmatic path. You know, not, not that, you know, like, I, I still imagine what my life would have been like as a person with a PhD in English Literature, and I think it would have been just fine. But, like, I chose one of my two equal interests. And then for a while I thought I was going to be a computer science professor. And at the last minute, this is where Bill and I intersected. Like, I decided I was a compiler optimization and programming languages person through years and years in grad school, and I got almost all the way to the end. I was like, I don't think I want to be a professor anymore. Like, I, I would work on these things where it was, you know, six months of effort to write a paper and you make some synthetic benchmark 3% better. And I was like, this doesn't feel to me like the way to have a lot of impact in the world. And like, I don't want to do this over and over and over again for the next 25 years of my life. And so I, I sent my resume cold into Google in 2003 and I got a email from this guy, Craig Neville Manning, who had just gone off to New York to open up Google's first engineer remote engineering office. And like, I had an amazing interview at Google. I don't know whether this was on purpose or not, or like, I just got luck of the draw. But like, it seemed like every compiler person who was working at Google was on my interview slate and I was like, this is amazing. Like, all these people, like, know all of this stuff that I know and, you know, we can have easy conversations now. I worked on nothing that was even remotely close to compilers at Google. And I was confused why all of these people were there, but it was a great interview and I was super stoked and I joined Google. And Google is yet another one of those things. Just like the PC and just like the Internet. Like, it was a phenomenon that was growing crazy fast with a bunch of smart people working there. And that resulted in this opportunity. I had to go join this startup, AdMob, when it was very early on, like right at this pivotal moment when mobile was taking off and you needed things in mobile, like advertising infrastructure. And I helped build the seminal company, I think, in mobile advertising, and then was back at Google and then I helped LinkedIn go public running its engineering operations team. And then I was at LinkedIn when we got acquired by Microsoft. So, like, none of that I think you can plan. It's just like a lot of right place at right time and, you know, trying at every point you can to do the most interesting thing you can do on the thing that's growing really fast.

You know, when you talk about your personal history, Kevin, I guess, you know, the focus nowadays is on AI machine learning. A lot of the practitioners are people with PhDs. How do you think about sort of practical teams for AI, since you're obviously doing a lot of that work at Microsoft and involved with partnerships with OpenAI and others?

Yeah, I mean, I think if you are building the really complicated platform pieces of AI, so like the big distributed systems for training and inference, the big, like networking and silicon and system software components, or the algorithms that you're using to do training and inference, I think a PhD is super helpful. Like, there's just a huge amount of prior knowledge that you need to have in order to jump into the problem space and be able to, like, go quickly. And like, you know, you need to be clever, but, like, you don't. PhD is. I like, yep, I know you have a PhD, Bill, and are far cleverer than I am. But like, usually folks with PhDs are clever, but, like, they're not the only people in the universe who are clever. So, like, I think it's mostly helpful in the sense that you've gone through like, a pretty rigorous training regimen where you get a whole bunch of prior art stuffed into your skull. And like, you, you know, demonstrably can a very complicated project. And the PhD projects look kind of like AI platform systems projects. Except the AI platform and systems projects are lots and lots of people working together. Whereas when you're getting your PhD, you often are working in relative isolation on your particular thing. So, like, that's the, you know, one of the things people have to learn is like, how to get yourself docked into a group and to be able to collaborate effectively with a bunch of other people like yourself. So useful. But, you know, there's so much else in AI that needs to be done other than building the, the platform.

Yeah.

And for those things, PhD is helpful, but certainly not necessary. You know, like figuring out, how do I apply this to education, how do I apply this to healthcare, how do I build developer tools around this, how do I do all of the million things that happen when new platforms emerge that you sort of complete the whole platform into a portfolio of products and A portfolio of middleware and a portfolio of all of the other stuff you need.

Well, speaking of which, Microsoft seems like it has about the most sort of far reaching or ambitious AI strategy of anybody out there. Can, can you just kind of say in a couple words what is the AI strategy for Microsoft? And then just for fun, if you're going to grade yourselves, what have you done particularly well? What have you done maybe not as well as you could have.

Yeah, so I mean we've been sort of talking about this strategy. Microsoft is a platform company. Like we, I think have participated or like helped drive a handful of the big platform waves in computing. Like we were certainly one of the pillar companies in the personal computing revolution. Like we had an important part to play in the Internet revolution. Although I think that one was a far more diversely contributed to revolution than. Yeah, than personal computing. Yeah, we kind of miss the mobile computing revolution. But like each one of those things, like we have thought about how do you go build a technology platform for this particular era of technology that allows other people to go build on top of that platform to make useful things for other people. And so that is our AI strategy. It is like how do you, from frontier models to small language models to like highly optimized inference infrastructure, you know, like hyperscale on both training and inference, like economies of scale, like making the entire platform more accessible because it's cheaper and more powerful with every turn of the crank and like all of the developer tools and safety infrastructure and testing and everything that has to be there in order to have robustly built AI applications. Like go build that and like listen to developers and listen to people building building AI as, as intensely as you possibly can so that you are filling in all of the gaps that you can for them as they are encountering problems deploying this technology to users. So that is our strategy. And so yeah, I think we're doing a reasonable job of it. I don't know, I hate to grade myself. It seems a little bit disingenuous.

Right.

I want some low lights.

Well, so maybe before I do that, let me describe something about my own psychology. I like, I am an engineer and I think most engineers are like short term pessimists, long term optimists. And so the short term pessimism is like you come in every day and you're like, oh my God, this is like a bag of crap. Like I just don't like any of this. And like everything's broken and like I gotta, like I got, I gotta, got so much stuff to fix. And I'm so frustrated. But you work on all of those things anyway because you're optimistic that all of the problems can be fixed and that they're going to be worth fixing at the end of the day. And so, yeah, I mean, the. There, there's a bunch of stuff that I think we're doing really well. Like, I think we have absolutely, along with OpenAI, made very powerful AI dramatically more accessible than it otherwise would have been to a larger group of people. I think you, because of that work that we've been doing alongside OpenAI, we're just seeing lots and lots of customers who otherwise wouldn't be building powerful AI applications. And so I feel like we're doing a good job in the way that we're partnering. I think we're doing a good job in having a really particular point of view. And it's not an immutable point of view, but, like, it's a point of view about, like, what an AI platform ought to look like. And we're trying to, like, make it as complete as we can. You know, lowlights is I think we were a little bit late to, like, some of the basic AI stuff. So it wasn't that we were not investing in AI at all. And like, you can sort of look at some of the work that Microsoft Research had done over the years. And like, MSR was an early AI leader. And I think, you know, the Bill. Bill knows this just as well as I did just from his time at time at Google. And, you know, where we overlap for a number of years. But many, you know, maybe most of the really important advancements in AI over the past 20 years have been a function of some kind of scale. And it's usually you got data, scale and compute scaling combination let you do things that weren't possible at lower scale points. And yeah, at some point that scaling of data and compute is so exponential that you get past the point where you can have fragmented bets, where you can literally, it just becomes economically impossible to bet on 10 different things that are all exponentially scaling or have the ambition or the need to exponentially scale simultaneously. And so I think one of the things that we were a little bit late to is we didn't put all of our eggs into the right basket. Soon enough we were spending a lot on AI, but it was fragmented across a whole bunch of different things. And because we didn't want to hurt any of the feelings of smart people or whatever. Right. Like, I don't even know what the diagnosis was because a Lot of that was before I was at Microsoft. We just weren't as quick as we should have been at saying, nope, scale is what matters. And here's how we're going to focus our investments on scale in a principal way.

When did you get religion that scale is what matters? Was there a particular event or a moment that really crystallize that for you?

Yeah, I mean, I was. So I've been at Microsoft for about seven and a half years now and like, my, when I, when I became cto, my, my job was like, take a, take a scan left to right across both Microsoft and the entire industry and try to see where, yeah, we just had holes in execution where, like, we were not doing things at that point in time, which was, I guess, 2017 early, where, all right, like, what are we not doing today that we're going to deeply regret in 2019 or 2020? So like two, three years out. And like the biggest thing on the list was like, you know, our rate of progress on AI was not fast enough. So I'd say mid 2017, like, I had religion that that was going to be a big part of my job was like, helping us figure out what the strategy was going to be. And then in 2018, like, if anything, the Bert. The publication of the Bert. BERT paper from. From Google was like a real crystallization of that belief. So like, everything that I had that was in my analysis, I was like this, this is as fine an example as anything of like, why we have to really, really accelerate on getting more serious here. And so, like, you know, very shortly after that, like, I restructured a whole bunch of stuff inside of Microsoft to get us more focused on AI. And then about a year later, we did that first deal with OpenAI. And yeah, we have been accelerating our investments and like, trying to get more focus, more crisp, more purposeful since then.

When you were very early to appreciating the potential of OpenAI, what did you see in them at that time when that first partnership was struck?

Well, we, we had like, or at least I had this like real belief that what was happening with these models as they scaled, as they actually became a basis for building a platform that like the big shift that was happening is it wasn't just, you know, and I ran one of these teams at, at Google where you had a pool of data and a bunch of machines and an algorithm, and you were like training a model and like the model was for a specific thing. Like in the case of the things I was doing at Google, it was like, click through rate prediction for advertising and a handful of other things and just outrageously effective. But most of the work before GPT was about those sort of narrow use cases. Like you were purpose building models for narrow things and it was just tough to scale. You couldn't, you'd invest a bunch of compute and like you couldn't amortize the cost of the compute across anything more than just the narrow thing that you were building the model for. And you had to have a lot of expertise that, you know, if you wanted to replicate all of this, it's like you had to have different data and like different, you know, AI PhDs and you know, different processes every time you wanted to go build AI into an application and what, what was happening was like yeah, you, you had these big large language models that were useful for lots of different things so you didn't have to, you know, have a separate model for machine translation and sentiment analysis. And yeah, like all of the different text things that you were doing. And I was like, okay, this is extraordinary. And like they were also becoming more platform like as a function of scale. So transfer learning was working better as things scaled up. And yeah, and this is still the general pattern. So like everything, everything that we understand that large language models can do plus or minus like will get better when you get to the next scale point. And on top of that they will become slightly or maybe dramatically more general in the sense that their capability set broadens. And OpenAI had that same belief and they also had like a very principled analysis of how those platform characteristics emerged over time as a function of scale and a bunch of experimental validation that said that their forecasts were right. And so like it just, you know, you sort of like look at what the forecast says and you're like this is how much money it costs to run the experiment to see if you're going to be on forecast for the next turn of the crank. And like, you know, it was felt like a big number at the time, like it was billion dollars but like relative to what was happening, it just wasn't a large amount of money. And then GPD3 was on forecast and GPT4 was on. So like it, it just was like finding a partner that had the same platform belief that you did and like a track record of being able to execute through, through these scale points like it was, it didn't. Like I, I've done a bunch of things before that I have way more reservation about in the past, like just in terms of investments like this one didn't you know, like there was a bunch of people who didn't agree with me. I had pretty high conviction.

You touch on investment, I guess, you know, there's a lot of trade publications now speculating about the cost of doing training and so forth and you know, rumors of billions and billions of dollars being spent and so forth. And I guess based on my own background, I, I think training is going to get dwarfed by inference here pretty soon. How you, how do you see the hope? So, well, yeah, otherwise we're building models that nobody knows what to do with. That might not be a great investment. But. Yeah, how, how do you see kind of computing landscape evolving and where's it going? You know, the, I think people are joking that all the money is going to Nvidia at the moment.

Well, look, I think Nvidia is doing a, Nvidia is doing a good job. So like the, you know, the two interesting things that are happening with these models just in terms of the efficiency of the scale up, is each hardware generation is better price performance wise, usually by an extent greater than Moore's Law used to work for general purpose computing. So a 100 was about three, three and a half times better price performance than V100H100 quite that much. But you know, close on paper, the next generation looks very good as well. And so like you've got hardware for a variety of reasons, like you know, part of its process technology, but part of its architecture. And like a lot of it is like being able to leverage. Narrower word size in the computation. So like, you know, Instead of needing 64 bit arithmetic, like you're, you know, you're doing arithmetic with much less precision right now. And so like that, you know, there's just an embarrassing amount of parallelism there then like we're getting better and better at extracting that architecturally in the hardware. And you know, there's a bunch of innovative stuff happening with networking as well. Like we're well past the point for the frontier models at least where you can do anything interesting resting on a single gpu. So for years and years now, like both training and inference have been multi gpu, multi compute node problems. And so like there's a bunch of innovation happening on the network side as well, which allows you to strap all of the compute together like at the chassis level, the rack level, the row level of data center level more effectively. Which is great because you know, for the nerds listening, like we, we haven't had effective power scaling or denard scaling since 2012 or so. So yeah, we're not, we're, we're getting more transistors but like they are not, they're not getting cooler, you know, and like we, we just, we have a lot of density issues just with power dissipation that we have to go deal with.

Do you see inferences driving different data center architecture?

Yeah, I mean look, we already architect our training environments and our inference environments differently. They just need different things. And like I think all the way down to silicon and like through the network hierarchy you need different things for inference. And like inference is, inference is kind of easier than training, like training. The way that we're doing it now is like we go build big environments that take a few years to build. And yeah, with inference, like if somebody came along with like a better, a better silicon architecture or better network architecture, like a better cooling technology, like it's a much easier experiment to go run. You just go swap some racks out. I mean like I'm like my data center people will yell at me like it's not quite that easy, but it is, it is easier than like having to go like do a big capital project. Like, like a training environment looks like. And so like you know, intuitively you would think that that is going to result in more diversity in the inferencing environment and more competition and more like a faster rate of improvement. Like that's, and like on the software side that's certainly what we see like the inference stack just because it's such a large fraction of the overall compute footprint and it's constrained because we have more demand than supply at the moment. Like you just have very, very powerful incentives to go optimize the software stack to squeeze more performance out of it. Do you think we'll be in an.

Environment anytime soon where that demand supply balance changes? Not necessarily at Microsoft, but it feels like we're seeing that at the market level as well.

Yeah, I don't know. I mean if we, if we continue to see the platform continue to expand capability wise and it just becomes more useful. Like I think demand increases if anything now the shape of the demand is probably going to move around. Like I think you're already seeing a little bit of that. Yeah, building, building a frontier model is like a very, very resource intensive thing. And as long as people are like building frontier models and making them accessible and like maybe they're not accept accessible quite the way that people want, you know, like they're only API accessible and like there isn't an open source thing you can go instantiate and you know muck around with, but like, it's way more accessible than it was, you know, six or seven years ago, where like, the only way to access some of this stuff is you had to go work for like, you know, two, two or three tech companies. And so anyway, like. But I think, you know, you do have to ask yourself and somebody else should do the asking. Right. Because I'm like all kinds of biased. Right. But like, I, I don't know how many frontier models you actually need if, you know, they're all, roughly speaking, in the same tier of capability. Yeah, that's an awful lot of money to spend for, you know, things that are, you know, roughly equivalent. You know, it's sort of like, you know, if you're starting a company right now and you believe that you have to build your very own frontier model in order to go deliver an application to someone that's almost the same thing is saying like, I got, I gotta go build my own smartphone hardware and operating system in order to deliver this mobile app. Like maybe you need it, but like, probably you don't. And so I mean, to Bill's point, like, I think the thing that makes sense, you know, for the market is like, you just, you, you're gonna wanna see lots of people doing lots of inference because that means you've got lots of products that have found product market fit. And those things are. But like, lots of speculative dollars flowing into infrastructure R and D probably ends the same way that many speculative infrastructure booms have ended.

On the scaling front, Microsoft published a paper some time ago pointing out that the quality of training data is maybe at least as important as volume. And I think one of the speculations you see now in the industry is that we're running out of sources of high quality training data. And you're reading at least some articles claiming that various partnerships are being struck to get access to training data and it might be behind paywalls and so forth. How do you see that evolving? Because it feels like we have more and more computation, but we may not have more and more training data.

Yeah, I mean, I think that was almost inevitable. It is, in my opinion, a good thing that quality of data matters more than quantity of data because it gives you an economic framework to go do the partnerships that you need to go do to make sure that you are feeding, creating your AI training algorithm or curriculum that is going to result in smarter models and honestly not wasting a whole bunch of compute feeding at a bunch of things that are not. And I think one of the, from an infrastructure perspective, One of the things people have been very confused about is like, a large language model is not a database. It's not a repository of facts. You know, it's like, important for it to, you know, quote unquote, know some factual things. But it is like the world's crappiest database, you know, honestly, like, if you need it to be your retrieval engine. And so, like, you just shouldn't think about it as like, hey, I got this thing. And like, it has to have everything baked into it, like into the model weights themselves so that you can, you know, recall a bunch of stuff. Because, you know, like, as you've seen, like, the, the recall is imprecise in the same way that human recall is imprecise. So they're just much more efficient ways to do recall. Yeah. So look, I, I think you, you are. I mean, the way that we say, see things developing is like, you have data that is valuable for training models, and then you have data that you need to have access to for an application in order for the model to reason over. And like, those are two different things. And I think they're probably two different business models around those things. So. And like, at the end of the day, this is all just, this is about business models, right? Like, people who produce data want to be compensated for use of that data. And so, yeah, we have all of this data sitting inside of search engines right now, not in randomized weights, but quite explicitly. It's like sitting in indices and being in Google and whatnot, just waiting to be retrieved and, you know, plus or minus. Everybody's okay with that because there's a business model there that makes sense. So you, like, you enter a query and like, you're either sending traffic or, you know, like there's SEO and like advertising, like a whole bunch of like, business model that surrounds that. I think we'll figure out a business model for that referral data so that like when an agent or an AI application needs to retrieve some information from someone so that it can reason over it and give the user an answer. Like, we will figure out the business model for that. Like, it'll either be subscription, rev share, it'll be licensing, it'll be like some new flavor of advertising. You know, I was just telling someone the other day, like, if I was in my 20s right now, like, you know, for all of your entrepreneur, like, somebody ought to be out right now figuring out what the new ad unit is for agents and like, just building the company because, like, it, it will have the same characteristics and qualities as you know, previous ad units. Like you have people with information and products and services who are going to want to get to the attention of someone who might want those data and products and services and like, quality is going to matter and, you know, relevance is going to matter and a bunch of other things. And I, I would be shocked if there isn't a auction model for, you know, that's going to be the right way to value everything. And you know, like, maybe there's referrals and like referrals will have some economic value. I think for training. It's just a little bit different because it's very, very hard when you're building that model to, at the time that you're doing the building to really be able to ascribe monetary value value to a particular token of input just because it's contribution to the model. Like the same way that a word from Moby Dick has a very diffuse contribution to your own human intelligence, even though you definitely read it at some point in your career or your life, like how valuable that is to like forming, you know, Bill Corin or Kevin Scott's, you know, useful intelligence. Like, who knows?

Speaking of which, this one of the things that we hear a lot of times is the value function is in some ways the bottleneck to broader reasoning capabilities. It's easy enough to construct a value function when you're playing a game with a known winner and loser like go or chess or poker or diplomacy. But it becomes a lot harder to construct the value function when you're going into broader domains and it's things like assigning the value of Moby Dick to Kevin Scott's life, that sort of thing. Are there practical solutions to this? Are there practical implications to this? I guess the broader question would be where do you see the overall field of reasoning and LLMs going?

Well, look, I think people are trying to get at this. So you've got a bunch of benchmarks like GPQA and mmlu and we're just sort of rolling through a bunch of benchmarking paradigms to try to come up with scores of performance for these models, like whether it's reasoning capability or. Yeah, and yeah, I think one of the interesting things we've seen over the past handful of years is like, we just are very quickly saturating these benchmarks, like where you like, one emerges and then, you know, within a model generation, like you'll completely or like get very close to saturating the particular benchmark and then you got to go find something else to help be your guiding Light. So, but like, let's just sort of assume that like, you know, you'll have some interesting benchmarks that are correlated with the reasoning capabilities you want models to have. You know, then the question, it's just an expensive experiment to run. Like, you can run an experiment where it's like, okay, like, I'm going to train a model like, you know, with this information in and or out and like, does it get better or worse at, you know, performance on these reasoning benchmarks? And, you know, like, I think all of us have done different versions of those experiments. Like, they're just extraordinarily expensive to run at the most granular scale you can imagine running them. Part of that paper that Bill referenced, like textbooks are all you need is like a, it's not the full story, but it's like part of a story that is like, you know, just sort of evaluating like token contribution quality to model performance. So yeah, I like, I think it's. And everybody's got every incentive in the world right now to try to figure out like, what that is. If for no other reason than you, like, in a world where you're synthesizing data, you're literally spending computer to generate synthetic tokens for training you. You really want to make sure that the tokens that you're generating are actually useful or not.

Where, where do you think the models are at the moment? I, you know, I think Microsoft has introduced a whole bunch of copilots to try to help end users with, with their, with your products and so forth. On the other hand, I see lots and lots of companies trying to build agents that can be kind of autonomous actors. Now that's a wide spectrum of kind of expected performance of what the models can do. Where do you think we are? Where do you think we'll be in a couple of years?

Well, yeah, so I think that's a super good question. And there's even a philosophical thing there about what it is we should want to.

Well, there is the specter of everybody's job getting replaced.

Right. So, yeah, you know, somebody. Look, we chose the name copilot for the things that we were doing relatively deliberately because we want to, at the very least, encourage everyone who's building these things inside of Microsoft to think about how can I help augment someone who's doing some form of cognitive work. So like, we want to build, you know, assistive, not substitutive tech. And, you know, the good news is also easier to think about how to go from, you know, sort of rough frontier model capability to useful tool when you're narrowing it down to a domain. And so like I think that's been a reasonable deployment path and like we've got a handful of our co pilots that have real market traction right now and are like, you know, in daily use by like a lot of people doing real non trivial cognitive work, you know. And I think that will expand over time.

Just on that. What are some examples of copilots that have really like hit the bullseye already versus maybe copilot where the technology's not quite ready in terms of like jobs to be done?

Yeah, like I think you know, GitHub Copilot is like probably the, you know, the thing we've talked most about and like there's the most public conversation around. Yeah, like it's, it's been a, been a hit, you know, it is genuinely useful. Yeah, we, we've got some other co pilots that are like that, that are, you know, that are, that are super useful. But you know, I think the thing that Bill was getting at like the more general, the more general the copilot is like the, you know, the, the harder it is to have it actually take very high precision action on your behalf autonomously, you know, particularly if you know it's doing something where it's representing you, where there are stakes and consequences and accountability back to you if this agent makes a mistake. And we're trying to be very deliberate there because I think one of the things that you don't want to do is introduce a thing that's going to make a whole bunch of these sort of errors where the user's first reaction is like this doesn't work and like I'm not going to try it again for a good long while. So we'd rather have it be very good before we introduce it. Yeah. Which again means you're, you're sort of optimizing for use cases, not for like you know, super, super broad things. I mean there's a good, like we, we did a partnership recently with you know, with Devin which I think is another one of these like very interesting like you know, use case specific things where it's like you know, frontier model plus a whole bunch of other stuff that is like optimized for like giving humans high quality recommendations for actions that they can take. And then when you click accept on the action like you, you have reasonably high conf. You know it's going to work and you haven't made another set of problems for yourself. And so like I'm, I'm guessing too you all see this in your portfolio. Companies like there, there seem to be a bunch of companies out there right now that are like doing exactly this and, and that it's, you know, useful and working well.

It's interesting because we hear one of the things that we hear fairly consistently from the companies that are further on in their AI journey. You know, everybody kind of starts in the same way where they start playing with OpenAI and then maybe they start using some of the other proprietary foundation models and then incorporate some open source models and maybe they have some of their own stuff. There's a vector database in there somewhere.

Yep.

From an architectural standpoint, it feels like people tend to go on a not quite the same journey, but journeys that sort of rhyme. But then what we hear from them when they're 12 or 18 months down the road is there's kind of this massive 80, 20 rule at play. And maybe it's a 98. 2 rule, but you can automate most of a task pretty quickly and pretty effectively, but getting it to the point where it's actually end to end running autonomously in a way that is compelling and consistent enough that you can actually trust it. Kind of that last mile, that last couple percent that makes you really trust it. Yeah, it seems like that's been pretty elusive for a lot of tasks. And so one of the things that we're really curious about is, okay, well when do the foundation models themselves, you know, get good enough to knock out that last 2%? Or is that a domain specific thing and that's really the job of the software vendor who, who lives on top of the platform to figure out that last 2%.

I, look, I think it's going to be both for a while. Like the, the two things that I think you can trust. Yeah, I, I know you, you guys are probably going to ask this question at some point, but like we're, we're in. Despite what other people think, we're, we're not at diminishing marginal returns on scale up. And like I, I try to, I try to, you know, help people understand. Like, you know, there, there is an exponential here. And like the unfortunate thing is you only get to sample it every couple of years because it just takes a while to build supercomputers and then train models on top of them. And, and so the, the next sample is coming and like I, you know, I can't tell you when and I can't predict exactly how good it's going to be, but it will almost certainly be better at like the, the Things that are brittle right now where you're like, oh my God, like, this is a little too expensive or it's a little too fragile for me to use. Like, all of that gets better. Like, it'll get cheaper and like, you know, things will become less fragile and then like, more complicated things will become possible. Like that. That is the story of each generation in these models as we've scaled up. And so like, you know, we even think about this inside of Microsoft and like, one of the category errors that our own developers who are building these AI products can make is like, get too convinced that the only way to solve my problem is like, I have to go take the current frontier and supplement it with a whole bunch of things. But which you do have to do, but like, you want to be very careful architecturally when you're doing that, that it doesn't prevent you from taking the next sample when it arrives. So you just want to architect these, these applications where when the new goodness comes, you can go plug it in and like, you'll have to go optimize that as well. Like, I think that's just sort of the grind that we're on. But like, you just like the, the, the thing that was killing us internally is I would have teams inside of the company who would look at a Frontier model and say, oh my God, there's no way that we can ever deploy a product on top of this because, like, this is fragile and this is too expensive. And so, you know, like, please give me like giant pools of GPUs and like, let me go spin up a big team doing, you know, like, a very tailored version of this. And like, we're going to build a specific model and like, and yeah, they would go off and like, spend a whole bunch of money and like, the thing would be, you know, a little bit better cost wise, at the same level of performance as the current frontier. And then the frontier would snap to the new point and it would be just doomed. And, and so like, you, you just architecturally don't want to get trapped by that. I don't think. I mean, like, that'd be my advice I give to everyone is like, just give yourself the flexibility to snap to the new, you know, to the new frontier when it emerges. And, and like, that lets you preserve all of your skepticism. You can believe, you know, believe all you want that the new frontier is not coming. Like, go, you know, like, you know, read your favorite Twitter troll that says it's all over and a sham. And like, and like, but, but Just give yourself the option that, you know, may maybe what's been happening for six years now is going to continue.

Well, hearing that we are not at diminishing returns to scale, I'm going to count that as good news. And so let's stay on the theme of good news. I know that you're short term pessimist, long term optimist. Can you give us some of the optimistic point of view for where we're heading in this world of AI? What are some of the things that you're most excited to see in the world in five or 10 or 15 years or whatever you count as a longer term horizon?

Well, look, I think the thing that everybody ought to like spend some time thinking about is where are the gnarliest zero sum problems that we have in society? Like where, where are the things where like we just are fighting with one another or you know, like we are immiserating people because whatever it is that people need, there doesn't appear to be enough of it. And I think for a good number of those things, like what you have to have to turn them into non zero sum games to create abundance and to relax some of these constraints is you have to have technological breakthroughs. It's like the only thing that reliably that's ever turned zero sum to non zero sum in human history is like some tech has to come along that lets us, lets us have more, you know, and like, you know, whenever tech comes along and like creates more, it doesn't mean that the more gets, you know, equitably and uniformly distributed. And like, I think there are real conversations to go have about that. But like, what you do want is the more and you want it to be directed at things where like, we're just having a tough time right now. Yeah, like I'll tell this story that I've told a couple of other times recently, but you know, my mom, like I grew up in rural central Virginia. My mom's 74 years old and like she's suffered from this thyroid condition called graves disease for 26 years. And so, you know, when you have Graves disease, your thyroid is hyperactive, you're generating too much thyroid hormone. And so they go in and like irradiate your thyroid gland to reduce its activity level. And then you take hormone replacement therapy to like upregulate your hormones for the rest of your life. And so she was having some blood pressure issues and her doctor dorked around with her dosage of this hormone medicine. And then like, she just had like some serious health Issues as a consequence of that that landed her in the ER and rural central Virginia, like six times, like in a pretty short number of weeks. And yeah, the interesting thing there was the first time she went to the er, like she was presenting all of these cardiac symptoms and it was pretty clear they hadn't even read her chart. Like it hadn't registered on them that she had graves disease. And like the thing that they needed to do was like, go order a TSH panel to see what her like thyroid hormone levels were. Like, if they'd done that right away, like they would have said, okay, like we got to go, you know, adjust your medicine. Yeah. And like I, I'm not even ascribing ill intent. Like, you know, this is a health care system that is egregiously overburdened. Like, this is not a place where, you know, like, you've got this influx of, you know, talent, like coming into. Yeah. This part of the country and, and like, it's a lovely part of the country. Like, I, like, love it. Yeah. So like, I'm not criticizing anything. It's just, you know, they, they have an aging population and they don't have enough like young people coming in to be things like doctors to like, help this healthcare system keep up with all of the challenges that they have. If some of those doctors had had access to GPT4 and like it was an approved, you know, product use, all they would have, would have needed to do is like, put the symptoms that she was presenting in and her medical record and it would have said, hey, she needs a TSH test. And if you would put the TSH test result in, the recommendation it would add was like, look at the dosage of her hormone replacement therapy that she's on. And like, if, if, like this isn't theoretical, like I did this, I, it could have helped alleviate a massive amount of her suffering right now. And like, I think the only reason like that she got out of this tough situation that she was in was I had to intervene. Like, I, I sent her to a specialist that was, you know, 400 miles away that she couldn't have gotten into without, you know, special, you know, and like, it's, it's ridiculous. Like, yeah, there, there are so many 74 year old old southern ladies or old Midwestern ladies or like folks who are going through similar sorts of things who do not have someone who's going to go in and intervene on their behalf who are suffering unnecessarily because we're not even deploying the technology that we've got right now and like, it's just going to get better. And so like, that's the thing that I'm excited about. It's like, let's, let's go, let's go give kids a leg up in education. Let's go fix some of these crazy problems we've got in the healthcare system that is just, you know, absent technological intervention, is just going to get more strained over time. You know, let's equip our scientists with better tools so that they can, you know, find better carbon capture catalysts so that, you know, like, we can design safer modes of transportation so we can more quickly get to a, you know, like post carbon economy. Like, it's just so many things we can go do with this stuff. Like, I'm super, super optimistic about it. And so, you know, like, it just kills me. Like what we don't want to do is get distracted on, you know, like things that are just, you know, effectively noise in the ecosystem right now. You know, we're, we're getting so sideways sometimes with you know, like this model like said something that hurt my feelings or, you know, and like, and I don't, I don't want to dismiss like, you know, people's feelings matter and you know, I'm not trying to be a jerk here, but I do want to make sure that as we're thinking about how we develop and deploy the technology that we are always remembering like what the cost of not deploying the good is because that is a high, high cost.

Very well put.

Yeah, I'm here. Probably a good note to end on, I think.

Well, we have one other question that we like to ask people and it's a quick one. So I'm going to ask, I'm going to ask you this one. Who do you admire most in the world of AI?

You know, I was thinking about this, so I think it's Ray Solomonoff who was one of the folks who was at that Dartmouth workshop in the, in the 50s where you know, Marvin Minsky and Simon and you know, like a whole bunch of folks convene that summer and like they were all interested in machine intelligence and they, they coined the, the term artificial intelligence at that workshop. And like the reason that Solomon off is so interesting, like not, not many people I think know who he is outside of computer science is like, he was the one from the very beginning who was pushing on this whole notion that probabilistic methods were going to be very important for the development of AI. And you know, when, when I was in grad school in the 90s, the prevailing academic theories about how we were going to get to AI were all about like, okay, well you know, like there's some magic, you know, minimalist calculus about human intelligence and like, we've just got to figure it out and it's got to be, you know, rule based systems and ontologies and you know, symbolic reasoning and like a bunch of like, stuff where, you know, we, like we do in physics, we were going to have to like, you know, divine, you know, the, the inherent simplicity in the system, like figure out what the rules are. And as soon as we understood the rules, like, we'd be able to make software emulate human intelligence. And Slamanov is like, no, no, no, like this, this is just a. Intelligence is an extraordinarily complicated phenomenon and like, the only way that we're ever going to really get there is modeling it with probabilistic methods. And he was right. And like he was, he was judged wrong for a very long time. And so I, I really admire his contrariness. Yeah. All the way back in the 1950s and like, he stuck with his, he stuck with his beliefs his entire career. And I don't know whether Ray actually lived to see how right he actually was.

That's a great answer and a great story. Thank you, Kevin.

Yep, you're very welcome.

Thank you.

Sa.