Scale AI Leadership Summit 2024: Alexandr Wang Opening Keynote

Please join me in welcoming to the stage Alexander Wang, founder and CEO of Scale AI. Thank you all so much for coming. We have a pretty incredible confluence of people here. We have a mix of folks from government, from the intelligence agencies and DoD, and policymakers that we're very fortunate to have here. We have many researchers and folks from the AGI labs themselves that are building this incredible technology. We have folks from some of the largest enterprises in the world and other folks who are sort of like, really core to this entire project. This is our second year hosting the NAT and I are hosting the summit. So we thank all of you for coming and those of you who return. We wanted to start off this session with just a quick presentation about where we are, and we sort of named this the Calm before the Storm. Because, you know, I think to a lot of people in the AI industry itself, it feels sort of very clear that we're basically on the precipice of AGI, or superintelligence or whatever you want to call it. The technology is getting very good, and we'll get very good very quickly. So we sort of wanted to take a chance to sort of take stock. This will be kind of a whirlwind, and then we'll ultimately deep dive into each of the individual topics over the course of the session. So many of you know this. I'm from Los Alamos. This is kind of one of my shticks, so I'm just going to keep using it. My parents were both weapons scientists at Los Alamos National Lab. This, I think, was the last major project that had many of the qualities of this technological wave, this very advanced technology that really was sort of, kind of unknown whether or not it was going to be possible. And up until the path was very clear, sort of this sort of speculative thing. The atomic bomb. It had very, very important implications for national security and geopolitics. It was at a time when the geopolitical situation was worsening quite rapidly, and it raised a lot of key ethical questions and a lot of very big questions, frankly, writ large about AI as a technology. And I went back through and looked at various quotes from the scientists from the Manhattan Project, and there were these two quotes from Oppenheimer that really stood out. It is a profound and necessary truth that the deep things in science are not found because they are useful. They are found because it was possible to find them. When you see something that is technically sweet, you go ahead and do it and you argue about what to do about it. Only after you've had your Technical success. That is the way it was with the atomic bomb. You know, I think these ring particularly true as we sort of think about AI in its modern day, because, you know, we'll talk about this in a second. But if you look at the history of modern AI, you know, the deep, the original deep neural networks, you know, it had, you had no real idea that it was going to end up in, you know, these very powerful AI systems. And while there's been a lot of hubbub about AI security and, you know, AI safety, these are, you know, we are, we are here, we are where we are, we are at the precipice of this incredibly powerful technology with many open questions about how we navigate it properly to sort of walk through. Maybe it's useful to sort of first zoom back out on the, on sort of like AI in general as a technology. A lot of people in here know this, but AI is made up of three primary pillars, Compute, data and algorithms. And these three pillars sort of come together to drive all the AI advancement we see. One of the cool things about AI as a technology, which means that it, it's less likely to get bottlenecked than other major technologies like fusion or whatnot, is that any advancement in any three of these pillars results in advancements in the sort of end technology of AI. So, you know, if there's more compute added, that drives some advancement in AI, if there's innovations in data that drives AI, if there's innovations in algorithms. O, I think is the best recent example of this where, you know, we've been kind of at a compute standstill for some time actually. Like, you know, compute has been relatively stagnant for some time, but then algorithmic advancements mean that you get, you know, just dramatically better models and that this, you know, this sort of general paradigm where advances in A three drive and advancements in technology I think is one of the reasons why it's sort of, you know, all AI summers from here and then the pace of new models is pretty staggering. Honestly, it's pretty hard to keep up. You know, part of what we do is eval and new models and it's, it's pretty hard to keep up with all the language models that come out. 2012 was the first real large scale deep neural network. This was alexnet, trained for Imagenet. And the sort of 2012 through 2018 days seem quite pastoral by today's standards. You know, every, every once in a while, you know, you'd have a, you'd have a big model and then everyone would Go to the conference and talk about it. Now you have 20 models that come out between conferences and, and it's sort of this, this massive flurry of, you know, great open source models, great multimodal models, sora, great image models, great video models and new reasoning models. I like to sort of break down and to contextualize the modern era of AI into sort of three major phases. So the first phase was research, which really was roughly 2012 until 2018. So basically lasting, you know, going from the first deep, deep neural network, Alexnet, which was really trained to do basic image recognition. This was the era when all AI could do was tell you if there were cats in YouTube videos. And that was this incredibly powerful thing all the way up until the first GPT model. So the transformer and the first GPT model trained by alec Radford at OpenAI, that kicked off the next phase scaling. So the first six years, 2012 to 2018 was sort of this research phase. Then 2018 to 2024, really up until today is the scaling phase. You know, the amount of resources going to these models grew more than 10,000 fold over this time period. Last year or this year, I should say something like $200 billion is aggregate going into training these models. And this has resulted in incredible advancements in performance. You know, we've gone from GPT1, which was nothing special, to OH1, which is sort of PhD level in math and computer science. And then O1 I think kicks off this new phase of, you know, we sort of think about as an innovation for era, which is basically now until superintelligence. We'll see if that's six years or maybe even a bit less than that. The hallmark I think here is that, you know, we're spending $200 billion on the models. We probably can't spend a lot more than that, you know, probably can't spend, you know, 200 trillion on the models. So there's only limited amounts of scaling left in terms of orders of magnitude. And so we need corresponding innovations to sort of come alongside. Definitely advanced reasoning and test time compute is one of those. And we think there's probably a few more handful of others and will get us to superintelligence. And we have, I guess, like Shameless Plug, I think scale. We've been very fortunate, I think to really fuel a lot of this incredible advancement. Back in the sort of computer vision, image recognition days of AI, we were proud to work with a lot of the large autonomous vehicle programs, but which has now actually yielded Waymo has the L4 autonomous vehicles on the streets of San Francisco, which is a pretty great milestone. And then obviously worked with OpenAI and many of the labs in this sort of entire run up of these models. The first RLHF experiments on GPT2, all the way now to the incredibly powerful reasoning models or agentic models and whatnot. And then as this has all happened, we're seeing two additional key trends layer on top of this. So we, the first is really the sort of importance of additional testing for these models and frankly evaluation. So we worked with, we did one of the first public red teamings of the models at defcon, which is sort of some White House support. And then we continue to work closely with the DOD on testing of these models and launch CL to do sort of like public testing of these models. And then we've seen just huge amounts of adoption from the enterprises and governments, which we'll talk about over the course of the sort of entire session. So going forward to make this concrete, we sort of see maybe five challenges towards superintelligence and we'll sort of run through these like a buzzfeed listicle challenge. One is the data wall. This was sort of maybe peak memed six months ago, but is a real problem. Basically, you know, we're going to run out of all publicly available data to train these models at some point. You know, Epic AI had an estimate that 2027 to, so 2030 was probably the estimate. You know, if you talk to people, they say much sooner, sooner than that. And so what are we going to do about that? Well, the good thing is we have agency. And so there's sort of a few main solutions to this, I think, you know, breaking through the data wall with frontier data, which we think is an encompassing topic, including synthetic data of various forms as well as more advanced data types where you have greater sort of sample efficiency to be able to learn more advanced concepts like reasoning, multimodality, agent data. Embodied AI is gonna be a big area. We'll talk a bit about that as well as enterprise data. And then the vast majority of data out there is still private and proprietary. It's locked down. So, you know, GPT4's training dataset was about a petabyte, actually half a petabyte. JPMorgan's proprietary dataset, you know, all the random data they have is north of 150 petabytes of data. They're just one of many enterprises. There's like a crap ton of data that's just sort of lying around that has not yet made it into any major training run. The second challenge or challenge number two that we see is really around evals. This is one of these things that like, I think people in the AI world talk about and nobody else really knows why it's important. But evals are this sort of like measuring sticks and the sort of the incentives that we use to measure progress in these models. And so the fact that now evals are both overfit and saturated, so they're both over fit is in, you know, they're sort of gamed a little bit and then saturated in the sense that, you know, they, the models are already really good at all the evals. These are big problems. These are going to sort of, you know, they, they create an era of maybe more aimless research. And so it's very important that we set up the right evals. You know, this is a research result that we had around called GSM1K, where we found that a bunch of the models were actually meaningfully overfit to some of the open source evals. And so this is kind of a big problem. It's hard to measure progress if we don't have this. The other one is that the benchmarks are becoming saturated. So in the past years, if you were to look at mmou, math, gpqa, even the models seem to be plateauing. Well, that's not because the models aren't getting better. It's because the evals are just not hard enough. And so we're at this point where we're like, we're at this tipping point where the, where the models are getting really good. We need evals that are much, much harder to actually measure the sort of like path towards superintelligence or beyond from now. So we sort of, you know, at scale, we worked on two things. So first we partnered with the center for AI Safety, Dan Hendricks, on an eval called Humanities. Last exam, we, what we're doing is collecting super hard research questions from every field of research to be able to create, you know, the hardest exam that humanity has ever seen. We think that if, you know, this gets solved, if this gets saturated, we probably basically have AGI. So, you know, we have that to look forward to. And then we're also, we build all these CLE vowels, you know, private expert evals to be able to test, you know, the full breadth of capabilities of these models. This is one area where, like, explicitly we want to partner with as many people in the or in the audience as possible in really charting this path of how are we going to measure the sort of like super advanced capability of these models. The next challenge is that like, agents, despite being the thing that everybody talks about, are not here and are not reliable. You know, we see this very strong parallel between these autonomous AI agents and the sort of like L1 through L5 progression that we saw in autonomous vehicles. We think it's actually quite fitting. You know, L1 is maybe just a chatbot. L2 is a sort of assistant that you can sort of ask for help on various things. And then we're sort of charting this path, you know, the interesting path, I would say, of L3 to L5 autonomy. So, you know, L3 agent autonomy might be, you know, agents that you can use for discrete portions of workflows. So you can start using them as like, you know, things that you can basically start relying on. L4 autonomy maybe flips that. So you have autonomous agents that when they need human help, they'll like ask you. So it's sort of more a teleop paradigm. And then L5 is, you know, when we can sort of all just go on vacation and hang out. And this path even more discreetly between L3 and L5, we think it's even important to, to discretize those. But you know, this is kind of the path that we're on and there's a bunch of things, you know, there's a few things that we think are required in this or sort of like, you know, paths forward in this challenge. The first is just reasoning across every domain. The reasoning, advanced reasoning capabilities of these models are just going to work, we think, across basically every domain right now. They work really well for math and physics. They'll work everywhere. That's going to. Cause that's going to be a huge, huge driver of value, of economic value writ large. So every field will have advanced reasoning capabilities. The second thing that we think is actually really, really important is agent scaffolding and really like teleop infrastructure for agents. So this is something we're really focused on at scale actually, which is okay in the future when you have these agents that are, can do most things but will get stuck, how are they going to interact with humans? How are they going to sort of like call up to humans? And I think it's like not an exaggeration to think that in the future most of us are just going to be tele operators for AI agents rather than anything else. Um, so this is sort of a key step in the tech pathway. And then the obviously mainline stuff is better rl, better reinforcement learning and new data types to ultimately fuel sort of agency as a core capability of these models. Challenge number four, I'm really not the best person to talk about this, so I'm gonna be very brief. But obviously chips and energy are like a big bottleneck. I think a very conservative estimate of the amount of power that's needed to fuel these, these Data Centers is 100 gigawatts. In the next five years, we'll have a bunch of people talking about this. So maybe that's way off even, but that's roughly 20 Chicago's worth of energy, which is trillions and trillions of dollars of capex. So this is like clearly a big challenge. I have no solutions here, just a challenge. And then the last challenge, which is, you know, we figure might as well call out very plainly, is that China is a near peer competitor and somebody that we need to be pretty careful about. You know, obviously there's a fundamental ideological misalignment between the US and China and that, you know, could very easily manifest in, in how this technology plays out. You know, right now we're ahead on algorithms, probably behind on data, probably ahead on compute, and if we can maintain that, we'll probably be okay. If any of those slip for whatever reason will be like a Taiwan action or whatnot, it'd be pretty bad. I'll kind of end with a few things that we're excited about beyond the horizon, or just like things to be excited about in general for AI, I think the, the first is obviously embodied AI and robotics. Elon's event was cool, I thought so. I think that it's very clear there will be a pathway towards robots that can do a lot of the manual tasks in the world. Long horizon thinking, or better put, just the ability to get these models to ponder for a very long time to solve extremely hard problems. The models can just think for a few years and then solve cancer, cure cancer, or solve the millennium problems. That something is very exciting. And then something that I think, you know, we'll, we'll see with the new MLE bench evals, but might be around the horizon are AI researcher agents. So models that can actually perform as AI researchers, you know, who knows how many years away this is. But that is basically the first form of real self improvement that I think we're gonna see, which would be kind of a big deal in the, in the sort of development lifecycle of these models. And so if we sort of like sum it up, there's these very exciting tech milestones on the horizon, a lot of exciting upside for humanity and then the path to get there. There's all these things that we have to get right. You know, we have to get compute and energy infrastructure, we have to get the frontier data correct, we have to unlock all the private data sources, we have to get the challenging benchmarks right. There's a lot of problems on AI policy which we'll hear about through the panels today. And lastly, this sort of like looming threat of competing with China. This is like the greatest time of human history. So you know, I think the point of this is really that we are all participants in this project to sort of work together to drive superintelligence and get it right. So with that, thank again all for coming. We have a great set of sessions today. We're going to talk about frontier capabilities, we're going to talk about chips, we're going to talk about the eval wall, we're going to talk about enterprise adoption and a few of the things that we're missing. We'll also have a surprise panel with two of the four co creators of ChatGPT on the creation event of ChatGPT. So get excited. Thanks so much everyone.