“The Future of AI is Here” — Fei-Fei Li Unveils the Next Frontier of AI

Visual spatial intelligence is so fundamental. It's as fundamental as language. We've got these ingredients, compute, deeper understanding of data, and we've got some advancement of algorithms. We are in the right moment to really make a bet and to focus and just unlock that.

Over the last two years, we've seen this kind of massive rush of consumer AI companies and technology, and it's been quite wild. But you've been doing this now for decades. And so maybe walk through a little bit about how we got here, kind of like your key contributions and insights along the way.

So it is a very exciting moment. Right, Just zooming back. AI is in a very exciting moment. I personally have been doing this for two decades plus, and we have come out of the last AI winter. We have seen the birth of modern AI. Then we have seen deep learning taking off, showing us possibilities like playing chess. But then we're starting to see the deepening of the technology and the industry adoption of some of the earlier possibilities like language models. And now I think we're in the middle of a Cambrian Explosion in almost a literal sense, because now, in addition to texts, you're seeing pictures, those videos, audios, all coming out with possible AI applications and models. So it's a very exciting moment.

I know you both so well, and many people know you both so well because you're so prominent in the field, but not everybody grew up in AI. So maybe it's kind of worth just going through, like, your quick backgrounds just to kind of level set the audience.

Yeah, sure. So I first got into AI at the end of my undergrad. I did math and computer science for undergrad at Caltech. That was awesome. But then towards the end of that, there was this paper that came out that was at the time a very famous paper, the CAT paper, from Hong Nak Lee and Andrew Ng and others that were at Google Brain at the time. And that was like the first time that I came across this concept of deep learning. And to me, it just felt like this amazing technology. And that was the first time that I came across this recipe that would come to define the next more than decade of my life, which is that you can get these amazingly powerful learning algorithms that are very generic, couple them with very large amounts of computer, couple them with very large amounts of data, and magic things started to happen when you compile those ingredients. So I first came across that idea, like around 2011, 2012. Ish. And I just thought, like, oh, my God, this is going to be what I want to do. So it was obvious you got to go to grad school to do this stuff. And then sort of saw that Fei, Fei was at Stanford, one of the few people in the world at the time who was kind of on that train. And that was just an amazing time to be in deep learning and computer vision, specifically because that was really the era when this went from these first nascent bits of technology that were just starting to work and really got developed and spread across a ton of different applications. So then over that time, we saw the beginnings of language modeling, we saw the beginnings of discriminative computer vision where you could take pictures and understand what's in them in a lot of different ways. We also saw some of the early bits of what we would now call Genai generative modeling, generating images, generating text. A lot of those core algorithmic pieces actually got figured out by the academic community. During my PhD years. Like there was a time I would just like wake up every morning and check the new papers on Arxiv and just be ready. It was like unwrapping presents on Christmas that like every day you know there's going to be some amazing new discovery, some amazing new application or algorithm somewhere in the world. What happened is in the last two years, everyone else in the world kind of came to the same realization, using AI to get new Christmas presents every day. But I think for those of us that have been in the field for a decade or more, we've sort of had that experience for a very long time.

Obviously I'm much older than Justin. I come to AI through a different angle, which is from physics, because my undergraduate background was physics. But physics is the kind of discipline that teaches you to think audacious questions and think about what is the remaining mystery of the world. Of course in physics it's atomic world, you know, universe and all that. But somehow that kind of training thinking got me into the audacious question that really captured my own imagination, which is intelligence. So I did my PhD in AI and Computational Neuroscience at Caltech. So Justin and I actually didn't overlap, but we share the same alma mater at Caltech.

And the same advisor at Caltech.

Yes, same advisor, your undergraduate advisor, my PhD advisor, Pietro Perona. And my PhD time, which is similar to your PhD time, was when AI was still in the winter in the public eye, but it was not in the winter in my eye because it's that pre spring hibernation, there's so much life. Machine learning, statistical modeling was really gaining power. I think I was one of the native generation in machine learning and AI. Whereas I look at just this generation is the native deep learning generation. So machine learning was the precursor of deep learning. And we were experimenting with all kinds of models. But one thing came out at the end of my PhD and the beginning of my assistant professor time. There was a overlooked element of AI that is mathematically important to drive generalization. But the whole field was not thinking that way, and it was data, because we were thinking about the intricacy of Bayesian models or whatever kernel methods and all that. But what was fundamental, that my students in my lab realized probably earlier than most people, is that if you let data drive models, you can unleash the kind of power that we haven't seen before. And that was really the reason we went on a pretty crazy bet on imagenet, which is, you know what? Just forget about any scale we're seeing now, which is thousands of data points at that point. NLP community has their own data sets. I remember UC Irvine Dataset or some data set in nlp. It was small. Computer vision community has their data sets, but all in the order of thousands or tens of thousands were like, we need to drive it to Internet scale. And luckily it was also the coming of age of Internet. So we were riding that wave. And that's when I came to Stanford.

So these epochs are what we often talk about. Like imagenet is clearly the epoch that created or at least maybe made popular and viable computer vision in the Genai wave. We talk about two kind of core unlocks. One is like the Transformers paper, which is attention. We talked about stable diffusion. Is that a fair way to think about this? Which is like there's these two algorithmic unlocks that came from academia or Google and like that's where everything comes from. Or has it been more deliberate or have there been other kind of big unlocks that kind of brought us here that we don't talk as much about?

Yeah, I think the big unlock is compute. Like, I know the story of AI is often the story of compute, but even no matter how much people talk about it, I think people underestimate it. Right. And the amount of. The amount of growth that we've seen in computational power over the last decade is astounding. The first paper that's really credited with the breakthrough moment in computer vision for deep learning was Alexnet, which was a 2012 paper where a deep neural network did really well on the imagenet challenge and just blew away all the other algorithms that fei. FEI had been working on. The types of algorithms that they had been working on more in grad school that ALEXNET was a 60 million parameter deep neural network and it was trained for 6 days on 2 GTX 580s which was the top consumer card at the time, which came out in 2010. So I was looking at some numbers last night just to put these in perspective. The newest, latest and greatest from Nvidia is the GB200. Do either of you want to guess how much raw compute factor we have between the GTX 580 and the GB 200?

Shoot. No. What?

Go for it.

It's. It's in the thousands. So I ran the numbers last night. Like that two week range, that two week training run, that of six days on two GTX 580s. If you scale it comes out to just under five minutes on a single GB. On a single GB 200.

Justin is making a really good point. The 2012 Alexnet paper on Imagenet challenge is literally a very classic model and that is the convolutional neural network model and that was published in 1980s, the first paper I remember as a graduate student learning that. And it more or less also has six, seven layers. Practically the only difference between Alexnet and the Convnet. What's the difference is the GPUs, the two GPUs, and the deluge of data.

Yeah, well, so that's where I was going to go. Which is so I think most people now are familiar with quote, the bitter lesson. And the bitter lesson says is if you make an algorithm, don't be cute, just make sure you can take advantage of available compute because the available compute will show up, right? And so you just need to wise. On the other hand there's another narrative which seems to me to be just as credible which is like it's actually new data sources that unlock deep learning, right? Like imagenet is a great example but like a lot of people like self attention is great from Transformers. But they'll also say this is a way you can exploit human labeling of data because like it's the humans that put the structure in the sentences. And if you look at clip, let's say, well like we're using the Internet to like actually like have humans use the alt tag to label images, right? And so like that's a story of data, that's not a story of compute. And so is it just. Is the answer just both or is like one more than the other or.

I think it's both but you're hitting on another really good point. So I think there's actually two epochs that to Me feel quite distinct in the algorithmics here. So like the imagenet era is actually the era of supervised learning. So in the era of supervised learning you have a lot of data, but you don't know how to use data on its own. Like the expectation of ImageNet and other data sets of that time period was that we're going to get a lot of images, but we need people to label everyone and all of the training data that we're going to train on, like a person, a human labeler has looked at everyone and said something about that image and the big algorithmic unlocks. We know how to train on things that don't require human labeled data.

As the naive person in the room that doesn't have an AI background, it seems to me if you're training on human data like the humans have labeled it, it's just not explicit.

I knew you were going to say that, Martin. I knew that. Yes, philosophically that's a really important question. But that actually is more true in language than pixels.

Fair enough. Yeah, right? Yeah, yeah. But I do think it's an important distinction because clip really is human labeled. I think attention is. Humans have figured out the relationships of things and then you learn them. So it is human labeled. Just more implicit than explicit.

Yeah, it's still human labeled. The distinction is that for this supervised learning era, our learning tasks were much more constrained. So you would have to come up with this ontology of concepts that we want to discover. Right. If you're doing in imagenet. Fei Fei and your students at the time spent a lot of time thinking about which thousand categories should be in the ImageNet challenge. Other data sets of that time, like the Cocoa dataset for object detection, they thought really hard about which 80 categories we put in there.

So let's, let's walk to Gen AI. So, so when I was doing my, my PhD before that you came. So I took machine learning from entering and then I took like Bayesian something very complicated from Daphne Caller and it was very complicated for me. A lot of that was just predictive modeling.

Yeah.

And then like I remember the whole kind of vision stuff that you unlocked, but then the generative stuff has shown up like I would say in the last four years, which is to me very different. Like you're not identifying objects, you're not, you know, predicting something, you're generating something and so maybe kind of walk through like the key unlocks that got us there and then why it's different and if we should think about it differently and is it part of A continuum, is it not?

It is so interesting. Even during my graduate time, generative model was there. We wanted to do generation. We nobody remembers, even with letters and numbers. We were trying to do some, you know, Geoff Hinton has had generated papers. We were thinking about how to generate. And in fact, if you think from a probability distribution point of view, you can mathematically generate. It's just nothing we generate would ever impress anybody. So this concept of generation, mathematically, theoretically is there, but nothing worked. So then I do want to call out Justin's PhD. Justin was saying that he got enamored by deep learning, so he came to my lab. PhD. His entire PhD is a story, almost a mini story of the trajectory of the field. He started his first project in data. I forced him to. He didn't like it.

In retrospect, I learned a lot of really useful things.

I'm glad you say that now. So we moved Justin to deep learning. And the core problem there was taking images and generating words.

Well, actually it was even about, I think there were three discrete phases here on this trajectory. So the first one was actually matching images and words. And we have an image, we have words, and can we say how much they align? So actually my first paper, both of my PhD and ever, my first academic publication ever, was the image retrieval with scene graphs.

And then we went into the taking pixels, generating words. And Justin and Andre really worked on that. But that still a very, very lossy way of generating and getting information out of the pixel world. And then in the middle, Justin went off and did a very famous piece of work. And it was the first time that someone made it real time, right?

Yeah, yeah. So the story there is, there was this paper that came out in 2015, a neural algorithm of artistic style led by Leon Gaddis. And it was like the paper came out and they showed these real world photographs that they had converted into Van Gogh style. And like, we are kind of used to seeing things like this in 2024, but this was in 2015. So this paper just popped up on Arxiv one day and it like blew my mind. Like, I just got this like Gen AI brainworm, like in my brain in like 2015. And it like did something to me. And I thought, like, oh my God, I need to understand this algorithm. I need to play with it. I need to make my own images into Van Gogh. So then I like read the paper and then over a long weekend, I reimplemented the thing and got it to work. It was a very, actually very simple algorithm. So like my implementation was like 300 lines of Lua because at the time it was pre, it was lua, this was pre Pytorch. So we were using LUA Torch, but it was like very simple algorithm, but it was slow, right? So it was an optimization based thing. Every image you want to generate, you need to run this optimization loop. Run this gradient descent loop for every image that you generate. The images were beautiful, but I just like wanted it to be faster and.

Justin just did it. And it was actually, I think your first taste of an academic work having an industry impact.

A bunch of people had seen this artistic style transfer stuff at the time and me and a couple others at the same time came up with different ways to speed this up. But mine was the one that got a lot of traction, right?

So I was very proud of Justin. But there's one more thing I was very proud of Justin to connect to Gen AI is that before the world understand Genai, Justin's last piece of work in PhD which I knew about it because I was forcing you to do it.

That one was fun.

Was actually inputting language and getting the whole picture out. It's one of the first Gen AI work. It's using gan, which was so hard to use. But the problem is that we are not ready to use a natural piece of language. So Justin, you heard he worked on scene graph. So we have to implement, put a scene graph language structure. So you know, the sheep, the grass, the sky in a graph way. It literally was one of our photos, right? And then he and another.