Growing up to become a world-class AI expert

Bandwidth for Changelog is provided by Fastly. Learn more at fastly. Com we move fast and fix things here at Changelog because of Rollbar. Check them out@rollbar.com and we're hosted on Linode Cloud Servers. Head to linode.com changelog this episode is brought to you by DigitalOcean, the simplest cloud platform out there and we're excited to share. They now offer dedicated virtual droplets and unlike standard droplets which use shared virtual CPU threads, their two performance plans, General Purpose and C CPU optimized, they have dedicated virtual CPU threads. This translates to higher performance and increased consistency during CPU intensive processes. So if you have build boxes, CI CD video encoding, machine learning, ad serving, game servers, databases, batch processing, data mining, application servers or active front end web servers that need to be full duty CPU all day every day, then check out Digital Ocean's dedicated virtual CPU Droplets. Pricing is very Competitive starting at 40 bucks a month. Learn more and get started for free with a $100 credit at DigitalOcean.com changelog again DigitalOcean.com changelog. Welcome to Practical AI, a weekly podcast about making artificial intelligence practical, productive and accessible to everyone. This is where conversations around AI, machine learning and data SC happen. Join the community and slack with us around various topics of the show@changelog.com community. Follow us on Twitter. We're at Practical AIFM and now onto the show.

Hello there and welcome to another episode of Practical AI, the podcast where we try to make AI practical, productive and accessible to everyone. This is Chris Benson, your co host and today I am at the Nvidia GPU Technology Conference in Silicon Valley. It is Tuesday, March 19th. I happen to be operating solo today. Daniel was not able to make it out here. I have a pretty amazing guest today to talk to. With me is Anima Anand Kumar. Did I get your name okay?

Oh yeah, you did an amazing job.

Okay. Anyone who listens to me on this regularly knows that I screw up names all the time. So I'm glad that you are the Director of Machine Learning Research at Nvidia and you're also the Brin professor at Caltech, is that correct?

That's right. Yeah. I wear both the hats and I think it really gives me an opportunity to bridge industry and academia in many interesting ways.

Absolutely. So in a few minutes you were giving two separate talks here at gtc and I would like to kind of delve into both those. But at first I would really like to talk to you and I know that you talk to people often. You've been in front of audiences quite a bit about AI and the role you play there. But I actually want to go back to the beginning and kind of find out how you got into this, what that journey has been like and talk a little bit about some of the challenges that you may have faced along the way. And I know that you have spoken to some of those in the past. So can you tell us kind of how you got into this at all?

Yeah, certainly. I've had a pretty amazing childhood in terms of the people around me were always encouraging me in my passion for maths and sciences, but also for dancing and arts and everything. So there was a lot of. My mom is an engineer. My dad has a small scale industry that manufactures all kinds of components, builds machines. So I would go there as a kid, both my mom and dad would take me there and to see how those machines operated and how it just seemed so magical that you could automatically build these components and they would be in all kinds of shapes. You had these machines going at this really high speed. And so that was fascinating. And I would ask my parents how is this possible? And they would be like, oh, it's all math. So I had this very friendly. Math was a friendly thing to me from early childhood.

How old were you about that time period? Like what do you think that, that what age were you at at that point?

I think when my earliest memories are either me like kind of, you know, solving some puzzles or some toys. Right. And going to industry. Like my parents always like, you know, took me when I was maybe three or four or so.

Wow, that is young.

Yeah. That is so cool.

So you had this fascination at that really young age.

Some of my earliest memories are like me trying to solve math problem and wondering there is this addition and suddenly we're subtraction. Why is there these two symbols and how are the two related? So I somehow remember being very fascinated with it. And my grandfather is a math teacher and so he was teaching me.

You had a secret weapon in your family?

That's right. And my grandma would give me all these puzzles and. And she tells me that I apparently had memorized the calendar for a 10 year time period. And they would quiz me on what day is 12th of August for instance.

You gotta be kidding me. How old were you for that?

Apparently three.

Oh my gosh. Okay.

This is what my grandma tells me. I can't do it anymore, so don't quao.

Okay, well you're setting a high bar for the three year olds out there. I know that. I'm sure we have Three year olds listening to the Practical AI podcast. But in case for that one or two out there, you've just set the bar very high. My daughter, who's about to turn seven, has no excuses at this point.

No, no, I think every child's development is different. Right. But at the same time, there is so much fascination in all aspects too, of development. For instance, I love dancing. I started dancing when I was three as well. So I wasn't put into a box. I wasn't told, oh, this is math. You're only good at that. Do.

Do you think doing dance and other activities that have nothing to do with technology, do you think that made a difference and that having that diversity of experiences made a difference in how you progressed?

Oh, certainly. That's what makes us human, right? The artistic side of us and the humanities and the liberal education is very much a part of our growth as human beings, as a society. And to me, I would also argue it's highly mathematical. Dancing is all about rhythm, right? And you count 1, 2, 3, 4, 4 steps, and then you progress to saying, it's no longer these discrete steps, it's a more continuum of movement. There is flow, but there's also sudden peaks and sudden changes to it that may, you know. So as you progress in dancing, to me it's highly mathematical. It's like a wave.

I have this vision of you at 4 years old going through a dance and then dissecting it mathematically across the board there.

No. But I think the earliest memory that I have that very much relates to what I'm doing with AI today. I remember suddenly stopping and questioning myself. Huh. I feel so differently right now. I feel like there's something that's me. Like I had never. This kind of what they call self actualization. And again, no one had told me about it. Right. Like I had no idea. But I was just playing. And I remember this moment very vividly where I felt, oh, there's something known as me, even though maybe that's not the language I used. And you know, and for us taking baby steps in AI, right. That's so fascinating. How do kids learn so quickly, learn so intuitively, come up with all these developments in their personality?

Yes. There's so many things that we kind of take for granted as humans. And ironically, those of us now in the AI space look back with kind of incredulous in terms of saying, wow, you don't think about that. But there's so much there that we're trying to discover now in the field of AI. So that we can do amazing things.

Yeah. And I would say we are still quite far away from getting anywhere close to that. Right. But that's what will keep me employed for a lifetime.

There you go. Job security right there.

But, yeah. So going from those very early childhood memories and, you know, like, getting fascinated about math and all the way through, like, high school, Right. I was just, you know, getting into, like, math. And all the time it was, oh, let's, you know, what's the next thing, you know, like, I always remember, like, oh, there were, you know, number systems and then there was suddenly complex number. And this is imaginary. You can even make up things in math. Right. Before that, you think math is just concrete and you can't make up things in math. But imaginary number is something we make up, but it's so powerful in the way we use it.

So as a kid, the tangible aspects of math, looking at machines working and realizing there was math involved and having your parents and grandparents influence that. But then it sounds like as you grew in mathematics and learned more and more, the abstract nature really appealed to you. It sounds like.

Exactly right. So there is structure in the way we build math. We first start with a specific goal of counting numbers. But we said, oh, this is not enough. Right. We need to expand our number systems. We then had rational. And then we said, we also need irrational numbers and real and then complex. So it tends to get increasingly more abstract. But then it has all these applications that wouldn't be possible without it. And that's what fascinated me.

Me.

It's almost like the math, even though I cannot directly relate to it, I can't visualize it, I can still use it in many interesting ways. And that's when I thought, abstract thinking is important. I shouldn't always try to get it into something physically that I can relate to. I don't need that.

So as you're growing up through this process and becoming more and more proficient in mathematics through school and through home life and such was were you. Did you retain that passion for the engineering that you had as a. As a child? Did that drive, or did you get more into just the pure mathematical passion that you were discovering?

No, that's a good point. So my mom and dad would take me to different industry meetings. So we'd go to, like, these big trade shows, Right. Like gtc, but for manufacturing. And so I'd be seeing all these fascinating. The latest machines, the computerized, numerically controlled machines. Back then, if you remember, it was the green screen and it only had some simple programs. You could write in terms of the capabilities. It was like, ok, which axis do you go? What would be the set of movements of this turret that would go, and which tooling do you use? But to me, oh wow, you can program and now you can change material. And this is done in such an automated way. And that was my introduction to programming, which is very non orthodox way of thinking of programming.

And about how old would you have been as you really started expressing yourself in programming?

I think this, I would have been more around eight or so.

Yeah. So you're still quite young at that point in time. Okay. And so I mean by that time did you pretty well know that you were, this was the path that you were on as you grew up or were you still kind of finding yourself?

So I was still dancing, I always loved dance, I should say. Much of my family is engineers and mathematicians, although my uncle is a biotechnologist, so I would hear from him advances in biology. So no one really put me in any one path. For me I was generally fascinated with science and technology, math and engineering. But. But at that point I was looking at documentaries, for instance, about NASA and space. And that's how Caltech actually I remember thinking like, oh, that's such a fascinating place. So for me to kind of go from there to actually be at Caltech and be at Nvidia, that's making these cutting edge technology to enable AI is.

I guess. So you got to Caltech as an undergraduate then?

No, I didn't.

So how did you make that shift? Is your. As you're coming through your teenage years and it's time to pick universities, where did you go and what caused that decision making process?

Yeah, yeah, so I was in high school and you know, at that point, so I was back in India, right. So. And you know, I was looking at the local colleges. I think at that point I was still not ready to come halfway around the world. And you know, these Indian Institute of Science Technology, sorry or the IITs had this really difficult entrance exam to enter them because they're highly selective. It's like, you know, I forget the exact number. It's some huge number, maybe 500,000 students take that and you had to be in the top 500 or so or 600 at least when I was doing it to be like in computer science or electrical engineering or some major that you would like to pick. And you know, so I remember hearing about this exam in high school, right. And my first reaction was why is that so difficult? What makes it difficult?

Right.

I mean why? Oh, it's like, oh, it's because it's very. Somebody would tell me, oh, because it has difficult math. I'm like, really? What is difficult math?

You know, just in my own experience, I think that is a fairly unusual reaction to these tests. Most of the, most of us just go, oh my God, I have a hard test I gotta go figure out and study for and everything. So it's interesting you're already analyzing the test itself in terms of whether the difficulty and how to apply yourself to it.

Yeah, but also I was like, oh, I really want to learn something where I would be challenged to be honest. Because maybe because I had a lot of help from my family and I was reading books beyond my schooling requirements. So I felt honestly like, oh, I need to do more. That would really help me get to things I would see in the real world, like these machines that work. Or I'd look at these documentaries about the space. I was having this huge gap about the math I'm doing in school and how is this possible today?

So I guess which school did you start into your undergraduate work and did you select mathematics as your major?

So this was iit, so they only had engineering as most of their majors. Right. So I did take electrical engineering. That to me was a nice sweet spot of where I would be using a lot of math. Like I would be analyzing signals, I would still be connected to the hardware.

You're going back to that three year old seeing the machine working right there.

Yeah, and I would still be programming, I would be doing. And I was a minor in computer science. So I'd have all these different elements put together. And so that's where this entrance exam I felt was an opportunity for me to really go into. When we look at physics, what are the basic principles? How not only know about the laws of physics, but how do I apply to different systems? Because what made this test challenging was at least at that point those were near to Olympiad level questions. And why are those questions challenging? That's because they're not the usual ones you would solve during your assignment or during your usual schooling.

So did you dive heavily into physics at your university schooling at that point? Had you been exposed to it prior to university at all or was this kind of a new area that you were.

I mean, this was the entrance exam to get into the university.

Oh, to get into your undergraduate. I misunderstood that, I apologize.

It was physics, chemistry and mathematics. So all these three, you had to dive deep into it. And so when I was faced with these difficult questions that's when I had to really go back and understand the principle. Right. If I had misunderstood it, I could not apply it to solve the problem.

As you're approaching university, that is a fantastic story, even up to that point. It's truly inspiring and I hope people will share that with their daughters like I will. As you got into university, how did you evolve at that point? You know, you're in it now, you're in your schooling. At some point I know you would have come to the United States and ended up at Caltech. Tell me how that process happened.

Yeah, so once I was at iit, you know, it's just surrounded with a lot of amazing people. Right. Like the faculty, the students, you know, they've all come through, all the other students have come through the difficult test as well. Right. But I found so many amazing friends that I still keep in touch. And what fascinated there was, we all shared this common vision to ask how technology could help society, could help us have a better future. So I remember being very involved in this tech fest called Shastra, that's the annual festival that at IIT Madras where I went to school. And we were asking, okay, what would it, you know, how do we bring in more students here? What kind of contests would be good? Where do we get the best speakers? Right. What are the best researchers today? How do we get them here? And so to me, that was one of the highlights on seeing this community come together and ask, how do we showcase today's technology? Where is that going in future? And so that was that kind of a community togetherness is what I experienced at iit and part of that was also asking. At that point I was doing research both in my undergrad school, also in Indian Institute of Science, which is in Bangalore. And that's when, oh my God, I have so much freedom to think and to ask what I could be doing innovatively, what's missing today and what I could.

You can make a difference.

Yeah, and create something myself. And I can make a difference. And so during my junior year, I did this undergraduates research experience, an equivalent of what we call surf year and that summer undergraduate research fellowship. Right. So there's a similar program back in India. And that's when I decided I just needed to do a PhD. I had to, you know, dedicate that time to go deep into something. Right. And you know, really being, you know, and who knows, that was the adventure, Right. Because what I could do there, I didn't know. But the possibility seemed so exciting.

Did you feel, did you feel that you Wanted to go. I mean, what Caltech being the world class school it is, did that, did that draw you at that point in time? Was that the natural next progression in your own growth?

Yeah, so I applied to quite a few schools, so I ended up at Cornell University. So Caltech is where, you know, I think I did apply there, but Caltech is a smaller school. Right. So we are highly selective. But I'm glad I'm there now.

Your idea? Yeah, your idea of, you know, schools. They're all world class schools that you're naming here. So it's good to have that kind of a problem to figure out which of those schools you're going to go to. So you got to Cornell first then?

Yes, that's right.

And so tell us from there.

Yeah, so, you know, when I arrived at Cornell, it's just this beautiful place, right? It is in the middle of nowhere, but it's nice. It's a lot of natural surroundings. It's called Ithaca. It's gorgeous.

Good for thinking, right?

Just birds and I mean, I did arrive in early fall, which. But yeah, I mean, it's like to me, a place where, you know, like there's a lot of close knit community. Like my advisor Langtong just was just such a wonderful human being. He would like, you know, give me that freedom to think and grow and say. And he wouldn't say, oh no, this doesn't work. He's never told me that. Right. He would be like, go figure it out. And then he would be like, oh, you know, like if it didn't work, he would be like, oh, why not? Right? Like, so we'd have this very open conversation and very. And so honestly, he really motivated me to be a professor because when I saw him, like how he deals with students and the kind of lifestyle he has in terms of really being able to think openly.

So you were in your PhD program at this point?

That's right. Okay, that's right. At Cornell. And you know, so that's when I was going to conferences, I was meeting other researchers and to share these ideas openly. That's what, you know, I decided at that point when I was graduating that I wanted to be in academia. Because back then AI really hadn't taken off the way it is today. There were very few, almost no industrial research in machine learning.

And AI as we talk about that, because we're now turning explicitly to AI. What was it? At what point in that process? Because we were talking about the math and the engineering for a while. At what point did you realize AI was the area that you wanted to focus on. When did that happen in that process?

Yeah, so in fact it happened during my early PhD itself. So Milan Tong is an expert in wireless networks and wireless communications. And back then we were thinking about all these challenges of how to communicate over wireless channels. But the question is also, what do you communicate? And during that point the question was, what if you just didn't blindly send all the data? If you could make inferences, if you could make decisions on what's really relevant, you could really reduce how much you communicate. So I came into it from that viewpoint of how to reduce requirements in communications, which is not a traditional way of getting into AI. But I've never taken a traditional path in many ways.

Clearly.

That'S where it led me to asking, okay, now we need to then do some inferences. Right. I need to now think about machine learning techniques. We called it statistical inference and estimation and more, this traditional signal processing community. But the tools are. That foundation is the same.

Yes.

Right. And so that was my first project on how do we do distributed learning. Like where there are a number of sensors and they don't need to communicate all the raw data they collect, but they only communicate after making some inferences. What's truly relevant to your problem? Let's say if it's measuring the temperature and there's something anomalous, maybe you only want to communicate that anomalous behavior and not just keep communicating all the time.

Did you get your Ph.D. from Cornell? And then specifically what was the, the Ph.D. in by the time you finish this long process? Actually, finishing is probably the wrong word, but as you got to this point in the journey, what was your PhD in?

Yeah, so my thesis title, if I recall it, it's been a decade now, is Distributed Statistical Inference. So my first project led to many other projects of how do you route this in an energy efficient way in a network? And how do you model correlations across different sensors, sensors in useful ways. Right. And that's when it got into this large scale machine learning, because we said, oh, there should be like dependencies, like what do these correlations mean? There should be an underlying, what we call a graphical model, a probabilistic model that connects all these measurements. And now that's when I started getting fascinated into all the literature of machine learning that's kind of considered core machine learning. And you know, I met Alan Wilsky, who's a professor at MIT now, he's retired, at a conference and my advisor said, oh, just go visit him. Right. So I Took the. My whole last year was actually spent at MIT because he's the expert in graphical models. And I wanted to dive deep into that. And that's when I truly started publishing in machine learning conferences, you know, at the end of my PhD and when I had a faculty. So I kind of made a switch in terms of which communities I was publishing, although the core math was very much connected and related. So I didn't feel. It's a big switch, but a lot of people felt really surprised. What, you're starting your faculty career and you're suddenly going to these new conferences and back then, these main machine learning conferences back then was called nips, and now it's called newrips. And ICML were tiny and signal processing events were much bigger. And so many people were wondering why I made the switch. And for me, it's always been about where the core intellectual ideas are right and where there's a lot of potential. To me, I felt this machine learning revolution has to happen, it's just a matter of when. And so I'm happy I went into core machine learning before it all took off.

The Data Engineering podcast is a weekly deep dive on modern data management with the engineers and entrepreneurs who are shaping the industry. Go behind the scenes on the tools, techniques and difficulties of data engineering so you can learn and keep up with the knowledge to make you and your business successful.

Can you give a bit of an outline about the motivation for choosing Jupyter notebooks in particular as the core interface.

For your data teams?

Yeah, and actually, actually when I first joined Netflix, it was sort of tossed at me and I was definitely like, are we crazy? And the answer was like, we might be a little crazy.

Go to dataengineeringpodcast.com to listen, subscribe and share it with your friends and colleagues.

So let's. Now that you finished your PhD, let's turn towards. You're now out there, you have your PhD, your faculty, and you are doing work in the space. How did you. How did you arrive at Caltech? There's a couple of questions I'll ask you about when you get there. And then ultimately at Nvidia. And then after all that, I'm going to turn you toward your talks itself down the road, but kind of tell me, how did you. How did you move over to Caltech? What made that.

I'm still not at Caltech.

I'm sorry. Okay.

No, no, that's what makes this a fascinating story. Right?

There's still suspense, so there's still a ways to go.

Yeah.

I see I'm rushing your story. Terrible of me.

I actually started at UC Irvine, which is not far from Caltech. Right. So again, Southern California, like, you know, again, a beautiful place and, and a lot of really amazing.

If the listeners could see my face right now. I'm blushing for having done that. But anyway, keep going. Sorry about that.

No, but yeah, and as I said, I was getting into this new field in the sense of publishing. Right. Like these new conferences, new people and yeah. So there was a bit of me that was stressed, oh my God, I'm starting this faculty life and doing this. But they were just truly fascinating problems. I was looking at these probabilistic models and asking questions when can you learn these models at scale? And what do you mean there's a correlation between these models? Are there few variables that can summarize the effects? And that's when I got introduced into latent variables. The idea that we can't measure everything in the this world there's always going to be something hidden, but it's the hidden thing is what maybe we are really after. That's what we'd like to learn, but we can't directly measure it.

That seems based on your inquisitive nature, you know, talking from childhood up, I'm not at all surprised to know that that's what you were going after.

I'm always after the hidden things.

Next thing, what makes that work. So keep going.

Yeah, so I was looking into these, what we call latent variable estimation. Think about, you know, like understanding text. Right. In the last few years we made a lot of progress. But you want to ask like what do I really want to extract from this text? It's the meaning, it's like the topics that are discussed in the text. Right. But you know, as human beings know, even if that word is not even there, you can extract what the topic is. Right. And that level of like extraction is hard because we don't understand all the nuances of the language.

Now we're struggling with that. Now that's a big part of research in AI with a couple of interesting things that have happened in NLP news lately. But keep going.

Yeah, exactly. Right. So doing it in the fully generative way, like generate language and all its nuances is hard. And that's where I think simplicity, there's something to be said about that. So back then deep learning was not there. And with these models and with the compute we had, and even today actually, in fact that's the state of the art results when it comes to categorizing large documents. You have these lengthy, let's say, news articles or your reports, and they could be in the millions, and all you have is this raw text. Maybe you have a bit of metadata, but not much. So you can't go laboriously, even provide examples. You can't go annotate each word and say, oh, this is representative of topic. Let's say justice. This is representative of education. This is representative of geography. Or, you know, you can't go manually annotate each document of what it represents. Right. And then ultimately even each word, because you want representative words for each topic. And so this is called unsupervised learning, where we don't have these exams. Like, no one's telling you what a topic should look like and how you should categorize your document into multiple topics.

Even now is fully into the age of deep learning. That is still a huge push. That's where so much of the research is right now. So keep going.

Yeah, exactly. And so back then, the question was, what's the simplest thing you can do there? And that's always like I. Like I tell my students and I tell my colleagues and all the researchers I work with is first think about the simplest thing you can do and if it doesn't work, then go to more and more complex. Because one of the dangers, which I'll come to later with deep learning, is you may just overtly get it complicated and not understand what's going on.

You sound like an engineer when you say that. I can totally see that background there. Start with the simple.

Yeah, no, I have a lot of respect for the traditional engineering that makes aircraft fly today and all the transportation, structures, everything. There's a lot, I think, which will come to my latest part in the end about my talk. It's blending the old and the new together. So anyway, coming back to how to extract topics from documents and automatically categorize documents at scale in an unsupervised way, a simple intuition is, what if you look at frequency of words, let's say the word apple occurs a lot. If I just told you this, then you'd be like, huh, it could be about a fruit or it could be about a company. Right. So it's like not enough. Then you could say, I tell you that what if word apple and orange occurs in the document?

It changes the content. You suddenly have a little bit of understanding.

Yeah, you have more information. But orange is also a company. But maybe both of them may not occur that much together. And so then you could go to three Words you could say like apple, orange and banana. And then now it's like, oh, it is really a fruit. Something about fruits. Right. So the more what we call co occurrences. If you now look at how multiple words occur in the document together, then you can make better inferences of what the document is talking about. And note that no one is telling the algorithm. No one is writing down every triplet and saying, oh, if it's apple, orange, banana, call it fruits. I mean, we can't possibly do this in full scale. The size of English vocabulary is more than 100,000 and it's cubic in 100,000. So there is just no way we would ever be able to do this manually. Now the question is, where does math help us here? The intuition of. And this has kind of been there underlying almost all of machine learning. Right. Where is all the energy? Where is the strong signal? If I can extract that strong signal, then I can really do this without writing down all the rules. The the most basic algorithm, or I wouldn't call it basic because it was sophisticated for its time, is principal Component analysis or pca. That's where we say, oh, this data could have a lot of noise. I'm going to filter out the noise by looking at only the subspace in which majority of the signal lies in. And so it's the same principle that underlies a lot of what I did at that time. But it was now taking a turn more dimensions into tensors.

And can you identify for the audience what a tensor is?

So a tensor is an extension of a matrix. You can think of it as a multidimensional array. So just as matrix has rows and columns, now it has more dimensions in a tensor. But just as you multiply matrices, you can also now multiply and operate and tensors. And so there's a whole algebra you can build around it, which gives you gives a rich set of operations on which you can build algorithms.

Sure. So at this point, where are you working as we're. Because we kind of talked about the work. What institution are you at at this point?

Yeah, so I'm still at UC Irwine, although I took a break. This particular work when I first started doing tensors was in Microsoft Research and you know, Microsoft Research, like even now in the east Coast. This is headed by Jennifer Chase and again an amazing role model, both for women and men. And she wants to encourage open research. And I went there, I gave a talk, I gave a talk on one of my earlier topics that works on statistical physics. And her background is physics and she was so fascinated, I used it in machine learning. There was not a connection that many people made before. And so she was like, oh, come over, just hang out, talk to researchers, see what comes out. Right. So we had no agenda. And there was Shyam Kakaday and Daniel Chu and many other researchers around. And one day we started bouncing ideas about, okay, there is pca, right? This kind of algorithm on matrices now, you know, this topic modeling, which is now looking at trying to extract topics. Is that enough? And if not, why not? Why should we need more? And that's kind of how. I'm not an expert on tensors at that point. Nobody is, because we don't really study that in undergraduate. But this kind of, I think asking questions is where you suddenly lead into a whole new direction and whole new area that's completely unexpected.

And is that what happened to you?

Yes.

So what, what was the next step then?

Yeah. So once the tensors we realized, oh, there is a rich history. In fact, it was back in 1900s that Spearman asked questions about intelligence and tried to use tensors to solve them, making it very rough simplification of what he did. But there is a very interesting connection where he said, oh, let's probably. People have different sorts of intelligences, verbal intelligence and mathematical. Can I use this notion of tensors to try to separate the two? It's again, separating these signals and finding these different directions. And it's similar to separating topics in our documents. Right. There's this common underlying math.

I love how you just connected that back there.

Yeah. So there's math everywhere that connects things together. And that's what, once I got into that history and got into like Richard Feynman, Albert Einstein. Einstein, right. In quantum networks. And there's just very much core of that. And then signal processing, like for blind source separation, this was used. So it's looking at all this history and asking, okay, what's different now? What's different is we have a lot of data. Right. We can now scale up our computation and we can, you know, and we are now looking at, you know, even collecting different kinds of data, different modalities of data. Whereas in the earlier generations when tensors were used, there wasn't enough data. And so I felt like this is the right timing to really think about tensors.

It is. So I guess at this point, are you into deep learning full. Or is this still.

All this happened before deep learning took off? So there was like these few years where we were thinking about, oh, how can I now apply tensors to these different probabilistic models, more and more challenging dependencies. Like how do I learn communities of people? Like I look at friendship links. How do I know who's interested in what aspects? Right. Like in social media and sense.

So how did you make that leap into deep learning? At what point did you realize that was what you were going to be doing to carry your workforce?

Yeah. And as this was happening. Right. So the early deep learning results came about. I would say like by around 2012 and 13 there were results. And you know, so I was thinking about like analyzing optimization, like non convex optimization. Right. And you know, these deep networks have highly non convex optimization surfaces. But so much of my, even my theory and my experiments and the experience was telling we shouldn't be afraid of non convex optimization. In fact, the tensor methods that we use are also highly non convex. Right. And that was kind of in the beginning, people were shooting us down, oh, this is not convex. How do I know it works? And we're like, we're showing you the results. It's like, but still there's no proof this works. And I'm like, oh, there's a proof, but it's under some conditions. Right. And so it was. So I think that revolution was happening that convex is great, but so many things are not convex and we can't just try to force things to be convex when they are not. Right. We'd be limiting ourselves so much.

Understood. So for the audience here, if you could see her, she is so passionate about what she's talking about, she's waving her arms around. It must be a delight to take classes from you for your students. So keep going. We're both smiling a lot because it's. This is such a fast. And you've made my job so far as an interviewer so easy because you're so good at carrying it on. So tell us how you delved into it.

Thanks, Chris. And I love talking about it. And by the way, my most recent class, I will make all the videos online, so you're welcome to go check them out.

And we will put that into the show notes as they're available so that after you listen to the episode, you can go find and watch those.

And to give a brief intuition of convex versus non convex, think of the convex as this parable. Any pebble you slide from the top will only go to the bottom. It may oscillate a bit, but ultimately it'll settle all the way in the bottom. I mean, that's what essentially all these algorithms, optimization algorithms, are analyzing. But now it's non convex, it's many peaks and valleys like the natural landscapes. And so then you don't know where it's going. And that's what makes this challenging to analyze. And to me this is again like where between math and engineering. Right. I don't want to solve problems just because they're easy for the math to solve. Like for the math, I understand, I can express them. I don't want that to be the reason to solve a problem.

Understood.

And so when deep learning started taking off that, that mirrored my experience that what I'd seen with tensors, which is also both a nonlinear and both are non convex, but things work very well in practice. And that's when I was right from the beginning, I was, yes, this should work very well. But at the same time I think parts of it that I'm still hesitant on is it's highly black box, it requires a lot of data, but that's, those are the opportunities. So I was like, oh, there are, you know, this is great, like what results we've seen now. But there is so much more we need to solve there. And so one of the first things we did there was to ask, okay, now what does it mean to have, you know, what is, where do tensors and deep learning come together? Right? I mean that's natural because I understand one field very well, I know the other and what is the connection? And you know, and that's where we said, and I showed it in my talk on Monday was asking, you know, like if you look at the current neural network models, right, they are processing, essentially it's matrix operation, you're multiplying two matrices. Convolution is still a form of like linear algebra. And that's because this methodology has been developed for last, whatever 60 years or we have linear algebra libraries that have been highly well developed. That was the reason, one of the main reasons to build it as a foundation even for our deep learning.

So since you mentioned your Monday talk was the role of tensors in machine learning. Right. And so go ahead and share with us a little bit about what you did there because I think you're starting to go that way anyway.

Yes, exactly right. And one aspect of the role of tensors was what I described earlier with these probabilistic models and how do we extract useful latent variable modeling in this new one with deep learning we asked, ok, why should it be only linear layers and linear algebra? We can do this higher order ones. And so what does that add? And that's when we Found like, you can get very high rates of compression of those networks and still get good accuracies. And so the intuition is if your data is in many dimensions, you'd rather also process it in many dimensions.

That makes sense. Yes.

So why do we then limit all to two dimensions and just matrices when we go through the layers? And that's kind of the basic intuition. If you think about the image, its width and height, and if it's a colored image, it's now also RGB channels. And if it's going through these convolution layers, it's collecting more layers. It still has spatial structure. Then when it comes to fully connected layers, you destroy all that and you just do a matrix vector multiplication. So one of the first things we did was let's retain that 3D information throughout until the very end output and design operations that we understand very well from 10. And that showed a very high rate of compression. And that's when we said, oh, this is a natural things. Things should come together, because a deep network is nothing but a tensor operation. It's a big tensor in the end.

Okay, you heard it here, folks. That was a great explanation for that.

Yeah. And, you know, so at that point, you know, I'm still not a 10vd, I'm still not at Caltech, but I'm getting closer now. So I was at my point of sabbatical, so I got my tenure at UC Irvine. And that's when, as the field is taking off, there's so much happening in industry and AI is really getting built into practical applications. It's being deployed in the real world and in industry, there's so much activity. And so I joined Amazon Web Services as a principal scientist in. In the AI team. So it was almost from the beginning when the AI services were getting launched into aws. And that was again, an amazing time because it's like, what would an AI service look like? And we are also doing all this in a very short amount of time because there's a lot of demand for AI services from the public.

I am sitting here, even though you haven't even gotten to Nvidia and Caltech yet. I'm just amazed at how much of the pioneering work that you were doing in space. You really stand out in that way.

Thank you, Chris. And I think I've had amazing mentors and amazing support as well from the communities to enable that. And at Amazon, that's what I found. I learned a lot in terms of how do we think about the customer, what are the needs, how do we have the short timeline to the product but still make the customer happy? What would the requirements be? And like PR faq, which is the press release with the FAQ is how we first think about the product. Right. And so that's to me so non traditional. But again, you know, I like that. So, yeah, so that was again an amazing and a very busy time there. You know, that's when I launched the topic detection algorithm that I described earlier to categorize documents. It's running today in the AWS cloud, really in the comprehensive service. So we are going from this basic theory all the way to something that's.

Working to applying it in a very real world scenario that is touching a lot of people.

Yes. I was also managing an engineering team and looking into all the processes of the Sprint and how do we ensure good software engineering process, which was new to me because I was in academia before. Right. So it was a big learning process and that's where there are a lot of great people who helped me learn new things. And yeah, so seeing both the engineering side of it from a big production viewpoint and all the pain points. And this is where the SageMaker machine learning platform that got also discussed in the keynote yesterday when Jensen Huang invited Matt Garman from iws. Right. I was part of the launch, obviously the speaker.

So I now know.

Yeah, so I was very much involved right from the beginning to the launch. And it's. Yeah, so all that was great learning lessons. And that's when at that point I got an offer from Caltech, when there's so much happening in Amazon and in industry in general. And that's when I asked, okay, what do I do? Because I'm in a great place right now, but I do think for the longer term research and especially like going to the fundamental sciences. Right. That's where Caltech in particular has such a strong strength. And for me, like machine learning and artificial intelligence for me would make a true impact. I mean, not to downplay the other ones, the others are important to, to me, a holy grail would be what basic scientific discoveries can we make with that, if we can enable that, that can just have such transformative effects.

So you're at Caltech now and I'm going to throw a little bit of a wildcard in. Before we started recording, we were having our kind of introductory conversation and we were talking about some of the challenges that women face in the field and our mutual desire to remedy that. And I know that we were talking about the fact that there was the controversy. I know Anyone who's been in the space for a while will remember that there was a conference that used to be called nips, and there were connotations that may not be appropriate to that. And it was rebranded as Neurips, if I'm pronouncing that. Right. And so that originated out of Caltech, did it not?

Yeah. So the conference actually started at Caltech and it's very much, if you think about. Let's talk about the name in a bit. But the purpose of that conference was Neural Information processing.

Yes.

So it's highly multidisciplinary and this was back in 87. So it's truly visionary to think about. Let's bring neuroscience information. That's like information theory and processing, which is computation.

Yeah. It is still one of the top conferences, if maybe even the top one, depending in the world, in the area. And so I totally get that.

And actually the origin to that conference goes back to a course that Richard Feynman, Carver Mead and John Hopfield taught together in 81, before I was born. I'll send you a link. It's just so fascinating. I mean, can you think of these three luminaries coming together and saying we need to bring all these fields together? And ultimately that kind of resulted in new option we have for PhDs and a new division, in fact, that's called the Computational Neuroscience or CNS at Caltech. So we admit students today to that discipline and it's highly multidisciplinary. And. And so all that led to the formation of this conference back then called nips.

Understood.

But yeah, back then was a small event. People went, they liked to ski and also discuss topics. Right. So at that point, I mean, the name had unfortunate connotations, but no one kind of wasn't explicitly there. And so as long as. Because to me, even the beginning years of. When I started attending the conference in 2010, before deep learning took off and the field just expanded, I never thought, oh, funny name. But I didn't do much about it and I never got bothered or harassed by it. But once the field started growing, that's.

When it became just thinking that when it's small enough in a very small community, people may not be thinking that direction, but after it grows to what it had become and you have those negative connotations, maybe just as the conference had matured, maybe it was time for some of the branding around it to mature as well.

That's right. And, you know, now we have to have higher standards because all eyes are upon us. Right.

Absolutely.

And now it's in the public, and they're asking, oh, what is this name? And worse than that, you'd have, like, these few bad actors used the name to gain notoriety. And because any news is news, right? Any publicity is good because everybody's competing for talent, competing for attention. And so you'd have, like, these T shirts, like, with things like, oh, you know, my nips are np hard. And all these, like, kind of, you know, like. And then, I mean, honestly, like, I've been in these house parties that we'd have back then when the conference was small. Right. You'd think, you know, they were wild, but I felt so safe and so included in them. And suddenly these corporate parties are where I experienced really toxic environment. Right. Like, there was this infamous party where, you know, barely clothed women were brought in just for the purpose of, like, you know, like entertainment to an almost exclusively group of men.

Yeah. So, you know, just as an aside, when I hear things like that, it's something that I just find revolting. Having grown up in a family of strong, technical women, it wasn't until I was an adult that I realized that there were challenges for women to face because I had such amazing women in our family that were. That were every. You know, it never occurred that there could be inequality. So when I hear about bad behaviors like that, perpetuating those kinds of stereotypes and other things, I find it disgusting. I know, though Daniel's not with me today. I know he does as well. And so I am glad for people like you stepping up and doing what you can to correct the behavior that.

Needs it, because I really worry as the field is growing, that's when these kind of things can have a big effect. I mean, if you look into the history of programming, and there was a great New York Times article where in the beginning it was all the women who were programming.

I actually was sharing that on LinkedIn when it came out last week, I believe.

And they were literally driven out. Right. By the forces, by, like, how, you know, like. And I see the same happening today. And to me, that's a big concern. Right.

So call to action to everyone listening to this is we all together need to put things right in this way. We need to make it. We need to have a global AI community that is welcoming to all and is fair to all and doesn't accept such bad behaviors.

That's right. I mean, democratization of AI.

Absolutely.

In so many different ways. And that's the thing. I mean, honestly, the name didn't bother me before, but once these elements started appearing. It was a big issue. And then for the younger women, it was even a bigger issue than me, than it was for me. Right. So. And that's why I decided to speak out and so many others did. And it was just this huge collective community action. And I think it really helped us grow as a community better because it brought us together, it raised awareness. I mean, so many men said, oh, I had no idea it was this bad that you guys were having so many issues.

On behalf of the community, all the listeners that will be thinking the same thought, thank you very much for being proactive and taking care of a problem that was developing in that area. And let's not let such things happen in the future.

Yeah. And this is, to me, just the beginning. And there's so many efforts like AI for all that brings in underrepresented high school students or fast AI democratizing AI to everybody. Right. And there's like the human perspective of AI that was launched at Stanford just a few days ago that specifically saying ethics should be at the forefront. I think these are all things that we as researchers, we as people having a stake in AI in whichever different role should really think about.

So I guess I'm going to turn us a little bit and as we start to wind up, I'd like to get. You've talked a little bit about your first talk and role of tensor and machine learning, tensors in machine learning. But you also had a fascinating talk. And I know Daniel is going to be incredibly jealous that I was the one because Daniel came into AI from physics and I did not. But your talk is infusing physics into deep learning algorithms for stable landing of drones. And I was wondering if you'd share a few minutes of what that was about and maybe also talk a little bit about how you came to Nvidia as well.

Yeah, absolutely. And GTC is a place that really brings all this together.

It sure does. This is a wonderful place to meet people and to have fun and to just really enjoy this field.

Yeah. So this project was done at Caltech before I joined Nvidia. So here it's very much of the Celtic nature because we are saying these are. If you look at the author list, it's from many different areas. So there's people like me in machine learning, people in controls, in aerospace. So how we collectively ask to solve a question. Right. It wasn't like, oh, I'm going to use my one tool or this is what should work, but really trying to solve the problem here. And so talking to the Domain experts. Soonjo Chang is, is the main professor from aerospace who is an expert drones and all the drone flights. The main challenge even today if you look at the commercial drones, is they take a long time to land. And that's because the aerodynamics efforts is very hard to model. So they're being extremely conservative landing. And so the question was, can machine learning help us do this? Right. On the other hand, most of the machine learning, like deep reinforcement learning, is about learning from scratch. Right. You're doing it on games and there is a certain interesting aspect to it, but for most practical applications you wouldn't do it from scratch. And the question is, what is that? Right. Blend between existing knowledge, existing physics and what should you learn from data? Right. And I think this is broadly a question for us to figure out how much in most fields you have some existing knowledge, existing models, but they're not perfect. You also have data, but that's also not a lot. How do you bring the two together? It's not going to be purely deep learning based approach, at least not the standard way it's done in this work. We said, okay, let's now learn this unknown ground effect through deep learning. But we'll keep the existing controller and try to cancel this as a residual. But at the same time we want to guarantee stability. So it's not going to be a standard neural network. And we did try that in the beginning because we start with simple things and it just crashed the drone. Right.

You started with a simple thing, true to your engineering, didn't work and you did then the logical thing about.

Yeah, and it also shows like we shouldn't apply deep learning blindly.

Yes.

And then once we stabilize the network and we also have guarantee theoretically that but this will not crash, meaning it has Lyapunov stability from control theory perspective. That's the technical term. Then that also worked beautifully. So I have videos of it where you see very quick speed up and still a very fast but smooth landing with deep learning. And to me this is just even the first step. Ultimately what we want to ask is these autonomous drone flights that you can certify they are safe and that's still today given the recent incidents. We have so much to uncover there and I think machine learning broadly can help us a lot. And that also speaks to many of the projects now I'm doing at Nvidia which looks at using simulations. Like Nvidia has just such strong simulation tools like Physics Flex and all the graphics rendering and all that expertise. The the first part of the keynote, if you saw and how that blends in with the second part on machine learning and AI? Absolutely. How do we bring those two knowledge together and those frameworks together for robotics, for drones, for autonomous driving? These are all questions that very much connect the two.

And just as an aside for the audience, she's talking about the keynote yesterday, which was Monday afternoon. Jensen Huang, who is the CEO of Nvidia, gave his keynote for this conference and was covering the topics that she was just talking about. And as always, it was a very impressive, very impressive talk. It was one of those things that you don't want to miss.

It's a marathon, but it does so many. Each part you have to keep attention because it's so informative.

2 hour and 45 minute keynote, and yet it will actually keep you riveted all the way through.

Yeah, and that's, that's kind of my broad philosophy today when it comes to deep learning. What is the next things? I mean right now we've shown with lot of data labeled data and computation thanks to Nvidia GPUs.

There you go.

We are able to get good accuracies on deep learning in some tasks, but there's so many others where there's not going to be enough labeled data. But maybe you have unlabeled data and then you also no prior knowledge about what structures you should import and how it should behave, what constraints it should have. And that's what I think is the future in order to make it truly applicable in a diverse set of applications.

Well, that is fascinating. So just out of here, when did you join Nvidia? At what point in the process did.

You join the organization? It was roughly at the end of September.

Oh, so it's fairly recently. Okay, so it's funny, as you have taken us through your story, that seems like such a natural place for you to end up, given the work that you've been doing.

That's right. And to me it's at all the levels. If you go back to tensors, the basic primitives of these tensor operations, how do we speed them up? They'll come at the level of CUDA or even below that today. Most of them we've sped up for matrix operations. So we need to rethink on what would be a new way to have primitives. And there's going to be a talk tomorrow with the title Cu Tensor or Cu Tensor. That's going to be a new library for those tensor primitives. Now there's all through the layers of the stack. How do you truly take advantage of these new operations and parallelize them even more effectively. Like you can get better speed ups. Sure. And build better neural networks. Build better probabilistic models of all different kinds too.

And we can probably. I know that Nvidia publishes these, so by the time this comes out, it may very well be out. And we can put that in the show notes as well so that right after you listen to this, you can go look at that.

I mean, the library tomorrow after the talk will be available for the public.

Fantastic. Okay, well, I guess as this has been just an absolutely fascinating conversation, listeners may note that we went a little longer than we usually do, but I think everybody will agree this was well worth it. Just truly an inspiring story that you had to share and I hope that you will come back again sometime if people want to reach out to you and talk to you about this kind of work. Is there a preferred way that you have of people reaching out?

Sure. You know, I'm on different social media as well as my Caltech email is public. So you know, my Twitter handle is Anima Anant Kumar. That's a lot, but that's my first and last name. And my Caltech email is Animaaltech Edu.

And we'll have all of those in the show notes as well. So Anima, thank you so much for coming on the show. It's been a truly fascinating conversation as I'm sure everybody will agree and keep up the amazing work that you're doing. Looking forward to seeing what you do next.

Thanks a lot, Chris. And thanks for coming to gtc. Truly great to see you.

Thanks for having me.

All right, thank you for tuning in to this episode of Practically a High. If you enjoyed this show, do us a favor. Go on itunes, give us a rating. Go in your podcast app and favorite it. If you are on Twitter or social network, share a link with a friend. Whatever you gotta do, share the show with a friend. If you enjoyed it. And bandwidth for Changelog is provided by fastly. Learn more@fastly.com and we catch our errors before our users do here at Changelog because of rollbar. Check them out@robot.com changelog and we're hosted on Linode Cloud Servers@eleno.com changelog check them out. Support this show. This episode is hosted by Daniel Whitenack and Chris Benson. The music is by Breakmaster Cylinder and you can find more shows just like this@changelog.com when you go there, pop in your email address. Get our weekly email keeping you up to date with the news and podcasts for developers in your inbox every single week. Thanks for tuning in. We'll see you next week.