The Evolution of the NLP Landscape with Oren Etzioni

I think too much money is made over things that really ought to be free to benefit humanity. Our mission is AI for the common good. So that's why I say universities can't, but companies won't. Companies appropriately, right, have the mission, their for profit mission. And our mission has always been to bring to the fore and release systems, data sets, open source software that help to bring the field forward.

Hey, what's up, everyone? This is Sam. In today's interview, part of our guest host series, you'll hear a conversation led by longtime friend of the show, John Bohannon, director of science at Primer AI and former journalist for publications like Science Magazine, Wired and others. I'm sure you're going to enjoy this conversation, so let's jump in peace.

So for those listening in, I'm here with Oren Etzioni. He is the founder and CEO of the Allen Institute for Artificial Intelligence. And the acronym for that is AI. AI. So it's often shortened to AI2. Oren, how long have you been doing this?

Well, it all started for me about nine years ago. There wasn't an AI2, but the late Paul Allen's team reached out to me and said he wanted to create an Allen Institute for AI. And it was up to me to come in with, write a plan and make it happen. It sounded like an incredible challenge. I remember saying to people, people ask me, why are you doing this? You're fat, dumb and happy. As a professor at the University of Washington, tenured professor, you've got a good thing going. Why would you do something crazy like this? And I said, the sky's the limit. With Paul Allen's vision, with his resources, with his commitment to AI, we could do amazing things. And fast forward nine years, I feel like there's a lot still to do, but we have done some good things.

So I want to dig deep into what the Allen Institute has been up to, because it's kind of amazing how much you've accomplished in nine years. Just the impact and the unique way you've done it. But first, I want to give people listening in a sense of who you are. So I had my Orin moment. I think it was 2018, 2019 at latest. And you came and you gave a talk in San Francisco and you did a live demo, which is pretty unusual already, of natural language processing, nlp. And the room was packed. And I was one of many people at startups trying to make this stuff work and hoping to make it big. And this legend walks in and gives a presentation and you had it live on the AI2 website, a bunch of models doing stuff. We take that for granted today that you would have models just running live, powered by GPUs. To my knowledge, you were the first doing it. And what was absolutely mind blowing was you would do what we all do, which is you'll show some cherry picked examples. But then you did something that no one ever does, not then, not now. You then broke it in front of us. You would just say, yeah, looks impressive, right? But let me just show you what happens when you change the wording of the input just a little bit. And you said, this stuff barely works. And those words like, seared into my brain. And they did a lot of good because it made me very skeptical and pragmatic. So that when you actually got something to work, you didn't just say you didn't overhype it. And so that is you in a nutshell. Straight shooter. And do you feel like things have changed so dramatically over the past four years? A lot of people, you know, the hype is still here. But before we dig into that, just tell me like what you were feeling back in 2018, 2019. You were halfway through this grand project, you had laid the groundwork and you had contributed a ton to it. People don't realize that this whole obsession with Muppet names really began with Elmo. Right, right.

Well, John, thanks for remembering this and the demo and so on. I do feel like the principles that we have are what guides us through hype and turmoil and ups and downs, and that's the history of AI, but that's also the future of AI. So let me take the rich things you talk about and break them down. First of all, we have now phenomenal language models that do quite remarkable and certainly very impressive things. But our colleague Kate Metz of the New York Times says, never trust an AI demo. So you're absolutely right.

Yeah, right on.

If you don't get to kick the tires, if you don't get to ask the right questions, then you don't really know how well it does. And by the way, the best demonstration of that is actually in all of our living rooms with or are in our phones. Right. With Siri and Alexa and so on. You ask Alexa for something, you can get a phenomenal answer and then you change the wording slightly and it says, I don't understand that. So we have to be very careful with what we impute to these systems. And of course, there was a recent brouhaha about is this Google AI system sentient and so on. And of course it's not. So I do think it is very important, as you say, to be a straight shooter. And it's also true that one of my favorite sayings, because sometimes people tune in and they're like, wow, this is amazing. Another favorite saying of mine is our overnight success has been 30 years in the making. So if you look at a model like GPT3 or Lambda, or the latest of the bunch, they do have a long history that goes back to Bert, goes back to Elmo, which you kindly remembered, which was invented at AI2, won the best paper award of 2018, goes back to Word2VEC, which came to Google and actually goes all the way back to the 50s where a linguist, if I recall correctly, named Harris, said, you shall know a word by the company it keeps. Right. It's almost biblical in the way he phrased it.

Right, Yeah. I love that quote. And it explains most of NLP today.

Exactly, exactly. That's the underlying principle, that we can understand the meaning of words and from there the meaning of sentences and even. And beyond simply by looking at their context and looking at a large number of contexts. So in some sense, all of NLP today, if I were told, stand on one foot and explain all of NLP today, I would say, ye shall know the word by the company it keeps. But multiply that by a billion or 10 billion companies context, and you'll have NLP today.

But it's pretty revolutionary, right? Because there was a whole period where we thought grammar mattered, like encoding the rules of grammar. We thought that was really important.

I think that grammar does matter. But the remarkable thing about this technology, particularly when it's played out with large amounts of data, right? A billion, 10 billion sentences and large amount of CPU power, is that that data processing can recover the rules of grammar, nuances of semantics, et cetera. So it's not that grammar doesn't matter, is that this technology is remarkably good at at least approximating very, very well those rules that we have. And of course, by the way, we know that people only approximate those rules too. Right. We often say things and write things that are ungrammatical but kind of sound. Right. So it's really doing probably a better job modeling language than the rules of grammar.

Before you got into institution building, how would you describe yourself as a practitioner, scholar in the lens of today? You weren't an NLP guy, necessarily. You weren't. How would you describe yourself?

I've always been fascinated with two questions. The first one is one of the most fundamental intellectual questions across all of science and philosophy. What is the nature of intelligence? How do we build an intelligent machine? Over time, I've also added the ethical question, which maybe we'll have a chance to get into. Should we build an intelligent machine? And what would that mean for humanity? What would it mean for society? But that's one part, and the second part of me that's a lot more practical. The part that's founded startups and that delights in technologies, has asked, how can we use AI to build valuable technology in search, in software agents, in natural language processing.

So what was that conversation like, that early conversation with Paul Allen where you were making this pitch? Or was it he making this pitch? Did you come to it together? How did it come about? And for those who, there might be some in the audience who don't know. Paul Allen is the co founder of Microsoft, sadly passed away pretty recently, but an intellectual maverick.

He absolutely was an idea man. And that is the title of his autobiography, which I really recommend to people, it's really worth reading. And I think that his key role in Microsoft, particularly early on, was to have that vision of the PC revolution and what it would mean. It's hard to imagine that, right, where we've got a computer in every pocket and in our eyeglasses and 200 computers in our car. But back then, right, computers were far from ubiquitous and the idea that we'd have a computer on every desk was completely revolutionary. So Paul Allen was a visionary and I found talking to him incredibly inspiring. Right? And I'm not paid to say that the poor man has passed away, but, but he is and will always remain one of my absolute heroes. And not idle, but inspirations, mentors for his relentless focus on, you might call it the prize. And the prize not being a billion dollars or a trillion dollars, the prize being how do we understand intelligence? And of course he had a whole other institute, the Allen Institute of Brain Science, that is dedicated to understanding the brain.

It's like the wet lab side of this.

Exactly. Somebody once asked him, do you think that the neuroscience approach, the wet lab, is going to be successful in the long run or is it going to be the more software oriented approach that we use in AI? And he said, look, to me it's a horse race and I've got to bet on, on both horses.

So, so what was the race though? Did he want artificial general intelligence? Did he want to just crack the scientific mystery of what it is? Did he want to harness it? Like what did those pitch meetings look like?

Paul was fascinated and I Continue to be fascinated by two related questions. The first one is absolutely the most hairy, audacious big question you could ask, which is, what is the nature of intelligence and human level intelligence? No consolation prizes. The real thing. And so he was always asking us about that. He was always relentlessly looking to the future and saying, okay, we, what would it take to get there? How can I help you? Does this scale the second thing? And I think it comes from his fascination with human knowledge. His mother was a librarian, so he was fascinated with how do we collect human knowledge and how do we get a computer to understand it. Back in the 70s, I believe, early 70s, he said, look, it's one thing to take a book, collect all the words in the book and put them in an index and effectively what today we call a search engine. And it's quite another thing to understand the meaning of the book and answer the questions at the back of the book. Think of exercises at the end of each chapter in a textbook. And so even in the 70s, before a lot of this technology was around, he understood that meaning. Understanding the meaning of text, of knowledge was very, very tough for a computer.

I mean, have we even gotten closer though, or are we fooling ourselves? You know what I think about is I use semantic search all the time just at work. It's a great tool, it's really powerful, but it's so easily fooled. You sort of crack through the shell and you realize if this thing understands scare quotes, what it's reading, it's doing it in a very different way from me because I can just change a single word inconsequentially to me and it just falls apart. It clearly doesn't understand it the way I do. So are we chasing up the wrong tree when we say we're chasing text understanding, or is it all just performance based? We don't really care if it understands we care about getting jobs done. Find me documents that are about four legged animals that love to bark. If I didn't know the word dog, but I knew I could describe what I was after, we would love a computer system that would just find the right stuff even if it had no idea. And I don't care if it has any idea, let alone feelings about what I'm searching for.

Well, John, you're asking the most profound question at the heart of this field. I'm not sure I can answer it in 25 words or less, but let me take some shots of goal and it'll be more of a dialogue. So to the question you ask, is it Performance based. The first answer is that our performance has gone way up. Right. So if you take any objective measure, and there are many, and back in the day, we were interested in can a computer answer an eighth grade test. Right. The Regent Science exam, the SATs. And initially the answer was a resounding no. It did little better than chance on multiple choice questions. It was getting close to 25%. And fast forward now, it can get 80 or 90%. It's better than the most high school students.

And I really wonder how it's doing it. I really wonder.

So we know we have a lot of insight into that, and I'll get to that in a second. But to, to take the performance question, we can check the box. We now have exceptional performance. Agree. But now we're debating, as you're raising the question of, okay, so what does that performance mean? And there's a famous saying from Herb Dreyfus, the late philosopher from Berkeley, who said, look, we've run up to the top of the tree and we're shouting that we're on our way to the moon. Right. It doesn't scale, and it's not really a way to do space flight. We're just, you know, climbed a tree.

Yeah, but a big tree.

Right. So again, that metaphor is not drawn to scale. Right. It's more than a tree, but still, the technology that will get you to the space station to kind of riff on this metaphor may not be the technology that gets you to Mars and certainly not the technology that gets you out of the solar system. Right, Right. So I think that when we talk about competence, when we talk about genuine understanding, there's a real debate in the field. And there's some people like Gary Marcus, who is brilliant and pointing out how this technology falls short. And we can see that these large language models do things that are called hallucination.

Yeah.

You ask it questions that are meant to trip it up, like who was the President of the United States in 1492? And it'll answer something like Columbus. Right. It won't realize that the United States didn't have. Didn't exist in 1492, didn't have a president then. So there's hallucination, there's lack of robustness. You paraphrase the question, and if you ask me the same question, different words, most likely I would say, hey, John, that's the same question. I'm going to give you the same answer. But AI technology will not.

And by the way, that was a perfect demonstration of ye shall know a word by the company it keeps. The machine sees the string. 1492. It basically has seen enough. It knows you want to look for a person. Columbus pops right up.

Exactly.

And so that's. That's a case of the dumb pet trick with data failing you.

That's exactly right. The remarkable thing is every time we identify a trap like this, a phenomenon, a place where AI trips up, well, our colleagues who are deep learning gurus just get more training data, just modify the training regime, and they solve that one.

And they whack that mole.

Exactly. So is it a game of whack a mole, or is there a fundamental paradigm that goes all the way to human level intelligence? I would say that that's the question of the age. And I would look to people who are a lot deeper into deep learning, pardon the inadvertent pun, like Jeff Hinton and Yann Lecun. Right. They're Turing Award winners. And I would say that they themselves, while they're very much enamored of deep learning and this kind of paradigm, say that the current underlying algorithm, the current algorithms, I should say, back propagation, supervised learning, current neural network architectures, don't take us all the way there. They see the limitations of the current technology, but they do see that the paradigm, this distributed computing with simple computing elements and weight updates on edges between them, is the foundation for a much more sophisticated architecture, will get us all the way there. And of course, if we look to the brain, right, neural networks are a gross, gross simplification of the brain. But we do have an existence proof, right? We do know that there's an n of 1.

N of 1.

Exactly.

So here's a great quote from you. That is a good seg to dive into the AI2 impact. So you said AI2 is the place to do work that companies won't do and universities can't. So I think that really, to me, captures the weirdness of this thing you built. It is neither a university nor a company. What is it?

Well, first, to give credit where credit is due, which is always extremely important in academia. Right. We don't do it for the money. That is a quote that I repeat from my colleague Noah Smith, who's a professor at uw.

And hat tip to Noah. Hat tip to Noah.

Yeah, a leader at AI too. But it's a wonderful characterization of what we do.

What is it like? It's this strange hybrid. Sure, it's had all this impact, but without any of the benefits or problems of being a company, nor any of the benefits or problems of being a university. It's like, I don't know many things like it. Also, you've tagged on an incubator to it now, so it's like, definitely unique.

So I'm a product of the university system. I was a grad student, I was a professor for more than 20 years. And I love that system for intellectual exploration, for intellectual freedom, for. For the kind of debate and surprises that it produces. But it does fall short when you're trying to build systems. Some problems require a sustained effort over some number of years requires engineering sophistication, and it's hard to do that with students who need to graduate. And actually, it's not even fair to ask students to play engineer for years on end. Right.

Because you have to worry about their education.

Exactly. That's the primary goal. So over time, over my 20 plus years at the university, I did rue sometimes how, gosh, we really want some things to go into the real world and get more sustained investment than just tossing them over the transom, writing a paper, writing a research prototype, and hoping that somebody will pick up the ball. So today too, where we have researchers and engineers working shoulder to shoulder, and that's an important part of it. It's a egalitarian community. It's not the case that the researchers up on top of Mount Olympus and they're cracking the whip to mix the metaphors, Italian engineers do this, do that. It's much more the case that they're collaborating, the engineers are telling them, look, here's what you need to do to build a working system. We have Semantic Scholar that John, of course, you're intimately familiar with.

You've written, yeah, I love it, use it every day.

And you were one of the folks to first announce it to the broader community when you were writing for science and so on, which was wonderful for us. So. So something like Semantic Scholar, which, to those who don't know, it's a free search engine for scientific content. It has, you know, it's approaching 100 million users a year. It has 200 million papers in its corpus. That sort of scale. And running AI at that scale requires a lot of engineering. And we have a very strong engineering team, folks who came out of Amazon and Google and other places to be able to do that. And you just could not build Semantic Scholar, build it, sustain it, iterate on it in a university. You could build a prototype.

Actually, you know what a good comparison is, is Arxiv. So archive grew out of the university system, it's sustained by the university system. And you can see how far you can get with a system. Arxiv and Semantic Scholar are like worlds apart. Semantics Scholar is a full product with incredible amounts of hand engineering in IT and maintenance and users and I think Archive is about as far as you can get a preprint server that puts.

PDFs on a website and even Archive is an exception. Right?

Yeah, it's a gem, absolutely.

But it's rare, as you say, that you have a university that can operate at very large scale and then on the other hand, we have no profit motive. Semantics Scholar does not have a business model. I don't think there's a good business model in that space because it's meant to be free.

Yeah, I know you have opinions about academic publishing.

Yes, yes, well, we can get into that if you want, but I think too much money is made in for over things that really ought to be free to benefit humanity and maybe to bring this full circle to the late Paul Allen. Our mission is AI for the common good. So that's why I say universities can't, but companies won't. Companies appropriately. Right. Have the mission, their for profit mission. But Paul Allen is a major philanthropist and he won philanthropist of the year award a few a few years before he passed away. He wanted to make the world a better place. The Allen Institute of Brain Science released the free Brain Atlas. There was a tremendous resource catapulting research in that realm. And our mission is always been to bring to the fore and release systems, data sets, open source software that help to bring the field forward.

So what's up with this incubator then? From what little I know about it, you have added a startup incubator to AI too. So that ideas I presume can spin out and have a chance to be nurtured. Is that you hedging against like sometimes actually companies are the right way to solve problems. Or is it like what is that?

Sure.

Is it for the future health of AI too?

Not really. So from day one we had an incubator in recent times.

Oh, I didn't know that.

Yeah, yeah, it started our very first startup kit. AI was actually started in, you know, 2014, 2015.

How did I miss this? It's just not well known.

Well, because it was very small and then once we got the right leaders in place, it's grown and grown and now we're approaching three quarters of a billion dollars in the total valuation of companies founded and acquired are company Exnor, which was a computer vision at the edge company, was acquired by Apple and we've done now more than 20 companies in the pre seed stage. The analogy is to think of a university and their commercialization centers. So University of Washington, where I was, has one, Stanford has one, of course, the famously created Google and Yahoo and other other major companies. And it's actually in my mind a natural part of the life cycle of universities that, that some ideas and technologies that are created in a very nascent, incipient form in the university, it makes sense to transfer them to a for profit context. And that's where you both get the resources to make them shine, right, to take them to the next level. And also you get the opportunity to create value. Value creation in my mind is a great thing. I'm not in any way a socialist thinking, oh, we should all just be working for the common good. I think some of us should be working for the common good and I feel very privileged to be in that position. Some of us should be in startups figuring out how to revolutionize the world and make a killing at it. Even though I disagree very strongly with say, Elon Musk's views about AI, I very much am blown away with his success with Tesla. Right, so we wouldn't have Tesla if we didn't have for profit startups.

Hang on, which part do you disagree with? The robots are going to kill us?

Oh yeah. Elon Musk is famous for having said, with AI we're summoning the demon. And I think that's hype and actually the worst kind of hype. It's hype from somebody who you think would know better. Right. He's such a brilliant man if he gives a lot of credence to statements that are just not rooted in, in any data.

Although he's not alone, there are a lot of cautious voices. Yes, he's just the biggest on Twitter.

He's the biggest and he's the most articulate. But I do agree with you. There's an interesting conversation around this issue. I feel very strongly that we don't have any basis for some of these fears about AI. I've written about this and like you mentioned earlier, right, you've had the own experience. Anybody who's built an AI system knows just how much blood, sweat and tears we put in to eke out the modest level of performance that we get, let alone this AI that's free form, you know, taking over humanity can't be turned off that we see in Hollywood movies. So I think it's really important to distinguish science from science fiction and hype in Hollywood from the reality. Other people just extrapolate more Strongly for the future. They have ideas like, hard takeoff. Sure, AI is not very powerful now, but what if you turn your back on it? What if all of a sudden there's a sharp increase? You know, you leave for the weekend and you come back and on Monday, this fat AI, Your toaster is.

Your toaster's in charge.

Exactly. It's smoking a cigar and saying, I've been expecting you, Dr. Etoni. And that's just. It's not realistic. But to understand that, you have to get a lot more technical. I do want to share two metaphors that I think help. That one is these technologies that we're talking about where we tune the edges on the. Sorry, the weights on edges in a neural network, which is what our deep learning technology is doing. That technology is the moral equivalent, if you're not a technical person, of adjusting the gain and the equalizer and various buttons on your stereo.

Except you have billions of dials.

In this case, you have billions of dials. You're adjusting them automatically. But after you've adjusted them really well, it's still just going to be a stereo. There's no way that you find the right adjustment on lots of dials on your stereo that'll become the Death Star. It's still going to be a stereo. The same with these large language models that, again, as I mentioned, there's a lot of. Are they sentient? And so on. Those large language models are basically mirrors, okay? They mirror by collecting this corpus of all these words in the company they keep, they mirror that collected discourse back at us. And when we look at the mirror, we can see glimmers of intelligence because we see reflection of our own discourse.

Yeah, I agree with that.

The thing that's important to realize about a mirror technology is that you can scale the mirror. You can have a very large mirror. But a very large mirror is not going to turn into a Death Star either.

I definitely agree on the second point. I think of these large language models as data telescopes. They're just amazing devices to look back on all this amazing data we ourselves created with language, which is its own mystery. So really, we're just looking at our own mystery. But on the first one, I would say biologists. So you said, hey, it's just a big stereo. It may be impressively large, and it may be twiddling its own dials, but it's still just a stereo. And I think a lot of biologists would say, well, I can show you a cell that. An amoeba that is really just running around Trying to gobble up food and make more amoebas. And it's not that different from a neuron. It's really just a lot of very strange natural history that led to the job of being a neuron as a cell. And yet when you add them all together, you know, you get walking, talking goofballs like you and me. And so either you admit that we're not that special or you admit that there's something special in the system, but that's basically kept philosophy grad students in business for all time.

But I do want to address your comment because I think it's an important one. And where I would take exception is with the word at. So cells are the basic building block of life, guaranteed. Neurons are the basic building block of the brain. We have neural networks. The units in neural networks are actually very, very much simplified relative to a neuron. But never mind that for sure. I would accept that perhaps we have discovered some of the basic building blocks, but it's not the case today that I can give you a cell and say, here's a cell, make me a human. Right? Far from it. And that we understand.

Unless of course, that cell is a fertilized egg.

Sure, sure. But it turns out to be pretty easy. Well, that's the natural process, Right. And we're going to keep this PG rated, John. Right. So what I'm saying is that we don't know how to artificially produce a cell. And even if we did, we wouldn't know how to turn that cell into a human.

Absolutely agree. Absolutely.

And so. So even if we had a neuron, right, Even if we could build, simulate a neuron in a computer, we don't know how to turn that into a brain or into human level intelligence. So the organizing principles are still what's lacking. And one last point, because this is something I'm so passionate about and actually gets lost sometimes in all the hype and all the excitement about the technology, the one other point I want to make is even if somehow we came up with a recipe, a mechanical process to produce a human by cloning, right? To produce an intelligence by doing the AI equivalent of cloning. We still want to understand. We want to understand the organizing principles of how do you build a human? We want to understand the organizing principles of how do you build an intelligence so we can fix problems, so we can go beyond it, so we can use these technologies for the common good. Right. To cure diseases and also to know.

Ourselves in a deep way.

Exactly.

Let's swerve for a Sec. I really don't want to run out of time before we dig into some of the cool stuff that's actually happening at AI2. So back to that point about AI2 being a place where you could do things that companies won't do and universities can't. Let's dig into a couple of them. So over the years you've been absolutely swinging at the fences with attempts to make an AI system that can solve math problems where that's not directly to your point about companies won't do it. You're not going to make a buck, at least directly out of an AI system that can pass 8th grade math tests. It's not relevant. No one's going to pay you millions of bucks for that. But a university can't do it because taking a look at the papers you guys are producing, the infrastructure required to get there is monumental. So what are some of the big swings at the fence that you've been doing that excite you lately?

Well, so Semantics Scholar is the biggest one, right? Where we have a scientific search engine built from the way down. We are in the process of releasing a sub project of that headed by Dan Weld, who was a professor for many years at the University of Washington, now joined us to lead this project. It's called a semantic reader. And that is basically when you're reading papers, right. I feel like if you want to think about the history of reading scientific papers, okay, we had the cave wall, then we had the printed page, then we had PDFs you can read online. And not much progress since then. Right. We still kind of labor over PDFs. Well, the semantic reader allows you to seamlessly look at citations while you're in text, to look up definitions for terms in line, to do a lot of things I don't have time to describe skimming things that make the process of reading a scientific paper that much more efficient.

So is this like a machine reading over your shoulder and like taking notes for you?

Not that sophisticated. It's much more of a tool. Right? So think of Acrobat Reader. It's souped up to make it easier for you to read. So here's a very concrete example, something that we're very proud of is we've used language models to create TLDRs, one sentence summaries of papers that are really quite high quality. These have been published and measured and they're really quite good. So often as you're reading a paper, there's references to other papers and you're like, what do I do? Do I click on that and suddenly I'm reading another paper and they have a reference and I go down some.

Kind of infinite rabbit hole?

Exactly. Infinite rabbit hole. Or do I note it down, but then forget about it? Well, with a semantic reader, you can hover over that reference and get a tl. Doctor says, okay, that's what that paper is about. And you can make a quick decision. Hey, let me make with one click. I'll save that in my library for future reference. Or no, that's not really what I'm missing. I'll ignore it. So just little affordances, little tricks that are enabled by AI that allow you to focus better and just be more efficient at reading the paper.

Is the secret mission here to make AI researchers better at doing AI with the help of AI in a flywheel? Is this a virtuous cycle?

It's meant to be, but it's to make scientists across all disciplines better at their job. So if we can make scientists across biomedicine, people working on climate change, what have you, if we can make them 10% more efficient, that is significant. And potentially we can make them a lot more efficient. Right. If I give you a tldr that spend, you know, saves you an hour of groveling through the text, or even better, allows you to pursue something that you might have missed. Let me give you actually another example. We all now use adaptive feeds. We just don't call them that. Our Twitter feed. Right. Is automatically organized by an AI that study us. Our Facebook feed. Some people use email readers.

And of course, in that case, the motivation behind the algorithm is to make you click on ads.

Exactly.

And spend money. You're doing the same thing, but with a higher purpose.

That's exactly right. So we have a feed for scientific papers, and you can train it. It'll show you new papers that you might have missed that might result in amazing breakthroughs, and you'll tell it, I like this, I don't like that. Yeah, that's interesting to me. And it'll automatically compute tomorrow's feedback when new papers came up in Arxiv and elsewhere to help you in what's ultimately a needle in a haystack search. Right. Finding that key result that you, with your human intelligence, connect with another result and have this amazing breakthrough. So, yeah, case in point of what you're saying, the entertainment fees we have have a profit motive. But who has the motive to help you be a better, more successful scientist in whatever your field of study is? AI too, does with AI for the common good. Cool.

All right. Give me a second. Big bet. Something crazier.

Well, we have scientists working to fight illegal phishing using computer vision. So there's a lot of satellite data, but smaller countries don't have the resources to analyze that data and identify illegal fishing boats that are impacting their country's livelihood and so on. So we've saddled up to help solve that problem recently.

What's this called?

It's called Skylight. And we just won a national competition that was actually run out of the government to have the who's got the best tools for analyzing the satellite data. And AI too, came in first in the U.S. we're very proud of that. That just happened a few months ago. We are engaged in using deep learning for climate modeling. We're very interested in the problem of how will precipitation rain, right? How will that change as climate changes in an unprecedented fashion. That's incredibly important for agriculture, for irrigation, for making decisions about the sort of infrastructure you need to keep us fed right as the climate changes? Well, we're using the same types of models to help make these sorts of predictions.

I was just going to ask. I had recently heard that deep learning had found its way into weather modeling, and I didn't read enough into it to understand how. It kind of baffles me, why would you use a neural network to make such a model? But at the end of the day, it is just prediction. And deep learning is the ultimate prediction engine.

That's exactly the answer. Whenever you have a lot of data and you want to make a prediction. We've learned that deep learning models are almost invariably really, really strong. But I want to get to the craziest project, and maybe this is what you're alluding to, and that's the problem of common sense. So that's a problem that's been a holy grail for AI. How do we build a machine that has common sense? It's been a holy grail of AI for decades, but there really hasn't been much progress on it until recently. Where Yejin Choi, who's a professor at the University of Washington, shares her time with AI too. She's leading a team that works across both organizations to figure out how to endow computers with common sense, how they can, you know, if I ask you, can an elephant fit through a doorway? You would say, probably not. If I ask you, what's bigger, a nickel, right? The coin or the sun? You would say, oren, you're being silly. Why? Why ask me these questions? But if you ask that question of most computers, they don't know. Right. They don't have the kind of human experience you have.

I think it actually goes deeper than that. I think that's just a great demonstration of the lack of common sense. But this thing that the devil's NLP work every day of, you change one inconsequential word, and the model just has no clue. Suddenly, it all maps back to lack of common sense.

And I want to highlight again, to go back to this fundamental question about should we be worried about AI I think that common sense and common sense ethics are actually really important here. So one of the fanciful scenarios that people love is the notion of you tell your computer to produce paper clips, and it goes crazy. Kind of a magician's apprentice type of scenario. And it produces. It kind of takes over all of humanity's resources to maximize paperclip production, and we all die in the process. Right. There's no food, there's no energy. There's just paperclips. Well, what is that if not a tremendous lack of common sense and of ethical sense? So if we want to work towards having machines play a better role in our lives, it makes sense to start working on these problems now, but in a constructive fashion, not in a philosophical fashion. Oh, my gosh, you know, chicken little, disguise falling fashion. But to say, okay, how do we build into computers the sense to not cause harm?

And this is the alignment problem that people often talk about.

Yeah.

How do we align AI with what we should be caring about?

Yes. Although it's an important twist. So the alignment problem really comes from traditional reinforcement learning, where ethics and values are reduced to a number. And you say, I've got the number 15. For some world, John's got the number negative 15. How did John and I, or how the computer and I align our numbers? But that, in my mind, is actually a gross oversimplification, because how do you build something that figures out what are the right actions, figures out how to evaluate a situation. Right. We often find ourselves in moral quandaries. We often make mistakes and then recover from them.

So you say common sense is the first mountain to climb before all others.

It's certainly a necessary mountain to climb. You know, I never want to, like, say, the problem that I'm working on takes primacy on other people's problems. But I would say that traditional value alignment and reinforcement learning is grossly oversimplified and ultimately inadequate for common sense and for moral reasoning.

And so Ye Jian is. Is tackling common sense. What are the angles of attack on this?

Well, so One of the huge questions that we touched on is, are neural networks enough? Do you also need to create symbolic knowledge? You know, thou shalt not kill. Right. Does that have any value? Can you just use sentences from the Internet which can be, as we know, toxic, full of sexism, racism, xenophobia, anti.

Gay sentiment, and also mutually exclusive claims about everything?

Exactly. So is our moral sense going to come from just a large and arbitrary collection of sentences, or do we have different ways to build a moral sense in a more responsible fashion? And so those are some of the questions she studied. And again, it's a very rich project. Is language enough? What about should we put in robots? Should we put in computer vision? Can we learn from videos on YouTube? Right. There's a lot to learn. Language is just a limited data stream. So a lot of the work is now becoming multimodal.

So what do you think is the best bet we have today for making any progress on common sense? I mean, so far, I'd say the most impressive work has just been in creating better benchmarks to reveal how far we are from true common sense understanding. That's actually been a great project across the world. It's just showing our laundry with benchmarks that are actually challenging enough to show that. No, no, we really are miles away. We're at the top of the tree, nowhere near the moon.

So I think that there is a lot of value in that, and I think that continues. There is a funny phenomenon that when you build a benchmark that's large enough and the community kind of demands. Right. We learn arguably from relatively few examples, but here they say, hey, if you don't have, I don't know, at least 100,000 examples in your benchmark, it's not worth thinking about. But then the benchmark becomes kind of its own narrow task. And then you find we train a deep learning system on 90% of that data, we test it on the remaining 10%, and lo and behold, it does well in that kind of narrow task. And you're still left with this kind of doubt. Yeah. So we solved the task, we solved the benchmark, we solved the data set. But did we actually solve the underlying problem? And often we find the answer is no. Right. It's brittle. Then we make a little change and all of a sudden it falls apart. Right. So I do think we need to go beyond this one data set, one problem at a time, to build something that cuts across multiple problems.

Yeah, absolutely agree. But where are we going beyond benchmarks? Who's actually doing something that you think has A possible chance of being part of this near future system that will have common sense or something approximating it. I take it you're skeptical that it's going to be a bigger language model.

So again, Yejin Choi and her team with a project called Mosaic, is building a massive resource of common sense knowledge repository so you don't have to relearn it every time.

Is this like Doug Lynnette's like big collection of statements?

So it's analogous to Doug Linat's Psych project, which went over many decades. But there are several key differences. First of all, Psych was a heavy logical system and this is a much more modern system with elements of crowdsourcing text.

But it is still a big collection of common sense statements, right?

It is, it is. So in that sense it's analogous. The second thing is the Psych project, at some point, I think it was in the 90s, gave up on the academic community, on careful experimental measurement, whereas the Mosaic project continues to produce new algorithms and innovations and to be both measurable and open. Another thing about Psych is it was always hidden from view. It was a little bit like the wizard of Oz. This thing is amazing. Trust me. Trust me. But no, you can't look behind the curtain. And I realize these are strong statements, but. But again, if you.

But I do, I do just want to give a nod to the fact that it was the right idea, at least in your mind, of collect common sense as a very literal sense of statements about the universe.

Absolutely true. I think Douglannade and his team, the Psych team, deserve a lot of credit for their courage to tackle this Holy Grail problem.

Yeah, that was outrageous.

Yeah. In the 80s. And they did it with the methodology at the time. I think they kind of lost their way over the years. And so we've picked up the baton and other people in the community. I also want to just mention that another data set that we have, which is called, I think it's the norm bank, is a data set of little kind of vignettes or snippets with questions like is it okay to mow your lawn at 5 o' clock in the morning? Or you know, is it okay to kill a bear? Is it okay to kill a bear to save your child? Is it okay to kill a bear to amuse your child? All kinds of little short scenarios like this. And a label that says, yes, it's okay, it's not okay, it's not desirable, et cetera.

And where'd the labels come from?

So they've come from people and also from collecting efforts done by other people. We're always trying to amalgamate and bring in resources created by others and then of course give them back to the community. So we've created the most powerful resource for starting to train ethnic ethical AI systems.

So let's dig into that a little bit. This is really interesting. So I can imagine you can have. What you're generating is gold label data, you know, like we know and love across all of AI. But it has an unusual property which is that at the decision boundary there are going to be ambiguities where people disagree and there's no amount of consensus that will get you to agreement. There are statements that people simply disagree on and they always will. What do you do with that? That's actually a really unique kind of data has built in permanent ambiguity.

You're exactly right. Right. With a science question or math questions, there's one right answer typically, certainly when we're doing with grade level science, not the case here. And actually the system that we built on this, which is called Delphi, it's available actually a demo@delphi.islandai.org so again, open it up and you can see with some effort, it's quite easy to trip it up, get it to say the wrong thing. Well, when you ask Delphi a question, it can actually relativize its answers. It can say if you're a conservative, you would think this and if you're a liberal, you would think that.

It tells you that.

Yes, yes.

Wow.

Neat, right? It tells you that and you can pose the question. You say, I don't want to get into controversial or painful topics, but you take abortion, right? And it has learned a model of the conservative view of abortion, of the liberal view. Again, it has a long way to go, but it's exactly a platform to study the ambiguity that you were talking about.

I'm about to ask you a question. In knowing full well that you've been sort of dragged through hell and back in relation to the Delphi project, but zooming out just a little bit, how do you make productive progress on areas like this that you know are just fraught, you know that people are going to be upset. Anything where you have a language model saying things like this is right or wrong, according. And if you're a liberal or a conservative, someone's going to get upset. How do we make that okay to do that research knowing that you're treading into a bit of a minefield? Like I can imagine. One extreme is we just don't ever touch that stuff. But I know how you feel on that topic. That's. You're, you're leaving gold on the floor.

It's not just gold. I think that science is really hampered if there are questions that are third rails, right. We're not allowed to study how do we build ethical AI systems because people will get upset. I think that's highly problematic. And you're right that when we released Delphi to the public, and we probably could have done better in terms of putting warning labels on it, make sure you know that this is not the be all and end all. This is a research prototype meant to for open inquiry and so on. But people did get upset. And I would say two things. First of all, this is a great illustration of the adage where companies won't. If we were Microsoft, Google, Amazon, worried about our brand, we wouldn't do that. Look what happened with Tay, right? It was taken off and there hasn't been Tay 2.0 and so on. You know, Microsoft.

No, Microsoft hasn't touched that.

Right. Well, they have a brand to protect. I respect that. Our brand does not need to be protected. It needs to, to be the spirit of honest and open inquiry. And if we are alarming people, actually I think there's value to that. If you look at what neural networks do and you conclude, hey, this really needs to be controlled better, then we've done part of our job, right, that that's a good thing. So I don't think that we court controversy, but we are steadfast in our support of open inquiry as opposed to some kind of cancel this, don't do it, it's too fraught. I do want to remind people in the audience, their perspective is about the technology and about the effort to remember that behind this technology there are people, grad students, researchers, and those people have feelings. And I have to say when all the negative energy towards Delphi came across, I felt bad. But I didn't feel bad because I was involved in releasing the project or people were upset. I felt bad for the people at AI too, who are the recipients of all this energy. And that energy, I think could have been more constructively targeted.

I think anyone nowadays is very cautious about putting a language model out behind a text input portal anywhere on the Internet. And maybe that's one of the practical outcomes, is that you just have to be very careful because it's all too easy to elicit offensive stuff out of this language model because it is a mirror of ourselves and we, we are offensive to each Other, and that's all baked into the language it learned from. And so it just seems like there's just a lot of caution around doing what you do, which is to just be open with your work and put it out there as a prototype, warnings and all. I think fewer and fewer entities in this space are willing to take that risk.

Well, I really hope that we, over the years, remain willing to do that appropriately. It needs to be done right. But there are people for whom it almost seems like a sport, right? To use your phrase from a previous conversation, a sport to come and bash these sorts of efforts, and it's all too easy. I don't think it's sporting, and I think you can always do it. So I would not recommend to the Olympic Committee to include large language model bashing in the next Olympics. I would instead encourage the people who are worried about that to engage with building better models, with. Building better controls, with. Because these models are being built and AI is taking an increasingly participatory role in society and decision making. So we need to figure this out, not to bury the issue because it's too fraught.

Last question, Oren, if you could time travel back those four years to when you said in that packed room to all these people, including me, you know, this stuff barely works, what would you say today to, you know, this packed room, some tens of thousands of people listening right now, a lot of them hopeful and taking part in this revolution which is absolutely underway. I mean, it's unbelievable what you could do today compared to four years ago. And who knows, four years hence. Has your advice changed?

I would say that I, along with many people, have been surprised with the progress of the technology. So I would say, this stuff barely works, but I would add the proviso, but it's moving super, super fast. And then I would still add the cautionary notes. Never trust an AI demo. And even if this looks very impressive, think about what's under the hood. What are the implications for society? Don't get caught up in the hype. Not the negative hype of the sort that Elon Musk spouts, but also not the positive hype of the sort, okay? We have achieved sentience. We have not.

Thanks, Oren. Great talking with you.

Thank you, John. A real pleasure.

All right, everyone, that's our show for today. To learn more about today's guest or the topics mentioned in this interview, visit twimlai.com of course, if you like what you hear on the podcast, please subscribe, rate and subscribe. Review the show on your favorite podcatcher. Thanks so much for listening and catch you next time.