TMRE 2018 Debate

Thank you very much. Zeynep will be back on stage in a minute for the debate, which we're getting ready now. This is going to be fun. Slightly chaotic, I'm going to warn you, but super fun. What we wanted to do was to pick a really hot topic and really show that both sides of the debate are really valid, you know, panels. Sometimes they're a bit like, yeah, we're all going to agree with each other. This is going to be a lovely conversation. We're not going to be too controversial, not this afternoon. We're going to try and tease apart different perspectives on this topic. As such, some of the people we're going to get on stage maybe not necessarily representing their own views and certainly not the views of their employer, but they've generously volunteered to take positions to show you the range of views that are possible to show you both sides of this really important topic. So, hot topic right now, just in a minute, we're going to do a vote to see where you are on this topic before we bring the panelists on stage. Then we've got some great minds who are going to debate. I'll go through the running order of that in a minute. Part of that will be questions from you as well. And at the end we'll do another vote, the same question, the same structure. We're going to do in just a moment. And the goal of each of the different teams is to move your view, to move the view of the room in their relevant direction. So vote before debate, vote at the end. Let's see who is able to move you in which direction. Sound good? Great. All right, so if we could get slido open and then go to slido, type this code in, I'm going to need you all to vote. This succeeds and fails on whether all of you take part right now. If a huge number of you wait until the end and then suddenly form a view, it's not the fairest way to judge. So I'm going to take a minute. I'm going to need all of you to vote. And if you don't vote now, I'm going to need you to stay quiet at the end and not vote at the end as well. So do you agree or disagree with this statement? AI will replace researchers. Do you agree or disagree? Some people backstage are getting kind of nervous now, some are getting kind of excited. 80% disagree, 20% agree. 19%. Some of you are not reaching for your phones. I can see that right now. I need everybody to take part. 99 people, 101 people. Sorry. It's loading slowly. Thank you for that good intelligence. We'll give it another couple of minutes. It's loading slowly. 121 people hovering about 80% on disagreements. Notes are being checked and double checked backstage. There's been multiple planning sessions today. 164 people. How's that doing? Anybody still struggling to vote? All right, 174, 177. We're going to have to take a call here. 77% disagree. Let's call that. All right. That's the challenge. Can we move it from 77? In which direction will it move? That is the question. Right. So why don't we welcome on stage each of the two different teams. I'll introduce them and we'll go over the detailed structure of how we're going to do this. Please give a warm welcome to our great minds for the debate. Come on up. Right, we're going to line people up in order of photos. It's going very smoothly. Wonderful. Right, okay, then in order, then we have David Ginsburg, vice president of research and at Facebook. Hello. Marina Koston, vice president of consumer research, International Theatrical Distribution, 20th Century Fox. You all know Zeynep. Thank you for joining us again.

My pleasure.

And this team is in opposition to the motion. So they're saying this is not true. And in proposition, generously trying to argue in favor of something that you do not think is true. Bruce Boston, senior theorist at Nest Labs, and Anthony, corporate strategy at Pfizer. Okay, right. So first, most importantly, I need a coin to toss so that we can work out who's going to speak first. Does anybody have a coin they can generously give to me? You may not get this back. I'm serious. I don't have a coin. Nobody backstage had a coin either.

Coin.

Coin. Amazing. Thank you very much. Yeah. There's got to be an app, right? Could we trust the app, though? So I'm going to toss a coin. Coin. Who would like to. Why don't you? Heads or tails? Heads. This is not rehearsed. Is about tails. I think I know American coins very well. I checked the heads on the back. So. Okay, so in which case. Okay, so then which case? You get to choose. Do you want to go first or do you want to go second?

I'll go second.

You want to go second. All right, you guys are up first. So that's the most important thing. Thank you. I will keep that. Right, here's the running order. We've done the vote. Next, opening statements. So this team will go first because You've got three people. We're going to every turn, pick two that you want to speak so that it's fair. So two of you will do opening statements, two minutes each, then you will both do opening statements, two minutes each, then rebuttals. Two minutes, two minutes, two minutes, two minutes. Then questions from the floor. So I'm going to need you during that first section to go onto slido in the question section, start asking and voting on questions so that when we get there, it's preloaded, we'll flip it on, pick the top three questions from you. For each question, we'll do 1 person, 2 minutes, 1 person, 2 minutes, then on to the next question. Lightning round, then closing arguments again. Two minutes, two minutes, two minutes, two minutes. Then we'll do the vote, the same vote we just did, and we'll see whether it's 77, 78, 79, 60, 50, and we'll award. Well, there's no actual prize. You can have 25 cents. How about that?

Very good.

Okay, great. Sound good? Okay, so opening statements. You're going to be going first then. Who would you like to go first?

I'll start.

Fantastic. Thank you very much. So there should be a timer appearing on the middle screen any moment now. Fantastic. So David's going to be up for two minutes. Over to you.

Excellent. Well, first of all, I want to thank David for having us here for this important and momentous debate and keeping us going along with this very simple structure that you've put together. I appreciate that. Let me start by saying, I think we, speaking for the team, we believe that as companies rely on and build more AI systems and build more machine learning into their systems, that in actuality the importance of research and human understanding actually increases, doesn't decrease as companies use this more and more to make decisions. In the past, when you'd come to research conferences four or five years ago, some of the conversations were always about sort of the impending death of market research as big data systems and other things kind of overtake our industry. And the fact of the matter is that not only has that not come true, frankly, the opposite has happened, which is that we're in a world where research is blossoming and there's a reason for that. I'll speak a little bit just from my own experience in the year and a half that I've been at Facebook. At Facebook, we have some amazing data scientists with. We have amazing machine learning experts, we have amazing AI algorithms and stuff that work on our platform. And at the same time, over the Past four years, our research teams have exponentially grown. And there's a reason for that. Wow, that went fast. So I'm going to go faster. There are two quick reasons for that that I want to mention. The first is that at the end of the day, no machine can tell us those intimate details about what is happening and why. The data tells us the what, but it doesn't get give us the why and that intimate emotional understanding of what's the why behind things, and more importantly, what's next. In a lot of situations, that's what researchers do when we're at our best. The second thing I'd say is, as Zeynep talked a lot about the black box nature of a lot of these systems, make it so that you might move something, you might move an outcome, but you don't actually know what drove it. And if you feel good about what it was, that drove it. So we spend a lot of time as researchers at Facebook trying to get to what we call ground truth, which is really understanding what's happening and are we actually creating value for people? And that is what we measure the machine learning algorithms against. Not whether they're driving successful outcomes, but whether they are providing value for actual end users at the end of the day. Thank you so much. And so for those reasons, research will not be overtaken by AI.

So well done. Okay, one more speech from the side. Great. Two minutes, over to you.

I'm going to say every other word that I intended to say just to get it all out. So nobody does it today believe that we generate a ton of data, right? There's constantly data getting generated based on what we watch, what we pass by, what we react to, what we don't react to. Some companies capture all, all of it sell back to us for efficiencies. But data exists. And in order to interrogate all of that data, we humans don't have the ability to process it in an efficient way. So what AI is great at, there are systems which master complex computational tasks based on clear directions or by interrogating unstructured data. They're functional, they're efficient, they're beyond any of the abilities that we could have. However, what we do for a living is performing a very, very, very fine, balanced act that blends science and art. We take the data, we interrogate the data, and we put it into stories, into insights to help our organization actually drive the business. So my stance is that AI will continue to get better and better at many exponentially efficient processes. But as a function, we have to make sure that we do better and better with the art of what we do with that information. So let's talk about the art part for one second. An insight is a capacity to gain accurate and deeply intuitive understanding of a person or a thing. This is what differentiates us from AI. And if we're really, really good at our jobs, that's what differentiates us from being data providers and actual quantitative researchers who storyteller, qualitative, for that matter. This is our art. We take the data, we use that intuitive ability to turn it into insights, and we do that by this art of storytelling that has defined us as human beings for a really, really long time. That's what makes us very different from all the other animals in the world. We relate to life through stories. We understand our environment through stories. We build empathy through stories. We teach each other how to basically survive through stories since the beginning of time. So no matter how good your data is, you still have to get somebody to buy into what you're saying. You still have to know how to tell them the story to make sure that they're on board with getting your point across.

So thank you very much. Compelling argument from the opposition. And they've got the room already. So over to the exciting part next. Then let's see what the propositions say about it. Who's got. Who would like to speak first? Anthony, great. Over to you.

All right, well, David, thank you so much. I am honored to be part of this debate, and I love being part of this conference. The program here not only rivals some of the best programs out there, not just by content, but by the incredible intellect of its participants, of course, all of you. That's why I am confident this audience has the foresight to see that the role of the researcher will fundamentally change and that the traditional role as we know it today will be replaced by artificial intelligence, the traditional role. I also feel lucky to be up here with such wonderful debaters, David, Marina, and Zeynep. They all have great perspectives, even though their side of the argument is so difficult to defend. So as we start to debate whether AI will replace researchers, I just want to clarify two key points that I think are critical to framing this discussion. The first is the definition of artificial intelligence. And broken down simply, artificial intelligence is the ability of computer systems to take on human tasks. And the second is the role of a researcher. So a researcher is defined as one that systematically investigates information, materials and sources and establishes facts to reach new conclusions. So it's broken down into two simple steps. Now, I approach the Research perspective from a scientific side, you might consider it from, of course, market research and someone else might be doing it from a quantitative research side. So step one, you collect and analyze data. Step two, you make recommendations based on those findings. Collecting and analyzing data. I think we could all agree that there are computer systems today that can not only ingest huge amounts of information, more than any one person can, can consume, but that they can retain the knowledge and they can continue to get smarter. We have that today and it can do it in a semantic search capability. Step two, making recommendations based on findings. It's simply putting data into a format that is readable and can be evaluated. We're doing this today, rules based, and it continues to get better. So as far as I can tell, the basic specific roles, those two could be replaced.

Thank you very much. Thank you very much.

Go for it.

Hi, I'm Bruce Boston. I'm going to be arguing that AI is probably mislabeled. What we call artificial intelligence and the way that we use it ought to be thought of as automated intelligence. Where most people look at these black boxes and don't understand what's coming out of them. The cutting edge practitioners of the day do understand. We see coefficients and weightings and biases. We see data that was employed and data that was ignored. We see how these outcomes affect the systems that we put in front of millions of people, billions of people every day. And what we're including in these algorithms isn't random computer generated. Nobody understands inputs. But these are real data sets collected by real people who thought that data was important and when they thought it was important. We now have the tools to test that data at rates we've never seen before. I explained it to my dad the other day. We all would probably agree with the statement that the cranes have replaced the dock workers, that we no longer see people with bags of grain on their shoulder coming off the boats. But the revolution was actually two sided. The shipping container, the ability to lock and secure the goods you wanted shipped, refrigeration, the ability to have standards that could be moved all over the world by train or truck, was just as much the revolution as the crane itself. And so for this one, I'm suggesting that if you have two corporations, one that uses bags on the shoulder to do the research, and one which employs the economics of the big data machines that can crunch the data that we have, the second will be replacing the first.

Thank you very much. All right, each team has now heard the opening statements. For the team. We're going to go into rebuttals. The same with 2, 2 and then 2, 2. In the meantime, I noticed there are no questions yet submitted. So eight minutes from now, we're going to need at least three questions. How does that sound? There are. Alright, brilliant. As long as there's at least three, we're in great shape. Okay, rebuttals. Who would like to go first on your side? Okay, so two minutes, Two minutes over to you.

Well, artificial intelligence is not just automated intelligence. The way a crane is doing the work that the dock worker is doing. And that's where our opponents try as they might have failed to make their case. Because what you're doing here is you're looking at the data that exists. And the data that exists is going to mislead you. And you're going to need human researchers to figure this out. For example, to give a classic example from criminal justice system, we know from surveys that there's really no big disparity in marijuana consumption among different kinds of high schools. White, black, rich, not so rich. There's a certain amount of marijuana consumption in high schools, but the people who get prosecuted tend to be from poor and minority backgrounds because that's just the way it works. Or if you look at something like neighborhoods with more crime, just the existence of crime means that somebody was there to record that it was a crime. It doesn't necessarily mean that there's less crime. It could be a measuring issue. And the dock workers, it's more like the dock workers are picking some of the bags and not the other bags. And that comes from the way society works. So it's new, needs somebody to understand, wait, our data isn't perfect, it's coming in these deficient in ways and it's got this feedback cycle. Now the answer could be let's collect all the data. And that would be worse than having impartial data because that would be a surveillance state. So if we are not going to turn into a surveillance state, we're going to have imperfect data. And as long as we have imperfect data, we, we don't just have automated intelligence, we have intelligence that's faulty in some of the ways because of the way our society works. It's faulty because of the way data is collected. So we're going to need people to keep at their research game.

Great, thank you very much.

I'll go next. I am all for Industrial revolution and thank God that I don't have to carry bags on my shoulders anymore because it wouldn't work in my shoes. So I'm definitely In the camp of needing AI, however, algorithms do act out. We can give you some examples of that. And somebody needs to actually make sure that the data that they generate or the outcomes that they generate are meaningful and useful, not just in delivering on what you asked, but they are not taking any sort of loopholes around the actual process and coming to an end and coming to gold to their result in a faulty or nefarious way. So I'll give you some examples of them. An AI project which pit two programs against each other in a game of five by five Tic Tac Toe eventually had one of the bots.

Lost my.

Place, requesting moves that are so extremely long and complicated that the memory addresses would crash the opponent's computer and award it a winner by default. A program that was designed to stimulate efficient ways of breaking an aircraft as it landed learned that by maximizing the force on landing, which is the opposite of what it needed to do, it will actually set the value that they needed to zero by overflowing it, creating a practically catastrophic, even though functionally correct result. My favorite is that in a survival simulation, one AI speech evolved to subsist on a diet of its own children. And in some of the algorithms exploited flaws in the rules of the galactic video game elite dangerous to invent powerful new weapons. I am with Elon Musk on that one. Somebody needs to be checking things. So as great as AI is, I still stand by the fact that a human being needs to make sure that the outcome of the AI's inquiry is brought to life in a way that is meaningful, relevant, correct, and empathetic in ways that we understand.

Thank you very much, David.

Thank you. In rebutting, I shouldn't concede, but let me concede something from the beginning. The current generation of algorithms are not up to the task of the future. The current generation of algorithms aren't driving the cars today. The current generation of algorithms aren't landing the planes today. And the current generations of algorithms shouldn't be making the decisions of tomorrow. And if we thought that innovation was going to stop today and never proceed, I would concede to the arguments of the past. But we're not looking into the past. We're looking into the future. One of my favorite things to do on the weekend is just take a silly question and run the numbers. One night I was sitting there and I thought, I wonder how much energy a person could produce. So I looked up the Olympic record for a cyclist and how much energy they were able to produce in an hour. And I thought, there has to be economic value to this. And if there's economic value to this, perhaps as a business person, I might arbitrage what a person could produce in electricity with what machines can do at a later time and date. But what I found out was that an Olympian cyclist in an hour of peak performance produces approximately five cycles. Sense of work. And now I tell my children there is no manual labor. What we're paying for when we pave the streets is not work. We're paying for human dexterity. The reason the algorithms aren't driving the cars is because they can't do dexterity. But they will. And when they figured that out, we will ask ourselves in the future, how much does it cost to go through gigabytes of data to make sure that we're not making critical errors? And the computer is the only way we'll be able to do that task in the future.

Great. Thank you so much.

Yes. So, to start off, I want to bring up three main points. One about bias, one about learning, and one about the information of the past. The first about bias. We keep talking about machine bias. And based on information that it's inputting and it's finite and it has certain constraints on it, and certain, yes, machines at times will fail. But we aren't comparing the opposite side of human bias. Machines do have an inherent bias based on how they're being programmed. But we can see and we could learn into how those machines are making the decisions. These aren't the machines of the past where it's a black box. People are creating and building accountability because they are seeing decisions like this being made and not understanding. But let's test that again against what we have today, which is human bias. How long does it take to dispel the myths of the decisions that humans have been making? You talk about racial barriers and, and building neighborhoods in certain ways. We still, after 60, 70, 80 years of understanding policy and how neighborhoods were segregated, we still are not able to overcome that. But when we run that through a machine algorithm, we're able to understand how it is making it. And it's not taking 60, 70, 80 years to analyze it. It may be taking a couple of weeks. So we're getting faster in that space. Similar to talking about how the airlines are crashing. When you think about in the infancy, how many planes crashed, how many people tried that out. And again, that took 10, 20, 30 years. We're able to analyze that data in a matter of weeks, and it's only getting faster. And Zeynep talked earlier about IBM, Watson, I think it was an incredible show of the information that was out there and making very simple errors. But remember Watson played Jeopardy. In 2018. The systems today are building upon that and taking into account all the errors and building into that accountability. So today I can actually go into a front end system on my desktop and ask it, where did you get that information from? And it could drill me down and find the sources of that information and tell me by weight how it made that classification. So the information is present, it is transparent, and we have come a long way.

Thank you very much. You'll be pleased to know just as soon as I turn my phone off airplane mode. There were in fact questions. So that's very good. Thanks for the call out. Right, so we're going to do three questions now. Quick fire, though. So we're going to have one person from each team for two minutes, respond to each question super quick. So the first question, top question is 29 votes. Only recently have we begun to understand systematic errors in human judgment. Humans are black boxes too. Why a higher bar for AI? Who would like to take the question from your side?

So I'll take it. Sure. I don't know that we're saying it's a higher bar. I think there's two fundamental things that I think we're missing as we talk about this conversation. The first is the question is not. Not is AI useful or not? I don't think anybody up here or machine learning would argue that AI is not useful. The question is, is AI in a place or will we see it get to a place where it could overtake or replace any human research or element? And that is where I think this question sort of misses the mark a little bit in that we're not saying that there aren't biases with people. Of course there are biases with people. We're all researchers. There are biases in every methodology that we use. We're whether you're doing a qualitative approach, a quantitative phone, Internet, in person. The point of a researcher is to know those biases, leverage it and bring it into what you're doing. The problem with much of the AI systems today is that those biases may not be known, we may not know what they are until we're past the point where we can actually sort of fix it and get in there and tinker and correct the problem. So that's one. And the second piece of this that I would argue that we're sort of scared by in the conversations that the other side has made is the assumption that all information and all knowledge can be reduced to numbers and data that a machine can operate on. I think as we think about the different disciplines of research, whether you're talking about qualitative research, ethnographic research, social science research, these are approaches that are an important part of how we approach our research discipline, beyond just pure quantitative data, because you have to understand the cultural context of different societies in which you're researching. You have to understand the long term implications of these things. That is something that in any time in the near future, I don't see AI being able to displace researchers for. And I have 14 seconds left.

Amazing.

So I agree. There's no reason to hold the AI to a higher standard. 50,000 people a year die in car accidents. I can sit here and that's what the 50,000 divided by 365 is going to give us, about 1500 a day and put that into hours. And now you're at six an hour and then you're at one every ten minutes. I couldn't even finish the story of a single person's life, what we lost in a single accident, before I got to the next accident. At the rate that we're killing ourselves just in America, just with vehicles on the other side of that, you can't stop the news feeds when Tesla hits a bus in one of their testings. And there is this higher standard. There's also what I'll call the fear of the shadow, the fear of the unknown, the fear of the not me, the not us, that often happens in AI conversations. And we're often afraid of what's behind the curtain if it doesn't include us. These are reasons why I still go back to what AI is fundamentally going to be doing in the future is automating intelligence. There's the difference between what we might call fluency. When our economics and our culture and our businesses were done through words and we moved into an era of literacy where our communication was done through the written word. And those were fundamentally different skills. The order skill and the writer's skill was different. We're moving into a new era and it's the numeracy and it's the numerics. And the question is, how do we put all of the things in the human experience? The same way we had to move it from an oratory tradition into a written tradition. We're going to have to move from a written tradition into a numeric decision. And I don't see anything in in that that would forego including the data that David Says is getting excluded today.

Thank you very much. Next up, how do you think we as researchers and humans need to change to compete with AI?

We can take it. I'm not sure again that anybody here is saying that we're competing, competing with AI. I think fundamentally, at least our side of the equation saying that we need to complement each other, we need to figure out how to optimize this relationship. We need to be skilled to understand what AI is doing. We need to be able to process research not from maybe a small project based point of view, but from understanding consumer journey from every touch point of data that they generate. And that requires thinking about things from a different point of view. So I think we should embrace, understand, optimize, make the best of what AI brings to the table. But still, at the end of the day, it is up to us to know what to do with that information, how to convey that information, how to move people around us to do something with it. And I think that's the fundamental thing that will never, ever go away.

Great, thank you.

We have time.

Yeah. You want another minute? Anybody?

Oh, we're fine, please. No, my colleagues have made the case.

Confidence right there. I see. All right, we can clear the clock then and put two minutes up for the next team.

Sure.

So if we go back to the steps again of how we're defining the role of a researcher, so it's collecting data, collecting large amounts of data, finding it from all different sources, trying to distill it and understand it, and then making recommendations, that was the, again the traditional role of the researcher. We should take that and move that role to a little bit more of the evaluation and bringing that human element into it. So what as researchers, can you do be able to understand how an AI algorithm is looking at the data, how it's manipulating it and how it's presenting it? Brush up on your data science skills and understand how can I get to the root of it before you're figuring out typical regression. Now understand relational information and how it's being presented and shift that role from going out there, spending time collecting things and trying to make relationships yourself and then trying to evaluate it later. Spend the time on the high value activity and thinking about how to incorporate that with a human base into action. If we think about the role of a doctor and a nurse in the health setting, the role of a doctor to figure out all this information, what they're doing is they're learning all the different background science around how disease affects your body. What you can do is you could train A system to help understand all of that. And you put in all of your symptoms and present an outcome or a treatment regimen that you should be following, that role is going to change. The doctor is no longer going to need to diagnose you. You can be diagnosed by a system that has higher confidence. And the role of the nurse or the nurse practitioner will shift and adjust. They're the people who are going to be interacting with you. They're going to be providing that human element, telling you what it is that you can do in your lifestyle to change. So similarly to the role of a researcher, you don't need to spend the time going and collecting, spend the time shifting on that value added activity of how you implement it into your life.

Thank you very much. Onto our third and final question from the audience. It's a great segue from where you were going there, which is do you think an entirely new research specialty will become the norm? Maybe an AI interrogator?

I hope so. All right, well, I guess I'll take that if I have a clock. Well, we should at a minimum have AI interrogators. I also think we should have AI, maybe jailers at times. So things that put the AI police and AI accountability and all those things. Because once again, one thing I agree with my opponents is that it's just begun. It is going to do lots of things and it's going to be powerful, but in the end it is not going to be a human. And the fact that it's not. We're not just like lots of intelligence. In fact, we're kind of barely intelligent that it takes a lot of effort to hold it together. The things that we do as humans, we like to think of ourselves as the rational creatures. We are not rational creatures. We are group animals. We have lots of weaknesses, we have social weaknesses. Yes, we have biases. We don't make the best decisions all times, but that's the human condition. And automating that away and reducing decision making to just automated intelligence will be a loss, will be a loss for the human aspect of everything. And you can come up with examples from all fields that if we do it the way that AI would have us do, it'll just be running lunch. It will be as if we empowered a bunch of little very smart bureaucrats to make every decision. And, and they'll be very smart and rational and good at it. And I'm not sure that sounds great just because they're really, really smart. So I would argue just on that side, I would like them to remain more toaster like and less even if they're very good at what they do.

Thank.

You. One of the reasons I call myself a theorist while many of the other people in the room that I'm practicing, practicing with are data scientists or machine learning experts is because I think that's what we're actually doing, is we're creating a methodology of getting good theories and figuring out what makes for a good theory, and figuring out how we test those theories and figuring out what data we need to see the strength of a theory and how that theory can later be employed both in the workforce and in politics and in our everyday life. While I think AI interrogator is a little too close to exterminator, I think there are softer new rules that AAMI may take over, and one of them might be an agent. Think of a PR agent, Think of a representative. Think of an AI that really, really knows you and ask you a question when it doesn't know you. And imagine what that would do to our everyday work. What if you didn't have to attend 50 meetings a week? What if you could attend 500 meetings a week? What if you could attend a thousand meetings a week? What if you didn't have to wait for the other people in the room? What if you could all speak at once? What if you didn't have to process what you're saying and what the last 10 people said? But that could be done in a very, very clean, documented, and representative way of everyone in the room. What if we didn't have to limit our meetings to 20 people? What if 1000 people could attend these meetings and 1000 people could give their input for all the decisions that we have to make in all of our lives, especially in the workforce and throughout. An AI agent could do just that. An AI agent can say, bruce Boston believes these things to be true. And here's the body of work that backs up what why he would believe those things. Other people have different bodies of work, so they had a different set of theories, but they could all come together in the same place.

Compelling vision. Thank you very much. Well, I'm still thinking about 1,000 people meetings.

I'm sorry.

Right, that's audience questions done. Thank you very much. We can lose the questions from the side now for closing arguments. So same as before. Two minutes, two minutes over here. Two minutes, two minutes over here. And then the vote. Start to think about your decision. Who would like to go first?

I think we're going to volunteer our two plus two into a solid four.

For Zainab, we're going to do a four minutes.

All right, so two minutes and then two minutes for Zainab.

Great.

So I just wanted to say, what if you could attend 1,000 meetings and call a vote? It sounded like a perfect argument for. For our side. What if you could attend 5,000 meetings and just be done? But here's the reality is that what AI does, even when it's great, is it works to rule. And work to rule is actually a kind of sabotage. So work to rule is what people do if they're forbidden from striking and they want to harm their employer or wherever they are. They bring down the rule books and they find every rule they're supposed to obey and they obey every single rule to, you know, they cross every T and dot every I and do everything. Everything then halts. Because in every system, you actually never, ever really work to rule. There's a lot of tacit knowledge. And there's also times in which you break the rule because of human reasons. One of the most striking examples for me is what happened in 1983. About three weeks after the Soviet Union had shot down a Korean airliner. This Soviet military officer, Stanislav Petrov, was informed by his radar system that United States has just launched an intercontinental missile, nuclear missile, aimed for the Soviet Union Union, and then five more were on the way. At the time, the Soviet policy was compulsory, mandatory, immediate retaliation. The way ICBMs work is that you have maybe like a 20 minute window and if you don't respond, you get wiped out. That's the insanity of nuclear war. Now, I will concede that the fact that humans have set this up doesn't speak much to human sanity. But that's what we had set up. So Stanislav Petrov, sitting in that room, watching what is plausibly nuclear missiles coming towards his country, decided to do nothing. He didn't notify superiors. He just didn't do anything. He assumed and acted like it was a false alarm. He did not know it was a false alarm. It was a false alarm, thankfully. But that's the kind of thing where, yes, humans set up this insanity, but in the end, it's also the human side that breaks the rules when the rules should be broken. There will be points in time when we will set up the most perfect AI and we will unleash it. And we'll be like, wait, wait. We didn't really mean it because we're human. We don't mean every rule we write down. And if we lose that discretion and automate Our intelligence, without the kind of human discretion, we will be locked into the past because it's going to look at the past data and just going to replicate it and replicate and replicate it. And there will be a thousand of them in a room having a meeting in a cyber room. And they will come up with things that look really smart to them because they will be backward looking. You sometimes see this already in Wall street with flash crashes, when one algorithm is like, oh, wait, let's sell. And the other one's like, oh, the other one is selling, let's sell, let's sell. And they all start selling before a human can blink. And then the stock market has these stops exactly for this reason. 10% down, nothing happening. They just stop it. And I guess they do a proverbial kick the sort of unplug the thing and say, okay, no, you guys are stuck in a feedback cycle. We're not doing doing this. That's the kind of danger we face if we get blinded by how good they are for 10 seconds and how they could all be in a thousand meetings and be making decisions because they'll be working to rule. And that's not a good way for us to be governed.

Thank you. All right, two minute rebuttals, closing arguments on this side.

Yeah, I think at the end of the day, I still go back to the question. If you have two companies, one full of researchers and the tools that we have today, and the second with fewer researchers and the tools that we'll have tomorrow, which one, as a society, do we want to trust in the future? We'll have the algorithms to ensure that civil liberty violations aren't happening, and we'll be able to show that as a prerequisite for presenting any proposal that we have internally. The black boxes of the mind that was brought up in the questions is not something that can do that today. And as a CEO, if you're having to make a decision on pure human with some data to back it up, or lots and lots of data, where all of the theories of all the different people in the company have been integrated into this, where the machine has gone through it over and over to make sure that the math lines up to all the input that everyone had and that the decision is in line with, with the most number of people that could possibly agree with the decision, then I think the future that we want is that second one. I'll again reiterate the economics. The economics of a single person doing a dollar's worth of machine work is in the tens of thousands of dollars. And on the argument of economics alone, I can't imagine that we'll be throwing away the power, the richness and the pure beauty. I'm after Babbage. These are adding machines, but they do it in such a beautiful, wonderful, exciting way without the bias of the human mind, without the bias of the past. And that's what I'm suggesting we go for in the future.

The final word of the debate, sir, is yours.

Thank you, David. In closing, I want to acknowledge the great arguments of our opponents and just want to also acknowledge some of the far fetched stories that were presented here to create some sense of doom and gloom. But understand that I think we'd all agree when you break it down just to the the fundamental tasks as we've highlighted below, which is collecting vast amounts of data and making recommendations, we can all appreciate that machines, and we all understand that there are several nuances in that and differences that we need to incorporate into it. But those are rules that we can start to build into the algorithms that machines are going to be taking. The learning curve of machines is rapidly speeding up and it's compounding because it retains the knowledge of those that have come before it. I think for those of us who are fighting this notion that artificial intelligence will not replace many roles while creating the need for so many more, highlight one of the things that we hold back, that we hold in humanity, and that's a desire for permanence. Dr. Steven Rice writes about it in his book who Am I About. Humans need to remain stable and to be felt, needed, permanent. Although it's a basic desire, it goes against the progress of human existence. So we should no longer be asking the question if AI will appraise researchers, but when and what shall we do about it? Let's come to reality. Equip yourself with the decision making powers needed to understand, evaluating, to evaluate these complex recommendations. Brush up, as we mentioned, on your data science skills and keep adapting those and you'll certainly be needed again, considering the origins of the word research, from the French meaning to search, to explore. These basic functions of searching and exploring can be completed by a computer. And the role of the valuable component will shift to the evaluator, which also comes from the French, to find value. No longer will researchers be simply finding information and collecting that, or they'll be the ones to decide the true human value.

Thank you. We've heard about nuclear war. The only thing worse than nuclear war that came up was the concept of a thousand meetings. We heard about the tragedy of car deaths in the US Alone. And the vision for augmented experts and the potential of the new skills and technologies available to us. Ladies and gentlemen, time for you to decide. Have you been moved by these great minds if we could put the poll up again and hopefully clear it and start again, or is that the new numbers? Okay, yeah, we've cleared it. These are new numbers.

Great.

34 votes and 37 votes. Wow. Oh, it's very different already. 46, 54.

Not a readable base size.

Not a readable base size. Thank you. Who remembers how many votes we settled on last time? 169, 180 something. Okay, we are not resting until we get to 180. Come on, people. 74, 82. I think we can see how this is going to go unless the holdouts at the end are very, very biased. Do you want to tell me when you're happy statistically that this is a good result?

Now.

I just want to say that I don't just chat about nuclear weapons all the time, so it's okay to talk to me in the break unless I get the wrong impression.

Six percentage points at 95 basis, statistically significant. I think we win.

Ladies and gentlemen, we have a view. Hang on. You guys were opposing it though, right?

Yeah, we oppose the statement.

Right.

Yeah. And you had 77% before.

Yeah.

So these guys have won, Right, because they've got.

Yeah, we persuaded more.

Yeah.

In Oxford style. It's who persuaded.

We won the question. We lost ground.

Okay. Yeah, that's right.

Sorry. Yeah, great.

So you guys stole some of our.

The answer is that this is the correct side of the debate. But in terms of the debate, the rules at the start were who moved. Moves you the most. And you guys moved a very large number of people to a great degree. You lost the question, but won the debate. Congratulations. Thank you so much for your support in this very experimental Big hand to everybody who took part. Amazing, amazing, amazing. Let me reiterate in case it's not already clear, the views expressed in the debate, not necessarily the firm views of particular participants. The most common thing I heard in calls preparing for the debate was, well, I could take either side. Which one do you want me to take? Really? Do not hold people to the views they expressed here today. I've got two brief announcements and then I'm going to send you on your way. First of all, clear pending invite requests for meetings in the app. Please, please, please, let's get to 700 by the end of the day. I'll tell you in the morning if we got there. Secondly, please go over to the Facebooks and booth right at the back of the hall if you haven't been there already. Bunch of super smart people doing really interesting work using technologies that I've never heard of to do research. Go and use this chance to pick their brain on how they do that. Pick their brain on the tools and techniques they're using and find out more brilliant people. Not only that, you've got five, they've got five opportunities to get a photo op with Peyton Manning for which you have to go to the Facebook booth. Extra reason to go do it, go to the booth, talk to them, get one of those five photo op opportunities and then in terms of sending you on your way over to the exhibit hall now for drinks and then 6:45 kind of in that direction I think Marshall's Outpost for the Qualtrics bonfire. Have a great evening. I'll see you bright and early in the morning.