Making Technology Less Manipulative [Entire Talk]

Thank you guys all for coming. Yeah, it is very weird for me to be here. I did come here as a student and listened to many famous entrepreneurs come and give lectures. I remember the first one that I came, I had a venture capitalist, or it was like a real estate guy, come up to me and say, well, he gave me his business card and he said, here, if you ever need office space, call me. And I was like, what do you mean? And he's like, well, You know, you're going to start a company someday, so when you want to, just get some office space. So yeah, my role in the world now is very different than I actually thought it would be when I was at Stanford. So I graduated here in 2006 as an undergrad in computer science. I took mostly symbolic systems classes. I was very interested in cognition, in the mind, in neuroscience, in psychology. And I got particularly interested in a lab here called the Persuasive Technology Lab. Does anybody here know the Persuasive Technology Lab? Some of you, okay. If you didn't know, the Persuasive Technology Lab is basically taught by BJ Fogg. He's a psychology professor, and he pioneered this field of persuasive design. How do you persuade people? How does that work, and how could technology persuade us for good? That was the question. And this lab was filled with young engineering students, mostly computer science, business, psychology students, many of whom went on to join the ranks of Facebook. And actually, my project partner in the class, Mike Krieger, was here from Stanford and founded Instagram, actually using a lot of the techniques that we designed and learned about. And the thing that caught my attention in that class was this last semester, the very last class of the semester was about what is ethical persuasion? What does it mean to be able to ethically and personally persuade a human being? And we actually talked about this thought experiment. Well, what if you could, get the perfect profile, let's say every amount of information about how this particular human being's mind works, and you could use that information to persuade them towards anything? So this was sort of a thought experiment that we were running, and it turned out that that question became 10 years later the thing that I now do basically for a living and I'm concerned about. And that frankly is one of the most important hidden and invisible topics of our time right now. If you're aware, in the last election, including today, where the hearings in Congress with the major technology companies Facebook, Google, and Twitter testified before Congress specifically about the ways that their platforms were used to persuade the American public in the election and what Russia what Russia's role was in that, in using these platforms. So, and if you didn't know, Cambridge Analytica, which is a company that basically the Trump campaign used, and also for Brexit, to basically create a personalized profile of what would persuade people. So these questions that I was really interested in became really the foundation of possibly one of the most important things that's going on in the world right now. So why does this matter? Why does ethical persuasion matter? Really, this is a conversation about values. When you're in this class and you're thinking, what does it mean to ethically persuade people? You think, well, who's to say what would be good for someone else? If you say, well, we want to help them, we want to make their world better, we want to make their world more open and connected, we want to help them sell stuff. So we're starting to define these values, right? Like, we want to persuade people to be more open and connected. But what I want to talk about today is where our ideas about making the world better meet the— rubber meets the road with reality, and what it would mean to actually make the world better with persuasion. So I want to tell you a little story about kind of how I got into this and got concerned. When I was a kid, I was a magician, and I basically studied from a very early age that this instrument that you're seeing me through right now, your conscious experience, it can be manipulated. There's limits to our attention. There's limits to what we can see and not see. There's limits to how we can think about something. If I split this room in half and said, "Is the number of countries in Africa greater than 50 or less than 50?" and I asked you that, and I asked the other group, "Is the number of countries in Africa greater than 150 or less than 150?" just by anchoring these two groups on different numbers, you would get different results. And what you start to realize as a magician is that the human mind is living inside of a 24/7 magic trick. You're basically inside of an instrument that you can't see, that you're experiencing me right now, and that there's certain levers and strings you can pull on to get people to think about things in a certain way. But I say this because this was a childhood interest and I didn't really do it for that long. I gave like 1 or 2 magic shows, but it gave me the kind of foundations of that thinking. And I want to explain that I later came to Stanford, I took the Mayfield Fellows classes, I got very interested in entrepreneurship, and I started a small company called Appture, which was about learning and helping people learn things on the internet. And we thought, what if we could apply all that stuff I learned at the Persuasive Technology Lab for good, let's use it for good. And oh, I know what good is because I'm a 23-year-old smart kid from Stanford. So we built this thing that basically let the economist would be one of our customers and you would highlight something on the page and we would persuade you to basically go deeper and learn about something. We were just making, we called it lighter fluid for sparks of curiosity. We thought what could go wrong? We have pure good intentions, we're just trying to help people learn things, we're persuading them to learn. We used all these little persuasive design techniques. And as I was building the company, we raised venture capital, we had a team of 12 people. As I was sitting there, and what got me waking up in the morning was, gosh, I really want to help people learn things. That was about all I could think about. But at the end of the day, everything that my company did was not actually about helping people learn. What really mattered for the business was whether or not we could tell The Economist that we were increasing how much attention or time people were spending on its website. You guys following? So I had this goal of, hey, on the web, we're going to provide this technology that's going to make it easy to learn about things. But the way we sold that to the publisher, The Economist, was by saying, we're going to help people spend more time on your website. And so I had 12 people that worked at the company, and I recruited them by being really passionate, by being just classic, like, we're going to change the world, we're going to make people learn about things, it's going to be great. And they really believed it. I was very persuasive as the CEO of the company. And I honestly, as the founder of the company, couldn't admit that there was a gap between my positive intentions as a human being waking up wanting to do this good, and the ultimate thing that we were measured by, which was to capture human attention. And when you're young and you're raising money and all your friends are starting companies, there's this temptation to compare yourself, right? They raised this much money from DFJ, and oh, I only raised it from this other VC. Or they recruited this brilliant guy from LinkedIn, and oh, well, we recruited these other people. And there's all these dynamics when you're running something after you've built together a team and you're basically out there in the world with your company and your product, that how deeply will you be able to examine your own motivations and beliefs and see, Maybe what I'm doing isn't about helping people learn at all. If that were true, would I be willing to see that that was true? Now think about it, if you've got, we raised $4 million, we had a team of 12 people, their families, their livelihoods depended on me. I had a huge amount of responsibility, I was just trying to make the business work. And here I was starting to question whether the entire premise, the values, the core, of what I was doing every day. Were we actually doing that? Some friends of mine who are founders of tech companies as well got together one weekend. And we got together because we all talked about how there's all this pressure that you face when you're an entrepreneur. This pressure to succeed and to talk about how you're growing, how things are going better, that you can never express your doubts. Every time someone says, how's this going? You say, oh well, we just raised this much money, we just closed these 10 more customers, it's going great. You can never just say, I don't even know what I'm doing. I'm not sure that I know why I'm doing this anymore. Where is it safe to utter that sentence? So my friends got together this one weekend, and we had a group called Doubt Club. And Doubt Club was basically acknowledging that it's not safe anywhere, to talk with other founders about your doubts about your company, your product, and your life. And we basically went around in a circle. It was hyper confidential. It was very safe. We basically said, look, the commitment here is even above your connections and other employees who might or might not work for you, or if other investors are looking at your company. This comes first in a small private group. And we basically shared our doubts about what we were doing. And it was in that moment that I realized that there was a huge gap between my positive intention, that I really wanted to do good, but there was a gap between this learning goal with our company, Appsure, and the actual delivered thing that we had to do every day, which was capture human attention. I was able to see that, and then I had to figure out, what am I going to do? So Tina's told this story that makes it sound like I'm some, oh, he sold his company to Google and what a successful entrepreneur, but I'm going to tell you the real truth, which is I was terrified about what we were doing. I felt really lost about why I was doing the thing I was doing, and we had to figure out what to do. And we ended up being in a situation, we could have raised more money, but we were also shopping the company around and seeing what would happen. And ultimately, we soft-landed the company at Google. And so we had a nice out, we had two, Two of the other people in Doubt Club ultimately, one got acquired, talent acquired for their company, we got talent acquired. And the other actually just shut down his operation, decided to move to Berlin and do what he was passionate about. But I say this because it's incredibly, incredibly hard to question what you're doing. Upton Sinclair, the writer, wrote that you can't ask someone to question the thing that their salary depends on. It's even harder to get someone to question something that they deeply believe is their purpose. Talk— think about questioning someone's religion or something like that, or questioning someone's identity. And the reason I'm talking about this is because all of us in this room, I imagine, have good intentions for where we want the world to go and what it means to value certain things that we care about. But we're only as good as the way that we can self-examine our own values. So here I was at Google, and I was kind of recovering as an entrepreneur. I felt honestly like I had failed. I needed to get psychotherapy. I went to Burning Man. I had to kind of let go of the life that I was living and realizing there was more to life than just starting companies, right? I still had all these friends that had started companies. And then I was sitting at Google, and I had actually joined the Gmail team and working on some future-looking products. Personal assistance type projects. And I was working with Gmail, and I was in the room with the people who make this product. I was fascinated. Here's this Gmail product that literally, I don't know, a billion people actively use. And that hundreds of millions of knowledge workers, that's the window, it might be open on some of your laptop screens right now for all I know, right? We people live in this product. And it has this extraordinary influence on the thoughts that arrive in people's heads. right? Like, just, you're sitting there, if an email comes in right now, it's going to push thoughts into your mind, and you don't get to choose whether that happens. It just happens. And so I was sitting in this room, you know, thinking about, you know, I really cared about products that really delivered a positive benefit in the world, and I was in the room with the people who would be thinking the most deeply about what it would mean to truly benefit people's lives, what email should do, like what are the values behind email. Like, of all rooms to be in in the world, to be in that design room is like, that's the room. And I was, I don't want to say disappointed, but the way that the conversations were had were about let's make it really fun to use, let's make it engaging. What if we make it bounce up when you scroll? What if it expands vertically instead of slides horizontally? And there was all these design questions like that, and I felt that there was something missing. I couldn't put my finger on it for a while, but I realized later, it's like when is email actually adding up? To a real net positive difference in your life? Think about the emails that you send, whether it's love letters or your apartment searches, where you're actually getting some real actual delivered life change, a real benefit, a real value. And then think of all the other stuff that we're just kind of shuffling messages back and forth. And I felt like of all rooms to be asking this question, we were not asking the biggest question. And after a year at Google, I was kind of burnt out. And I decided that I was basically going to leave, at least I thought I did. And before I did, I kind of took these concerns and I made a presentation. And the presentation was basically, never before in history have 50 designers in California at 3 tech companies influenced what 2 billion people will think and do right now. And we have an enormous responsibility as a company that shapes this screen in terms of what we are causing people to do. And the reason I was even thinking about that is because I had this background as a magician and in the Persuasive Technology Lab where I learned that minds really are steered by forces that they don't see. So I have to say that in making this presentation, I actually thought I was going to get fired. Alarmist or angry or upset, but it was very critical. And it was very existential, like what does it mean for actually email to be a benefit to people's lives? And I was surprised because I sent it to about 10 people, and when I came into work the next morning, I got some emails back and I clicked on their comments. And I went to the presentation, Google Presentations, that showed the number of simultaneous viewers was 150. And when I clicked on it later that day, there was 450 simultaneous viewers. And basically I saw that it spread virally around the entire company, went all the way up to Larry Page. He was in 3 different meetings that day where people brought it to his attention. And suddenly I was in this moment where I felt like the whole company was woken up to this question of, are we actually influencing the world in a positive way? And that led me to the next 3 years, and actually an executive at Google saw this and basically generously offered to host me in a little in his lab in New York, where I could study ethical design, which I basically self-titled this role of design ethicist and this new field of what is ethical design or ethical persuasion. What does it mean to ethically influence what 2 billion people are going to be thinking and doing? And so that's basically what I did for the next 3 years. And I went really deep into understanding, first of all, if you take the human evolutionary code, You're living inside of a meat suit, mind, body, right, that was tuned millions of years ago. So our predilection for sugar, salt, fat were tuned millions of years ago when they were scarce, but here we are now when they're abundant. So we've got all these tunings, and you're living inside of it, and it can't really change. You don't really have much optionality. So you're living inside of this thing, and if you made a map of every single string you could pull on this mind-body system to persuade it, If you could make a map of how could you addict a human body or a human mind, how can you pull on its sense of belonging, how can you make it feel like it's missing out, how could you get it to do certain behaviors, how could you get it to think about certain things, how could you get it to make certain choices? So if you had a map of every single way that a human being could be manipulated, that was the first part of the study, the task. Second part of the study was What would it mean to ethically persuade? So what's constituted in ethically pushing this human animal around in the world? And then the last question is values. What are we pushing it around for? And who's to say, how do we know we have values we can stand on, that we can actually persuade people in an ethical way? So I basically studied this topic, and I want to say that I didn't know what I was doing. I was basically trying to figure out this answer to this question of whether we want to or not, Google's going to bump its elbow, and Apple's going to bump its elbow, and Facebook's going to bump its elbow, and a billion people are just going to go in these different directions, right? Because when you wake up in the morning, it's important just to set the context, 2 billion people, they wake up in the morning, and the first thing they do is they check their phone. And we check it 150 times a day. In the bathroom, coffee line, going to sleep. We spend a lot of time on these devices, and even when we're not looking at the device, the thoughts that are in your mind right now are still partly set by the time you did spend looking at the device. So we had this kind of 24/7 immersion in this environment. And I didn't know what I was doing in studying this question. I just found it to be fascinating and important and interesting. And here I am literally, Whatever it is, 3 days, 3 years later, it's November 1st, 2017, and the US Congress is questioning Facebook, Google, and Twitter about exactly the stuff that I've been interested in for the last 3 years. And I honestly find myself right now at the center of one of the most important and invisible problems, I think, in history. Which is that it's not just that there's this system that's kind of bumping its elbows into people's psychology, but we gave this system a set of goals that are causing tremendous harm. So what do I mean by that? The reason I told you this story about why I had these positive intentions, and with my company, Appsure, we wanted to help people learn, and how there was this doubt club process to try and figure out what that might have been, what really would have been underneath there. And that I had to actually examine the core beliefs that I had is because I think that just like everybody in the technology industry, all these companies that were up there today in Congress have very positive intentions for the world. Everybody at Google and Facebook and Twitter that I know really, really cares about delivering the best possible world that we can create. And yet, I would say that something like the statement, our mission is to make the world more open and connected, is just the same as me saying, I want to help people learn about stuff. Because what is that actually about? What it's actually about is capturing human attention. Mark Zuckerberg actually faces this exact same dilemma that I faced, except he raised a lot more money and his company's in the public stock, you know, stock market, and one of the most profitable companies in history, and controls what 2 billion people will think every day. But how could someone like Mark Zuckerberg question whether or not everything that they do every day is actually about making the world more open and connected? If you think it's hard as a founder when you're comparing yourself to your friends, and whether or not you hired the good people, you raised money from some impressive people, Talk about what it would be like to run one of the most powerful corporations in history, influencing what 2 billion people in every language think and believe, the terms of people's social relationships, and to actually question whether or not the thing that you think you're doing— because you know you're good, you know that you're trying to do good— but how that might be different than the actual result. And the first thing you have to do is pay attention to incentives. And I think this is really important as an entrepreneur, because all of us, I'm assuming many of you are going to go off in the world and start companies, is pay attention to who is paying who. What is the actual thing that you're beholden to? With my company, it was The Economist. They were our customer, and we had to do one thing for them, which was keep people on their website for longer. Let's look at Facebook. Their business model is advertising. No matter what good they want to do in the world, their stock price is dependent on how much attention they capture. YouTube's, Google's stock price part of YouTube, is dependent on keeping people's attention. Twitter's stock price is dependent on keeping people's attention. Everything else they say is an intention that's outside of that. It's a dream, it's something that they'd like to have happen. But at the end of the day, the thing that they're beholden to is capturing human attention. And I want to talk about why this situation— I'm going to take you down a little journey which might leave some of you more alarmed than you intended to be. But I promise that I'll turn it around at the end. I want to scare you for a moment about where we are, because I think that we're in a much more dangerous situation than people tend to recognize. The goal of capturing human attention becomes this arms race for who's better at being a better magician and pulling on the strings of the human mind, right? So you have these products competing, and everyone's trying to figure out, how can I get more attention? And then my friend Mike Krieger at Instagram says, hey, just like with Twitter, let's add the number of followers that you have to our product. If we add the number of followers we have, then everybody has to log in every day to see how many they have. And they want to get more, so they have to come back to the product every day. So this one tiny design choice, right? These products just evolved like an organism that evolved this new hand. And that hand is really adaptive for the adaptive environment, which is what's good at capturing attention. We just invented this new persuasive thing. But what could the consequences of that be? And how would you as an engineer or designer be thinking about that? You might say, well, we're helping the world because now people know who's following them. And now people know that they can connect with certain people that are interested in their interests, and you can see the list of who follows you. And there's all sorts of benefits, positive things that could come from that. And when you're a human being living inside of the eyes and mind and beliefs of somebody who makes Instagram or Twitter, You're thinking about it in terms of those positive things because that's what you're trying to do. But how would you know that it might cause a whole bunch of other externalities? Because right now the number of people who define their self-worth based on the number of followers they have— David Brooks wrote a book called The Road to Character, and he talks about the World Values Survey and how people value different things over time. And one of the big things that's changed is that people went from valuing fame as number 18 on the list to valuing fame as number 1 on the list. So, and that's in the last, sorry, that's in the last, I think, 10 years or something like that. And I would argue that the reason that literally billions of people now value fame higher on the list is actually because of little tiny seemingly innocuous things like putting the number of followers that you have in the core of our software interfaces, right? So we have this natural situation where everyone's competing for attention, and we are evolving these systems to get better and better at extracting human attention. And then it gets really competitive, and we have to add something else. So what do we add? We add AI. So now instead of just offering a product to you, I actually have to predict with big amounts of data and machine intelligence what's going to keep you on the screen. Instead of just offering some stuff you can click on, I'm going to actually predict from millions of things I could show you, if I'm YouTube, what's the video that I can put in front of you. If I'm Tinder, I'm going to pick from millions of people I could show you, what's the perfect reason to not be with the person that you're with. If I'm Facebook, I could say, of the million things I could show you in the news today, what's the perfect thing It's going to get you to like, click, or share. Okay? So you have an AI that's basically been given this goal. Now, I want you to put this in context. When we think about AI, we have to remember, you know, when you point the AI system at chess, right, first it kind of wiggles around and it makes some kind of funny-looking moves. And then it starts making some smart moves as it gets better. And then it makes some surprisingly smart moves, and then it beats Garry Kasparov. And when it beats Garry Kasparov, it doesn't unbeat Garry Kasparov. It's now better than all human beings at chess, right? So you take that same AI and you point it at the Go game. And that took 30 years. It made these funny-looking moves. And now with AlphaGo, with Google's AlphaGo, it beat all of Go players. And when it beats all of Go players, it doesn't unbeat all of Go players. So we built these AIs, and then we actually invisibly gave it a new target. We pointed it at this, and we said whatever gets this human being, play chess against this human being's mind, and play 20 steps ahead of where their mind could even possibly see, and show stuff to them that is either that perfect next video, on YouTube, the perfect outrageous news story on Facebook, which controls what 2 billion people will think every day, the perfect reason for you to cheat on your spouse with Tinder, the perfect political message. So we can vary 60,000 political messages, and we can actually combine word choices and different contortions of politicians' faces and colors of buttons to perfectly animate a response. From your brainstem? Because we're playing chess against ourselves. And we have every click that we give this system, it gets stronger, right? Every time you click or you share, you're feeding it attention, which feeds it more dollars, which feeds it more resources, which feeds it more computing power, which means it's better at playing chess on your mind. The reason this should be alarming is that these systems are not neutral. We have a tendency to say, you know, we've always had computers, smart— I mean, computers, video games, radio, TV. We always worry about them. So why should we be so concerned this time? And there's a few different reasons that this is so different. And the biggest one is this AI enhancement that I just mentioned. Couple others are that no other medium could pull on your social psychology. So no other medium could show you an infinite set of reasons of why other people are living better lives than you are, an infinite set of reasons for why you should feel like you're missing out, other people are having fun without you, an infinite set of reasons why you owe people responses. You didn't open your TV and it said, you know what, you owe like 100 people responses, you better start getting back to them. Right? So we're doing such enormous, I don't wanna say harm or damage, but we're doing, there is so much that is now put on the human evolutionary animal, right? More than we've ever put, and especially when you add in AI. So now the question becomes, what is the goal of that AI? What does it actually want? So if this is not a neutral product, if this is not just sitting here, but it actually wants something from me, well, if you're Mark Zuckerberg, you think the thing you programmed the AI to do is to make the world more open and connected, which you translate into, let's engage people. Let's show people whatever engages them. And that's going to be very persuasive to someone who's living inside of that mind. But we have this problem because the actual thing that this is all based on is attention. It's the same contradiction that I felt. And we now have this system that's, you know, we always talk, I don't know how much you talk about it here at Stanford, but there's all of this discussion about runaway AI. What if in the future we were to build a runaway AI, like a paperclip maximizer, and we give it this goal to make paperclips, and it turns the whole world inside out just to create paperclips? And what if that would happen and how would we make sure that wouldn't happen? So there's all these people working on AI safety. And the amazing thing is that this basically already happened. Because we already built a runaway AI that's steering what 2 billion people's thoughts are, and we hid it from society by calling it something else. We played a magic trick on the human mind. Because if you call it a news feed, or you call it YouTube recommended videos, or you call it Tinder recommendations, people won't even notice. That's how easy it is to fool the human mind. So, let's see, where do I want to go? What we're really dealing with is we have systems that have exponential impact. Right? Facebook just disclosed in the hearings today and yesterday that there are more than 5 million advertisers on Facebook.

Okay?

And that means that there are also potentially hundreds or thousands or millions of campaigns per advertiser, so you get this combinatoric explosion. And so if you have China or North Korea or Russia or someone else who's trying to spread manipulative advertising, not even for a politician but just for divisive issues or conspiracy theories or lies. The problem is we've created an exponential system of persuasion without exponential guidance or exponential ethics to control it, to govern it. And so where we find ourselves is basically the situation that many of the people who worry about runaway AI have been talking about forever. And that's the situation where we're at. And I don't mean to depress you, but that's actually where we are, like, right now, like, today. And the Congress is just now waking up to basically the reality of what this system has the power to do. And the cat's out of the bag. Now, you could say, well, hold on a sec. When we talk about these future scenarios, where if you have runaway AI, there's always this discussion of, like, well, let's put it inside of an air-gapped computer. And it won't, like, get out, right? Or we can always pull the plug. Let's just shut the thing down. We can always pull the plug. It's not going to be a big problem because we can always just turn the computer off if we have this runaway AI in the future. But the problem is, if you think about what that would be today, that would basically be like turning the lights off at Facebook. Now, it's not as if, from an existentialism perspective, as a human being you can't, like, get out of your chair and walk out of the room in space and time and tell the board of directors, you know, we're going to shut this thing off. You could physically do that, but there'd be consequences. And we can't turn off this system that we've created. So the only way to solve a system that's run away pursuing its own goals is not to manage your relationship to it. I'm saying this because I want to contextualize. A lot of people think of, if you've read any of the work that I've done, that it's about we got to better manage our relationship to our phones. Let's be more mindful with our phones. That's like saying when chess is kicking your butt, let's manage our relationship to the chess that's just totally overpowering where we are. So the only way to solve this problem is actually to put the AI on the same side of the table as us. We have to be really honest about what the lines of power are. So right now, the thing is pointed at us. It's extractive. It basically says, "I need to extract as much attention out of you as possible." And the classic line is it's a race to the bottom of the brainstem. So it's not enough that I offer you the product, I have to reach deeper down into the brainstem and create an unconscious habit so now you actually pull for the phone more often. That's not enough, I have to reach even deeper down to the brainstem and own your social psychology, so the way that Snapchat basically owns people's social relationships. That's not enough, I have to reach deeper down to the brainstem and get to your self-worth and control your sense of how often people like your self-worth. So I told you I'd go dark, I apologize. So we have this extractive economy, and if you'll notice, it doesn't look that different from how the broader economy is structured when it's extractive. It's the same as the environment. We have a runaway system that makes more money the more you extract without putting something back in balance. So the fundamental situation here is we need to find a more ergonomic relationship with the boundaries of our architecture in the case of how it's extracting us. And if you don't care about us, let's talk about our kids because this thing is just— eating kids alive for breakfast. So we can do that, but I think that what this is about is just like the point where we were where we realized after we extracted coal and we created this incredible economic prosperity after generating energy with coal, but we also polluted the external environment. And no one really saw that at the time. Good-intentioned people created this bad stuff, and we just realized it later. And we're at that point where we're just realizing this extractive attention economy where we're capturing people's attention. Is basically polluting the external environment. It's also polluting the inner environment. And now we need to invent solar and green, basically, solutions, like things that are regenerative or ergonomic or replenishing to the human being. So you can make all the same metaphors of the environment, right? And that's gonna mean that things don't grow the same way. One of the reasons why advertising is such a great business model is because it grows incredibly fast and is super profitable. I mean, who wants Facebook to be regulated when the stock price is through the roof right now? But also, who wants the changes that we're going to need to make for climate change to be abated, given that we're going to have to make huge sacrifices? And so the conversation actually comes back to values. It comes back to, am I willing to value something else besides money? Am I willing to value something else besides status? Now, this is an open question when it comes to corporations, who basically, they don't have control or choice. We created these robots called corporations that are just maximizing profit, and they have to. There's no way they can't do that. So we have to acknowledge when that misalignment exists, and it's the same for our own lives. Just because I can start a company and my friends, I have friends personally who've sold a company for $1 billion. Now that's very seductive. We can be pulled, it's a magic trick. We can be pulled by basically other people's success or incredible amounts of money. Right? But the conversation always comes back to, what do we care about? What is this for? What is the problem that this technology is the solution to? Why am I doing this? And if we ask those questions, that's really the essence of ethics. I have this fancy title of having been the former Google Design Ethicist. I will tell you, I didn't study that much philosophy at Stanford. I think the deepest ethical question you can ask is simply asking questions and asking why. And so with that, I will open it up for some questions. So thank you. Okay.

Sure.

So, Kristal, this is really fascinating and really important. Are there examples of companies that are doing this well? Doing this well? Yeah, are there examples of companies that are doing this well? You know, I think that, yeah, so we can get into the theory of change. There is a whole bunch of companies whose business models are aligned with us. Anyone you're paying, right, the money is going directly, you're paying the person that's serving you. So, you know, we actually on Time Well Spent's website, we have a bunch of different products that we basically show you pay for the product and they basically help you. Calendly helps you schedule meetings efficiently. Basically name any product that you pay, it delivers value, and you pay on a monthly basis. So anything like that. I don't want to give, for some reason these examples are sort of fleeting my mind, but there's all sorts of products that are like that. I think the main thing we have to do is recognize that the advertising supporting products, are the ones whose incentive is to extract from you. And it's always hard to acknowledge that because we get also benefits from advertising. We get the fact that we enjoy getting an ad for Nike shoes that we really wanted to buy. But the problem isn't the ad, it's the misalignment of incentives. So yeah. And what if we paid for Facebook, for example? But question in the back.

[INAUDIBLE] So I see your talk as a magic trick in itself. And there's the first part where you present something ordinary, this humble CEO of a startup in Silicon Valley, and then you have a turn which is, and here's what's behind the curtains and the thought club and your journey of discovering the extraordinary on top of the ordinary. But then the third and most difficult part of any magic trick is the prestige, is how do you get it back? How do you come— how do you bring this extraordinariness into the ordinary? And so, and your magic trick was about your journey and your insights and about how this journey allows you to create kind of magic through design or through whatever it is you're doing. But my question is, is it the purpose of this to create people that believe in magic or is it to create more magicians? Because what I don't understand is that we're spending all this time trying to create the perfect system, the perfect Gmail, the perfect platform, which is in itself a magic trick so that people can continue using it so that people believe in its magic and adequacy. But my question is why don't we spend more time creating magicians and decentralizing design or allowing people to design their own experience in different contexts?

Well, so I think you're taking— I appreciate the question. I think you're taking the magic metaphor further than I was intending. Yeah, it's not about the magic of ethics or values or these kinds of things. I think the point of the magic frame is that it exposes the fact that we're inside of a system that can be pulled or drawn. It's more delicate or vulnerable or manipulable than any of us would like to admit. And right now, all of our design institutions and our metrics don't account for that model of human nature. They're governed by simpler ideas like, we're giving people what they want, or if you clicked it, that's what you wanted, which is very persuasive when you tell someone that, but then you could imagine if you said, well, are there any cases where someone clicks something and it's not what they want? That they were outraged, they were upset, they were that kind of thing. Is there another question? All right, student? Yeah.

Yeah, so you said that these features are implemented by companies who compete against each other, but for instance, if you were to tell one of them to just remove, like for instance, how would you persuade a company a company to remove one of these time-consuming features without them jeopardizing their competitiveness in the market? So how would you tell Snapchat to remove Snapchat streaks? How would you do that?

[Speaker] Yeah, so the question is really, in the attention economy, where everyone, doesn't even matter if you're building Snapchat or you're building a meditation app, even meditation apps need your attention. You have to figure out how to put a habit inside of your body so that you use it every day. How do you tell any company, not to get people's attention, or to subtract some of these manipulative features from their products, is your question. So the answer is you cannot obviously get any person whose business model is to maximize how much attention they get from you to not do that. You can't tell YouTube to not try to show people the next videos. You can't get Snapchat to now subtract the streaks feature. So the way to do it is you have to go up a level. And you go up a level by going to the device and the platform. So in other words, we can't actually ask Facebook to do something against its business model, but you can ask Apple, Microsoft, or Amazon, or Samsung, who are companies whose business models set up the choice architecture, which is the interface between you and this army of things that wants your attention. And so right now that choice architecture, because we have this model of human nature that says, people are clicking or choosing freely, is just Wild West, right? Like basically when you get a phone, it's like Apple and Google give totally open door access of all those apps to just reach into you and do all the stuff that they want to do. So there's a bunch of ways they can clean up and mediate that relationship. So it's like if we're jacking people into the matrix, do we want to jack their impulses, or do we want to jack basically the top part of their reflective minds? And there's ways of designing the choice architecture on a phone, like a home screen or notifications, that are basically asking the most reflective part of ourselves what we want. And to make that concrete, that's the theory, the concrete version are things like conversational interfaces. So Siri, the AirPods, watches, things that have supplemental interactions that are peripheral interactions in which you basically are doing something with the device but you're not giving people a bottomless bowl of things to scroll through and explore in an infinite world, which is what they're all based on now. So that's one theory of change is the platforms, and the other one is governments. We can talk about that more later. Yeah. So that sounds like you're talking about further up the chain. I'm just wondering on the other side, if you move down the chain, I guess I'm wondering if you're seeing culturally an acceptance that this is a real problem.

Is it widespread, or is this just a small group that feels this way today?

And is that important to driving change? Yeah, so is it important for driving change? So is there a large group of people who agree or see the problem, and is that important for driving change? I'll say a couple things on that. One is, I felt so nervous making that first presentation at Google. I thought, I was like, man, I can't be, I honestly thought if this problem existed, someone else would have thought of this and said it already. And I felt really nervous putting the idea out there, because I was basically saying humans don't choose freely, here's all these ways that we don't. And I felt uncomfortable saying that. I felt very vulnerable. And it actually has been this process over 4 years of being repeatedly validated that people understand that you really— this really does happen. This really does work this way. That I felt more comfortable stepping into it. And I'll say since then, the acceptance from more and more ex-alumni of the big companies have come out, at least privately to me, in agreement. And so basically, I was on 60 Minutes earlier this year with Anderson Cooper talking about this problem. We call it brain hacking. and it was mostly the addiction part of this problem. And since that interview, Mark Zuckerberg's personal mentor Roger McNamee, who actually convinced Mark not to sell the company to, was it Yahoo or Microsoft for a billion dollars, and now feels incredibly conflicted about his role in creating Facebook, has actually partnered up with me and we've been doing all this great work together. And then there's been more and more people who've been coming out of the woodwork because they understand this is actually what the situation we're in is. So I would actually say if you don't see the situation this way, and for someone who doesn't and they're in the industry and they actually work on these products, what's happening is more of the Upton Sinclair can get people to question the thing that they're doing thing. And we always want to look for the positive. So, yeah.

Maybe you can add a perspective on this.

Is it true that the executives of a lot of the companies send their kids to schools where there's a ban on Yes, the question was, is it true that— are we okay? Is it true that the executives send their schools to places where they don't have phones, there's limits? The answer is yes. And I think that's always a really telling signal, is when the CEOs of companies don't feed their own kids with whatever it is that they're making. It's an extension of the golden rule. It's like, don't just do unto others what they would do to you, do unto others what you would do to your own children. And I know Steve Jobs limited his kids' phone and tablet access. I think Sheryl Sandberg doesn't let her kids use social media, as I understand it. I know there's people at Apple that are very high up that send their kids to Montessori schools, and I think Google is very similar. And I'll just say, in parallel, when the CEO of Lunchables food line, which is a billion-dollar food product, would not let his own children eat Lunchables. And you know that there's a problem when that's true. So, women, yeah.

So a lot of your talk was like the company has this responsibility of ethical behavior, but what can we as consumers do? Like what do you suggest we do if we don't want to have our minds being pulled at all the time?

So what should we as consumers do if we don't want our minds to be pulled at? The first thing to recognize, and this took me a long time to kind of get across, is I interviewed all sorts of people who were like the best experts in the field on persuasion and behavioral economics and these magicians. And I had lunch with, years ago when I started working on this, with Danny Kahneman, who was actually one of the founders of the field of behavioral economics. He's the Nobel Prize winner in behavioral economics, which is how we can kind of fool people. And in his book, Thinking Fast and Slow, he says, look, even though I know how all these techniques work, and how these cognitive biases work, it still works on me. Now the important part to realize that it's like, okay, if you told someone I'm going to give you access to know what your DNA looks like, then you don't think, well, then now I can change my hair color from red to something else, right? So I think the challenge is that we really are inside of this experience. We're really inside of cognitive biases. If I did that Africa number of countries thing and I anchor you on one number versus another number, Whether you want to or not, there's all of these invisible influences that are guiding us. And so the question becomes, how do you ethically use those forces? And then again, for good and for what values? In terms of being concrete, what do you do? You can turn off all notifications on your phone except for when a person wants your attention. So one of the big things in the industry is just about everything that's a notification that comes at you is actually generated by a machine. And the machine's goal is usually like what will get your attention right now. And so it's automatically sending you stuff. If you ever, if you tried the following, you don't use Facebook for like a week, and you watch, suddenly they send you like 3,000 emails. It's like a drug dealer who wants you to come back. So I'm not trying to be, I hope when I say this I don't sound like I'm just trying to be against something. It's more that there's this model that's just not aligned with all of us, right? I don't think anyone wants a world where this is going. That's hopefully what I communicated. It's like, this is so dangerous where we're headed. And I want to say one more thing about what danger that I didn't mention. I think the deepest existential risk from this is that the thing that's best at capturing a human being's attention, a single person's attention, is going to be to show them an individual reality that confirms their worldviews. That's going to be so different from the thing that would confirm our shared sense of reality. So you can think of Facebook innocently, like they gave it this innocuous, naive goal of like, let's help people, let's engage people the most. And then that thing is taking entire societies and putting them through a paper shredder, where out the other end you get basically filter bubbles and echo chambers where people don't have shared facts and agree on the same reality. That's like, I think, the deepest part of this problem. So if you weren't convinced by the kids threat or the national security side, I think that is a sort of step one for good society. You.

So from a design ethicist perspective, I was wondering if you could give an example of like maybe one of these persuasive techniques, and how do you even begin to apply an ethical framework to it given the scale at which it's deployed and the diversity?

[Speaker] Okay, so the question was, if you take a persuasive technique, how would you apply an ethical framework for deploying it? Okay, so let's take, I love this example. So let's take streaks in Snapchat. Sorry if you've heard me talk about this, but it's really important because if you didn't know, Snapchat's the number, how many of the people How many students here use Snapchat as your primary? Okay, so a good number, okay. So, and it's obviously invented by a Stanford alum who has good intentions. But they use this feature, which is a technique, is a persuasive design technique called streaks. If you are older in the audience, it shows the number of days in a row that you've sent a message back and forth with every contact. So why is that? Is that ethical or not? I wanna actually hear, what do you think? If I put the number of streaks in a row, the number of days in a row that two kids have sent a message back and forth, What makes that, is that wrong, is that good, is it bad, why?

I'm trying to think of like from a utilitarian perspective, if the ad increases your quality of life and helps you feel more connected with your friends.

Okay, so if it increases your quality of life and helps you feel more connected with your friends, then that would be one reason why we say it's good. So let me tell you actually what the people at Snapchat, when they made this feature, what I've heard through the grapevine is the way that they justified this was that unlike these other social apps that connect you to all of these friends all at once, you have all your relationships all at once, they help you focus on the relationships that really matter because they show you the depth of your friendships. Do you see what I'm talking about? Again, you have a belief and a narrative and you're stepping inside of this belief where you see through these rose-colored glasses where this is deepening friendships. According to the human being that believes those thoughts, they're inside of a magic trick and they believe that this is actually about deepening friendships. And there's definitely evidence for that. I'm sure that there's someone somewhere, two kids who feel like, man, my friendship is way deeper because I can see the number of days in a row I've sent a message back and forth. There's someone who's experienced that. Just like there's magic moments on all these other applications, but there's still something that's not quite right about it. And I'll just jump to the chase. With SnapTreaks, there's first of all an asymmetric situation. So the children don't know that there's like 1,000 or 100 engineers on the other side of the screen. And that they're deliberately choosing this technique against them. So that's one thing. So there's an asymmetry between the knowledge that the persuaders have and the knowledge the persuadee has, okay? Second one is that the, just had it, the goals of the persuader are different than the goals of the persuadee. So when the people who make Snapchat do this, their actual goal is how can I hook you to use the product every day? Cuz that works really well, like the streaks feature is super addictive. That's their goal, and the goal of the person who's using it, usually the challenge with ethical persuasion is people don't know their own goals. So they actually just sit there, and then the persuader's goal infects the persuadee. So now the persuadee is like The Matrix. There's a drilled hole in the back, and this goal went in of like, now I need to keep up with these streaks. And now they actually want that. That's something that they intuitively, independently want. That's successful advertising. The persuader's goal has become the persuadee's goal. And now they define their friendship based on whether or not they're able to keep up their streaks. And if they don't have their streak, they're no longer best friends. And by the way, that's actually true. So there are children walking around who think that their terms of their friendship are the streaks that they have. But I want to just do one more thing here. It's a great example. I'm so happy you asked me that. You could do the same thing, a streak feature, on a meditation app. Same feature. Now we're putting on a meditation app. The number of days in a row you've meditated. So why is it feeling— like intuitively it feels better there, right? So why is that? And I'll just say quickly, it's because theoretically the person who's trying to meditate actually cares. Their goal is aligned with the persuader's goal. They actually want to sort of mark off the number of days in a row that they've done the thing that they're yearning for. Whereas in the first case, that's not true. So there's a whole field here. We should write like a 600-page book about it, but that's a great question. Yeah.

So we were recently at a group visit to Google Ads, and they were talking about YouTube. And I was saying how YouTube's changed. I'm spending so much more time, but I'm seeing all of the like huge videos. And my experience on YouTube is more frustrating now.

More frustrating for more time.

I used to look at the most viewed globally, and it would be such an interesting mix of news and viral videos and other things. But now it's optimized to take most of my time. The response that I got was, yes, but you're seeing more advertisements and spending more time on site. So essentially, we're doing our job. But I now dislike YouTube when I used to like YouTube. So is there a way for us to capture our actual attitudes towards the service being provided versus just the time being spent on site? And do you think there's a role for pointing to the flaws of measuring just on ad view or click-throughs? Because I'm very hesitant to buy anything that I see. I'm not sure that that's functioning.

Well, so the challenge with advertising is they make money whether or not you buy it or not, because they still make money from just the impressions for seeding the ideas into your mind. That's still a a success case for advertisers. And I know that we're wrapping up. No, no, no, finish please. But the challenge here is there are the metrics. My first TED Talk went through some of them. The challenge is that any business model, any business or company, technology company, whose business is to capture attention, they just have to do that. Like, there's just no other choice. YouTube just has to do that. But if you look at Vimeo as a counterexample, they don't autoplay the next video. They don't show you the related videos and try to get you into an infinite world. And it's because their business model's a little bit different. And so you can imagine a version where you pay, where these choice architectures are not trying to bottomless bowl you into infinite watch time. So, yeah.