Crowds, Computation, and the Creation of a Globally-Networked Mind with Michael Bernstein

Thank you for joining. Yes, this is my 10th reunion. So it's a little surprising that I'm actually up here with this label on my name. I still can't quite believe it, but I rejoined Stanford in 2013. as a professor. After graduating, I went off to MIT, did grad school, interviewed back here. I was lucky enough that they gave me a job. And this, what I'm gonna talk about today is gonna be sort of a mix of what I would say maybe the TEDx version of what's going on in my world. So this is gonna be a little bit of what's the phenomenon that's interesting and why am I so excited about it? And also, what are we doing here at Stanford about it? So I'm actually gonna start out with some participation. I know they just told you to turn off your phones, but I'm gonna tell you to turn it back on and go to this URL. It's a Google Form. We're gonna see how wise this crowd is. If you don't have a phone or a computer, I'm not sure why you would bring a computer other than me to reunion, unless you're a nerd like me. I'm not a nerd. We're asking you to guess how many minutes it takes to fly from, I believe, what did I say, Phoenix to Detroit. So that is like the time printed on your ticket. How many minutes would it take? Why am I asking you to do this? We'll come back to that in a minute. But for the moment, just see how close you think you can get. No Googling allowed, by the way. It says on the form, I think it says, can you read it? It's fee— No service. No service, oh gosh. Boom, all right, well then maybe we won't do that. Can you give us your Wi-Fi password? Yeah, no, is the— Stanford visitor. There is a Stanford visitor Wi-Fi. I'll put it over here if you wanna come back. It still requires a password. Really? It's asking for a Google password? I got in, okay. All right, so some people are working. Okay, well you have an entire 20 minutes or so to get this to work. Don't worry, even if it doesn't, the rest will still happen. Okay, so yeah, Stanford Visitor Wi-Fi should give you access. That's new, that's, welcome back to Stanford. Okay, so So, as I mentioned, I'm in the computer science department. And as a computer scientist, I reflect on the relationship between people and technology. And really, from my point of view, this relationship has been pretty static for the last few years. That is, let's say you have an idea, or maybe you're working for someone or with someone who has an idea. And you wanna get something done. What you do is you turn to this computer and you fire it up. Maybe you turn on Excel or whatever you're doing, and the computer accelerates the doing of that work. The computer is sort of an object that helps you accomplish whatever work you're trying to do. But this relationship is shifting. The computer is now no longer the thing that we're just doing the work with. It's actually connecting us to other people who are also trying to do work or get work done. And if you wanna see any evidence of this, look at things like Uber, like Lyft. You might have ordered food through some of these services recently. You may have heard of platforms such as Amazon's Mechanical Turk, is connecting people who want to make money with people who want to pay money to get stuff done. This is a vastly different relationship. This computer is mediating labor, it's mediating work and And it's mediating volunteerism, right? So, so far what we've done as computer scientists given this huge opportunity is, well, we've sort of let them talk. And don't get me wrong, this is actually incredibly powerful. It's led to the largest encyclopedia in the history of mankind, right? We can through wikis and through chats and all these kinds of things get a lot done. But there's also a lot that's left unsaid here. There's an entire field Elsewhere on campus, known as organizational behavior, they study how people and organizations work. And they will tell you that we are really, really bad at organizing ourselves to get this kind of stuff done. Teams are very, very poor at, for example, knowing how to integrate work. We're great at splitting up work, saying you do this, you do this, you do this. We're really bad at integrating it again later. Empirically, we've seen that most teams, this feels like the high school problem. Most teams get almost all their work done in the second 50% of the time they're allotted. So they're sort of just churning around for the first 50%. Second 50%, sprint and getting it all done. There are a lot of errors that we make, just as we know in cognitive psychology that people have judgments and heuristics that they make that are not always the right fit. Maybe you saw Mehran Sahami talk yesterday about somehow how we can use statistics better. So too are we really, really poor at making judgments about how to organize ourselves. The things we think will work won't. But here I see an opportunity, because all of a sudden we have a computer mediating this relationship. We can actually design that layer, right? Just like I could design Excel better to make your doing of your work more fluent, we can design that computational layer in the work to make it more effective, to help us get to come together to do things we couldn't do before, to really work on a massive new scale. Putting that into words, I'm really curious about this question. How could computing connect really, really large groups of people to tackle bigger and harder problems than we could do today? Now let me give you a few examples just to sort of warm you up, and then we'll get into some of the meat. So we can build classifiers. These are sort of machine learning classifiers that integrate human judgments. All right, so this is gonna be a very audience participation-y kind of class. All right, so I'm gonna show you 2 videos. These were both from YouTube. YouTube had a meme a little while ago where teenagers mostly were telling 2 truths and a lie. And I'm gonna ask you to guess which of these is the truth and which one's the lie. Can we pull up the audio a little bit? We're gonna watch 2 about 5-second clips. Okay, and which one is he lying? Here's the first one. I broke my hand beating someone up cuz I punched him so many times in the face. That's great. All right, second one. I've stolen things from a store. All right, so if you think the left, raise your hand. That he's lying? Okay, which one is true? The left, raise your hand. If you think he's telling the truth on the right, raise your hand. Okay. Oh. So we're really bad at this. People in general are really bad at this. But, and a major reason for that is that while we're good at noting potentially informative features, so to speak, we're very poor at integrating that evidence. We don't know how to weight things. But it turns out computers are really good at this. So we can go online and build sort of an aggregate human lie detector that's way more effective than any of us would be in isolation. Even taking a vote, we would have gotten this wrong. But what we can do is get potentially informative features. Oops, it looks like the right side of the screen is getting clipped off. We use short sentences when we lie, we use few details when we lie. These are things that you might have noticed if you were comparing them, but you might have actually flipped the bit on which one was actually the lie. But again, we can train these things to be hugely more effective than we would be in isolation. So we can also think about new kinds of platforms for crowdsourcing. So you have your phone, how many times do you think you check your phone a day? A lot. A lot, right? When you unlock your phone, what are you doing? It's basically like a little CAPTCHA, right? You're making sure that you didn't unintentionally unlock your phone. Well, that's a lot of time people spend. What if we actually overrode that unlock screen on your phone and used it to get information about the world around you? How many people are around you? Or we might ask, how much energy do you have, or what are you wearing, or how many people are around? These kinds of questions. And we can launch this kind of thing, this is out on the Android App Store, and get hundreds and thousands of people's unlocks over time and build essentially a Google Street View of where people are, what they're doing, all these things that robotic cars driving around can't get about the world around them. This is on the left, does anyone actually know where this is? Cambridge, Mass. Cambridge, Mass, that's right. Yes, we also have a number of, it's actually worldwide we have a number of these kinds of things. But you could see, when do the suits leave the downtown and get replaced by the t-shirts and so on? Okay, we can even talk about how computation can make people make judgments way more quickly than they might otherwise. So machine learning researchers essentially need to build huge labeled datasets. I collaborate with Fei-Fei Li in the AI lab. And they basically pay people to do this all day. Which of these has a man riding a motorcycle in it? Takes on average, I think, 2.5 seconds or so per image, or no, about 1.5 seconds or so per image. That's what this looks like. Okay, well, that's fine, but I don't know, I think it's kinda boring. So, turns out we can do way better. Hold up your hands. I'm about to ask you to clap. I'm gonna show this at about 100 milliseconds per image. When you see a man on a motorcycle, just clap, okay? And this is gonna loop, see how well you can do. Okay, this is a little difficult, right? You can stop if you want, all right. Let's go back. This is difficult, but as you notice, if we got enough of us around, we were delayed and we had error, but the error was sort of normally distributed. And if we monitor enough of us, we can actually get this right. In fact, we can get the same precision on this task as someone who's going sort of slowly and carefully at about an entire order of magnitude faster, essentially. So this thing that my machine learning colleagues are spending $1 million to do, I can do it for about $100K, right? Or if it took you $100, I can do it for $10, etc. All this comes under this heading of crowdsourcing, where we're taking an open call out to the internet usually, and we say, hey, help. And that's really the topic of today's talk. So crowdsourcing, sort of rewinding significantly, started out with this guy. So Sir Francis Galton. In 1906, I don't know whether he knew Leland Stanford, but he had really, really cool facial hair. That's essentially the— Yeah, that's right. So he was at essentially a county fair or state fair, think of it that way. And at this fair was, of course, because what else would you have, an ox. And Sir Francis Galton was was interested in this ox. He wanted to find out how much it weighed, okay? And of course, this ox, having an insufficiently developed Wernicke's area, wasn't really able to answer this question. But what he did is he asked everyone around him to make a guess. And he made a table of these guesses. And what he learned was that by taking the average, It was within 1% of the true value. Wow. And in fact, the average was more accurate than the single most single accurate individual, right? So the best person in the room was still less accurate than the average. So I started out by asking you to actually try this yourselves. Let's see what we got. Did we get any responses? Let's see here. All right. We got some. All right. So the true flight was 227 minutes. Here we can see what everyone said. All right. The sorted tab is not playing with me here. That's fine. I think it's because you're not sorting by time. What we can do is we can just take an average here of B2 to B30 and see what we get. Median? Oops, sorry. Sorry? You could take the median too, yeah, absolutely. Actually, Galton took the median, but either way it tends to work. What did we say? B2 to B30. To B30. Why is it disappearing? Equals. Ah. Was there— did I forget the equals? No, we got something else going on. I'll go to the source. Now we're on B2 again. 193. 193. All right. So who was the most accurate here? Let's actually sort this. So we said 227. So Jennifer and Corinne, you're the closest. And in this case, it looks like it's maybe slightly more accurate. Than our crowd. 10 minutes. Oh, Testy McTest, you're throwing us off here. 200. All right. So a little closer. We're about off by a half hour. All right. So we got a pretty wise crowd here. I don't know. Went to a decent school. So what happened was that computation started playing a role. This sort of, This was interesting, but it stuck around by itself for, I guess, about 100 years. And then computation came in and allowed us to essentially amplify these effects far beyond what we saw before. So computation can sort of scale it. It can help us do things at larger complexity. This was actually a friend of mine ran this study. I don't know if you can read this text. Okay, but I'm gonna show you, actually, can you figure out what the first word is? You. You, all right, what about the second word? Yeah, I don't know either. Okay, so what my friend did, Greg Little, this was part of his PhD thesis, was write an algorithm. And the algorithm was very simple, it was this sort of iterate and vote thing, where we'd have someone do whatever they could do, and they got, for example, that first word, you. And then they handed it off to someone else and said, okay, take that and try to improve it slightly. Then they took those and they had another set of people vote on it and say, okay, which of these do we think is better? And then they iterated further. And it was sort of like a big complicated game of telephone that was Turing complete. And what they ended up with was a nearly perfect transcription of what that messy text looked like. Wow. So next time you get a doctor's note you can't read, know that you can write an algorithm that will help you do it. So this is sort of all coming under a heading of what we call crowdwork. And I mentioned earlier this has to do with things like Amazon's Mechanical Turk platform, Upwork, freelancer.com, Uber, etc., DoorDash, whatever you would have. There are crowds of professionals and non-professionals online who are doing work for money. So all of a sudden, computer science has been thrust into this world Where it's deeply embedded in something that's very ethically laden, policy-ridden, and is clearly having an impact on how businesses are run. So this opportunity is immense, right? You could say, what if I could snap my fingers and bring together the smartest minds in my generation in 10 minutes, right? I have an idea, my startup is not gonna take 2 years to start, it's gonna start right now. And there are gonna be 30 of us, and we're gonna start. But of course, we have these perils, like what does it mean for an algorithm to be your boss, to make decisions about your promotion and your pay, right? We have to think very deeply about this. Let me start with some of the opportunities. One interesting application of this, our group said, well, what if we started— This is gonna be a turn-taking. We can actually mute this now. There we go. What if we started thinking about smarter user interfaces, right? What if, well, it takes forever to rewrite your text. What if we hired people on demand to make lots of little tiny tweaks and we wired them all together algorithmically so you were just using Microsoft Word, you could say, grab this text, make it shorter, and the crowd is sort of coming into your user interface and giving you a slider that you can move up and down and it'll make your text longer or shorter or whatever length you need. Likewise, imagine your text having a smart copy editor, Microsoft Word, just built straight into it. Or my colleague at Carnegie Mellon has been looking at interfaces for the blind. So you know that blind individuals are actually, these phones are great for most intents and purposes. They have wonderful accessibility features. But there are many things about the world around them that blind people can't actually see. For example, necessarily What is on this label, or what denomination is this bill? But imagine if you could now reach out on demand to people over the internet. So what this VizWiz application does is it allows blind individuals to take a picture of something in the world, speak a question about it, and within about, oh, a minute or so, get an answer. Is there a bench across the way? Should I bother going? So huge benefits potentially for accessibility, and he's gone on and done things that have had crowd workers go and transcribe audio in real time without actually them knowing how to be stenographers, etc. We can even do this really in real time. What I'm about to show you is a camera that doesn't take a picture, it takes a short video. And then we bring in these crowd workers on demand, and in about 10 seconds they go from a video to the best moment in that photo. So here's the video. We're gonna capture it, and what you're gonna see is these crowd workers jump in, and they're gonna be sort of scrubbing around like you would on YouTube. We call them right now, and within 2 seconds, they arrive. So 2 seconds to bring someone in, and they start scrubbing around. We do a sort of binary search, and we find our way down to that, which is the moment that you would keep on your phone. So from beginning to end, we can do this sort of real-time, 10-second in and out. Crowdsourcing, which allows us to do all sorts of things. So you're taking pictures during reunion weekend, you're trying different angles, you're potentially trying different poses, maybe action shots, people jumping off of a bench or something like that, or just being silly. And again, within just seconds, you have on-demand human intelligence wired into your phone, right? That's a different world. But let's push this even further. What if we could create these work collectives that were not just doing things that, in that case, nearly anyone could participate? It doesn't matter what kind of expertise you had. But what if we were really focusing on the ability to create expert work? So I teach in the Human-Computer Interaction program here, which means we largely teach students how to apply design thinking to computers and computer science. So we design interactive technology more effectively. And so there is this thing that we do in this class, like a project, it takes 10 weeks. And I said, okay, sure, how about 1 day? Could we crowdsource the creation of a user-tested, iterated mobile application in a day? The answer is no. Until, okay, so basically everything that we've been talking about so far has been using microtask workers. Microtasks are these small, tiny tasks that people are getting paid sort of piecework rates in order to do. But why do we need to limit our imagination to this? Upwork is just one example of a platform where through API access you can find programmers, designers, I found opera singers on there. We could then think about using computation to draw these people on demand together, scaffold a process that they're following, and for example, take in some input like, These are sketches that happen in maybe the second week of my course. In fact, every example I show here I may or may not have taken from an undergraduate project in my class. And then put it through one of these what we call flash teams and produce a working user-tested app within a day. So these workflows, as you might call them, look something like this where we're sort of unwrapping the user-centered design process. People are going through an expert evaluation, they're revising these lo-fi mockups. There's software engineering, they go do a user test, they revise the software engineering, drawing on demand just the experts they need at just the moment that they need them. So because these things are sort of modularized teams, if I have one such design team, it would be really straightforward for me to create 3 or 10 or as many as I need. We can just draw on demand from this large group of professionals. So if I have 2 different sketches, I'll get 2 different working applications. A really much more esoteric quality of these is that you saw each of these tasks were sort of abstracted into individual blocks. And those blocks each have little inputs and outputs. So you give me a sketch, I'll give you a mock. If you give me a mock, I'll produce a heuristic evaluation of it. You give me a heuristic evaluation and the mock, I'll revise the mock, et cetera. So maybe you have a sketch, but you don't want a mock. that working prototype. You like video prototyping these days. You want some animation, you wanna put out your Kickstarter and, I don't know, get lots of interest. Now we might not have such a team that's ever existed that does this, no modularized predefined team. But these little blocks in between could conceptually be woven together to produce such a team, right? The first few elements of this one, then we might bridge it over here to someone who could write a script, etc. And what we can do is take all of these blocks and use some really old-school artificial intelligence techniques known as action planning languages that then allow us to basically synthesize a team that's never before existed, but does exactly what we want. So again, each of these flash teams never knew each other, have never met before, but come together and run this sort of scaffolded computational process to very quickly achieve what they're trying to do. Now there aren't very many project management approaches that really work with this. Basecamp isn't exactly ready for what we want, Asana not really, so we've been building our own called Foundry. Foundry is essentially this online application that allows people to author these flash teams, like you could create anything that you want, or it turns around and becomes sort of like a runtime manager. So just to give you evidence that this thing all pulls together and works, here are a few examples of the kinds of applications we've created. Now you're not gonna launch these on the App Store tomorrow, but imagine as a designer you have an idea today. Being able to come back tomorrow at this time and actually play with it and interact with it really tightens that loop for you. Now, but of course, this is something like I know how to do. How about something I'm terrible at? Let's say making videos or animations. So we worked with a director off of this Upwork marketplace to produce another flash team that would essentially take a prompt for a script and then produce an animation in about 2 days. So you can see that now we're dealing not with engineers, but with people who are animators, sound engineers, scriptwriters, illustrators, et cetera. And in particular, We needed to give them some input to work with. I don't know if any of you ever interacted with my colleague Terry Winograd. Yeah, so Terry was sort of the founder of the HCI group here at Stanford. He was actually my undergraduate advisor. He's retired now. He was retiring at the time we were doing this work. So I said, okay, let's make an animation celebrating his career. And in particular, he tells a story of building a computer when he was just a kid. So if you haven't seen Terry, he looks somewhat Einsteinian. And what I'm about to show you is what this flash team created in about 2 days from nothing but this description. So that's Terry, a young Terry. Good morning, son. Oh, hey, Dad. What the— Harry's dad looks like Harry's dad. Look at that. It's the future. It works. Hmm. The arms always get me. So I'm still awaiting my Emmy. But I want to just step back and say, these people, again, never met before. And we're able to produce something like this in 2 days. A really, from my take, pretty incredible accomplishment. So if we can create these little teams, you could start to think about what if we go bigger? What might it look like if we had entire organizations that were drawn on demand from the crowd, just the experts you need, just when they need them, could reconfigure themselves into new forms as they face new issues, could prototype different kinds of organizational forms, could perform organizational data science to figure out what's working and what's not. It's a very different model, right? And it may not be exactly what, say, Google or Microsoft or Apple want to do tomorrow, but it's offering a very different vision of how labor might work in the future. So if we're building these kinds of new production models, what does that mean for the rest of us? We need to think about what's that future of education and lifelong learning? One issue would be, well, how do you learn new skills, right? If I'm working for Apple today and for Microsoft tomorrow and for my research on Wednesday, how do I learn new things? One concept might be to say, well, we have tasks that are already out on this marketplace. What if we actually took advantage of them and sort of paired you with someone who had more experience in that space? So now you're an intern essentially, but an intern for like 2 days, and they're a mentor for about 2 days, and you can, work together through this platform that might look something like this that we've been building called Atelier. And so what would happen is that someone comes in and they post a job, like you want, maybe Chris wants to post a new application to get prototyped. Okay, well that gets connected with a mentor, maybe Yoyo knows something about that, and then I come in as the intern and we're gonna work together. So I'm gonna get mentored in a very tight loop while I'm working on this. And then ultimately, the mentor says, okay, this is good, and passes it back. So now you have this sort of distributed mentorship model where I'm able to pick off tasks that I might not otherwise be qualified for, but work with them in order to produce something interesting and creative. Okay, so maybe I can educate myself, but what about this history of collective action, right? This is an issue as well. These workers are online, they can't necessarily talk to each other, how do they push for changes to their work environment. We could start to think about what those might look like. Here's a platform, Dynamo, that we've been creating where crowd workers are coming together to really push, in this case, Amazon and Mechanical Turk to make changes that they need, right? What does it look like to actually come together and push for change? Think back to the last time you tried to come to a decision over email. Doesn't go well, right? So we need to think about how we would actually design these platforms to enable people to come together, figure out what they wanna do, and then actually take purposeful action, right? What's that future of labor action going to look like? In this case, Dynamo was able to do things like write public letter campaigns to the Amazon CEO Jeff Bezos. Usually the media says, poor turkers. They wanted to actually take ownership of their own narrative. And in fact they did, getting all sorts of coverage on this kind of thing. Notice the non-use of the word union here, you see things like Things like guilds. Guilds. There's an interesting interplay. Just as a moment of reflection, you may have heard that Uber drivers are really struggling with this question of how and when and whether to unionize. And what's really at the core of this, as we've seen from sort of the qualitative, more anthropological work we've been doing on this, is that there's an identity challenge where people view themselves as a freelancer. If you're on Uber, if you're on Mechanical Turk, you're doing this, you're in charge of your own destiny. I'm a freelancer. And yet, all of us freelancers are somehow also still under the thumb of this platform, whether it be Mechanical Turk or Uber or whatever. So you have this identity challenge where it's like, am I working for the man, in which case I should be unionizing, but no, I'm a freelancer, right? And I think this is really an identity issue that these groups are still working through. So I'm gonna pause sort of this chapter, or I'm gonna end this chapter, excuse me, What I hope you're taking away is that, first of all, there's incredible wisdom in crowds, certainly this crowd, and many others. But by combining design and computing, we can actually amplify this, we can guide it, we can help us collectively achieve more complex and interesting things than maybe we could do if we just tried to organize ourselves to do it. So that's sort of an existence proof of what's going to happen. But there's a big open question about how the heck do we even design these things? And the real challenge here is that as a designer, right, I teach at this intersection of computing and design. We know how to design interfaces for individuals, right? If you told me to design, I don't know, whatever, take a calendar app, it's relatively straightforward. We can understand what people's needs are and the context in which they're using it. generate a number of ideas, we can iterate, we can prototype, etc. The challenge here is that all of a sudden, I'm trying to design an interface for you that actually isn't about you. It's about everyone and how this all comes together collectively, right? And I mentioned at the beginning that we're really bad at thinking about this, right? We are not good at thinking about how to design, how do I influence you and you individually such that we as a collective all exit in an orderly fashion? For example, right? I can only touch individuals. So this is really actually the boundary between social psychology and computer science, right? And what this is, I'm gonna take us through a few examples of how I think interestingly in industry and in practice, we start to see these kinds of issues and how people are solving them. So let's start with an example. This is an entry from— This is an entry from the F My Life website, read it, and tell me what you would intuit about this individual. What do you, like, tell me about this person. There's like 2 sentences there, right? Who wrote it? Yeah, the person who wrote it. Short-tempered. Short-tempered, okay. A male. Male, all right. One more. Doesn't like football? No, he likes football. Oh yeah, different kind of tailgate, I think. Right, okay, these are all, I think, totally valid inferences. They are all, and I do this too, unfortunately, also an instance of what's known as the fundamental attribution error. Did any of you take social psych when you were an undergrad? Yeah, so this is gonna be a bit of a callback. So does anyone remember what the fundamental attribution error is? The tendency to— Sorry, say it louder. The tendency to attribute something on personality when it's actually caused by the situation. Yeah, exactly. I'm gonna repeat what you said because it was that smart. Basically, we have a tendency to blame an individual, right? The question is, when this happens, who's at fault? Is it me or is it something around me that caused it? And usually, I will blame the situation. So imagine that I run in late to this lecture. What do I believe? It's because, well, my son was a little bit hard to deal with this morning, and then there was traffic or something like that. Okay, everyone else is like, no, Michael, you are late. You are a late person, right? And so this often happens that we associate sort of observations with the person rather than with the situation that generated it. And what's crazy is that this doesn't even, This still happens even if we know it's forced. Now I'm gonna date this study because it involves Castro. So the study actually in psychology that was run that demonstrated this was that we took people in, and now imagine that I randomized and I said, you're gonna debate pro-Castro, you're gonna be anti-Castro, pro-Castro, anti-Castro. All right, we get up here and we have the debate and the rest of you are listening. Even though all of you know that I randomized it, You're still gonna believe that the pro-Castro debaters are pro-Castro, right? This is very hard for us to get around. Where does this happen in social computing systems? Here's an example off Facebook. So what has my friend Winter Mason done, right? If you use Facebook, what he's done is he's gone in and liked this movie by Jon Stewart called Rosewater. Now in my feed, I see this. Right? So what does it look like Winter Mason is actually endorsing? Hint, it says sponsored, right? It looks like he's endorsing this exact piece of content, which he's probably never seen, right? This is something that we are associating, like we assume, yes, Winter is actually endorsing this. sort of endorsing this when he didn't actually, he did none of that, right? Now he's not lying, he does like rosewater, and neither is Facebook quite lying, but the way we interpret it is a little bit interesting and almost insidious. So here's another example. Actually for this, I'm gonna need to wrap around there. Can you hold this for one sec? Thank you, okay. So here's a second example. Can you hit the right? Button, there's a little right-click. Yeah, okay, stop there. Yeah, so that's known as the foot-in-the-door technique, what I just did. What this involves is essentially giving people something really easy to agree to first, like maybe say holding the clicker. And it's much easier to get them to convince something, to do something else, like running your slides once they've done that. So for example, hold on, hold on, hold on. So for example, if I just asked in this study— hold on, whoa, whoa, not so fast. If I just asked someone to put up a huge Drive Carefully sign in their yard, nearly no one consents. But if I then say, here's a little 3-inch thing, right, like 3 inches is dinky, would you put that up? Almost everyone complies. All right, now you can press the button. Yeah, okay, but once we have this 3-inch sign up, About 3/4 of people now agree to this big thing that we originally wanted. This is known as the foot-in-the-door technique in psychology. Go ahead and press next. So here's an example from Instagram, right? I've put a huge arrow next to this like button, cuz I think the like button is one of the most successful foot-in-the-door techniques anywhere. What do these platforms actually want you to do? They really want you to be posting content and commenting and engaging in really pro-social activities. But most people don't really like doing that, at least not off the bat. So what do they have? A button that's really easy to click, right? It's very easy to sort of get in. Go ahead. We're gonna see how long this lasts, right? Okay, one more. Oh, you hit the black button. Are you— all right, thank you. Thank you, Brad. Amen. Okay, so who here has heard of Asch's conformity experiments? So another really classic experiment. So what they did in this one was ask people, they brought people into a room together and said, okay, this row of people, we want you to tell us which of these 3 lines on the right is most like the source line. Right? So you think about it, and most people will agree independently on what the right answer is. C. C. It's not straightforward, it's not super easy, but you can figure it out. But what Asch did was he had only one actual person in the experiment. Everyone else was known as a confederate. They were sort of on the experimenter's team, and now imagine That what happened is, it's gonna be Brian, he doesn't know that he's in this experiment. And then the 3 of you all in order and I say, which one is it? And you yell out, A, very confidently. And then she says, A, and then you say, A, right? And then it comes, and we're like, Brian? And Brian is very likely to say A despite the answer actually being C. So in fact, about 3/4 of people can be convinced to say the wrong thing this way. This is known as a conformity effect. Where do conformity effects show up in these sort of online major large-scale systems? One that I see all the time, here's Reddit, is essentially upvoting, right? So you have these effects where large groups of people who are already, say, on the top liking some piece of content on Facebook or upvoting something on Reddit influences us to say, yeah, that is good. Right? So we need to think about what effects this is having on sort of this wisdom of the crowds. If we're influencing each other, one of my favorite studies of all time was done by Salganik and Watts, in which what they did is imagine you had Spotify, but instead of a single Spotify where it would show you how many listens each song has gotten, they split it into like 8 separate Spotify worlds where the numbers couldn't impact each other. And they had little randomizations in the initial number of listens that everything had. What they found was that each of these different worlds ended up with totally different rankings of what was the most popular and the least popular, just because of little nudges like this initially sort of spiral up and a little bit out of control. It's sort of like a system that's not closed loop enough, right? It's sort of feeding back on itself. So there are real impacts of this. Here's another one, social loafing. Also known again as the high school team project effect. Right. So the concept here is that when you get more and more people together, sometimes people are gonna do a little less work than they would otherwise. So that is, when we know that there are other people contributing, we individually will contribute less. So here's another experiment that folks ran, and then I'll again show you sort of how designers are thinking about this. In this psych experiment, they brought people in to play tug-of-war. Now it was just like the last one in which, in this case, you thought you were on a team, but they would blindfold you, okay? So they're like, here's a rope, you're blindfolded, okay, here's your part of the rope. You're gonna be playing tug-of-war against that other team, and there are some other people on your team. Now of course, there's nobody else in the room. They've tied the rope to a wall that, with a force meter that can measure how hard you're pulling. And then they just manipulate how many other people they tell you are on your team, right? So do you believe you're alone or do you believe there's a ton of people there? And people pulled about 20% harder when they thought they were alone than when they thought there were a bunch of other people on the team, right? And I'm sure they would have said that they all pulled equivalently well. Okay, why does this matter at really large scale online? Well, this one's maybe a little easier to see, but here's one that really matters, I think. Wikipedia, right? So here's a test I would use. How many people have used Wikipedia in the last month? Yeah, really useful, right? Keep your hands up, hands up. Okay, now keep it up if you've edited Wikipedia in the last month. Okay. Keep it up if you're an admin on Wikipedia. Yeah, okay. So Wikipedia is actually being driven by a very small number of people. And in part, this is because it's so easy for us to say, yeah, it's already good enough, there are other people around, they'll take care of it, right? It's relying on this aspect of someone being like, no, I'm going to take ownership of this. And you can bet that if you didn't think anyone else was gonna do it, you'd be more likely to actually do this. So we need to think about how we design for systems that emphasize that your value is high, right? That it's not just that there's a lot of other people around, that you matter. One more, I'll talk about reciprocity, and then we'll close. So reciprocity is interesting because it means that you're much more likely to do something when someone else has already done something for you. So this can be social capital, it can be all sorts of things. But what's crazy is, again, even if you didn't ask for the favor, you're more likely to be nice to the person. So again, the psychology experiment works something like this. You and I were teamed up, but again, the team member, never trust a psychologist, the team member was a confederate. And at some point, they walk out of the room ostensibly to go to the restroom, and they come back and bring you a soda. They're just like, hey. You didn't ask for one, but I got you one. And so they bring back the soda to half of the people. The other half of the people, they just sort of come back and they went to the restroom and you continue. And they later asked these people to sort of buy lottery tickets for these partners. And if you got the soda, you were more likely to buy more lottery tickets again for your partner. Okay, so you're like, thanks, I didn't ask for a soda, but you know. That's enough, that's fine, good. Now I'm actually, I feel indebted to you, so I'm going to actually do more for you. I think Facebook is the master of making use of reciprocity. Here's an example. What's the best way to get someone to like your photo or comment on your photo? Comment on theirs? Yeah, or tag them, right? Put up a photo of them. Do something that's nice for them, right? I think the birthdays stuff on Facebook is brilliant, As a form of reciprocity, right? You log in, you have 100 or more friends who've all said happy birthday. You're gonna start actually replying to them, you're engaging in conversation with them, you might be using Facebook more, right? They're taking advantage of this notion of reciprocity. And I don't think that this is like anyone being evil. I think that this is actually a very pro-social thing for Facebook to be doing, right? But it's really tying into these issues, this function of reciprocity. So what I hope I'm getting across here is that there are a number of aspects of how individuals are impacted by the social environment around them that are really being written into code and rolling at huge scale out to potentially, well, Facebook's over a billion users now, right? Right. Right, and it's having huge effects. So if you are engaging in this kind of design, Or you are being engaged by this kind of design. I think we need to be informed populations about this. So zooming the furthest out, crowdsourcing is a thing that you may have heard of before now, but it is really only going to grow. It is going to be taking on increasingly more complex tasks. I hope I've given you a glimpse of some of what that might look like with being able to summon sort of on-demand teams and organizations that Have exactly who you need and when they need them. We can start to think about how large groups can start to advocate for pro-social outcomes. I think this is especially relevant as we see an especially caustic election season. Think about what the future of campaigning or of trying to get the word out about things is going to look like, right? Crowdsourcing is in some sense no different than democracy. It's a bunch of people coming together trying to make sense of the world, right? So how would we design that if we were able to and able to start over. And solving open-ended challenges. I think much of what we've seen with crowdsourcing thus far has been things where the goal is already set. We're gonna label these images, or we're gonna find the best picture here or something. But imagine, Rome was not only not built in a day, it was not clearly specified with a street plan when they started, with the exception of my hometown in Irvine. Southern California joke, sorry. But increasingly, what we're going to see is the next wonder of the world is gonna be something that's iterative. It needs to be created, we don't know exactly what we're doing when we start. How do we engage huge groups of people when we're still figuring it out as we go? Big unanswered question. So I'll close here. I think we have about 10 minutes if we wanna chat. You're also welcome to dash out the door because you have better friends. That's totally cool, I don't mind. But I'll stop here, and thanks for your time on this weekend. Yeah. So I'm an old person, and a lot of what you've said is very new and a little odd to me. But one thing that seems really critical is how that mentor, in your case, Essentially defines those subtasks and how does that occur? Are you talking about that being sort of a new way of doing that or is that something that's really kind of already established? I think at some level what we want to do is take advantage of the sort of effective roles that already exist. So one thing that a good manager will do is help an individual understand how to sort of plan their time and break down a big task. And all we've done is we've sort of made that explicit here in the code. What this mentor does is they come in and they say, okay, you've been asked to build a Ruby on Rails application. It doesn't really matter what it is. It could be make a portrait of my partner or something like that. And that's big and sort of unwieldy to start with and people tend to get lost. But the mentor has done this a number of times and they can sort of say, well, the first thing I would do is this, then this, then this, and let's check in at each of these milestones. You can imagine that if enough people start doing this, you can build a sort of Wikipedia of these processes, right? So then the mentor could even sort of scale out their abilities even more, spend more time critiquing and giving feedback and less time sort of saying, here's the plan. But yeah, absolutely, I think at the heart of your question is how will mentorship work? And imagine maybe now no longer just being part of a hierarchy within some company where you're expected to get mentored by by your manager, but imagine a mentorship network where globally you're getting mentored by other people who do what you do, maybe in a very niche area. I think it could look very different than it does today. Yeah? When you have these writers and musicians and music editors, sound mixers all coming together to work on a video, How do you give them credit? Like when you have 50 to 100 people, how do they get credit once that's published in the World Wide Web? And then who receives what percentage of the profit? Great question. So I'll just repeat it so everyone can hear. She's saying when you're drawing these sort of on-demand groups together with very complex and interrelated outcomes, how do you assign credit, right? This matters, for example, because payment might matter. Also, a number of these systems use reputation systems. So imagine Yelp, you get 1 to 5 stars on various restaurants. People in these platforms also essentially get 1 to 5 stars based on how effective they are. And you want to be able to identify, remember the fundamental attribution error, the thing might be bad but this person was good, right? You need to be able to separate those. And so I can tell you, let's start with payment. Everything that we did, people, what happens is that these freelancers basically have hourly rates. So in order to be hired, they're sort of getting paid hourly for whatever they do. But then I think that there's the broader question, so let me zoom out to the much bigger one of how do we sort of over the long term develop reputations and credits and credentialing, right? All of a sudden, someone might no longer be able to say, I graduated from Stanford, right? They're a freelancer who's doing good work, but what does good work mean? How do we know? And what we've been exploring largely is actually using some of these techniques from sort of networks and graphs to think about this. Imagine that, for example, we had 100 people and you can sort of, you have 100 points and you can give it out to various people in the group who you've interacted with based on how effective you think they were, right? So you've got this crazy world where now Maybe this is you, this is me, there's a bunch of other people, and this is slowly moving up for— This is why I use slides. And you've now, we've created this network that says here's who knows who, and in this direction, this person thought this person deserved 5, this person thought this person deserved 1, et cetera, et cetera, right? And we can build that all out. And what we can start to do is use things like PageRank, which you may have heard of, is sort of like, was the initial impetus behind how Google figured out which web pages were actually good versus not. And we can start to use these sort of distributed trust mechanisms. Now, these are gonna look very different. I mean, right? I'm just gonna let that be. Right now, your credit is sort of determined by your manager. That's sort of, if I work in a company, that's how it works. There are known problems with that, but there are also some benefits. This will have other problems and other benefits. It is essentially a big open unsolved problem. The reason it hasn't come to be a huge issue yet is that most of these sort of crowdsourcing interactions have been dyadic. That is to say they're one-to-one. So if you go on Uber, one rider works with one driver. So it's very clear, the rider rates the driver, the driver rates the rider. But even this has issues. John Horton at NYU has found that there's huge reputation inflation on this. This I call the Yelp effect. It's everyone's like 4.5 stars, making it extremely difficult to know who's actually good. 'Cause everything's amplified up toward the top end. On Yelp, basically, if something has 5 stars, there's this great XKCD comic that says if it has 5 stars, there's only 1 review. If it's like 4.5 stars, it's amazing. If it's 4 stars, it's terrible. If it's 3 stars, then it probably killed someone, right? So I think there's this really open-ended and challenging question of how we design for reputation more broadly. I would also imagine, sort of a final footnote here, I've sort of given this view where people are coming together, they don't know each other, they never have, they never will again. But there's nothing that says it has to work that way, right? You could belong to a small group of people who you really like working with, or several small groups depending on the day of the week or something, and always work together as a unit. And right now your unit is getting the reputation and you all know amongst each other who's actually doing what and who's responsible for what. So you can imagine, rather than just saying this is gonna be an issue, we can think about, well, if it's gonna happen anyway, how do we design for it to be as effective as possible? I don't have the answers, but I have ideas. Yeah? Even though there are some benefits of the rapid production of a professional, acceptable product, I mean, this is excellent, but still you have to think of the alternatives. Yeah, so I think there are multiple layers of questions there, so I'm gonna try to wrap out and probably miss something, so please stop me. So with respect to accessibility, I agree that Blind people have developed strategies to deal with it, but there's an incredible amount of the world that is still inaccessible. Just as an example, look at nearly any microwave in any workplace. At the home, microwaves, people will tend to sort of Braille-ify them. They'll put little dots on them so they can feel. But when was the last time you saw a workplace that did that? Right? When was the last time you saw a touchscreen on the copier that did this? Right, so being able to essentially make anything in the world accessible, I think, does bring significant benefit. For better or for worse, the ADA doesn't apply to copy machines. Moving back, you were talking about autonomy, and I think I'm trying to locate the individual in your question. I think what you're saying, and correct me if I'm wrong, is that if I rely on a flash team, that I'm going to lose the ability to do it myself. The other thing you could have been saying, and I think this is not what you were saying, is that if I'm in the flash team, I lose autonomy to make decisions about how the process works. I think the first was really the question. I mean, if an individual initiates the project, they should also be able to analyze it and think it through themselves. In the absence of a flash team, even though the flash team has come in, Sure. I think this is a— I think this is an open question. It depends, you know, sometimes on whether you're in a training mode or if you're actually doing. So, you know, one could claim the same thing about any calculator, for example. You know, I can calculate logarithms. The last time I did so by hand was probably a while ago. So I have lost some of that autonomy. For sure. Sometimes that is worth it. Sometimes Excel does things for me that while I could do it, are much more effective here. Sometimes I don't want to know how to program an operating system in order to use an operating system. I was thinking more in terms of creativity, authorial creativity as much as computational, like the circle. I see, yeah, so I think that the biggest issue I see that is related to this, I know huge problems with what I've been doing, One is, it essentially follows a recipe, right? And creativity often doesn't follow recipes, it needs to be open-ended and iterated and these kinds of things, and if you're not being reflective about it, absolutely, you sort of, you get what comes out the end, and okay, but you don't know how to fix it, you don't know how to make it better, et cetera. So absolutely. One thing that you can think about is how you might integrate yourself into the process. I sort of pitched this as something that's sort of, put in the coin and go, but imagine that you were actually able to interact with the people or shadow them as this was happening. This need not be a one-size-fits-all. I'm happy to chat more about that afterwards. Someone over there has a question. Why don't I swing around? Yeah. I have a reframe for— Yeah. What about putting it this way? So you're both telling me in principle there that the group mean median is going to beat any individual, even the closest individual. Well, that's fine, but the counterargument would be, what about regression to the mean? You start out with a great idea in the first person, then you put it through the process and it goes to the mean. Yeah. Okay, it's okay. Yeah, I love that for every, for every like, there's wisdom in crowds, we have the same saying that's the opposite. It's like design by committee, right? I think what this actually has to do with is we don't know Scientifically, we don't know when this is a good idea or not, right? We think it works with democracy, but that also sort of has a regression to the mean, right? It makes it slow, it makes it sort of, we can only pass things that everyone sort of agrees with, right? If you're looking for an outlier idea, but then you expose that to the crowd and then get the crowd to vote on it, And now you're, even though the idea is way different than the consensus, you're getting back the view of the consensus to see, yeah, this is a breakthrough and really deserves high votes or something. Yeah, this doesn't always work, but it can sometimes. So what he's essentially saying, and then we'll wrap 'cause I think it's 1:30 and you have cool things to do, is it's possible to, crowds are often, not as good at generating content on average, but they are good at recognizing something that's good when they see it. Think of it this way, you probably can't make Game of Thrones, but if you watch Game of Thrones, you know that it's good or maybe bawdy or whatever, but you can recognize high-quality TV when you see it. And the reason it doesn't always work is that if I came and pitched to you maybe the idea behind Game of Thrones, you might be like, I don't know, that seems kinda weird. It took someone with vision to actually get it far enough that the crowd could appreciate it, so to speak. So what I take out of this is that sort of at a scientific level, there are unsolved questions about who you should ask and when and how many. Imagine, you don't wanna just sort of become the thing that's the average, because the average is literally going to be average, not very good. You want to be able to become the best of the people in that crowd. And that's a huge opportunity and one that we don't know how to solve yet. So I'm happy to chat more afterwards, but thank you all for your time and attention. Thank you.