Michael Bernstein: Crowd-powered systems

Great to see all of you. It was mentioned, I believe, earlier today that I drew a lot of inspiration earlier from The Media Equation, which really started to point out ways in which the relationships between people and technology began blurring. And I'm really excited to talk to you today about some really interesting new ways in which I think we're blurring this line between what is human, what is technology, and who's in control of what. And I specifically want to talk to you about ways in which we can reach out to large groups of individuals to help actually create interactive systems that are really unlike the kinds of things we could have created before. So when I talk about interactive systems, let's start with the most basic one you've probably ever seen. It's the word processor. It's the white lab rat of human-computer interaction. And that's more or less why, because it's the most heavily designed system, I would argue. And like many of these kinds of designed systems, it helps with complex cognition tasks, complex interaction tasks. You can think about how Microsoft Word and its brethren help out with layout, with spelling, with grammar, many things. But at the same time, they're really not helping with the core activity of writing, or even, say, editing. If you think about it, the ideas are still completely your own. There's nothing else that Microsoft Word can do to help you here. And this hasn't always been the case. Think of these kinds of situations. I'm faced with this all the time as an academic. I would assume many of you other ones, especially journalists, face word limits all the time. And this is exactly the kind of thing that we just can't help with as software. AI is not good enough. The interaction is not good enough. But historically, we didn't even do this ourselves. We actually turned to other humans. If you were a writer, a good writer, you would have an editor who would help you actually fit yourself onto that 10th page if you needed to. And even today, we pass text back and forth to our colleagues. But this was never something we could really build into an interactive system, because if you wanted that, you would need people to be available at any time at large scale, and that's really never been possible. Until recently, we've really started seeing a huge profusion of what's started to be called crowdsourcing or crowd computing. Crowds gathering together on the internet to do things all ranging from data collection, training advanced machine learning systems, running social science experiments, writing an encyclopedia that we all use daily. And this isn't even new. It goes back to the 1800s when the British Royal Astronomers started sending out spreadsheets through the mail to help calculate the positions for nautical sea charts. And— but all throughout this historical lineage, this has operated what I would call a batch platform. This is something where you have a lot of work, you put it over the hill, and you wait a while, and hopefully it comes back. So what would it take to take a situation like this one and think about how we could actually have crowds interact with you through technology? That is, rather than having human editors support— a single human editor— what if we could have tens or hundreds of individuals all look at your text, make suggestions, and we could wire it back into your software such that you could deal with it in an interactive way? That's what I mean when I talk about a crowd-powered system. It's interactive software that is really powered by crowds. And I think that there's a lot to be done here and a lot to be said. So I hope to just spark your imagination with this brief presentation. Let's say that I have sparked your imagination already. You're going to run into a couple challenges, the first one being quality. I asked 1,000 people online to flip a coin and type H if they got heads and T if they got tails. So hopefully we have a large enough sample here that if you all flip coins, we get about 50/50. Turns out on the internet, it's about a 2 to 1 ratio of heads to tails. Now this isn't because the internet is a biased coin. It's because people are satisficing. There may not be a whole lot of actual coin flipping going on. And we know when people try to generate randomness, they do so in very non-random ways. In fact, you may notice this doesn't add up to 100%. That's because fully 7% went sort of out of grammar and typed things that were either misspelled or the entire word or the enigmatic F. These are people who are trying to contribute but are making mistakes that our software might not know how to understand. Also think about speed. I'm talking about interactive software here, like a word processor. When you started seeing crowds being used for all sorts of things, people were really excited about how fast it was. Except when you looked at the time, it was like 2 days, 5 days. This is much better than survey research was up until a couple years ago, but still quite slow when you think about the fact that interaction happens on the order of seconds. In fact, depending on how much you pay on some of these paid labor platforms, you can see anything between 2 days and 12 hours of turnaround. So what I want to convey to you today are the ways in which we can build interactive systems that reach out to crowd intelligence to make that interaction smarter, more natural, more powerful. And I want to make an argument here that we have the opportunity to lead through technology and computation, to think about ways in which technology can help crowds gather together and work in ways that they might not actually succeed if left to their own devices. So I'm going to focus right now on something that's called paid crowdsourcing. How many of you have heard of Amazon Mechanical Turk? Wow, okay, well maybe this is less necessary. On Amazon Mechanical Turk, Amazon as in the bookseller, there's a large amount of tasks being done that take very little time and very little money. So label this image for a couple cents, people do this and make up a fair amount of money. It's, now there are many of these markets, one for example oDesk, you can actually I believe make a living as a professional on. This one I think is a little bit more fraught with labor concerns, but it's interesting because if you look at it across the population, it's actually balanced in terms of gender, education, income. It mirrors the United States distribution, as well as the Indian distribution for about the 40% of workers on the system who are from India. So there's a lot of people who are participating. I wanna convey to you the ways in which this can really lead to some exciting opportunities for interaction. I'm going to start with one called Soylent. Soylent is people. It's a word processor with a crowd inside. And the idea is that we can think about how we might actually change, again, this white lab rat of interaction by adding in some elements of crowds. So rather than telling you about it, I'm actually going to show it to you. What I hope you'll take away is that we're actually thinking about crowds as sort of a first-order element of software. People, software, the line is blurring. And that we can think about ways that technology can help decompose these problems to make them more solvable. So here's Soylent. It's a plugin for Microsoft Word. It's running on the Soylent research paper, which is a little bit meta, I admit. But let's say we have this exact situation I came up with before. This paper's a little too long. Let's say that I think the conclusion in particular is a problem. I'm going to send it off to shorten. Shorten is one of Soylent's particular features that it provides. And when I do that, we're actually going to be recruiting help out from the web. Now this is all offline, pre-baked Martha Stewart style. But what you see is that people are actually coming in. They're marking up shortenable sections of the text. They're making edits. There are some votes going on. And when all of this comes back, we can consider all of these contributions that people have made. You see everything here in purple has been marked as shortenable. Each of them, as you can see behind it, has an alternative. And then we can consider this space of possible paragraphs that got generated, and I can give you a direct manipulation interface, and you can sit there and drag the slider, and as you drag it, your text gets shorter or longer. Or anywhere in between, and hopefully now down onto the 10 pages you needed in order to submit your paper. So we're really wiring together all these contributions from many different people with many different perspectives to create a new kind of experience. Now, we can also support existing AI systems. CrowdProof is a proofreading service that really layers on top of Microsoft Word's built-in proofreading, and the idea is that we're asking people to actually catch things that Microsoft Word wouldn't. You notice that it made a couple changes here, Again, these were run beforehand. One that's interesting here is a parallel sentence structure error, which got past, I think, something like 6 authors, 6 reviewers, until CrowdProof caught it. And the reason why is that this is down at the bottom of page 5 of the paper. And as much as we care, we're still going to miss things and our eyes are going to get a little tired. But people, again, coming in with diverse perspectives, maybe not having seen this before, might have a lot to say. You can also think about sort of opening this up, asking for anything. When I write, I tend to leave citations open like this, in brackets. And we might want to ask for help. So we could go out to something called, say here, the human macro, and say, hey, can you help me find BibTeX, which is sort of structured citation for the citations in brackets? And just so you believe me, this is the kind of input that people who use the system make. Kind of unclear. It's not necessarily grammatical. But we can send it over. And when it comes back, you get what you would hope. Here's that structured language. They went off to Google Scholar. They came back and plugged it in. So that, in short, is what Soylent does. And I want to talk about how. Because the how is non-trivial. We've really worked with many, many, many people as a part of this. And if you were to do the naive thing and just ask for support, you'd get a lot of errors. In fact, we have a rule of thumb that about 30% of what comes back, if you were to just ask on open-ended tasks like improve this text, really is not something you'd want to give back to a user. And I'll give you an intuition for why that's the case. And the reason I'm doing this is that I want you to think about not only the opportunities, but the challenges, and the ways that we can work together to solve them. So here's a paragraph I got off of a high school essay website. It's really horrible. Don't turn this in. You will fail the class, I guarantee you. I'm just underlining a few of the errors here. And we see 2 kinds of canonical problems. One of them I would term the lazy worker. This is someone who wants to send a clear signal that they've done the work, But they also just don't want to do much more. So given this error-ridden paragraph, they'll make that one character change. And it's actually not surprising they made that one, because it was the only thing underlined in their browser as being misspelled. They make a clear contribution and say thank you. On the other hand, you have eager beavers. These are people who also want to send a clear signal that they've done the work, but they actually go way beyond as well. Our algorithm may not be expecting this. So these are equivalent to the people who would write out head or heads. In this context, they might think that the paragraph would read better if you inserted new lines, or returns, between every sentence. I happen to disagree. But we might not have expected this, and it would be difficult to actually protect against it. So as a result of these kinds of personas, we have what you might call low-quality work. And from my perspective as a computer scientist, What we really need to think about here are design patterns. These are sort of reusable ideas that we can apply that say, okay, our goal is to take something in an open-ended fashion and refine it. How do we go about doing that? How do we actually get a computer to play a little bit of a role of a manager and to get the crowd itself to manage itself successfully? So here's one idea. There are many out there. We call it find, fix, verify. And the idea is that we're actually decomposing some of these very difficult open-ended problems into things that are actually a little bit more directed. As you might presume, we're actually splitting it into 3 stages here: find a problem, fix it, and verify. So I'll give you an example. Here we can actually say instead of fix this text, like proofread it or shorten it, we're actually gonna say find an area that can be shortened. Don't make any changes. We're gonna collect a bunch of these, look for independent agreements, And anything that people seem to be agreeing on, we split off in parallel to a fixed stage. In the fixed stage, we ask people to edit that section, and we gather a bunch of these options, randomize their order, and feed it to a quality control where we try to say— effectively enforce invariants. We want to make sure we don't include new grammar errors. We want to make sure that we don't change the meaning of the text. And anything that survives this particular gauntlet, we can actually then send back to the application that can choose to render it in any way that it wants. This is one example of a kind of design pattern we might have for crowd computing. Okay, so does this work, you might ask? In fact, we asked that as well. What does the quality look like? How long does it take? How much is it going to cost you as an end user? I'll give you a very high-level sense here. I'm happy to chat more. So we fed Shorten a whole bunch of different inputs ranging from academic papers to TechCrunch blogs to my favorite, which is a rambling email from the Enron corpus. And you would see edits that look a lot like this. Overall, what we tended to do was cut about 15% of the original text without changing the meaning. If you step back for a moment, what that means is that you could take an 11-page draft of a college paper, hold constant all the boilerplate, the title, the figures, Soylent would give you back a 10-page paper with no major changes to content. Same meaning. How are we doing that? Well, mostly our contributors are focusing on what I would call wordy phrases. These are things that are sort of unambiguously changeable. The phrase are going to have to can change to just have to. Sometimes we're actually merging sentences, right? This is something we can actually feed to a machine learning algorithm and start to learn to do slightly more automatically, though this is a more difficult problem than something that's just, say, a dictionary-based approach like what you just saw. This doesn't always work, of course. Here are a couple interesting failures. In one case, if we're not part of the community of practice, the workers may actually choose to change something. There's an academic signaling phrase, in this paper we argue that, which workers find incredibly boring. Again, you may or may not disagree, but it did in a sense change the academic context of that sentence. Also interesting is parallelism. We have workers nearby each other who may not actually be able to see what the other ones are doing. See, here it renders the resulting sentence almost meaningless, because you have the workers in one phrase cutting the main clause, and in the other one they're cutting the parenthetical. Okay, so we have hundreds of people actually contributing to these kinds of things at about $1.25 per paragraph. Depending on how long you're willing to wait, that number can go up or down. So how long do you have to wait? Well, we have to post a task out and ask for help on Mechanical Turk, and it turns out that if you sum the median find, median fix, median verify, these are sort of log-normal, so it can take much longer, it can stall, you're waiting about 20 minutes. Now in a moment I'm going to show you how you can actually get that much shorter. And in fact, if you can get that much shorter, the work time is much— is actually just 2 minutes, a median of 2 minutes. So how do you do that? Again, find, fix, verify. You could, in the limit, ask for help and get a response in 2 minutes. So this will hopefully open your mind to the kind of possibilities that are— that's there. I'm going to turn slightly more briefly to showing what happens if this happens in real time. Real-time crowdsourcing. Why do we want to do this? It's because the kinds of things that we can design interaction for are really constrained by latency. In fact, people have started building all sorts of really fantastic applications that are collaborating with crowds to accomplish things like generating designs, helping people with health and nutrition, supporting robotics. But everything is constrained by that same limit that Soylent is, which is that, well, basically we can get one person to maybe get back to you in about a minute. And if that's the best we can do, we've lost already. Because it turns out that we know from some work, actually Stu Card, who's a researcher here now, back in the day found that people just lose focus about 10 seconds after waiting. So you pull out your phone, you say log in, and if it sits there for 10 seconds cycling, you're just going to put it away and start focusing on something else. So we need to do something much better. We need on-demand, flash, real-time crowds. So how do we do that? So I'm going to show you how we can do that. I'm going to show you an application called Adrenaline. In Adrenaline, the idea is that we're going to actually get these pictures to happen in real time. You're going to take a short video instead of a single frame, and we're going to choose the best frame. We want to do this because you want to actually see the result almost instantaneously. So here's an example. We're capturing a video like this with me and a participant actually just high-fiving each other. And as of right now, we've just made the request. People are going to jump in along the bottom. They're going to focus in. And just a few seconds later, they're going to identify the frame, and we're going to pass it back to the user. We're done. Real-time crowdsourcing. What kinds of opportunities does that open up? You can see the kinds of pictures that we get back. People trying different angles, different poses. People— action shots here jumping off of a bench. That's awesome. Or people just being silly, dancing around and letting the crowd actually contribute what they think to be the most memorable moments. So I'm going to give you a sense of how this is possible. One thing we can do is a retainer model, which is to say that we're actually going to ask for people in advance. They're going to come. We're going to say, we know that we're going to need help. You sign up. Go work on something else. We'll pay you to just stick around and come back. We'll throw up a JavaScript alert when we need you. And it turns out that when we do this, here on the x-axis, how long it takes them to get back, and on the y-axis, how many of them get back that quickly, we see that about 50% of them come back in 2 seconds. And in fact, you get about 75% of them in 3 seconds. This is incredible. This means we can get collaborators extremely quickly. In fact, if you have them waiting longer, they don't come back as quickly, but you can offer a bonus, and it takes something that looks like that and pushes it up all the way as if they hadn't been waiting at all. This costs about 30 cents an hour. I said about a half cent a minute in addition to the base task. And we can do all sorts of things. I showed you Adrenaline. I'm going to close by showing you a few other examples. Okay. A/B. Let's say I wanted to know which tie to wear today or which logo I should have for my company. Instant opinion poll. Here's a recorded piece from our dataset. I'm going to click go, and then individuals are going to appear here. Think about which one you like. Well, the crowd seemed to actually like the right one. That was a real-time playback. So we can get crowds in 2 seconds, we can get votes in 5 seconds, and hopefully you're thinking ahead to what's hard. Well, we can do votes between 2. How do we do it across 100 frames? Right? And for that, people take a long time to choose. So we're going to sit there and we're going to actually think about the fact that everyone's participating at the same time. And because they're participating at the same time, arriving together, and they can work together, we can actually get the crowds to work together collectively than even the fastest single member of the crowd. What we're going to do, as you saw before, is we're going to actually start noticing when people are likely to agree, and then focus their attention on that area. So we say, hey everyone, you're focusing over here, let's everyone pay attention and really help us out. So again, just to show you that video, it looks something like this. People hone in on that sort of center area, we reduce it, reduce it again, and we have a single frame. So just to give you a sense here, we can control— we can try a bunch of different approaches here, get asking them all to come in on retainer. We can try to bring them in retainer and vote slowly, vote quickly. And what I'd like to get across is that we're really actually getting real-time responses within that 10-second response time for the first frame and about 12 seconds for the final reduction. And the reason we can do that is we can take these people who are slightly a little bit more laggy and pull them along with the rest of the crowd. Those are the people on the bottom half of that orange block. So we can actually do all sorts of things in just 10 seconds by reaching out to collaborators out in the crowd. So let's think about one more example. Are you a designer, perhaps? This is Photoshop. It has a tool that allows you to sort of bend figures. Let's say that you're a designer and you wanted to populate a crowd of individuals in a poster for, say, a rock band concert. So you want to have a bunch of people looking excited. You can author these control points and then ask for the crowd to come in. This is what the control points look like. Here we put them— we call them off-retainer. You have people actually Manipulating the points, as you can see here. And then we can draw that back into Photoshop. And as you were seeing just a moment ago, the end result looks something like this. So here we've actually, again, closed that loop, real-time interaction in an everyday kind of application, where we got responses in just a couple seconds, the first figure in about a half minute, and then a new figure every 3 seconds after that. So hopefully I've now convinced you not only that this might be possible, but that we can also do it in real time by considering techniques that can actually help people walk together quickly. So really, here's what I hope to leave you with. It's possible for us to think about how social computing, broadly construed— Twitter, Facebook, Wikipedia, Mechanical Turk, its entire brethren— can really get us to rethink the core aspects of how we interact with technology. What is possible? And I would invite you to think with me about what we could create by working together. We could think about how we could train machine learning systems using crowds, using machine learning systems to actually help teach the crowd. We can create high-level tasks. How do we make sure that this is a positive eventuality for the workers embedded in the system? How would— what would it make— what would it take for you to be proud of your child if they were going to participate in this crowd labor force? You would need education, career advancement, reputation. There's a lot to be done. I think that we as the innovators have the opportunity to grab hold of these markets and of crowd computing and point it in a direction that will make it a positive outcome for all involved. So I'll close with this thought. We can create these crowd-powered systems now that can enable experiences that neither crowds could take on themselves because they would perhaps trip over each other or act slowly or in low quality, maybe not be able to collaborate well. But neither could machine intelligence do it by itself either. We're actually creating a hybrid system here that is aiming to take the best of both. And more broadly, I hope you think about the ways in which computation can actually be critical to this notion of the wisdom of crowds. Many collaborators, thank you to all of them, and I'd be happy to chat, or we can move on as per the schedule of the day. Thank you. Thank you, Michael.