Melissa Valentine and Michael Bernstein – The future of work could be “flash teams”

All right, so imagine you're a medical student and you work in the emergency department, and you have an idea for an app that uses GPS to track ambulances as they approach the hospital. Or imagine you're a group of friends who run a podcast and you have an idea for a game that can go with your podcast. Imagine you're a software engineer in a large enterprise firm and you have an idea for a workshop portal that will help coordinate workshops for your clients. So in any of these situations, are you imagining to yourself, I can crowdsource the production of my idea? So this research shows that yes, you can crowdsource the production of such complex ideas. And we think this is the future of crowdsourcing, and for some industries, the future of work. This research is an interdisciplinary collaboration. I'm in management science, My collaborator is Michael Bernstein. He's in computer science. He's currently on a plane to Tokyo. We have a PhD student who spans our department, Daniela Retalli. She's also working with us. So many aspects of computing are already changing the way that we work. The one that I want to focus on is how information technology connects workers to jobs in different ways. There are now online labor markets that pool millions of workers. You can almost costlessly reach these workers, any of these workers at any time. So what these massive pools of workers, sometimes they're called crowds, what these massive pools of workers enable is crowdsourcing through open call. The way that works is you have a job or a problem and you post it online, and whoever's interested and whoever's available can take that job and complete it. Sourcing work through open call is faster. You have millions of people who are on tap to to do the job. I do an exercise with one of my classes. I tell them, you have an hour to get a professional graphic designer to make a team logo for you. And even my very tech-savvy millennials are skeptical that it's gonna work, but I have never had a team so far fail to crowdsource a professional logo within an hour. Crowdsourcing or sourcing open call can also in some cases be better, not just faster, but better. So NASA has recently started doing open-call innovation contests. So they'll take some piece of their very complicated work, and they'll distill it down into some problem, and they'll post that small contained problem online, and issue an open call. Anyone online can participate and offer a solution. And what they're finding is that some of the solutions that are produced through open call are actually outperforming the solutions that are offered by their expert scientists in-house. So there are, of course, limitations with what you can do with workers convened through Open Call, and the crowdsourcing approaches to date reflect a lot of those limitations. So if you post a task and anyone online is eligible to participate, that means you need task structures that are very simple and very routine. Anyone with any experience, any skill, you know, any ability can show up and do the task, so it needs to be very— your task structure needs to be able to accommodate any of these sorts of people that show up. So as a result, the kind of jobs that are being done in crowdsourcing right now are the kinds of jobs that I think that Chris Wray will be after very soon, and these will be things that will be done by machines before too long. That's what crowdsourcing is currently being used for. So there's limits to what you can do with these microtasks, with these small workflows that are set up. So you have a small work task, and then the computer will algorithmically combine whatever the workers do. So it's things like image labeling, or, you know, caption— creating captions, or editing text. So the question we wanted to ask in this research is, could we crowdsource more complex goals? The kinds of goals that you can't accomplish with a workflow, the kinds of goals you can't do with an assembly line, the kinds of goals that require more interdependent, sort of innovative human ingenuity, more human creativity. So our insight was that we wanted to use a different computational structure The previous crowdsourcing approaches set up workflows, and then they issue an open call, and then anyone can come and participate in the workflow. So our insight was that we could, instead of using workflows, we could computationally represent organizational structures, and then issue open calls, and then people could come participate in these organizations. So there's 2 things to note about flash organizations. We call this approach flash organizations. 2 things to note about flash organizations. One is flash organizations are temporary organizations. They are convened by open call, do a project, and then they disassemble. There are a lot of examples of such temporary organizations in the offline world. You can think of disaster response teams. You know, you have hundreds of groups of people get together. These are relative strangers. They come together and they're able to coordinate really complex work emergently. So management science research shows that what makes that possible is role structures. When people show up to the disaster field, you know, say I'm a firefighter, you're a med tech, we both know what we're supposed to do by virtue of our roles. Our role structures encode our division of labor, they encode our responsibilities. So our idea was that we could computationally represent these kind of role structures. So you can see an example here at the top. So we do, the role structure tells you that you have an Android developer, like a backend developer and a frontend developer, so they know how to work together by virtue of their roles, even though they've been convened through open call. So the other thing to note is that changes the nature of open call. So instead of issuing an open call to anyone online, you're going to issue a role-based open call. So there are expert crowdsourcing marketplaces like Upwork that organize people by roles. So you can issue an open call to Android developers, an open call to backend developers. We issued an open call to poets for one of our projects. So these role-based open calls allow you to set up flash organizations with a computational division of labor. So Daniela and Michael built Foundry. This is a web platform that creates flash organizations and supports them. So this is a web platform that you can use to author the flash organizations. So you can see, here's an example. So you might create a task here. This is a timeline. This is a web portal with a timeline. When you create a task, then it will automatically issue an open call to Upwork. So let's say you created a task for the Android developer, then it issues an open call, and everyone in Upwork who is— everyone in the panel Upwork who's eligible to be— to fill that Android developer position will show up, and the first person to show up joins the Flash team. Foundry will then take the worker through a rapid onboarding. It tells them what their interdependencies are. It tells them their role responsibilities. It tells them how to communicate with each other within the platform. So one of the key features with Flash organizations is we needed them to be able to reconfigure as they worked. One of the major problems with workflows is that they're pre-specified. You have a set input and a set output. You can't adapt as you work. So we needed flash organizations to be able to reconfigure as they worked. The offline analog is, if you imagine, a disaster response team getting to the disaster site. Maybe they encounter hazardous materials and they need to suddenly spin up some hazmat team. Maybe they need to redeploy one of the firefighter teams to a different side of the field. So the organizational structure, the role structure, is very flexible. It can be redeployed in real time. So we needed to create a feature in Flash Organizations that was similar to that. So Foundry has a functionality for editing the role structures. This is kind of a simplified way of how it works, but basically you'll have, so this is the timeline again, these are the different roles. So when someone wants to suggest a change, then they make a copy, they add a role, they add a task, they submit the copy, they submit their edits for approval. What's cool about flash organizations is this kind of editing, this kind of suggestions of how to reconfigure the structures can happen top-down from the user, from the CEO, or it can happen bottom-up from any of the workers in the field. And this allows the whole organization to sort of update its understanding of who's doing what, who has what responsibilities, and what the interdependencies are. So now I'll tell you about the evaluation we did of this idea, of this system. This is a proof of concept research design, which is common in HCI research. We're now doing a field experiment with a control group. This is more common for management science research. But this, what I'm reporting now, is proof of concept. So with this proof of concept, we needed to prove 3 things to sort of show robustly that this idea works for crowdsourcing complex work. So the 3 things I'm going to show evidence of is, one, that they were able to successfully crowdsource complex goals, the kind of which have not successfully been crowdsourced in the past. 2, that the organizational structures were reconfigured throughout the time. And 3, that the flash organizations were able to automatically hire through open call in the sort of timeframe that you expect with crowdsourcing, which is very, very fast. So to do this proof of concept, we recruited the 3 people I was describing at the beginning. We had a medical resident, we had a group of friends who ran a podcast, and then we had a software engineer at a tech lab. And each of them had their idea that they were willing to crowdsource using our approach. So here are some of the deliverables. This is the slide that I'm hoping you most remember when you leave tonight. Each of these users was able to successfully crowdsource their idea in under 6 weeks, and they spun up entire organizations by hiring people in under 14 minutes. Using the crowdsourcing approach, using the role structures, they were able to successfully complete these complex deliverables. The deliverables were complex. They created 2 mobile applications, web applications. There was over 50,000 lines of code, 2 illustrated card decks. This was over 3,000 hours of work. The teams were diverse. They included engineers, designers, testers, and poets. You can crowdsource poetry from Upwork, it turns out. We had neutral experts review the quality of the deliverables, and all of them passed the quality review. So let me show you— this is kind of the fun part. Let me show you what they built. So these are screenshots from the app. So this was the idea that the medical resident had. What's interesting about this, this is a video. He actually decided when he made his organization that he wanted to include a marketing department in his organization, so he spun up someone who did graphic design and did videos. So this is someone from Bangladesh who was hired in 10 minutes and made this video for this organization. It's— you can see here, it's going to show in the video that the GPS is actually functional. So that was a fun moment in the project when we saw that you could actually see the ambulance approaching the hospital. The second one was a true story game. So what was cool about this is the crowd interfaced with the physical world. They sent the cards out for manufacturing. So now these card decks actually— these Card decks are now in Michael's office. We have the— this is an actual functional app. Here's screenshots from it. The last one is this enterprise workshop portal. So you can see here, this is the Accenture brand. Accenture was the company that supported this. So what was complicated about this project is that it had to integrate with Accenture's existing IT system. Okay, so here are the emergent organizational structures. So each of these started with just one user. who had the idea, and then over time they would automatically source, they'd automatically hire these people who joined the teams. The organizational structures you can see are complex. You can see that they're also custom, so this just depended on the preferences of the CEO, is what we called them, and also the nature of the work. These organizational structures map onto— these are the emergent retrospective timelines. So this is the trauma report. It was the most complex. So each little colored square represents one of the tasks that was done. So for example, look here. This is the true story game. So back here, these are the 12 poets that were sourced from Upwork, and here the poets are doing their work. They write the poetry before the cards are produced. So that's what you're seeing here. Notice that it was 3,000 hours of work, and people were working quite collaboratively throughout the entire time. All right, so during each of the projects, the organizations continuously adapt. adapted. You can see here are the different projects. Here's the average of all, or the sum of all the projects. The number of pull requests, the number of times that somebody asked for the organizational structure to be reconfigured. You can see that it was happening kind of through the course of the project. I wanna draw your attention to this. So 22% of the pull requests came from workers. 9% came from the CEOs. And we have a big role for middle management. So 68% of the— Pull requests came from the team leads. So here's evidence that they were continuously adapting. We also have evidence that they kind of did the crowdsourcing timing for the hiring. So each of them did automatic hires. In total, there were 75 automatic hires across all the organizations, and the median time to hire was 14 minutes. We gave the CEOs the opportunity to do manual hires, meaning they could interview, they could just talk to someone out of Upwork and interview. Those, in contrast, took 15 hours. So I want to, in conclusion, reflect on how this work advances the field of management science, and then on Michael's behalf, I'll say how it advances his field, the field of computer— human-computer interaction. So management science is a discipline that's interested in measuring, modeling, and theorizing important phenomenon. So flash organizations represent a brand new phenomenon that raises fundamental questions about what we understand about organizations. For example, flash organizations change the transaction costs that are core to how we understand the theory of the firm. They change the social psychology of collective behavior. So there's a lot of management research about teamwork, motivation, leadership, incentives. So those are all established for bureaucratic organizations, people interacting face-to-face. What we're talking about here is organizations that are, you know, flash organizations that are convened on demand and work entirely online. And lastly, these organizations, and this is what I'm most excited about, offer really unprecedented opportunity to do data science on organizational structures to figure out which are more— most effective. They also allow prediction markets, so anyone in the organization can kind of give information and predict how things are going, and that allows the organization to continue to adapt. And then this research also advances the field of human-computer interaction. So crowdsourcing kind of lives in this world, and so what flash organizations represent is a brand new crowdsourcing approach. So previously, crowdsourcing was very focused on workflows, as I said, so this is the first time that we've done a crowdsourcing study that has such a complex and alternative approach, and in so doing, we're able to crowdsource much more complex work than has ever been crowdsourced before. So for us, this represents a future of work that's mediated by algorithms that's mediated by these online labor markets. A famous sociologist, Max Weber, said that organizations were the most important social phenomenon of the 20th century. What we're talking about here with flash organizations is a new organizational form that's fluidly assembled and disassembled from globally networked labor markets. Thank you.