Michael Bernstein | Crowds, Computation, and the Future of Work

I think a lot about the question of how we work together to achieve really complex goals. I sort of view human cognition as essentially the most interesting and exciting thing about society. And you all are living this, and you know the strengths and the weaknesses. And you're not the only ones who have noted this. There's an entire field called organizational behavior that has, let's just say, dictated a number of problems that we have when we try to organize. We're not very good at this. We don't really foresee how hard it's going to be to integrate our work. We think that it's going to get better when we have bigger teams when it actually gets worse, and so on. The issue here is just fundamentally we as people cognitively are not great at reasoning about how all of these little micro interactions that we have sort of grow up into an emergent whole. So if I want to design for some emergent outcome, all I can do is sort of design the little individual pieces. And we're not very good at thinking about how to put those pieces together. to do it effectively. And it's not just that we're dumb, it's just really hard. But what I see an opportunity here is that all of a sudden work is getting mediated. Increasingly, all of these things are happening inside of a computer or through a computer. Work is getting distributed through online algorithms. People are coming together through online platforms. Their work is just getting mediated. And because it's mediated, as a designer and as an engineer, we can think about how we want to fashion that channel to help people achieve the goals they want to get done. And I'm going to give you 2 vignettes today at sort of this super ambitious organization scale and then down at the team scale to give a thought on that. So this first piece is a piece of work, as Mitchell indicated, collaboratively with my colleague Melissa Valentine in the Management Science and Engineering Department, actually also in collaboration with Alex and his group. So you'll be hearing a bit about that later. They're called flash organizations. And really the idea here is that we want to think about what it might be if it were possible to actually assemble an organization at the snap of your finger, to pull together distributed workers, maybe in a globally distributed workforce, for really, really complex collaboration. And just to cut to the chase, what I'm going to try to demonstrate is how software can actually help these, as we call them, flash organizations carry out really open-ended, really complex goals that were out of reach of anything that you might have called, say, crowdsourcing, or in some cases organizations previously. ranging from product design, software development, game production. And underlying all of this is essentially a computational layer that we've been building, designing in our groups at Stanford called Foundry. Foundry does everything from trying to help author these organizations, reconfigure them and adapt them very dynamically, and essentially run them. So you should be saying this would never work, and so would the organizational behavior scholars. The reason is that you have people who don't know each other necessarily just coming into these organizations needing to work together. And it's true that if you took traditional organizational approaches, that wouldn't work. Instead, what we look to are actually what are known as temporary organizations in the literature. These are things like film crews, disaster response teams. And it turns out that these kinds of organizations can coordinate really effectively through something known as role structures. That is, if, if Dan were the director of a movie and I were a gaffer, Dan may not know me, but he knows how he's supposed to interact with a gaffer, and I know how a gaffer is supposed to interact with a director. And by designing these organizations around role structures, we can actually start to make these things effective. So these flash organizations are essentially parameterized by the roles, e.g., the skills you need, grouped into teams, nested into hierarchy. You have essentially a centralized decision-making architecture inside of software. And because this thing is represented inside of the system foundry, what we can do is start to hire these individuals that you have in the roles automatically. We can start to onboard people and accelerate that. It's not perfect, but we can start to give you some ability to sensemake around who's on your team, what you're expected to do, what's been happening. Again, because we have this representation internally of what's going on. And you could say, well, OK, so you've got a bunch of people. But organizations need to change rapidly. Like, we're teaming. How do we do this? We start to think about ways in which people inside of the organization can essentially branch what the organization is about. These roles, the teams, the hierarchy, the tasks, any of this quote unquote source code of the organization, anyone can branch say, well, I think we should be trying this, or we should be reconfiguring ourselves in this way, or I need more time on this, and then issue essentially a pull request back to someone up their chain, which then allows, once merged by someone up the chain, the system to turn around, hire the right people, update everyone, keep everyone in sync. So we have this sort of digital layer that's helping coordinate this large group of globally distributed workers. And again, without going too much into detail, I just want to give you a sense that these We gave individuals who had no expertise with crowdsourcing, no expertise with the Upwork labor market, which is what we were using for these kinds of things, about 6 weeks. And they were able to create really, really complex outcomes, convening new workers on average within about 14 minutes each. And what do I mean by complex? Everything from— these 3 folks created 2 mobile applications, 3 full-stack web apps, 50,000 lines of code, 2 illustrated card decks, lots of tasks, tons of pull requests, about 3,000 person-hours of work. There was a medical resident who wanted to create an app such that folks could report en route to the hospital. They started just by themselves, and by 6 weeks later, they had everything from a marketing team to a HIPAA expert. You had this group creating marketing materials, creating the app, prototyping it, trying it out with doctors. You had a storytelling podcast team creating a card game, sort of like Cards Against Humanity, But for humanity, it's a little confusing. And Accenture worked with us to actually create an internal workshop portal that needed to meet enterprise branding standards. So what I hope you would take away from this is that there's an opportunity here to think about what I think Alex will probably call these hybridized workforces, groups that are essentially thinking about how we draw together just the right people at the right time and coordinate them effectively toward wonderful ends. Now, there are obviously lots of other implications of this. There are ways that it could go wrong societally and otherwise. I would love to have that conversation. But given less time, I want to give sort of a second project here to give a sense of where this is actually going when you start to wire in AI to this kind of thing. It's called Dream Team. This is led by Sharon Zhou, who's a PhD student. And this kind of comes down to this question of, all right, you've got people. This is, you know, flash organizations and the organizations that you all work at have this question of like, how should we organize ourselves, right? We're working in teams, so, but Are we supposed to be flat or hierarchical? Should we be a holacracy? Should we be really critical of each other? Should we be supportive? These are huge questions that every team in every organization has to ask. I'm going to refer to these things, like in the literature, as team structures. Again, think of it as sort of like the source code, the thing that defines what the team is and how it works. And we spend a lot of time thinking about what's the ideal team structure? What's the best way to do this? And then we build systems that try to actually encourage those kinds of structures. Turns out this is a terrible idea. Back in the '90s, organizational behavior recognized that there's this thing called structural contingency theory. And basically think of that as saying, it depends. Right. The best team structures depend on who the people are and what they're trying to do. You change either of those things, a new person on the team, a new task, and all of a sudden the team's going to be dysfunctional. So managers aren't great at this, unfortunately. The literature says even though they're paying a lot of money to be trained to do it, And I don't blame managers. It's just a really contingent, really, really complex problem space. So we were thinking about, could we turn this into a search problem? Could we think about allowing teams to experiment essentially on their own team structures to find the right fit rapidly? So intuitively, the notion would be that you have your team that you're working in. You work with our system Dream Team, and it adapts you to a particular set of structures. A different team comes in, gets a different set of structures. A third team comes in. And each one's sort of personalized to the sets of structures that work well for those people on that task. So in the kind of system that we've built, this is mostly working on Slack. You can imagine it as an email agent or anything like that. So you have your team come in. In this case, maybe they're working on what's— I'm calling it a puzzle, but any sort of intellective task, anything you're trying to do at your organization. There's some sort of feedback signal that comes back to the system, whether from the team itself, some sort of self-report, from a manager. In this case, maybe it's just automatically visible. Like you have a marketing team where you can see how well they're doing. And the system starts to suggest, hey, actually, why don't you all try this? You don't have to, but give it a shot. And behind the scenes, what the system is doing is comparing about something like over 400 different combinations of these different dimensions from the literature, where it might start with a particular set of structures. We get feedback. It decides to explore, maybe saying, hey, let's try to have a decentralized hierarchy here. And then over time, adapting the team through this self- Experimentation to the thing that seems to work best for them. We do this through something called a network of bandits. This is a reinforcement learning technique that essentially has a series of different dimensions that are all exploring at once. Unfortunately, you can imagine that trying to explore at once is like meeting with me as an advisor and me telling you to do like 20 things at once. It doesn't go so well. So one thing that we started trying to think about doing is trying to sort of constrain these bandits in order to explore a little bit less ambitiously. So they're telling you to do maybe more like 1 2 things at a time. We start to model this, for example, noting that teams are only open to change at certain points in their progress, or that certain dimensions like hierarchy are better to change at certain times and not at other times. Then we turn to something called Thompson sampling, which is how these systems actually work behind the scenes. And we do some math to essentially say, OK, make it less likely that you're going to move away. So we're going to make it maybe 1/4 as likely as that you're going to resample to some other arm Shift all the probabilities, so now we have a 0.8 probability of saying, whoa, stay where you're at. We do this across all of these things, and that makes sure that we can overall say, okay, everyone calm down. We're only going to change 1 or 2 things at a time here, and these are the things that are going to get prioritized at this given time. And it turns out that when we do this, we see some incredible changes in the team performance, something like a 40% performance improvement. in these teams just by adapting their team structures relative to basically any of the other control conditions we tried— having managers lead them, having the group lead themselves, having sort of unconstrained bandits do this, or just having no advice at all. So lots of work to do there. But just in closing, I want to think about how we talk about this. I think many ways we talk about the future of work and automation and all these things as this very routine work that's going to get taken over by AIs. And that's just not going to happen. That will not happen. The reason is that there's stuff we have high process knowledge about. You can sort of routinize it. But anything you don't have high process knowledge about, anything that's sort of open-ended, complex, the kind of work you're probably doing, we can't encode it into a workflow. And thus, we're not— teaming will remain the most important way to do it. And so I think what we should be talking about when we talk about the future of work is empowering teams and organizations to actually collectively work toward this. stuff. As a closing thought, organizations, when they originally came about with actually the Industrial Revolution and the railroad, when they were designing this, they looked around and they said, oh, what's the coolest thing around? And it was actually mechanical structures, like gears interlocking. And so this notion was that they were like, well, let's design organizations to work as awesomely as gears that interlock and work perfectly together. And so the entirety of how organizations work today was totally designed around this archaic notion. And in the same time, computing went from essentially a dream of Charles Babbage to something that's on my watch. And I ask, what would organizations look like if we stepped back and designed them around the affordances of computing, being data-driven, reactive, malleable, rather than those of mechanisms? I will stop there, and happy to chat on the couches. Thank you. Great.