HAI Seminar: Melissa Valentine and Michael Bernstein on Flash Teams

We're delighted to have Dr. Melissa Valentine and Dr. Michael Bernstein with us here. Melissa Valentine is Associate Professor at Stanford University in the Management Science and Engineering Department. She's co-director of the Center for Work, Technology, and Organization. And Michael Bernstein is Professor of Computer Science and the SC Microelectronics Faculty Scholar at Stanford University, where he's a member of the Human-Computer Interaction Group, and his research focuses on the design of social computing systems. As I noted, today's seminar is something of a celebration, as it's part of the book launch for, for their new book, Flash Teams. And in Flash Teams, the 2 scholars chart the opportunities of flash teams and navigate the challenges that teams and managers will face. They distill lessons from their own work and their scholarship in assembling and managing flash teams on demand that every manager can learn And so that they can successfully use flash teams in their own organizations.

All right, I'm going to begin with a story of one of our first users. So he's a doctor who works in an emergency department, and he has an idea for a pretty simple app. He noticed the same problem play out time and again in the ER, which was an ambulance would drive up, and as soon as the ambulance got there, they would start to gather all the needed doctors and nurses, And he noticed there was all the time that the ambulance was driving that they were not in communication with the ambulance. So his simple idea was, what if we made an app where the ambulance drivers could communicate with the hospital the moment they're going to get there and what the patient needs? So with this great idea, he goes to his boss, the ER director, and he says, I have this great idea. It's a simple app. It's going to save lives. His boss takes it to— his boss takes it to her boss, the hospital CEO. Who takes it to the IT director, who says no. Because— Thank you for all the disappointment, right? That's how he felt, right? Like, what a great idea, and how easy is that to build that? Engineers know that's not the hardest app possible. So he goes up the hierarchy, and he's told no. So here's where we meet him, and here's where the idea of flash teams comes in. Michael's gonna tell you about what he was able to build.

Okay, so at this point we have Mark. Mark has no one because his director just said no, said no. And he goes from that to a team of over 30 designers, user researchers, software engineers all around the world to a working user-tested iterated system In six weeks, this is a vision of flash teams. This is a vision of on-demand AI augmented teams. This is not a vision of AI replacing our work. This is a vision of how online platforms, communication, and AI allow us to connect to any expertise you need within minutes, and then can quickly adapt and shape that that movement as you go. In the book, we kind of— we, we pull this into a phrase, kind of experts in a click. And if I were to describe what this shift has been, it's been the availability of on-demand labor through online labor markets such as Upwork that make essentially any kind of expertise available through freelance work via a software API. So something that you would have needed to search, you know, in In the best or worst case, depending on your point of view, Craigslist for, you now actually have platforms that you can reach out to. Maybe that's Upwork, or Fiverr, or Artella, or Catalance. Or you're trying to get software engineers, you're trying to get writers, you're trying to get graphic designers, you're trying to get business consultants. They are all on these platforms. This has been a major shift in how work is happening. More and more people are working remotely, are working online. making their services available through these platforms. And at some level, the question of our book is, what is that leading to? And our answer is flash teams. Flash teams are essentially what happen when you can recruit any expertise that you need from a global online network within minutes, that you can right-size each person's involvement. I need 5 minutes of this person's expertise to help me shape exactly my approach. Some offer a moment of brainstorming, others join as full-fledged team members as you build whatever it is you're trying to build. It's a future in which you can reshape and adapt that team composition automatically as things go. And this is at some level a team as you've seen it before, but it's also a team unlike we've seen before, where you could win a Nobel Prize like Richard Coase for pointing out that the reason that organizations look the way they do is because of the amount of effort that it takes to pull them together and keep them bound to get something done. He asks this question of, why do firms hire people? Why don't we go contract out in the marketplace? Why don't we go to the, to the market out on the street and go negotiate with each individual? And the reason is that it costs less in some sort of both monetary and effort sense to, to just hire someone. That's because money isn't the only cost, that we have to be able to find this person, we have to negotiate a contract. And this is exactly what we think has been shifting with platforms of the sort that we've been describing. And as we do this, what I want to convey to you is the way in which we observed in our research flash teams carrying out these very open-ended and complex goals, the kinds of things that were traditionally out of reach for online collaborations, crowdsourcing, and so on. where basically what they needed to do was be nimble, to be adaptive. We, we have flash teams that are doing things like product design, software development, game production. We build software that helps coordinate these. Now, of course, you know, as labs at Stanford, we're building custom software because we do that. We just love to build software. But at the same time, in the book, we talk about the ways in which you can do this today. What we had built— and this is work that was led by our jointly advised PhD student Daniela Rotelny— was a platform called Foundry that helps sort of coordinate, recruit, and sync up the members of these flash teams. So you can say, this is what I want, this is how we're going to flow our work together. It's sort of like a supercharged Gantt chart. And when you do this, you have to start asking questions of what does it mean to have a team or an organization that has never met before and may never meet again after this? And for this, this is why it was so It was fun for me to collaborate with Melissa Valentine on this book, coming from a background in organizational behavior, teaching— well, this is not a completely new problem. There are teams today that do this quite well. Film crews, for example, disaster response teams, teams in the ER. And the way that they succeed is through this use of what are called role structures, where if you're a director and I'm a gaffer, I don't need to know you and haven't worked with you so much before. We just need to know what I can— what we can expect from each other. That if you're the director, you know what the gaffer ought to be doing. And if I'm the gaffer, I know what I ought to be looking to the director to help me with. So we start building these sort of role structures into the flash teams that when you author a flash team, you say, here are the roles I'm looking for: the Android developer, front-end development, back-end development, user interface design, user experience research, marketing, whatever it would be. Nest those roles into teams. teams into hierarchy so that we can coordinate this, and then essentially start hiring on demand. And what we demonstrated in the book is just how quickly you can get essentially any expertise to go from an idea to a team within minutes. In fact, in the research that we discuss in the book, non-experts, non-technical experts were using our system to essentially spin up and lead not just teams, but entire 30-person organizations over the course of several weeks, getting the expertise that they need within 14 minutes on average. And in this time, they built mobile applications, full-stack web applications, 50,000 lines of code without Claude Code helping, I should say, 2 illustrated card decks, lots of tasks, lots of different people coordinating, moving into different positions in the organizational hierarchy. All of these reviewed by experts in the domain saying this is pretty good. Let me give you a picture of what this looks like. He starts the story that Melissa tells. He's by himself. He brings in, first, an Android developer, builds a user interface design team, a front-end development team, a back-end development team, a user testing team. They figure out they need HIPAA compliance. They build a whole marketing organization. They build a QA organization. Again, over the course of days and weeks, they are building the airplane as they are flying it, quite literally. And at the, at the end of this, they produce working user-tested pieces of software that they, that they showed off at the hospital, including again this marketing team making websites and pitch decks. Uh, but not just this. Here's a team that was creating a card game. This is a team that was running a successful podcast and wanted to create a companion card game. So this group, uh, wrote poems for prompts on all of these cards. And in fact, as part of doing this, they hired a series of poets off of Upwork and then wound up promoting one to become the chief poetry officer. A week before they were done, they were like, you know what? We want a companion Android application. So they spun up an entire team to build an Android application. We worked with a large consulting firm to build internal portals for event planning. This was a single app, a single user within in this consulting firm that was essentially the CEO, so to speak, of this flash team, this flash organization, bringing it together and putting it to use. So let me hand back to Melissa, who's going to tell us about what comes next. Okay.

So Michael and I have been working on this research for about 10 years, we discovered. So we actually— the flash teams that he was telling you about, we actually did early in the flash teams research. And when he and I were working together, we started to recognize that these teams had special properties. Because everyone was working on software, it meant that they were developing massive datasets on all of the decisions they made, on all of the roles they took on, the role structure, the team structure, the hierarchy, the workflow. And we started to see that they were creating essentially like data models of the team. So I'm— my PhD is in organization science, and so I started to understand that we were starting to measure things about organizational design and team design in a way that we hadn't before. So my PhD from a fabulous business school out east taught us organization science that looked a little bit more paper-based. And what I was seeing is we were producing massive datasets that were going to support a different kind of organization science, and we needed to sort of update our paradigm of thinking about organizational design and how AI and data can support managers in how they design teams and organizations. So Flash Teams, the book, is in some ways about building quickly and solving problems quickly, but it's also about recognizing how platforms, tools, AI can help managers make better teams and make better organizations. So the rest of our examples are going to be systems that we or companion companies that we worked with have built that basically augment organizational design or augment team design. So here's one I'll tell you about. So a pretty common representation of how organizations work is that they will sort of— information will flow from the front line to a central decision maker. The central decision maker will select a strategy for how to lead the organization, and then they'll communicate the information out. So that's like organizational behavior 101. So let's do it again, right? So let's think that you have some strategic choice that you have to make. Let's say that you are now facing facing the pandemic, and you have to decide a work-from-home strategy. So WFH, work from home. Let's say you have to decide your WFH, work-from-home strategy. So it's like all of a sudden, it's like 2020, there's a new policy you have to change on the fly. So what you do is you get information from the front line, and then the CEO makes a decision, right? Oops, about what to do. And then they will communicate it out. So that kind of information flow, as I said, that's like the basics of how organizations work. And it's actually a pretty powerful property because it allows like large groups of people to adapt together. So the policy changes, suddenly everyone's in office. Policy changes again, suddenly everyone's at home. So having groups as big as 100,000, right, companies as big as 100,000 change altogether, that's an impressive property. Are there any problems with it? Is there anywhere that AI could help? So there are a lot of sort of known problems and information breakdowns that can happen with that kind of flow, information flow within the organization. My favorite one to joke about is, if I give my boss bad news, by the time it gets to the CEO, it looks like good news, because there's such a positive bias of information up. So it's sort of recognized that information flow is supposed to work like that, but there's a lot of opportunities to really help information flow within the organization. So I'm going to tell you here, this is a company that we write about in the book called Perceptics, and they've worked on this kind of information flow. So what they have is a personalized recommendation engine. So they will collect information from all the employees in the company, could be through surveys, it could be archival. And then they'll have a central decision maker make some sort of evidence-based strategy, make a decision about what should happen. And then they offer personalized nudges. So for example, you might be walking into a meeting and your phone goes off. And your little Perceptics bot might be telling you, reminding you of a goal that you have, or remind you of something you're supposed to be working on in that meeting. So what's cool about this is the kind of personalized engine at the heart of it. So the information that is going in, the selection of the strategy, and the information going out is far more sort of precise and customized to the person. So that's different from organizations of the past, and that's like what becomes possible when all of the work is done on a platform.

So where does this go? Once you have flash teams, suddenly all sorts of interesting questions arise where computation— because computation is mediating how the team is coordinating— has the opportunity, whether through traditional algorithms or AI, to get us to question how we think things have to work. So let's take this classic question. Of how should we organize ourselves? Should we be flat? Should we be hierarchical? Should we be nice to each other?

Should we be—

should we call it like it is? Should we make sure that we all enforce equal turn-taking? There are many folks in the audience I see who have actually worked on these kinds of questions. And, you know, the answer from the organizational behavior literature, at least, is there's no single correct answer to this. Essentially, it's all contingent. It's contingent on who the people are, how they're being, you know, what problems they're solving. So if the, if the task changes, if the people change, you kind of have to re-ask this question. And so what if we could— if this is effectively a search problem where you're essentially, every time things are changing, you need to figure out what's the right set of strategies— what if we actually turn this into a little bit of a search problem where as you are working with your team, you are getting— you're trying to do something, whatever the goal is, And at the end of the period, you actually give feedback about how you think the team is going. And based on that, this system starts to say, hey, I know this sounds a little funky, but what if you tried this? And what happens is you can take a set of the sort of design axes, the sort of source code of any team. It initially starts out in whoever the team is going to kind of set with its default. But based on the feedback, we give a nudge, maybe just a small nudge. Yeah, shift from having no hierarchy to a more decentralized hierarchy. And as the feedback comes in, this can start to evolve the team structures towards something that would actually work well for this group on this task. And we can evaluate this. You know, we've ran— we ran experiments where we would say, have, uh, people try this, you know, managers would try this, or the team would decide together. Turns out, you know, uh, loss aversion rears its ugly head. People under-explore. We try a couple things and we say, oh, it's good enough, don't, don't break what's, what's not broken. We can have full-on, you know, networks of multi-armed bandits that attempt to explore this space, and it turns out that they're changing like 18 things at once, and people just say, oh, just ignore it. It's just, it's just a bot. The bot is too noisy. But what we can start to do is think about how much and what kind of change can people absorb how quickly? What kinds of changes are appropriate at the beginning of the team, at the end of the team collaboration? How much change can we do simultaneously? And what we start to see when we do this is massive performance improvements. Now, would I expect your team and your business to get a 40% performance boost? Probably not. But the result in our experiment was really striking to me. And what's impressive about the state of the world that we're currently in is that you can just do this. This is the kind of thing where you used to have to download some Python code and get it running. You can today without installing any special apps, just Make a spreadsheet, fill in columns. Each week is a row. Here's how it's going. I'm calling that a reward here. You say, was I micromanaging? Did we have a daily scrum? Is our reporting flat or hierarchical? Just make a little Google spreadsheet. At the end of the week, you take that informal survey or you yourself just say how you think the things are going, call that the reward. And then literally just take that spreadsheet and throw it to a large language model with reasoning on. It can write code. You just say something like, blah, blah, blah, copy paste this, something, something, Thompson sampling network of bandits. And what you get out of it is a set of advice. So that's enough information for it to basically run the algorithm that took us an entire paper to write and to tell you what you might try next. And I put a little QR code there if you just want to see a demo of it. Over to you.

Okay, so you're getting the vision, um, that the ambition of the Flash Teams book is to ask the question, how does organization science change? How does team design change when you have AI in the loop? So far, we've been telling you about some cool use cases that are very sort of technology forward, um, but throughout the book, we also recognize there's ways that this perspective, that sort of recognizing recognizing the value of teams and recognizing how easy it is to find experts can actually be powerful in other ways that are not necessarily quite as technical. So I want to tell you about another use case that we found pretty inspiring that was similar to kind of the perspective of flash teams. So when Michael and I were trying to, you know, sort of like hashtag the book, as he was saying, we were like experts on a click, or something else he would say is there's like experts everywhere all the time. So if you sort of start to have that mindset, and working in this space now for 10 years, I think like every moment I'm like, who could I ask? Like, who could I tell? How could I ask? How could I crowdsource this? How could I flash teams this? But once you start to have that mindset, experts everywhere all the time, then you start to recognize all of the different value-adding ways that we collaborate with each other. So I think that this mindset and recognizing the ways that tools also help, tools can help with that. So let me give you one example that is a little bit less sort of tech-forward. These are all the ways we talk about in the book, all the ways of sort of recognizing experts everywhere all the time. There's lots of different kind of examples and stories of all the different ways you can work with people. So for example, maybe there's a known expert in the firm. Actually, a story that I tell all the time, this is from when I worked at Stitch Fix. So my— one of my colleagues from Stitch Fix is here. There was somebody in the data science department who was an expert in causal inference, and they needed her expertise on just about every team. So her— she was like a flash expert, so every team would like find a way to pull in time with this expert on that one specific capability. So across all of these, like, we can give you examples in the book of different ways to sort of understand the ways of drawing on expertise of, you know, either big groups of people like wisdom of the crowd, or experts one at a time. So the one I want to tell you about is sort of visualized here as group consult. So I was doing a study at an academic cancer center, and one of the doctors there recognized that patient care was way better if all of the experts who are relevant to complex patients talked together at the same time. So for patients who have cancer, it's better if they can hear from a nutritionist, dermatologist, palliative care, plus their oncologist, plus their surgeon, plus the radiologist. So all of those experts are very relevant to any of the decisions that get made for them. So what this doctor did is she basically created flash teams. She wouldn't call it that. In the hospital, we called it integrative care team. So she They created integrative care teams. So for complex patient cases, they found a way to sort of recognize who are all the relevant experts in the hospital who needed to weigh in on that patient case. And they would have sort of like quick meetings where they made sure that all of those relevant experts would come and weigh in on the patient case. So that's a flash team essentially, and it's good for patient outcomes. And it's hard to do with kind of a traditional mindset of the bureaucracy of the hospital, kind of the slowness and bureaucracy of the hospital. So it was very inspiring, cool to see this doctor sort of have that mentality that she could create flash teams of people who would make sort of decisions better for patients.

But once you start having— starting to see these things this way, and a lot of classic questions in computer science and network science and so on start again coming back up. Hey, can you help me recruit a super team, right? Like right now we recruit out of the manager's Rolodex. But what if we could find something better? What if it could help you find— what if AI could help you find exactly the right people given your goals, right? Maybe the right individuals and the right combinations of individuals who are going to work well together. You know, this is in some ways, you know, an ancient question from the lens of computational social science, HCI, and so on, but becomes freshly relevant Because suddenly we have access to potentially millions of people who can all— actually, in our case, we're all quite excited to work together. Many of the folks in these platforms had mostly just done individual freelancing and were quite excited to work together on teams with each other. And essentially, this boils down to, as kind of a distillation of what the best practices are here, you've got to start with what your objectives are. Maybe there's individual objectives. Do you care about people who have good reputations, how much you're spending? Whether they're likely to be available if you ask for help. Look, as much as I enjoy trying to get time with one of my colleagues, if I send an email and say, can I meet in a half hour, chances are low they're going to be able to. But there might also be team objectives. How familiar are they with each other? Have they worked together? What kind of intellectual or other kinds of diversity do they provide? What kind of time zone overlap? Just very practical considerations. And at that point, you then have to start asking, how do you weigh these objectives? Right? How much do you care about each of these? Because ultimately we need to put these together. Based on those objectives, you can do essentially a stochastic search over so many different kinds of combinations of people to identify what might wind up being a useful team. And again, this is something that you can think about in the abstract, or you can just try it today. Uh, so once again, the thing that, you know, took us months and custom code, you can now just throw into GPT. Start with a spreadsheet of your Rolodex, basically the people you care about. The columns just basically being scores on the things that you might care about. Again, how much is it going to cost me to hire this person? Are they familiar with each of the other members of the team? And you put your weights in another sheet, say, here's how much I care about each of these. Don't max out only one of them, or you're going to wind up with a bad time. At that point, again, you can just feed that spreadsheet to a model and say, hey, I'd like you to essentially find the optimal team according to these weights and those scores. And once again, it says that, uh, Niloufar and I should work together. So, um, Melissa's really good too, I promise. Um, so the, the goal here just being to point out that while we are at some level in the academic work that would— that underpin the book doing this, you know, bespoke creating these algorithms, at this point you can take descriptions and more or less recreate these things in real time and just use them. You could even ask, how do we make sure that this group, that this entire crowd of people we could reach into, intermixes ideas? One of the major challenges of teams is that you'd think that we bring everyone together and we bring all of our perspectives and we come up with the best idea. Instead, what happens is if I leave an idea on the whiteboard and then, Sandy, your team comes in and is brainstorming, you just ignore the whiteboard. What you would like be the case is that when Sandy rotates onto my team or something like this, suddenly Sandy's ideas have exert influence and start shaping us and getting us to think in new ways. So we can start to actually rotate this collaboration network over time so that people are moving from team to team to spread ideas out optimally. And you can imagine that on one end we can sort of keep only ever prioritize tie strength. So people who've worked together before, they jam together, they work well together, but there's no ideas spreading out through the network. On the other side, you might care about maximizing network efficiency, you know, the, the densest collaboration network as possible. But that turns out no one gets to know each other well enough to actually diffuse the ideas. So again, you can start to think of this more continuously and produce something that would be somewhere in this sort of Goldilocks zone in the middle that winds up producing better results.

You can hear this funny moment that we keep having working in this space. when we first started working on this and presenting it, we had what we called like our, our beer moment where we would like flash it up and be like, they did it in 6 weeks. And people would be like, whoa. And then like now, like that's not fast at all. And it would take our grads— we had worked with grad students and it would take them like a semester to like custom code something that, yeah, now is like you just QR'd it and like now you all have it from ChatGPT. So it is funny to be working in an area that is so impacted by AI. So what I wanted to do with this final example is I wanted to talk about flash teams in 2025. So some of the early, earlier examples we gave you were, you know, before 2025. So in 2025, now we're seeing companies use essentially flash teams. The flash teams are also working alongside agents that help the flash teams go even faster. So an example that I just thought was cool, I'm going to show you this. The example is coming from Microsoft. There's an engineering lead at Microsoft who gave a talk. I'm going to give you the QR code for her talk down below in just a second. So she— so Microsoft and her engineering team have sort of prototyped and demoed this idea. They're calling it hypervelocity engineering. But what I want to really show you or draw your attention to is they have a way of bringing together, let's call them flash teams, but tight multidisciplinary teams with deep domain expertise. And then they're bringing them together with, you know, AI agents or different tools that help accelerate the work of those small domain expert teams. So I love this description of what it looks like. So in the video, if you watch it, she talks about, you know, how it used to take them about 10 people, and she calls that like a Costco-sized pack of people. So she has 10 people would come together for probably 6+ weeks, maybe a month, and they would show up to solve a hard problem. So we used to show up with what we called a crew. She says it's 10 people. It's the business program manager, technical program manager with domain expertise, either in the industry or technology. You have the software dev lead, it's your engineers, it's your data scientists, it's your AI expert, and then some smart whatever. So it's like 10 people, the Costco-size pack everywhere you go. And she says now with this notion that they have of these small multidisciplinary teams plus the AI agents that they sort of set up in advance. Now they're working with smaller teams and they're getting more work done. So at this point, the way that it's working for them is they'll have, you know, UI/UX folks, they have the marketing analysts, the business analysts, and a handful of engineers. Let me just show you quickly what the agents are doing for them in these flash teams plus agents. So they'll come in and already have deep research on the customer and market research They'll already have a scenario brainstorm done. They'll always have kind of the engagement scoped out. So they're coming into a sort of sprint with this small multidisciplinary team. It used to be the case that these would be handoffs, like one group would do it, might do the market research, hand it off. Then a team of experts might look at it, hand it off. So all of that has now been collapsed to they'll do it in like a 24-hour period. They'll get together with all of this content, and that— and so they'll have the domain experts, the engineers are already there, the business experts are already there, and then all of this content is already gathered. So this is sort of the layer that they're doing all of this on, just if that is of interest to you. But going back here to key ingredients of hypervelocity engineering, as Michael was saying at the start, to us, this is not about, you know, job replacement or the end of interesting work. It's about how teams can get together and solve problems together better and more quickly with all of these really cool tools that are helping us, I mean, do things at flash speed at this point.

The kind of things that you imagine that flash teams may sort of get you to is the ability to ask, you know, is this team going to fracture? Is this team on a trajectory to do well, or is it quickly going to fall apart? Suddenly, if technology is intermediating everything that this team is doing, you can get a sense of that. Likewise, what might happen if we reorged? Ask these what-if questions. Could you simulate what would happen with this group of people? I think that there are so many questions that become possible here when suddenly you are able to actually connect between what I would view as organizational behavior and computer science. For me, I think one of the questions that I often think about is, at the time that organizations arose, I learned this from working with Melissa, it was sort of the dawn of the industrial age, railroads essentially. They needed to be able to coordinate railroads across great distances. That gave rise to the first organization in history, large-scale organizations as we would think of them today. To build those organizations, they took inspiration from the metaphors of the most advanced technology they had at the time for physical mechanisms. Then you go back and you look and they would talk about, we would like organizations to be as effective as these clockwork mechanisms where everything's so tightly coordinated. In the time since that has arisen, organizations have not fundamentally shifted in how they work. Yet computation has moved from a dream of Charles Babbage to something that lives on my watch. AI has moved from barely a dream to something that we are all using today, every day. How would you design organizations from scratch if you were looking not to the historical metaphors, but the modern metaphors? What does coordination look like? How do we work together? I think that remains, for me, to be a real interesting unanswered question. What's the operating system that underlies an organization of the future? Right? If it was an offline paper org chart before, what is it today?

I don't know what to say.

Oh, sorry.

One second.

Say thank you to our—

Oh, we're done. Oh, sorry. Okay. Oh, then I do know what to say. Okay, sorry. I thought you were— Michael has a cool— I'll just say this. Michael has a cool thing where you can use AI to simulate the organization. I thought that's where you were headed. Okay.

So— Talk about that if you want.

Yeah. Ask him after.

Yeah.

Okay, so you can hear from our point of view, Michael and I are very into sort of like social computing, collaboration, teams, and we sort of recognize that we are all, you know, better together. Collectively, we are able to do more. So we wanted to, at this moment, acknowledge people who have been huge mentors to us and very important to our book. So these are our endorsers, and 2 of them are here. So could I ask Eric Colston to come up and Liz Weissman? Will you guys come up? We have a thank you for you. So these are— so our endorsers, one is, I want to say, Fei-Fei Li, who is the founder of the HII Institute. So she endorsed our book. And then Liz— sorry, we have a gift for you guys. So Liz Weissman is an author. You guys probably already know this. So yesterday she was named one of the top 50 thinkers in the world by Inc. magazine. Is that right? Yeah. So anyway, please join me in thanking Liz. And then another of our endorsers, Eric Colson, who is former Chief Algorithms Officer of Stitch Fix. He's an alumni of our program and is probably one of the first people who helped me think about data science and organization science together. So thank you, Eric.

I also wanted to give a huge thanks to the PhD students and postdocs who drew Melissa and I together. Uh, so the, the researchers in the room know that faculty sort of go and spread in like 18 different directions, and it really winds up being the students who glue us together. So to the, to the PhD students who led the work that we've described here and all the other work that was— that, that's described in the book, we a hearty thank you, and we are such, like, a fan of you and look forward to everything you're doing going forward.

All right, so with that, I think we're opening it to Q&A. Is that right? Okay, so that's it. I just want to say thank you for a great talk and mind-expanding.

Um, you had a really cool diagram that was graphic, like an organism that was blowing up and changing. Can you describe what that came from or what it is?

Oh sure, how is that not self-evident? That's— yeah, um, that is I should have given it a voiceover. Thank you for asking. So there's a company called Autodesk, and they basically modeled their organizational chart like that. So what I find really interesting about that is it is actually the same idea as the org chart that is paper-based, but they've just put the decision— the CEO is like in the middle, and then every dot is an employee. And when somebody like changes teams, like changes managers, it shows kind of a little explosion. If they reorg, then it shows a little explosion. So it's a way of visualizing an organizational chart that just recognizes how dynamic they actually are. And instead of having kind of ye olde decision tree, it's just more of like a central— like the reason the organization can coordinate is because you have like a centralized decision maker rather than a top-down. Hi, Candace Mays. I'm a John S. Knight Journalism Fellow. I run a data journalism unit that frequently will rotate in and out certain people and experts depending on what we're building. And I've ran into the problem of that team chemistry. Can you speak a little bit more about the slide or the section of your presentation where you talked about AI being able to assess and provide feedback on communication in the various, like, chemical makeups and processes of the teams?

Mm-hmm.

Do you want to go first?

Sure. There's— at the highest level, you can think of sort of teams having a fairly well-recognized set of options. There's kind of a set of dimensions about how should hierarchy work, how should decision-making work, how often should we be in touch, anything that is a sort of a meaningful decision for you. And at the same time, you can imagine, are things going well or not? That's sort of a simplified version of this. And what happens in practice is that in most team settings, the manager explores a couple options and then it's like, yeah, but we got to get stuff done, and just sort of settles into it. And it turns out that if you were to have tried more options, you would've found something much better. And so what we observed was that essentially just if you think of the intervention at some level as being, smart about nudging into other spaces, saying, hey, what if you tried this? What if you tried that? If you make the right suggestions at the right time and velocity— now, you can obviously tip over into too often or not the right suggestions or whatever, but if you're at least relatively smart about this in the way I was— you can again just copy-paste what I was suggesting. It turns out that this can provide some real lift Now, there are some teams that probably will just experience substantial friction. And maybe there are just some people who are oil and water and just cannot work well together. They're great individual contributors, but cannot work well together. And in this case, what we saw— didn't discuss it here, but in the project we describe in the book, we can take small, what are called thin slices of the team interaction, something as little as 90 seconds of their chat interactions. And predict whether this team is going to basically fracture, which is to say, setting aside whether they get across the finish line, if you ask them later, if you're like, hey, would you want to work with this team again? Sometimes there's a high-performing team where you're like, hell no, I would never work with these people again. And sometimes they're like, well, that was really good. We can predict very accurately with fairly little data whether the team is later going to feel that way. I think the question to you would be something along the lines of, is this a fundamental issue where the team got off on the wrong foot and it could get fixed if we gave it the right nudges? Or in the book, we even talk about ways that we can blind team members to who else is on the team, sort of do team speed dating, except you're actually dating the same team in different configurations and you don't realize it. Then we say, here's your team. It was the same people the whole time. and they get off and do quite better. Is that the situation? Or is it a situation where you're just like, this team is going to fracture no matter what, in which case we can give you sort of an early warning signal. Maybe as a manager, you might dramatically shift strategies or change how you're approaching it.

Yeah, and I would say really practically, there's a chapter in the book where I'll just give the whole chapter away in the title. If you're dealing with something that's that dynamic, you have to get the launch right. Like, from the start, like, from the moment. So we learned that from a number of different founders and managers who work with flash teams. You really have to get the launch right, like, from the first moment that you're interacting. You have to have, like, role clarity, system clarity, goal clarity. So that's a place that you can really kind of practically, tactically focus if that's not working out. So I have a question. Congratulations on your book, Melissa and Michael. And, you know, it just seems like one of the most interesting insights for me. I don't hang out in large organizations anymore. I kind of got over that, mainly because of this premise of how organizations are structured in like this 19th century mindset.

Mm-hmm.

And then I spend most of my time now with startups, and it seems like there's this fractional life.

I'm a fractional CMO. I'm a fractional chief product officer, blah, blah, blah.

There's an incredible rate of change that's happening right now.

And so what is it that we have to unlearn? So this is like—

the question is, what do big organizations have to unlearn? And what are startups already learning in this fractionalness? Yeah, it does feel like there's a different kind of expected rhythm in the different ways of working. So I can just— so my mind is going to an example that's coming up a lot. People are worried about how entry-level people get jobs, given that a lot of those jobs are automated at this point. But someone said something that I— this is to me an example of what to unlearn. They were like, maybe we need to stop assuming that it takes 3 years to be expert at something. Maybe we could assume that people could actually get up to speed a little bit quicker on stuff. So that's an example. We have a lot of assumptions about how long things take, so maybe that could be something we could work on.

Yeah, I mean, one thing that we observed in a number of the flash teams that we either studied or convened and then studied was the rapidity with which they were reconfiguring themselves. Like, traditional organizations sort of like set an org chart, and then because if you get large enough, it becomes disruptive to be constantly reshaping yourself. But what we would see is that these flash teams, when most effective, were essentially changing multiple times a day. They would bring in new expertise. People would move from over here in the org chart to over here because they saw a hole and they suggested, what if I did this? Startups do this all the time, of course, as well. I think that my weirdest hot take at some level here is if you view If you want to see where things are going, look at the remote work revolution as something of, we're currently in a halfway zone in some ways, where we have a bunch of people who are now, they can be fractional or they're principally working remotely. And yet, what does that actually open up? I don't think we've quite realized yet at some level. Suddenly, my affiliation with this team or organization could look dramatically different, or I could work for many different projects. What does that mean in terms of the policies that we need to author to make sure that people would prefer to work this way than have a traditional job with how these things work? And so it feels to me like this is happening. And it's sort of like we We developed a rocket engine and then strapped it onto a horse a little bit, if that metaphor makes sense, which maybe by design it's not supposed to. We have suddenly this world where people are moving between projects, between organizations, and yet we're still sort of building organizations the same way we did in 1912.

Yeah, and so I can, like, the last thing that I would add to this, I think we're very comfortable with organizational design being like an art based on manager intuition. And now that we have a lot more data on how things are going and sort of like models of the organization based on the work that's actually being done, we could start to measure some of the theories that came out of like, so I graduated from Harvard Business School, we have all these theories from the 1970s about how organizational design is supposed to play out. So what if we could actually measure it? What if we could actually measure any of our organ— of our, like, team design theories or organizational design theories? It seems like we could create much more effective systems if we were actually measuring what works and what doesn't work.

Yeah, thank you for the presentation. It was really interesting. Um, I have a question actually related to the previous question about team chemistry and your answer on that. Uh, so follow-up question on that is, uh, you talked about, you know, the AI being able to look into the chat history and, you know, Predicting stuff. What other datasets besides chat do you think, or have you noticed or recommend, would be useful datasets for, you know, to be able to feed the AI to be able to make better decisions in the types of decisions you're talking about? I mean, in my ideal world, your organization keeps track of, like, think of maybe software consulting firms do this quite a bit. It's like, who was on what team? How did they work together and how did it work out? If you have a structured dataset of this, or if everyone were contributing to a management science Switzerland data pool, you could learn so much about which organizational strategies work within which cultural contexts on which kinds of projects and so on. In the book, we talk about companies, Gigster and others, that do exactly this. And I think that there's a lot of unobserved dark matter about how people work together that is— Yeah, that's right.

That's right.

That is essentially the stuff of not what people write on the web that we are pre-training our language models on, but the actual You know, meat of collaboration that often does not get written down. And I think that's like, that's the real unlock, in my opinion.

Yeah. And I think there's— so it's important to be human-centered with it and not do it in an exploitative way with like too much surveillance. But if you think about what you can actually record, I mean, it's at the level of like clicks. It's at the level of— you can get a lot of fine-grained data on what people are working on. So finding the right kind of aggregation of work that predicts something that's like useful, that's like human interpretable and actually predicts something. Maybe it's a task, maybe it's a workflow, maybe it's a role, maybe it's a team. So I think we now can kind of play around with the level of analysis, which I think unlocks a lot of cool theorizing.

And if you're interested in these kinds of things as a researcher, DeepMind has—

That's right. Yeah, we're doing something called the AI and Organization Grand Challenge. So if anyone's interested in advancing organization science in the era of AI, please look up our grand challenge. Did I already say sponsored with Google DeepMind?

Thank you.

Thank you. First of all, congratulations on your book. I'm very excited about it. I've been telling everybody about this book already, so I'm glad it is written. And I had a question regarding global countries. Have you taken to— can you share your thoughts on how this will apply globally, considering that, you know, team dynamics, global nuances, the implementation of that. And yeah, no, thanks again for writing the book. So let's see, from experience running our flash teams, so this is anecdotal, you might be asking something at a higher level. Something that we found interesting was how much solidarity people felt in kind of building together. solving the problem together. So there was sort of like a builder culture that kind of emerged a little bit higher level than some of the differences. I mean, I was not a worker on the team, so I can't say for sure what people's experiences was, but I will say working in this space, something that seems pretty prevalent is that feeling of solidarity and builder culture where people are solving problems together. So I guess you could hope that it becomes a superordinate kind of goal. I'm trying to think if we had—

We had very international flash teams. I think if you look at the composition of platforms like Upwork, you have a number of people who are well settled into Upwork, and they are themselves fairly international. But especially if you look at the new entrants, the people who are just getting started and haven't developed, we can have a whole conversation about the 5-star rating systems. Uber, Lyft, Uber Eats, these kinds of things. Those exist on these platforms as well. Now imagine the jobs that you get access to is determined by your 5-star ratings from the people that you've worked with in the past. There's a whole thing to think about there. But many of them are coming internationally and actually working with even domestic US or North American companies because these platforms become a sort of equalizing force that lets them access that work. There's, I think, all sorts of dynamics there that are quite complicated. Just to highlight one that I think many researchers in the area have thought about is, what does that mean about minimum wage? Whose minimum wage should we be respecting? Is it the person who's hiring or the local position where they are operating? How do we make sure that compensation is appropriately calibrated? How do we help these folks enculturate into a new organizational environment? So you have a developer from Eastern Europe meeting a designer from South America who's working with a user researcher from Australia, and suddenly these people need to align their time zones and their culture to work well together. This is where I think, again, taking inspiration from these temporary teams, the film crews, the disaster response teams, and so on, really, in our experience, was kind of like the key ingredient. Not treating everyone as sort of like unwashed masses that are all the same, but actually it's like, here's a role that you can use, you can see yourself in. What Melissa articulates in part of the book is, like, when we didn't do enough of that, people would start, like, foraging around for what their role was supposed to be. They would get massively confused. They show up, like, where do I fit into this picture? Thank you so much.

Great talk.

I'd like to ask about this organizational pattern of kind of when teams form and storm and norm and then perform. You know, that happens, and it's been seen for decades. What happens in this context? You know, because a lot of that isn't necessarily a failure point, like the team could never work together. That's calibrating trust and calibrating how to communicate with people and cultural things, a lot of stuff that's sort of human. I wonder if you just kind of elaborate on that in the context of flash teams.

Yeah, so the— it's forming, storming, norming, right? Did I get the phrasing right? So in team science, this is sort of like a well-known kind of phasing or progression that teams will go through, and it's pretty predictable. Something that I found really interesting about these teams is they did seem to honor the midpoint dynamics that are predicted in team science. So even— no matter how long the team was, the prediction is sort of like things change at the midpoint. When we were doing the Dream Team proof of concept, where the algorithm was helping them experiment at different times, it had to stop experimenting post-midpoint. I'm saying, so team dynamics, like the kind of thing that you're describing, still were like really important. I don't know that we looked— I'm trying to think. I think getting the launch right is maybe how I would say the norming is really, really important. Getting the launch right. I think when people show up and they don't immediately know who they are, who's their colleague, who's their manager, what's their first deliverable, then as Michael was saying, they tend to like forage and then you get a lot of like misunderstanding. So I think that one.

Storming, norming.

Um, swift trust. Oh yeah, that's right. Yeah, would you like to—

I mean, you're the expert. I was gonna talk about, you know, essentially many teams there's sort of an immediate, uh, what's referred to as, I'm sure you're already aware, is swift trust in the literature, where like team comes together and there's sort of like a blanket assumption of trust that can then be broken as people will, you know, whatever, not pull their weight or say something odd, but we would see this very much in the flash teams as well, especially these ones coming from online platforms. Now, they actually share— there's not even like a fracture point. They're all coming from the same online market. They have the same background. They've all shared experiences in doing this online work and so on. And so it was very much that. That was one thing we would see, swift trust. The other thing that I think what I would often— I'm less effective at theorizing because I don't have a good term around this, It was essentially the teams were in constant motion. So yes, you had to get the launch right, but the team we talked about lasted 6 weeks. And so it was just constantly accumulating people and people would move around to new positions. And then, oh, now we need a poet. Now we need an opera singer. Now we need a voiceover artist. Now we need a director. So it was constantly evolving and turning rather than just constantly confused. So once you got past that, threshold of here's how this team works, it became maybe more traditional dynamics for the evolution, just faster. Hi, my name is Chinat. Thank you so much for sharing your insights. I'm a researcher here trying to understand about hackathons. I'm just curious to understand how have you been examining flash teams, or how would you interpret hackathons through the lens of flash teams, especially since I've been seeing mainly primarily a lot of student-based hackathons. But I know that there's a lot of industry-based hackathons or other sectors such as government and outside of academics.

Yeah, I think that's a perfect use case. It's a perfect example. If you talk to me after, I can actually show you a research paper where somebody analyzed hackathons as temporary teams. So I think what flash teams could offer to hackathons is what I haven't seen happen in hackathons is sort of like a measure of the different sort of inputs and process and outputs that happens in a hackathon. Because I think they are so temporary, and they oftentimes are student-led and very prototype-focused. But if you do take the flash teams perspective, then I think you could sort of like learn what makes an effective hackathon, what makes an effective team within a hackathon. Do you have any?

Yeah, I think the roles seem to play an incredibly powerful predictive—

Oh, that's good, yeah.

Yeah, basically, Here's how you get an ineffective hackathon team. People come together and just sort of all start coding and bashing into each other. Whereas if there's clear role differentiation of like, okay, this person's going to own the vision of what we're doing and maybe the front end, this person's going to own the back end or something like this, the coordination just becomes much more fluid, much more quickly. That was one thing that we would see. Hackathons are interesting because along these dimensions on which Flash teams can vary. They often— the hackathon teams often are very high familiarity already. You bring your roommates and you already, you know, maybe you've even built stuff before. So that does change the dynamics somewhat, you know. Here, now let's imagine a hackathon team that, you know, actually in Melissa's class, you can say more about this, but like, we'll actually— there will be a local team that will then reach out to, again, millions of people online, like, oh, we need a logo designer, right? There's no one on this hackathon team that can like do the graphic design quickly. I don't know if there's anything you want to add.

So what I was referring to with my— my classes got too big, so I couldn't do this. It got too expensive. But early days when I was teaching, I would teach undergrads the concepts of flash teams, and the assignment would be come to class with your flash team role structure. We would push play, and then within 24 hours, they had to turn in a completed prototype. Again, this was pre-ChatGPT. Now that's easy. But so they would do that. So it'd be like teams, flash teams of students being able to pull in flash teams of logo designers. So that seems like a way of augmenting hackathons also, teaching them how to think crowdsourcing, think experts everywhere all the time, that sort of thing.

I will say that the publisher made us sign an affidavit swearing that we did not so much as consult a language model in the writing of this book.

That's true. We wrote it in like— we wrote it like 20— we started in 2023, right? And they made I'm curious if that would happen.

Yeah, I wonder if that's still there.

You want to go here? So I was really curious, you know, there's obviously like decades of research going into what are the qualities of like a good leader or a good manager in a traditional organization, but as you're thinking about the shift into a flash team structure, like what are the qualities of good leadership and good management Oh, I'm so glad you asked that. When I give this talk in exec audiences, this is always where I start. And I actually start with the story of a friend of mine who joined a venture studio that had spent $1 million paying for 5 different disparate pieces of work, but nobody had bothered to see if they integrated. So $1 million later, they had 5 pieces of disconnected work, and they did not have, like, a working product. model or prototype. But that person, the person who joined the studio, came in and was like, wait a second. So they brought in a very talented founder. So the founder came in and was like, wait a second, why is none of this working? And kind of proved that their business model wasn't working. Within 2 weeks, the person had shown the million dollars was not working. So what I'm trying to illustrate is there's something about founder energy, and it doesn't have to be startup founder energy, but there's something about integration and leadership So it's like the entrepreneurial vision to integrate, to make sure it's working, to have the end in sight, and to make sure that all of these, you know, these 30 people that you've pulled together have the vision, that everything's connected, that everyone's inspired, that everyone's like pulling together. That's pretty rare, but that is so key. If you do not have like the project manager type or like the founder energy, then it does not work as well.

Maybe to be even more kind of on the nose about that, one thing that we saw a lot in research underlying this book was, when we say you need role structures, you have to get the launch right, it's like, this is true in traditional teams as well. But because you have essentially a fully remote team, in an in-person team, sometimes you can lean on the fact that you're around the corner from each other to sort of supplement or catch each other. Here, you have a fully remote team, and so the leader really has to lean into those best practices. So one of the things that's been interesting to us, one of our endorsers, a colleague of of Melissa's, where he's like, everything's a flash team, every team's a flash team, this email chain is a flash team, right? And he—

And that's credit to Bob Sutton. So Bob Sutton is another— he's a great writer and he's my office neighbor and he's one of our endorsers. Yeah. And when we told him about this, he was like, literally everything's a flash team. And he kept telling us every— like, we were standing in the hallway, he's like, flash team. Yeah.

And so I think the point that I'm trying to draw out of that is that the leadership strategies that we saw in this almost like, in some cases, extreme spaces where it's like you have the HIPAA expert living in, I think it was Egypt, the same time as the software engineers were in India and the US and so on. You have to lead, right? You can't just rely on it just happening. And now, speaking as an interpreter myself, I see that so salient, like that the things that just work when it's you and your friends really don't work with these other kinds of teams, and you have to learn those skills. And there's nothing unique there, but it just becomes way more important because you have nothing to fall back on. Yep.

Thank you for the great presentation. So you touched on how receptive teams were to changes, and particularly when there were a lot of suggested changes, they would kind of ignore it. But once the types, or maybe the amount of changes were reduced, or, uh, more specific, they were more likely to adopt it. Could you elaborate on that and maybe provide recommendations on the types of changes or how to correctly suggest changes for teams to be more likely to adopt them?

That's a great question. This was a really interesting example of kind of an impedance match between, you know, organization science and computer science and AI, where, you know, the organizational science literature has articulated a number of theories, or if you want to be more pragmatic about it, rules of thumb about certain kinds of changes make most sense in certain periods of the team's dynamics. So at the very beginning, you can make these kinds of changes, but don't change the hierarchy in the last inning of the team, right as you're running up for the deadline. And anyone with an advanced degree in org sci can probably rattle those off. Most of them could probably give a lecture on it at the drop of a hat. And so we basically, in this project, I basically just sat and like took notes from Melissa for a while and then turned that into essentially, it's just kind of like where in the team's phase can this change happen or not? And something about essentially what you think of as change management, like how much change can a team absorb at once? And this was actually the biggest issue with you. You could have all of the right ideas, But if this network of bandits were just sort of playing them all simultaneously, imagine your team started and it was like, wait, change your leader and be more and have scrum every day and be mean to each other and do all these things. And you're just like, whoa, you just, people just, yeah, they, I mean, freak out, but then sort of just tune it out.

Mm-hmm.

And so it really became important To model how much change you were pushing the team to do in order for this to succeed at all. And if I were to boil it down, it was actually that. It was like, we could talk about where in the team, when in the team, and so on, but at some level, it was about being able to understand how do we nudge teams a little bit out of their comfort zone so that they explore a little bit more? And you could think of this at an individual level too. You're trying to, I don't know, finish your project or whatever you're trying to do, make yourself better at this sport or this skill or whatever, we tend to under-explore the design space. That's just sort of how people work. We suffer from fixation and other kinds of cognitive limitations, shall we say, that keep us kind of hemmed in and sort of only see the horizon around us. And so it was basically people buying in enough to try things and then making good enough recommendations. You know, if we weren't right, I'd figure it out the next round. But if it was always wrong, then people would ignore it. It was kind of that Goldilocks zone.

Thank you for a wonderful lecture and book project. I'm looking forward to reading it. And my question is actually about— but with going back to office, actually, looking at physical spaces and design of the space as a variable. And the example that I have in mind is startup, early startup companies adopting to a mid-century modern open space office design of 1950s and '40s with panels mainly in the open space. And you can use this space and sort of modify it for a conference and also creating flat team with, for instance, a CEO sitting next to the software engineer, junior software engineer, or move to a different space. So this sort of fluidity which in a way, instead of creating temporal teams, create temporal relations amongst permanent teams. And I was just wondering how that could potentially modify or affect this research. I think that's a really cool idea. You'll have to forgive me, where my mind went first was a friend of ours is a professor at Michigan, Jerry Davis, and he always talks about Pokémon Go. So, like, with an app, you can, like, get people together, like, quickly. So, I always liked that example, because that to me is basically combining the affordances that we talked about in flash teams, you know, internet, everyone has a phone, possibly some, like, I don't know, possibly some optimization, but you get everybody to the same place with the same goal, kind of having this in-person interaction. That's really very fun, by the way. Does anyone— Pokémon Go here, right? So you like it, you're convening, you know, you're convening in person and having like a social experience. So, okay, so that's Pokémon. That's what was making— I was like sort of imagining you're describing like startups in a warehouse in this space. Like, just similarly, could you figure out like different ways that people are coming together and like understanding what's the nature of the group, what's the interactions of the group, what's the outcome of the group? In ways where you're, again, drawing on data about what happens to really support the kind of in-person—

Yeah.

Totally. Yeah. Ooh, I love that. Yeah. So the in-person is a variable in it, I'll just say. Yeah, I love that. We didn't write about that, but I think that is a very cool idea.

And there's certainly research going back many years. The stuff I'm familiar with is Bob Kraut in the 1980s showing that as we get further away, our probability of collaborating drops off exponentially. And if you know that that's the case, I don't know if you remember, we had this graph of the spaghetti network of collaboration. It may be that you actually need to balance between people feeling like they're able to— they have a place they live in this physical space against maybe we should always be in constant motion just a little bit, just on the edge of it being uncomfortable in order to spark collaborations. across what Ron Burt would call structural holes in the network so that we can sort of see things that we didn't see yesterday. Yeah.

Yeah, cool. Oh, sorry, I just want to say, I'm thinking of, for instance, reinforcement learning, right? That it's actually you reinforce the data, but what if the space actually, with going back to in-office, the reinforced learning to be the real, you know, the reorganization from ground up through reorganizing people to create those, you know, non-hierarchical spaces. Yeah, yeah, that's a cool idea.

I mean, I think that my final thought on this would just be, you know, what's wild to me is that we, you know, most of the time we don't even stop and measure this or try to do it. Like, we just sort of let it happen. I think maybe one of the things that astounded me through working with Melissa on this book was how much that if you just actually kind of tracked it and said, what if we tried to do better? Even a very straightforward algorithm can do better.

Yeah. All right, last question. Who's got a good one? No pressure.

Oh, please. Yeah.

Hi, I'm Taju. Thank you very much for that. I'm curious about, since AI is essentially becoming a lot more competent, we've seen with the ChatGPT example there, one, what relational dynamics you're seeing change, but also I guess like marketplace dynamics with like Upwork, for instance, if AI can do a lot of the tasks on the flash teams that people will typically do. Yeah. How does that change the market dynamics at Upwork? Yeah, there's, uh, thank you for that question. There's, um, so Upwork is very interested in this question and then they've started, um, doing a lot of research on it. So I might have the statistic wrong, but it's like ballpark, right? I think they were figuring out something like 13% of the jobs posted on Upwork they think could be handled by agents at this point. So they are very focused on what is the future of work that they're trying to build with their contractors. And the thing that I think is great about their approach is they're looking at it as sort of like a community relationship where they have— they sort of owe something to their contractors. Like, they need to be transparent. They need to, like, let them know what does the future look like. They're thinking of things like how to help the contractors reskill so that all of them are using AI really regularly in their work. So I think, and then with that, the contractors want to stay. They stay because this is a place where they can grow. This is a place where reskilling is possible. This is a place where they still have high-value work available to them. So I think Upwork is doing a nice job of it in a way that is, I think, useful for any company that's thinking about this. It's like just being very transparent about people, the work is impacted, here's the reskilling that you need to do, here's the tools on how to do it.

Yeah, I mean, we saw this actually previously on Upwork. There was a moment, John Horton at MIT has this great paper about, he watched on Upwork the moment that Steve Jobs said iPhones will never support Flash. There were all these developers who were experts in Flash who very quickly reskilled over into JavaScript and other kinds of of areas. And so I think, you know, people see and react. I think there will be a shudder in the marketplace as people reskill, but at the same time, I mean, we see this in even the undergraduates here, they're sort of absorbing these tools very quickly and our ambition only gets, you know, our reach only grows as we can achieve more. So I actually think that it is, it's not gonna change what Upwork tries to do at a fundamental level, But it might change how quickly certain kinds of jobs on the market can take place. It's a really good question. And Eric Brynjolfsson here at HAI also thinks quite a bit about these kinds of questions. So he would be another brain to pick on this.