Michael Bernstein | Designing artificial intelligence to navigate societal disagreement

Good afternoon, everyone who's joining us today. Welcome very much to this talk. My name is Peter Loewen. I'm the Associate Director of the Schwartz Reisman Institute, and I'm also the Director of the Munk School of Global Affairs and Public Policy. Before we begin today, I do want to do as is customary and right, acknowledge the land on which the University of Toronto operates. This has for thousands of years been the traditional home of the Huron-Wendat, the Senecas, and most recently the Mississaugas of the Credit. And we're very thankful to be here on the verdancy of Tattle Creek and here on Turtle Island to work and to play and to come together. A few logistics just before we begin. This session is being recorded, customary thing, but you should know. And the second has to do with the format of Michael's talk. He will speak for about 50 minutes, and he does invite questions during the talk. I'll be monitoring hands if you want to use a raise hand function. or you want to put a question within the chat so Michael could focus on giving the talk. And I'll flag those for you, Michael, as they come in.

Okay.

So without further ado, it's really my pleasure to welcome here today, joining us from Stanford, Michael Bernstein, who's an associate professor of computer science and is also the STM Microelectronics Faculty Scholar at Stanford University, where he's a member of the Human-Computer Interaction group. He's got a remarkable record of work that centers around the design of social computing systems. He's won the Best Paper Award at top conferences in human-computer interaction, including CHI, CSCW, and UIST. And he's been reported in venues such as the New York Times, New Scientist, Wired, and The Guardian. He's been recognized with an Alfred P. Sloan Fellowship, a UIST Lasting Impact Award, and the Patrick J. McGovern Tech for Humanity Prize. He's got a bachelor's degree in symbolic systems from Stanford, in case you want to know that he was credentialed. And he's got a master's degree and a PhD in computer science from MIT. That's a long list of accolades, Michael. I'll just say that one thing I really appreciate is that you wrote a great paper on how to think about research ethics in AI, which has really, really changed a lot of the way I'm thinking. So I'm just so pleased that you'd join us today in Toronto, and I'll turn it over to you.

Thanks so much.

Thank you. And now I have to live up to that introduction. Excellent. I will in fact be hopefully talking a bit about that effort that you just mentioned. So yeah, I'm a computer scientist. I'm a researcher in human-computer interaction, which has a long history of really, really impressive work at the University of Toronto. So very excited to be participating here today. I'm gonna talk about how we deal with societal disagreement as we're designing AI systems. And when I'm talking about this, I bring this up because I spend most of my time thinking about how we design social computing systems. Think social media, crowdsourcing, collaboration systems, Zoom, all over the place. And the way that AI gets instantiated in these worlds has some very particular flavors that raise questions and challenges that I think when we just sort of think about AI in the abstract, maybe we start to overlook. And I wanna actually focus for a moment on ground truth. Ground truth, of course, is this stuff that you use to train the AI algorithm, and it's incredibly consequential. And the reason why I wanna talk about it is because there are so many spaces in which, as a designer and as a technologist, we're embedding now artificial intelligence classifiers inside of online social environments. This may be things like toxicity detection, which are being used to sort of flag and remove posts from online communities. Your Reddits, your Facebooks, or your New York Times comment section. Misinformation detection happening through AI. Many moderation decisions, hate speech, what qualifies as hate speech or not, happening in many cases getting flagged by AIs. You see products that look like this. This is Google's Perspective API. You can integrate it into, I suppose, whatever you want. You feed it some text like the following text and it gives you some probability that the text you fed it is likely to be deemed as toxic by its annotators. I hope none of you actually bang your head into a desk, that tends to be painful, but just to give an illustrative example. When we talk about ground truth, I bring this up because, well, that one may be clear, maybe not depending on your point of view. But there are many where actually what the correct answer ought to be, what the ground truth that the AI should be emulating is, is actually a big open question in my mind. Here's a comment from a large dataset that is used to train these models. Someone's commenting online that it's just absolutely ridiculous that anyone would still eat at Pizza Hut, that it's gross. Then they go on a long rant, which I've elided here, about how it's shameful that this country, blah, blah, blah. You can probably imagine how the rest of this goes or toss it into a large language model and GPT-3 will tell you exactly what happens after this. Now, if I were to pull this group Should this— this is clearly aggressive. This is clearly offensive in some ways. Should it— does it cross the line? Should it be removed? Is it toxic? If you all are anything like the surveys we've done on this, it's actually a split jury. There's about 50/50. About half of folks think this should be taken down and half shouldn't be. I bring up this example because that entire situation gets entirely removed by how we train AIs today. So I've walked in the sort of crowdsourcing and data annotation circles for about a decade now. I can say with a lot of confidence that what we tend to do is we take our big datasets, we tend to then go ask multiple annotators or labelers to say, what's the correct answer here? And then we're going to aggregate their results to identify ground truth labels. So let's say we took that Pizza Hut comment and we say, is this thing toxic or not? We give it to 5 different people, 3 of whom apply one label. Here I'm just saying gray. Another 2 give it a red one. We're gonna then aggregate it. How do we aggregate it? Well, turns out typically just majority vote. We're just sort of going up straight majoritarianism. And what comes out of this is a single ground truth label. Great. More people thought this was gray than red.

So we're gonna go with it.

Of course, we do this a number of times. We aggregate a ton of these things. We train this deep neural network, whatever it's going to be on top of it, and the network learns to emulate those labels on new unseen data. Okay, fine. But for many tasks, including the ones in which I tend to traffic, even experts disagree on what the correct label ought to be. In traditional machine learning tasks, the kinds of things that ImageNet-style AI tended to grow up on, this may not be a big issue. Is this a cat or a dog? I would argue that these tasks tend to operate at the perceptual level of our cognitive systems. They're not higher-level cognitive reasoning going on. You hopefully are going to say a dog. Otherwise, maybe I have some issues with my own perceptual system. But when we give examples like this in social computing, social media, many societally oriented tasks for which AI is getting deployed, there's a lot of disagreement. This is important because the aggregation process that we're using silences minoritized groups. Because when this disagreement is irreconcilable, it really matters who we're sampling to give those ground truth labels, right? Let's say that the 3 people who labeled it as not toxic, they look a lot like me. They're Caucasian, they're male, they look like the exact average computer scientist. And then Folks who see this as toxic will often come from minoritized groups. They're often subjected to harassment online, but because they're not a numerical majority in the sampling process, they're then going to get overridden by this majority vote, and it's labeled as a flat not toxic, essentially just entirely erasing that. I hope to just convey to you that this kind of disagreement is prevalent. It's all over the place in these spaces, and we've been ignoring it. So in this dataset that I've just been— drew that example from, If you just look at everything that's labeled as toxic, actually about 40% of annotators disagree with that classification on average. That's an average. There's lots of other literature in my own field and elsewhere that talks about how even experts tend to disagree with each other. Here you have moderators for one of the most popular websites in the world disagreeing with each other about a third of the time on whether something ought to be removed. Off the record-ish, large social networks, you may know, use fact-checkers. One of them had a bug at some point where they actually fed the same article to the same fact-checker by accident multiple times, like a few days later, and the fact-checkers were disagreeing with themselves a third of the time given the same information. Disagreements all over the place. It's also prevalent as we're deploying AIs out in society. In medicine, we're working with some groups, we're chatting with some groups at UC San Francisco, You know, when specialists disagree, whose diagnosis gets to move forward? When different instruments are sampling and they have different results, how do we make sense of like these different readings? In design, closer to my own field, and actually here an area that a lot of DGP folks have worked on, there's different designers are gonna have different schools of design that they come from. You know, if you're a Bauhaus designer, you're gonna be thinking differently about a visual design than someone who comes from a different kind of design school. What ought we to do about that? So today I'm gonna sort of unpack this in 3 different ways. I'm gonna focus first on what do we do in terms of the models we're building? Can we reconsider sort of how the AIs function and the metaphors we use to design them around this notion to more productively deal with disagreement? I'm gonna then turn my attention maybe a bit more briefly to what we do about evaluation, right? So maybe you can't change the model, but maybe we can stop some of these issues happening in their tracks by looking more critically at how we're evaluating these models. And third, as Peter actually sort of referenced, I want to step back for a moment and think about what kinds of institutional responses that we might have that can think through some of these issues and maybe ameliorate some of them upfront. So sort of going down from the model, zooming way out to the institution, which is how I like to live. All right. So let's start with how we design these AIs. This is work that was led by my PhD student, Mitchell Gordon. He's graduating this year. Everyone should hire him. He's great. It just appeared at CHI. We're going to talk about how we design AIs to navigate this situation. Now, generally speaking, you have 2 approaches that you can follow if you're a practitioner and you're trying to deploy an AI. One, do exactly as I said, collapse everything into a single ground truth. 2, instead of predicting a single thing, you predict a distribution. You say, well, about 60% of people are going to say this, 40% of people will say that. Unfortunately, the second one runs aground as well, because while that absolves the AI itself from dealing with the disagreement, 2 things happen. One, that predicted distribution is going to depend on who you sampled. If you didn't think critically about who you sampled, whose voices you're listening to, that's going to run aground. Second, most products at the end of the day end up making classification decisions. They can't show you, well, 60% of people think this is toxic, they have to decide, is this up or is this down? We tried to come up with another approach, one which we drew on the metaphor of a jury. We call this jury learning. The basic approach is that we want to shift the metaphor of how you think about whose voice the AI ought to be emulating. Where you would now specify a jury, it could be any size. Here, I'll draw 12 because 12 people fit on a slide. And you say what proportion of that jury should represent each group and intersectional identity. So you're gonna identify some set of characteristics of individuals who you might want to populate the decision-making system here. And then the model, we're not going to literally hold, run juries every time the model needs to make a decision. We're going to use that as a framework for making a joint decision that's essentially proportionally representing these groups. So again, given the running example I've been using, if you're dealing with online harassment, you might define a jury that's going to represent groups who are commonly targets of harassment— women, Black individuals, Black women especially, these kinds of groups. So the—

Can I ask a question, Michael? Sorry, can I ask a question just before you go there? I put my hand up and I saw it myself. Is the Is the composition of juries by ascriptive characteristics just used as an example here, or is there kind of a belief embedded in this that identity is going to actually get you to the primary differences that you care about? Because you can imagine that the minorities that you talk— when you say, you know, there could be a minority that's always excluded if there's a majority decision, If that minority is constituted by something else like ideology or belief or values, not things that you can see, how do you address that in jury composition?

Right. So I think what we're trying to do here is to get— create a structure by which the developers and the product managers, whoever they would be, need to think about those kinds of questions upfront. Whose voices ought this AI to be listening to? I'm here using identity groups as an introduction to this, but it's any identity group that— or excuse me, any group or set of characteristics that you would want to collect alongside the data annotations themselves. If you think that political ideology matters, or if you think that age might matter, or if you think that attitudinal measures of various sorts might be the thing that you would want to actually describe the jury using, this, this will work with that. It's essentially, at the end of the day, any— you would need to have the sorts of information that you need collected so that you know who it is that you're trying to emulate. And sort of, as I'll show, you'll need to be able to connect those characteristics to example annotations from that group. But that's sort of all, you know, the only assumption is you need to describe Something. Yes.

So why not just randomly sample from the population to compose the jury? Because then, because then, you know, in the limit at least, you're getting, you're getting, you're getting a population-level distribution that's, that's equal on even on things you can't see, right? On, on dimensions that you didn't know were relevant.

So I think there are 2 responses there. One is that, so that would certainly improve. It certainly is more thoughtful than doing what we tend to do in computing, which is go, you know, hire some some workers online or some firm to do this, and you would probably get more of a stratified sample in the sense that you do if you're doing polling or something like that. Now, you still may have the sort of majoritarian critique here where the minoritized groups will not— will sort of get suppressed out of the system if you're not careful. Especially intersectional identities could get easily sort of undersampled in this space. But you could do that if you felt that what you wanted is the AI to proportionally represent the population of some group, you could certainly do exactly that. The second reason is more of a practical reason, which is that in most of the literature on the design and development of artificial intelligence systems in society, it's not a waterfall model where we decide and then we go collect the data and then we train it and then it's deployed. In practice, it tends to be much more highly iterative. When we realize something goes wrong, we might have to go do some other stuff and retrain it, etc. As a result, we felt that it was largely unreasonable to expect that you would go sample the data once in proportion to the voices that you would want to have happening. In some sense, in the language of maybe like of like a survey researcher, we are doing the stratification, but the stratification is almost like post-stratification. It's happening after the data is collected. So we're giving more fluidity in doing that.

Great.

This is where I benefit from my wife having been a political science PhD student. So the output of an AI system here no longer just carries a different kind of interpretation than your traditional classifier. Instead of just saying this is the correct answer, essentially there is something that is more explicitly stating, you know, for the following jury composition here, I'm just giving an example, you know, 56% of them agree that this talk, that this comment is toxic. So the AI is now carrying some very specific stratification, if you will, of whose voices it's listening to. The real effect here is that rather than building AIs that model a single aggregated pseudo-human, like this aggregate ground truth, we're actually going to be teasing this back apart such that the AI's task is no longer to predict a single ground truth, but to take every annotator in the dataset, and to predict, or juror here, their individual responses. Each individual who we have in our training set, we will then be able to teach this AI what would they say, and then we can then aggregate that as a result to make, say, a classification decision. Again, this slightly changes the semantics of how the AI works. We think in a way that's fairly straightforward and interpretable. In practice, you would end up using our system in the following way. You would define first a set of juror sheets. Each of these are sort of strata you might care about. So liberals or parents who have high school diplomas, whatever you think it is, and you'll give each of them a certain number of seats on the jury. Here, a simple jury of just 4 people. This is not particularly balancing anything, but just to show the sort of expressivity that you might imagine. Following this, you get 4, you know, the jury seats. There are 4 of them in the one that was just described here. Again, it could be sort of as large as you want. For simplicity, I'm showing a very small one here. Now we have a very large population of annotators in the dataset, some of whom will match each of these characteristics. When you do this, we can then sample randomly from the set of individuals in the dataset from that population, people who match each of those characteristics to fill these seats in the jury. At that point, we use our technique to predict each individual juror's response. Aggregate that, and make a classification decision. Many AI systems are not classification. We can do the same thing with a regression outcome where it's trying to predict a continuous number. Now, one thing that's hard to do in real life, but we can do very easily through our system, is running many parallel juries to understand what would happen if we actually had many different groups consider this. You can bootstrap resample here with replacement, get many different jury compositions. Each of them comes up with its own verdict. This gives us a distribution of jury outcomes where each of these histogram points might be a different jury with different folks. We, for the purposes of our system, select the median here for the final decision. It's called a median of means. It has some nice characteristics, but you can imagine other kinds of decision rules from there. One thing that this helps with is it does solve some of the challenges with explainable AI. This allows you to do some interactive sensemaking around the nature of the disagreement. For example, there's often more disagreement within groups than across groups. Here we might be able to visualize the 2 folks who were sampled for this particular jury fall here on this histogram of opinions for this— predicted opinions, excuse me, for this input. I just want to, at a very practical level, call out that I'm not proposing something here that is going to require a huge amount of re-architecting for many deployed AI systems. In fact, all we really require is a bunch of annotators' individual labels on a subset of the items. It turns out that most of this requirement is matched by essentially most of the annotation datasets that are already out there. You can take nearly any crowdsourced dataset that you're using to train an AI and do it this way. I've sort of jumped over this, like the how does it work. I want to give just a very high-level flyover of what's going on here when the AI is trying to predict each juror's response. So we have a typical dataset here, and if it's a typical dataset, you're going to have a situation where you're going to have your jurors, each of whom is going to have responded to some subset of the training examples here. So like yes and no, positive example, negative example, and a bunch of unlabeled things. These are— it's a sparse dataset. There's also a bunch of folks who are not currently jurors who likewise have labeled some subset of this data, and there's a bunch of missing values, and we want to be able to make predictions on things we haven't seen before. So given something like characteristics we know about these individuals and their other votes in the dataset, we have some new example. Some comments on my talk, say, and we want to be able to predict what would this juror, what would this annotator in the dataset have said to this new thing. Without going too much into detail, we can really leverage a lot of ideas that are already out there, but then put our own twist on them by basically combining existing AI techniques with something that's called a deep recommender system. What's a recommender system? Think of stuff like Netflix. Amazon, all these things that are saying, like, people who liked this might also like that. We're able to learn embeddings of how folks of various groups or characteristics and how annotators themselves, patterns between how they vote. So it might learn, for example, that if Işık and Peter have similar opinions, and then I give 1 vote or 2 votes or 3 votes and I'm starting to look like them, it can predict that, well, my votes on the other stuff is probably going to be similar to how the 2 of them would vote anyway. There's a high-level notion here that essentially what we're doing is that the model is taking 3 things into account. One is the language, like, would this person— what topics does this person feel like are crossing the line? The annotators themselves. This is learning relationships between people who tends to vote similar to which other people, and then annotator characteristics, meaning the groups that we've identified. This all goes, gets thrown into what's called a cross-network, which is a technique for recommender systems, and makes a prediction. Now, lots of critiques you might have here. One, okay, jurors can be biased. Absolutely. Yeah, Istiak, do you want to— Yes, I have just like a clarification question.

So, you know, in the last slide you were showing this BART tweet, but, you know, so, but the BART tweet also needs annotation, right? Annotated level data. So how are we getting them?

Right.

So BERTweet, we're using pre-trained. So basically we're taking advantage of the fact that it has a sort of a pre-trained language model, which is— so it's already knowing that, you know, certain words are sort of tightly related to each other, and then we are fine-tuning it using our dataset. Did I answer your question? I think you're asking something different.

No, no, no, that's fine. I mean, I mean, the There are questions about the way BERT-Tweet was trained too, but I guess that's a different question.

Yeah, yeah. So it's a good point. In an ideal world— so let me start by saying I'm not tied here to BERT-Tweet. It was just sort of like the thing that was closest to what we needed, and we were able to rely on the fact that by fine-tuning it with our dataset, it would sort of get re-entangled in the ways that we would want it to do. But you can imagine Really, that should be replaced by any language embedding that you would care to use. Yes, jurors are biased. This is a common critique. I think that I want to unpack that a little bit. If you gave me a choice and you said, look, Michael, one option, you can use some of these algorithmic debiasing techniques for your AI. I'm sure you've all heard of these things. They're trying to enforce various group fairness requirements on your AI. That's on one hand. On the other hand, you could say your option would be to select and train appropriate voices. My vote is on the second one. The reason is at some level that once your dataset is dirty, it's really hard to clean it up in a way that you don't know something else isn't going wrong. In some sense, if you've got a pipe full of sewage mixed into the water, it would be much easier to actually just not throw the sewage in the water than it is to try to clean it up afterwards. Now, there's another sort of power question that's at play here. Who gets to decide the jury? So I'm making the voice explicit through this, and that, on the upside, opens opportunities for contestation. In fact, we're working currently on some methods for sort of participatory efforts for defining the jury and how well that works or not. My hunch is that— that we're trying to test— is that That sort of people will be more equipped to argue over jury composition than they are going to be to argue over precision-recall curves and confusion matrices and so on. At the same time, some big company might just say, okay, and hide the jury composition and do whatever they're doing. I just want to point out as a partial response to this that all datasets already have a jury. It's just implicit. It's hidden in that majority vote, in the sampling procedures they use, and so on. In fact, using some, a little bit of math, we can sort of backwards solve for what's the composition of the jury in these kinds of datasets. The one that I've been referring to here is about 74% white. So I think the hope is that by— it's much better to make this jury composition transparent so that stakeholders can deliberate over it. It's a sort of Transparency does not help all things, but at some level, I would rather that we at least need to think about it. We're forcing ourselves to think about it through this approach. There are other opportunities I won't really go too deeply into. Again, through some fun math, you can essentially create a counterfactual jury. You can say, what change to the jury composition would have made this decision flip? Okay, this juror, if the jury composition had been this, then maybe you would get— you have gotten a different result. This allows, again, sensemaking on like, well, why? You know, I thought that given this composition, this jury, clearly they would've found this toxic, but they didn't. So what's going on? Now, we also did a field evaluation here, relatively small. And oh, should I pause? Yeah, Mohammad. Hi, my name is Rifat. I have a question about the jurors themselves. So, Yeah, so if we have different jurors, they'll have different background and different ways of thinking. Otherwise, they have different kind of polity, right? Different kind of ideologies and, and the way they'll like to take the decision. So if they have disagreement within the jury board, how do we address those disagreement among the jurors? Is there any way to address this in the system? Right. One thing that you're sort of implicitly pointing out, and I just want to call out explicitly because it's a limitation, is that, you know, there's a bit of stress on the metaphor here. This is not a jury in the sense that they are not deliberating, which we would typically think of as sort of a core compositional characteristic of a jury. You know, this is essentially about, you know, sampling and voting. It's in that sense more, you know, an electoral process than it is a jury. We've done some work where we want to actually try to estimate how jurors might influence each other through deliberation. So you can imagine that each juror would then produce a rationale for why they voted that way. They can then see each other's rationales and then adjust their positions if interested. We've started work on that.

You—

it's essentially you need to collect data on that and what causes people to change their minds and not. You'd have to think about social factors at play. But I think that the Currently, the model does not really engage with that deliberative aspect of the metaphor very deeply. I think that's to its own detriment, but it was also just a practical limitation that we said, okay, that seems like an interesting question that we might want to tackle separately. So here's some questions we asked. We got— it's hard to recruit professional moderators or even— excuse me, amateur— not amateur. What's the word I'm looking for? Volunteer moderators. Thank you. volunteer moderators who spend a lot of time on these things. They're effectively professional, just unpaid. We recruited these volunteer moderators and maybe even some who are paid from platforms like Reddit and Discord, which are large online community platforms. We asked them to basically author juries for an AI that they would want to deploy in their communities, which again can be quite large. Tens of thousands, millions of people. We measured a few things. One, are the jury compositions representing more diverse opinions than the state-of-the-art classifiers? Which again, I'll remind you, those ones are 74% white. Second, we were measuring whether it changes the classification outcomes. There's representational diversity, but is this actually leading to changes in the behavior of the classifiers? Here's one thing we found. A way to interpret this graph is that the red lines are the representation of each of these groups in the training dataset that is sort of out there, this large training dataset. You can see the bar in the upper right. That's where I'm getting the 74% from. And the blue bars are these estimated— well, sort of, excuse me, mean and then confidence intervals around the representations in the juries that these participants created. In essentially all cases, they increased the diversity of minoritized groups, mostly by reducing the representation of white folks in the juries. Now, did this lead to changes in the classifier behavior itself? Yes. In fact, about 14% of the classifications changed, which if you step back and think that people are celebrating 3-5% improvements in the performance of these classifiers, This is a massive shift just by changing the definition of what the classifier is trying to do. Now, what was most likely to flip? Most likely to flip were topics that were essentially on this, that were contested, so that by changing the demographics of the groups that were making the decision, things like discussions of racism, death and suicide, LGBTQ+ issues, mental illness, the police, and so on, these tended to change valence. as compared to the state-of-the-art classifier. Then there were things that were very unlikely to flip. On one hand, things that everyone agreed were quite positive, doesn't really matter who you ask. Thank yous, happiness, hugs, weddings, that stuff. Then there were also some things that everyone agreed were toxic and changing the composition was not going to shift that general consensus. Stuff around human trafficking, various racial epithets. Those were never under contention that this might be okay. Changing the composition of the jury did not change those. I just want to very briefly mention that, as for AI folks in the room, this actually also produces better performance. The reason is that it's hard to predict an aggregate test set label because you don't know who's been sampled. So as soon as you start applying our approach where we can now model who's being asked to make these judgments, our accuracy actually goes up. It does better. So I will Pause here for any other thoughts and questions before I go on with the talk. As was mentioned, I'm happy to make this interactive.

Michael, I've got a question, which is just to— it's just really a practical question, which is how hard are these moderating tasks? And is a model here to try to actually kind of You know, recruit from— it could be random sampling or not, I understand it's not important to that, but recruiting large samples basically through like online surveying, having people then make judgments and then getting a large number of judgments from that rather than relying on the kind of slightly more precise expertise of a smaller number of jurors.

Right.

So that way you would get more kind of variation in baseline characteristics, right, that you could then sample from. when you're constructing your jury scores?

Right. So I think that that would certainly be a step forward. Now, there are— so there's community-specific norms. Ishwar at USC has done a lot of stuff here. So for example, if I'm in an LGBTQ+ group, certain epithets I may bring up and discuss with others in the group to say, like, I got called this, how should I be responding? Sometimes they're used in, like, as affectations and, like, sort of, collegially. And if you just have one sort of one-size-fits-all sample, you're not going to catch those. So I do think that at the end of the day, we want to be able to tailor these to specific community norms, what are referred to now in the literature as micronorms. And I think the way you do that is you might start with something that's trained broadly, but then through a smaller application of just, you know, some data within the community, can then do better. So I think you are asking a deep question, which is, at what— when should we be sort of, um, doing a kind of representational democracy for making these decisions, and when should we be relying on some, you know, like the elite few, like the sort of Plato's Republic style? And I think that, you know, generally that's going to depend on the forms of legitimacy you want to instill within the community, right? There are going to be some communities, like if I were going to create something for, say, Stanford, it feels to me that I would probably not want to rely on some small number of moderators, but instead actually have some sort of more random sample, like, just representing the breadth of the community. Or I might want to intentionally construct a polis here that is, like, you know, overrepresenting some of the minoritized groups. Like, I think that's the tension, and I think that that has to get resolved at the sort of individual community level, because they have the knowledge on the ground.

Yeah, thanks very much.

Matt?

You know, I didn't think this was the case, but I think this question is related. So I have sort of 2 related questions related to each other. So one is, I'm conscious of the fact that you're focused on content moderation, and I'm wondering how— but yet your sort of thought process around this is sort of broader than that specific dataset. So I was wondering if you could reflect, and this may be a bit speculative, on 2 changes or differences that you might engage in. One would be, what aspects of this work would change with different types of datasets? Not content moderation, but something like radiology data.

Mm-hmm.

How would this work be different in that kind of world? And as related to that, what aspects of this work would change with different identity characteristics? that you would draw upon, not, for example, ethnicity, but things like education level, expertise, field of study, substantive forms of knowledge difference. Just wondering if you could reflect a little bit on those 2.

Yeah. So the first one, domain differences, I actually— I can say a little bit more precisely because we've been chatting with folks in medicine who come to— I've spoken about this, and sometimes there's like a medicine faculty member in the audience who says, let's go talk. because we have some needs here. In these cases, I'll give you one example. There's a faculty member at UC San Francisco, which is a medical school, where they have different— different labs have slightly differently calibrated sensors. And so knowing which lab this sort of result, this estimate comes from, actually becomes very helpful. And so they want to be able to model as part of their prediction, where did this estimate come from? They want to, at the end of the day, be able to kind of, instead of creating an aggregate single result, what they want to be able to do is imagine glasses that had different lenses. I can sort of say, what would this lab's sensors have said about this thing? What would this lab's sensors have said about this thing? And our model enables exactly that kind of thing. So if we were in that space, I think we would have wound up with not something that's just you know, a jury, the jury metaphor itself might not have existed. Instead, you would've had sort of like this multimodal AI that's like able to put on different perspectives, 'cause that's really what it's doing under the hood is it's just being able to—

Sorry, if I could just sort of restate that and make sure I understand. So in this case, it's not identity per se that are the characteristics that serve as ways you select the jury, but sort of the technical characteristics of different instruments.

Yes. And then also brought up, so you're absolutely correct. They've also brought up, or different faculty have also brought up, the idea that we might want to be able to again put on different lenses to say, what would, you know, a radiologist see when looking at this? What would this other kind of specialist see when looking at this? So it becomes a sense-making tool that allows me to sort of see as someone else sees, kind of not exactly an empathy tool, but sort of like an expertise shifting tool. Your second question was around, you know, if you weren't You'll notice that we're very focused on identity characteristics. I think that's reasonable given the recent history of identity and race and ethnicity in the United States. That's kind of where we were starting. And there's nothing specific about the modeling approach that makes it about identity characteristics. I'm trying to think about, you know, yeah, as you say, age. I feel like there's interesting questions there around the sort of like paternalism of like who gets to make decisions at different ages that I think I just want to call out again that our Our approach is not gonna be able to solve like power and status issues, right? At the end of the day, someone is probably controlling what that composition looks like. And if, you know, and if it's a bunch of, you know, elder statesmen deciding, well, older statesmen should— elder statesmen should be making decisions, our model is not gonna fix that problem. Like, you know, we make them be explicit about it, which is, I think, a step forward. We're sort of like, oh, so who's your jury? Oh, Elder statesmen, all, only elder statesmen. Okay, maybe let's talk about this, but I think we can't go any further than that given the current approach. Cool. I want to actually— this is going to sound like a pivot, but it's something that sort of surprised me, and so I want to bring it up, that this project led to a second one, which is nominally about a very different topic, but I'm going to hit it very quickly. This work was led by Michelle Lam, And a highly overlapping rest of the team, including new faculty Denai Metaxas, who's at UPenn. And essentially, it turns out that if we have this model, we can empower end-user audits. So you may have heard of algorithm audits. These are typically conducted by technical experts to kind of test the properties of an algorithm and say, does this algorithm behave differently, you say, if I'm male than if I'm female? But at the end of the day, the things that we audit are limited to the hypotheses that the auditors can come up with. Groups of individuals who witness impacts that auditors might not. Again, because if you look at the population of researchers, we sort of tend to bias in particular ways as well. But typically, if I wanted to actually do such an audit, I'd have to label thousands and thousands of system outputs. So it turns out that the same underlying technical approach also allows us to empower end users to do this. So I can, as an end user, just go in and label like 20 items. As a result of that, our model can then actually project and predict what you would say on the rest of these hundreds of thousands of items in the dataset. This allows you to then look at how would this algorithm perform for people who share my opinion, to construct audit reports from it, and then surface this in a more structured way. Now, essentially all you're doing here is labeling some subset of the data. We're then predicting what the algorithm— through our algorithm, what you would say on a bunch of other items. Again, sort of collaborative filtering style. And this allows us to then construct sort of reports that make a broader case for what's going on. Here you can see some histograms of where this thing is agreeing and disagreeing with your labels. So I bring that up very briefly just to sort of point out that this idea of being able to annotate and model individuals, I think, has legs. Now, I promised I would hit evaluation. This one's going to be a little bit briefer than jury learning. But again, I want to sort of ask, well, okay, it may not always be the case that we can deploy, you know, change the AI that someone's using internally. But we might as well— we at the very least should be able to adjust how we sort of judge success. This exact— this is the same exact same running example I've been using. There are Kaggle competitions and so on. And if you look at the ROC AUC scores here, where 1.0 is like perfect and 0.5 is basically random, these things look like they're doing really amazingly, you know, 0.95. It's like, not that this is exactly like class grading, but, you know, they're getting an A roughly. But I want to make the point, this is— if you followed me thus far, you might be starting to see why I think this is maybe a problematic scoring system. There's kind of 2 cultures of evaluation going on here, and we're trying to find a way to bridge between them. In AI, typically the goal is to evaluate technical performance. You want to measure generalization error. You want to take things that the AI has never seen before and see how is it going to do on those. In HCI, in my field, then, you know, Istiak, you can probably say this better than I can. You know, the goal is to evaluate user-facing experience. We want to understand How does this system impact people? We wanna do things like measure attitudes, responses, behaviors. So this isn't— these are totally aligned in the case where everyone agrees that something is fine. In both cases, someone's gonna say, yeah, 100%, great. But in the case that I've been demonstrating, there is no HCI researcher worth their salt who's going to say, 100% of users approve, in a situation where 40% of people dissent, we would report that 3 out of the 5 people said that they approved of the system response and 2 out of 5 said no. But in AI, the evaluation measures are still going to tell you 100% in general because they're reducing this to a single ground truth. If this 0.95 is inflated, basically what we did is created a technique that would better, more honestly, user-facing sense, report what the metric might be. In this case, I'm going to show you how we get to 0.73, going from like an A to kind of like a C- situation. Or given Stanford grade inflation, it would be practically like, I don't know, failing. The students would get very stressed out. The reason we want to do this is that my theory of change here is that we will make different product decisions if we realign our performance measures, right? If we think— if you see something that looks like it's nearly perfect, you're gonna be like, yeah, ship it. But if it looks like it's really struggling, you're going to go look into what's going and maybe think twice. The technique we developed to do this is called the disagreement deconvolution. The idea is that you should be able to transform any evaluation measure to adjust for disagreements. The idea here is that even if you're not changing your model, we can actually do some thinking on how you measure performance, which is very easy to slip into a pipeline typically, to handle this. Again, the goal here is to give the teams a better sense of how their classifier performs when it's deployed to end users, not when it's being matched against some ground truth. Intuitively, what you would want, if you agree with me thus far, this should not be a very scary claim that if we have 3 of 5 people arguing something, that the test set labels at the very least should give weights proportional to the set of annotators who believe that. So that a classifier predicting not toxic here wouldn't get 1.0, it would get a 0.6 on this example, right? Then there's no way to get 1.0. Best you can do is 0.6. But it's actually more complicated than this. Turns out that, as you know, in crowdsourcing, we again know that some annotations can't really be taken at face value either. There might be some noise. Sometimes people misunderstand the task. Sometimes they're getting distracted because their officemate knocks on the door and they make a mistake. Sometimes they're accidental clicks. Basically, there's a consensus that we can't take label annotations at strict face value either. So how do we kind of thread this needle? Our approach is to sort of like deconvolve these, like to say, over here, this is things that you might refer to as noise. These are things that you might keep and hold on to as irresolvable disagreements. How would you do that? That sounds weird. That would be nice. But how do we do that? Basically, it turns out that again, Many datasets that are already out there already have the data we need. They do this thing by inserting what are referred to as attention checks. You do this in survey research as well, where sometimes you'll ask the same question twice or ask it reverse coded and see if people agree with themselves. Turns out, in crowdsourced data annotation, you have this fixed number of what are these things called gold standard tasks. And then they, like, you run out and they start asking the same ones over and over again. And because they're asked multiple times, we can actually look at how often—

excuse me—

how often someone disagrees with themselves later. So this allows us to estimate how often a label would flip given the same example, just sort of randomly. And we treat that as an estimate of how— of sort of the noise part of this. And then we do a little bit of math I'm not going to get into that basically allows us to suck that out. and be pretty confident that what we're left with is just the irresolvable disagreement. When we apply this, what we see is that many classifiers look way better than they actually are when you account for disagreement. So problems that appear solved, reality, really not. So again, here's sort of state-of-the-art toxicity. There's the 0.95 from before, that leaderboard. When we adjust it for disagreement, you get this 0.73 I was arguing for before, right? You go from your straight A to your C-. So if current state-of-the-art text-to-speech performances, you know, claim to be 0.95, our deconvolution reduces it to 0.73. The next thing I want to point out is that we can actually do the same thing, but now with what's called an oracle classifier. What that means is imagine you had a magic AI that always guessed correctly, that always guessed what the majoritarian view is. No classifier— we're able to show this— will ever do better than 0.73, because even if this oracle, this magic classifier, always guesses the quote-unquote right thing, there's a bunch of people who are going to disagree with it. So it turns out that we're basically maxed out on this task, given the inherent amount of disagreement there is in the population. We're never going to be able to do better. Across a bunch of other social computing tasks ranging from misinformation detection to adult content detection and so on, there's just large corrections, huge drops based on disagreements on what the correct answer ought to be. But in traditional AI tasks that are, again, more perceptual, word sense disambiguation, image classification, it's a much softer correction because people tend to agree on what the right answer ought to be. What I want to point out with this is that even if we're not changing the model, Simply paying attention to disagreement can really change how we think. So I will again pause here. The next thing I'm gonna talk about will be this project that Peter mentioned at the beginning. Thoughts, comments, disagreements? I welcome all. Silence is fine, too. I'm really good with awkward silence. Lynette, hi.

Hi, thank you so much. So coming at it from an— as a non-technical person, knowing nothing about computer science and limited knowledge of AI, I'm a political scientist. It seems to me that, you know, your talk is about mediating societal agreement, and I wonder what you mean by mediating societal agreement. Because I think that, that will have a lot of impact on how you actually select your jury. So this is like President Trump trying to select Supreme Court judges.

That's right.

And then it depends on what sort of decisions you're trying to make. I think for certain decisions, you want to have a random sample of your population. For other sort of decisions, you want to— like the first example that you showed, it seems that you are purposely selecting a sample that reflects the victim's characteristics, then that would have an impact on the sort of decisions that you're trying to make.

Yeah, that's right. So at some level, I think what I now sort of going around— I now sort of go around saying that computer science is also political science. Like, we need to learn from all of the lessons that political scientists and folks who study institutions and elections, all these things have learned over a long period of time. What I— what these projects at some level are pointing out is that we have tried to pretend that we don't have to deal with these issues to this point. And, and the sort of normative point of view of this project is to say that's wrong, we shouldn't be doing it that way. We need to actually ask, yes, what are these sort of decision-making criteria? Who should be empowered to make these decisions? In what capacity and under what under what conditions. How we actually construct that, it's sort of like a constitution-making question a little bit, is, I think, an open question that we as computer scientists have not wrestled with. And I think that you all have, you know, many, many years of experience understanding where, you know, where you would want to say, again, do a straight random sample, where you would want to do some sort of stratified sample to give weight to certain voices in your in the population. I think that what this work is doing is it's providing tools that enable that to happen. Previously, I would argue that was generally very hard. It was invisible, and it wasn't— in practical terms, it was not possible. What we're doing through jury learning and the disagreement deconvolution is giving hooks that actually let you ask that question that you just asked. And now we have to sort of do the hard work of deciding, Wait, if that's actually the case, that we're making these decision-making criteria and that we're authoring them, we were doing it implicitly, let's do it explicitly. How do we craft those? And what are the right ways of doing it? So I'm— I don't know that I really have a response to your question other than saying, yes, I agree. And that's in some sense the point we were trying to make, that you should be— that we should be thinking this way. Maybe to a political scientist, this would have long been obvious, but I would claim that to technologists, this was not sort of tractable or feasible previously in many cases. Uh, Atreisha, then Daniel.

Um, yes. Um, so my question is kind of, um, I want to know your thoughts on the effectiveness of this in terms of behavior change. So for example, here what we are doing here is labeling something as toxic using a certain set of compositional jury, and like the decision is that it's going to be removed. Now, but from the perspective of someone who wrote something toxic or said something toxic is like supported by a sort of a descriptive belief that saying something, saying that is okay, right? So unless we are able to somehow guard that across all of global internet, they're just going to go somewhere else in that just reproduce their behavior, right? And I kind of like want to know your thoughts as an alternative view of like having this through a normative way of like incentivizing participation of people who would otherwise not participate to kind of do a community-level or moderated intervention to say that was not right, regardless of the classifier. And that might change the descriptive belief itself to change behavior better than— I just want to know your thoughts in these 2 different lines of approach.

Yeah, so clearly this, this project's taking a normative position. Totally agree with that. Are they just going to go elsewhere? This is commonly a critique that comes up when, you know, if I say— I'll go on record saying I think that, you know, many online social network platforms should go further in terms of dealing with toxic and harassment, toxic speech, harassment, and so on. And then there's this rejoinder that basically says, oh, they're just going to, like, leave the platform and do it elsewhere. They're going to go on Gab and they're going to go on Truth Social.

Yeah.

Truth Social and these things. So there's actually literature studying this in human-computer interaction and social computing. There's some really great work out of Georgia Tech, UIUC, now Rutgers, demonstrating that when you do this kind of thing, when you sort of effectively deplatform folks, it actually has the intended effect. It detoxes— these folks are sort of in an environment that's allowing this to happen. And when you remove that environment, they detoxify elsewhere. They don't take it elsewhere. There's a little bit of an asterisk there. Some people will jump ship and go elsewhere, and they will— and those individuals will tend to further radicalize. But it's a very small group, and they have a very small audience. So it does tend to, on balance, have, I think, the intended effect. Now, there was another element to what you were saying. I'm gonna lose it. Hold on.

Um, yeah, I was talking about the descriptive belief about what the reason that they thought it was okay to say that, and like having it through a community moderation, very incentivizing a jury to kind of like, why don't you go and like say something, like somehow designing the incentives in that way.

Yeah, so, um, there's also a bunch of really great work in the field that demonstrates, uh, social influence, the very like classic social psychological effects that Actually, Paul Resnick recently had a project that demonstrated this, that as people enter different communities, like different politics communities online that have different norms, they immediately adapt their toxicity levels to the norms of that community. So I think that our spaces are far too— don't communicate norms nearly as well as they should. online. It doesn't matter whether this is a group that is— okay, let me start offline. Offline, there's like a coffee shop. That's different than like a tea shop. That's different than like a sports bar. That's different than a wine bar. That's different than a dive bar, right? These all communicate different norms, different expected behaviors. They influence our behavior. Online, all of those, if you're on Facebook, are like blue bar at the top. They look exactly the same. These spaces do not strongly communicate norms, at least in my opinion, nearly as strongly as we've learned to do so in offline spaces. I think this is a major sociotechnical gap we have to cross. Now, you asked about, like, why would I believe in this jury? I think that this question, again, CS being a political science, comes back to legitimacy. Like, Tom Tyler has all of these results studying, like, you have sort of perceived legitimacy of institutions. And, you know, when you're constructing this, the hope would be that if you have, you know, an electoral sort of random sample or some structured approach, that will attach higher legitimacy to the decisions than, like, some random mod made that decision. That's my design hypothesis behind this. And I think that there's some support for this. Now, again, asterisk, it's complicated. We've run some survey experiments online where we take content moderation decisions out of Facebook here, and we randomize the source of the conclusion. It might be an AI, it might be a random jury of peers on Facebook, it might be a set of experts, it might be paid annotators, the general models that Tarleton Gillespie talks about as doing this. There are differences in legitimacy and perceived legitimacy as a result. But unfortunately, like, the way larger effect is outcome alignment. Do I agree with the decision that was made? And so it's like legitimacy is going to be really challenging for any online platform to achieve, so long as we have sort of like a strongly polarized political situation. So I— that's sort of an unsettled— I'm not sure what to do about it. It's like, you know, people will attach more and less legitimacy to different approaches. But at the end of the day, as we're starting to see with the Supreme Court in the United States, if it's like swinging, if it's perceived as biased in one direction or another, you know, against you especially, you're not going to give it much, much, much listening. Daniel, do you want— let's pop over to you.

Hey, thanks. So yeah, I guess I wanted to just like— I've been thinking a lot from the beginning, kind of this whole idea of like, you know, the jury and kind of how the one vote, like one person, one vote thing is, is suboptimal in some cases where, you know, if you have a minority, then they kind of get— their voice can get erased. Um, but it also seems like it's tricky, right? It's also like, you know, there's like the, the saying, like, democracy is the worst form of government except for everything else that we've tried. Um, and, and I mean, I'm wondering, is there like a solution for this? This is— maybe you've already thought of this, but like, what if you did something where you still did like a random sample, but you did quadratic voting, and so that people would be able to have like a bigger say about the things that they cared a lot about, and so that then those voices that might not be majority voices could still have like a bigger impact.

So I think what you're saying is computer science is also social choice, which I agree with, right? Like, it's— I know, how do we combine voices? What's an equitable way of doing this? Absolutely. So in practice, we in fact could even do this in the construction of the algorithm. You could say, here's the jury. One thing that's interesting about quadratic voting is that we could further model what are the topics that this individual cares the most about? How many of their quadratic votes would they apply to this input versus that input? And this is something we talked about, we in practice didn't do. Again, just sort of like at some point we had to stop working and write a paper. But yeah, absolutely, you can imagine different kinds of decision strategies like quadratic voting, liquid democracy, any of these things, right? And I think they're gonna have different properties. And again, I think we should— we can and should be drawing on the social choice literature to understand the trade-offs between them. I have a colleague, his name is Ken Arrow. He's gonna like loom over this and be like, you're gonna have limits. And like, we know that, but I think we can do better than we do today. So I'm gonna pop onto one final thought here, and that is how we might change institutional practices. So this is not about disagreement qua disagreement, but it sort of zooms out a little bit. So this is work that I did with several colleagues, many, some of whom you may know. Margaret Levy used to be the director of the Center for Advanced Study in Behavioral Sciences. She just stepped down from that position. She's a political scientist, Skid Prize— Schütte Prize, excuse me, winner. And David Magnus, a biomedical ethicist. Betsy and Quinn, Charlotte, now Quinn, are program managers at the CASBS. And Deborah Satz, who's an ethicist, a philosopher. So, you know, this deep into the talk, I don't need to convince you that AI carries societal risks. That's sort of a horse that's left the barn. At this point, I think the horse has crossed state lines, it's raced halfway across the continent, it's attracted international headlines, and it's probably even landed a shoe company scholarship. You know, it's like, it's not hard to argue this. But we lack institutional responses. What are we able to do that's not at an individual level, but at an institutional level to address this? You know, we have this constraint where success requires that everyone needs to participate, and really early on in the formative stages of research. And we have, in particular, in the case of AI, some developing approaches that are opt-in, like office hours, design principles, checklists that are powerful, but it can only reach those who self-select to participate. And other approaches, like NeurIPS has a really successful experiment in getting people to produce ethics statements, now actually shifting that to be a checklist, and they're iterating. But it's all done after the research is complete, which you know, has some limits. Like, sometimes, not always, but sometimes I'm suspecting that those are getting written in the 2 hours before the paper deadline. So it's too late to really impact the research design in those cases. In the United States, you'd be like, wait, whoa, Michael, slow down. Like, we have the IRB. It's fine. Unfortunately, again, at least in the United States, and it turns out in many other countries as well, the regulatory frameworks under which the IRB operates prevent it. from helping here. It's only allowed to focus on risks to human subjects. It is not allowed to do any sort of judgment of risks to human society. In fact, if you look at the US Common Rule here, it just says that you cannot consider possible long-range effects as among those research risks that fall within the IRB's purview of responsibility. So throw in your swear word here. Basically, a lot of AI research can go from start to finish without ever going through any sort of IRB or ethics review. And in fact, at a place like Stanford, when I have colleagues who try to get IRB feedback, they return it and say, I'm sorry, we can't help you. We're not allowed to look at this. Sometimes IRBs will take a broader lens. I want to credit here Mary Gray with her work on the Microsoft Research Ethics Review Program. It's an internal IRB at Microsoft research, they basically take a more anthropological lens where they say, well, communities are human— are subjects of this research, so they should have a stake here as well. But unfortunately, that's not as widespread as we might like. So we're considering these sort of institutional arrangements at Stanford and elsewhere in sort of American universities and see what we can do about it. What we ended up creating was something called the ESR, or Ethics and Society Review. It's an institutional process. We collaborate with funding agencies here. We started on campus. Stanford has this thing called HAI, Stanford Institute for Human-Centered AI, and it's supposed to facilitate researchers in mitigating these negative ethical and societal aspects of their research. Basically, the main trick of it is that we design it as a gate to access funding. So HAI runs these seed funding things on campus that are available to faculty. And just like I cannot spend money on a grant until my IRB protocol is approved, So too with the ESR, you do not get access to your HAI grants until you've completed the ESR process for your project. So the funding is a gate here. That's sort of our moment of institutional leverage. So when people apply, they would write their grant just like they normally would. They add a statement at the end that goes to the ESR faculty panel. Basically, it's expected to do 3 things. One, What are the risks? 2, what are the principles that people in your field you think should be using to mitigate those risks? We're not trying to erase them, we're trying to mitigate them. 3, how did you instantiate those principles concretely in your research design? So this is to prevent people from sort of saying like, well, this is a risk, see you later, drop the mic and walk away. They have to say, what are we doing about it? So then the funding program goes through its normal merit review, and then the ones that they want to fund, they pass to our panel. which goes through a triage round. Right now we have some postdocs and graduate students sort of finding the ones that pose more than minimal risk. Many things are really, through this grant program, are like humanistic inquiries that pose very little ethical or societal risk. The ones that do go to an ESR panel where they get feedback and iterate with this interdisciplinary panel of faculty. Recently, our panel has included faculty from anthropology, communication, computing, history, management science, medicine, philosophy, political science, sociology, sort of all the groups that you might imagine as stakeholders here. And, oh, actually law recently as well. As this iterates back and forth, when the risks have been appropriately mitigated, we recommend that they release funding and the PIs get their money. So we provide scaffolding materials that help people understand what are common risks with this kind of AI research, right? representativeness, a very common one. Who's in your dataset? And many others. Informally introducing this kind of thing in its second year had a major positive impact on the quality of the ESR statements that we received. As an example, you know, there was a collaboration happening on campus between engineering and medicine where they were trying to create non-invasive stress sensors at work so that, you know, workers— like, people are stressed out at work. They wanted to be able to help people detect when that's happening and then step back and engage in meditation. And then they named a pretty powerful risk, which is like, wait, a surveillance state might result if my employer gets access to this. And then they said, oh yeah, that's a problem. But okay. And the ESR panel said, well, okay, what are you going to do about it? And then went back and forth. There was some meetings to discuss the feedback. And they wound up committing to a privacy-preserving architecture, basically doing it all locally on the worker's own phone that they own, so that the employer never gets access to it. And they made a commitment to use their bully pulpit as researchers and as Stanford researchers especially to say, this is why we made this decision and here's why it's important that you do it this way too. So we've— we're now in our 3rd year. This is going to put us over, I think, 100 proposals that we've looked at. About a 3rd of the grants are asked to iterate with us. So about half get triaged out and then at the end of the day, a 3rd go through iteration with the panel. So what have we learned? What works? What's not working? We surveyed a bunch of the PIs on these grants. About 2/3 of them got back to us. We did some semi-structured interviews with the researchers, again, ranging across engineering, social science, earth science, medicine. And then we actually went and did a content analysis of the feedback that we are providing to the PIs. One thing that stunned me I was like adding red tape. I expected everyone— I expected, especially at a university, it's very decentralized like Stanford, almost sort of leans kind of libertarian in its politics. I thought people would tell me, get out of town. Every single person who responded to our survey wanted to keep it around. They felt like the juice was worth the squeeze. They felt like they were actually gaining something. What do I mean by that? About 2/3 of them actually said that this process had impacted their research design. positive evidence, I think, that engaging in this kind of reflection at the front end of a research project rather than the back end can really powerfully shape what happens. Essentially, why is this working? It's because of scaffolding. It basically serves as this forcing function commitment mechanism to think about this. Here's a psychologist who wrote to us that this led me— or spoke, I think this was an interview— led me to engage with my co-PI 'Cause I wasn't even aware of some of these possible ethical implications as a psychologist, and it helped me engage with my computer science co-PI as part of the requirements. In fact, about, you know, 2/3 or so, if I'm reading that correctly, said that it raised new issues for them to think about. One of them here is a PI in the School of Engineering saying, we might flip our whole research approach to being about privacy. We don't have the answers yet, but it's helped us sort of ask the right questions now. It led to some things that I wasn't expecting. One that was cool is that there was a project that came through ESR trying to use machine learning to predict sort of non-toxic chemicals. And the ESR said, well, if you've got an AI that can predict how non-toxic a chemical is, people are going to use this to like go the other direction too and create more toxic chemicals. That seems bad. What should we do about it? They wound up like sort of launching an entire sort of research trajectory as a result. Wrote this piece in Nature Machine Intelligence about it. This approach is also getting recommended to be included in the National AI Research Resource, which is sort of a US government-wide approach. Chatted with some folks on that national panel. What is going wrong? Basically, people want more help, more scaffolding. Like, don't just tell us that, you know, don't just make us think about this. Like, it's like a blank page. What should I be talking about? Give us scaffolding to figure out what to, what to People wanted to have more detail and specificity. This is a major reason why we added that thing I showed you with example common issues and what prior researchers had done to, to address them. That seems to have helped a lot. People also want an ethical Richter scale. We get quotes like this. You're like, you say this is an issue, but is it like a kind of minor moderate issue, or is this gonna wind me on the front page of the New York Times? Help me understand that and help us make those trade-off decisions. Common themes in ESR feedback. This is the inductive analysis of the content analysis. Wouldn't surprise you. Issues like harms to subgroups. Who's negatively impacted if this gets widely adopted? Do you have all the relevant stakeholders included? Dual-use issues, representativeness. These are the most common things that tended to come up in ESR feedback. There's a bunch of stuff we haven't solved. The Dean of Research at Stanford is like, yeah, let's do this all over the place. We're like, oh my gosh, you're going from like tens per year to like 1,000 per year is not something we're currently equipped to do. How do we actually manage this at the level of our actual whole institution? We tend to try to have a coaching lens rather than a compliance lens like an IRB might. Are we going to lose that if we get larger? How do we measure the impact of the ESR? We have some approaches we're doing about where we're randomizing the deployment of the ESR. We want to measure impact on culture, not just the research itself. But these are all open questions we haven't solved. So with that, I will finally rest my case. Thank you so much for the invitation. Also, thank you to the PhD students who led this work, the colleagues and collaborators who contributed to it, the supporters you see here, Stanford Center for Advanced Study in Behavioral Sciences, and all of you for spending an hour-plus of your day with me on Zoom, which I know is always your favorite part of the day is the Zoom part. So I'm happy to chat for a few more minutes. I think we have a little bit more time. And again, happy to chat.

Yeah, Avery Slater.

Hi, thanks so much for that really fascinating talk. I'm still thinking about a question that Peter asked you about the dimensions you didn't know were relevant. How do you capture the dimensions that you didn't know were relevant? And, you know, I think what I want to ask you about is the sort of specific dynamics of toxicity and how it evolves in online spaces.

Mm-hmm.

Who's toxicity for? And I feel like that's something I'd love to hear you talk a little bit about. Here I'm thinking about online subcultures, and I appreciated you bringing up the LGBTQI community not being able to use and kind of reclaim certain slurs. And so— and that work is really important. People have like that. But I'm thinking about, you know, online subcultures that are really based around rapid evolution of the semiotics of hate.

Yes.

And thinking, say, in the Trump era, where we all got used to hearing this term dog whistle. I'm not going to give examples of some of these, you know, encrypted to the minute, even week by week, the evolution of some of these, we'll say, dog whistles or different codes that are constantly evolving. that are designed to spread toxicity, maybe for a subculture, maybe for those who know what that means. So can you talk about how these many different methods that you've described, how they are going to deal with that particular problem? I know it might be outside the scope of what you're doing here, but what are your thoughts on that?

I mean, regardless of whether it's outside the scope, it's an important limitation and, and like thing to consider. So here's the challenge sort of on the technical side. Is that, you know, these— every technique out there, as, you know, Ishtiyake pointed out— and sorry, I always worry that I'm mispronouncing your name— but they rely on language models, language embeddings, right? And when there is evolution in language, as there always is, There's no magic that allows a model to rapidly absorb some, you know, turn— new turn of phrase. You know, it would take a lot of data to sort of retrain these language models to understand, actually drag this over to, you know, toward the, like, the hate side of the spectrum. And we have tried, you know, there's all sorts of techniques you might deploy. But in general, we have not had success doing that. And so I'm not currently optimistic that a— on the sort of socio-technical spectrum, I would lean more on the socio side of addressing those kinds of issues than on the technical side, because I don't think that those— that rapid cultural change and linguistic evolution that's happening in short terms is going to be very easy for, or even hard, even maybe possible, given our current trajectory, to be able to address. And that's just because of the fundamental limitation of these language models needing huge data to train. And when language changes quickly, they're not being constantly fine-tuned. And like, maybe they could be, but they're not yet. And even if they are, they're not going to be fine-tuned fast enough to stop the spread of some new form of hate. So I, I, I very much agree with you, and I basically— my view is that this remains a totally open research question that I have not seen work that I think suitably addresses, including my own.

Last question goes to you.

Yeah, sure. Yeah, I'll not take much time. So, fascinating talk, Michael. Always a fan of your work. Um, I think the representationalist thinking that you were bringing. It has, like, you know, like, definitely some limitations that others rightly pointed out. But I agree with you, it's still like a first step for computer scientists toward embracing these political theories. And so I had some questions, like, even for this kind of representationalist thinking, there are some cautions that people in political science try to remind us. One thing is that we are assuming that we will be getting people for each of these groups, which in, you know, like ideally, you know, like it might sound like fine, but practically it's difficult. So that's number one. And then number 2 is that even like this kind of representationalist thinking, people advocate that there are vulnerable communities or vulnerable topics that we need to safeguard no matter what, like, you know, like a majority or, you know, like your poll, you know, whatever your poll says, you cannot cross a line. So that is also important here. So have you ever thought about like this or run into any, any of this kind of like questions in your research?

Yeah, let's see. Let's, let's take these in reverse order. So, you know, the second, the second part is getting a little bit towards sort of like deontological ethics. It's sort of like, are there lines that just never get crossed? I think it would be really— so I've not seen it done, but I think it would be really cool, and like, or helpful. I use cool as sort of a code here for like good research, to be able to specify what are those sort of like deontological lines, what are those normative lines that just ought to never get crossed, like the model that need not go further. We don't need to go ask people, you know, we have bills of rights, we have, you know, We have some things that are decided through majority vote or other kinds of votes and other things that we just say, this is a fundamental right. Don't even try. Typically, when social computing or AI folks try to get into that space, then we start making all sorts of false positives and false negatives. I don't know how well— if we'll fare better here or worse here, but I think it's absolutely a thing that is Not, yeah, not integrated here, but would be really, I think, powerful to be able to separate, like, you know, where are sort of like our constitutionally guarded responses, and where are— where is the sort of the polis deciding? Your second comment was that essentially sampling and surveying is hard. Yes. So I mentioned my wife's a political scientist. She has done a bunch of work, especially in the Middle East and North Africa.

Yeah.

Data is really hard to come by in certain places, and, you know, they end up hiring survey firms and so on. So I feel like that is not a thing to be overlooked or laughed at lightly, like, oh yeah, we're just gonna have this many, especially even domestically in North America when we start getting into certain intersectional identities. Like, even if you have a lot of, you know, certain groups, the intersectional group you may not have very much data on. In practice, the way that our model handles this is that if you try to articulate a characteristic or a group for which we don't— that the system does not have enough data to generalize well for, it just doesn't let you do it. It says, go get more training data from these folks. And then we just sort of say, all right, that's as far as we can take it. But yeah, so I agree at a high level with everything you're saying. So let's see. I think we're at about time, but Peter, I'll—

Yeah, Michael, we can have everything in the world but more time. So Thanks very, very much for joining us and fitting in 4 papers in that time. All of them provocative, and for someone who I confess is often a bit mystified by these talks, were very clearly presented. So thanks for thinking of the social scientists on this one, which is excellent. And thanks to the 50-plus people over the course of the talk who joined us. Thanks very much, everyone, for joining, and we'll look forward to seeing you at our next seminar.

Thank you. Thanks all.