The Human Veto: Who overrides the machine? | Dreamforce 2026

Please welcome Chief Executive of the Raspberry Pi Foundation, Philip Culligan, Chief Ethical and Humane Use Officer at Salesforce, Paula Goldman, and Editorial Director at Time, Ayesha Javed. Hello.

What a nice warm reception. Thank you everyone for joining us today for this session on managing the machine and the human veto. I'm delighted to be joined by, by our panel. We've got Phil Colligan, who runs the Raspberry Pi Foundation, and Uh, Paula Goldman, who is Salesforce's Chief Ethical and Humane Use Officer and author of Manage the Machine, which is out today, I believe.

Yay!

So, uh, AI, uh, used to wait for us to ask. Um, now, uh, it can recommend, finish entire tasks, can act on our behalf. We were just talking earlier about, about some of the agents, what your, your agent, uh, for that, um, and often there's no one checking along the way. Um, so every organization has to answer the question, um, of as, as AI becomes more capable, what should they actually be allowing it to do. And so I'd like to kind of talk to you about, you know, what, what we choose to keep human, as well as what we actually allow the AI to be doing as it becomes more capable. So coming to you first, Paula. We've spent years framing AI at work as a replacement, which, you know, thinking about which jobs are going to survive, which won't. What do you think that, that discussion kind of gets wrong?

Yeah.

Okay.

Well, first of all, I just want to say thank you. You all have a copy of the book here. I'm going to ask you to do me a solid because there's all these team members that helped with it. Can you do me a solid and can you hold Hold it up. I just want to take a picture of you all holding the book. This is like you're making my mom super proud. Thank you very much. Okay, so so why this book? It's really in direct response to that question. There's so much talk about automation in people out, and that misses, I think, the real value, the real potential of the technology. Yes, AI is incredibly powerful. It can digest terabytes of data. I could never. It would take me years to amount you know. Maybe a whole lifetime to do what it can do. However, just because AI can do something doesn't mean it always should. Sometimes that's because the AI has lots of failure modes, things that are, you know, we've all heard the visible stories about that. But there's an even more important reason to think about this question of how do we use AI to unlock human potential? It's because We're not the only ones that have AI. Our competitors have AI. Our customers have AI. AI can be used to filter out your messages. So then what becomes the competitive edge? It becomes what is possible that was not possible before for your people, for your customers, new customer experiences, new capabilities at work. And that's what I saw across thousands of examples and talking to people and researching for the book was That is the differentiator for AI, and we need to be asking that question more.

Yeah. And Philip, what do you think people get wrong about this kind of replacement?

Well, I'm an optimist around the potential for AI to open up more new, interesting types of work for humans to do. I think when it, you know, if you think about in my field where we're helping young people learn computer science, develop AI literacy. They're very worried about, are they going to develop these skills and then go into a labor market that doesn't need them anymore? And what I say is, look, AI is lowering the barriers and reducing the costs for software to solve problems that we haven't even tried to solve with software yet. So I'm a believer in Jevons' paradox. I think as the cost of something comes down, the demand for it goes up. And so I'm an optimist that there'll be many more problems in the world that we can use technology to solve, and we need young people to be ready to seize those opportunities.

Yeah. And just to stay with you on that, how— I'm quite interested in sort of deciding what should be delegated. And, you know, AI can do things now that just a few years ago we assumed needed a person. So how do you think that we should be approaching deciding what gets delegated and what should be kind of done by a human and stay as meaningfully human?

Yeah, well, I think about this question mostly in the context of learning and young people learning, right? And so, you know, all of us will have, we have experience of creating artifacts like an essay or even a computer program as part of your learning to show your teachers that you've acquired mastery and knowledge, right? It is now, of course, possible to stick in a single-line prompt into an LLM and get a fully fledged essay that would pass the test. And that's a good example where, you know, you shouldn't be using AI to do things in the realm of learning, um, which prevent you from acquiring the knowledge and memory that you're trying to develop, right? And so I think, you know, again, I'm optimistic about the role that AI can play to support young people learning, but I'm also deeply concerned about the inappropriate use of it. And I think it's important to flag one other thing, which is we're all hardwired to use less energy. So we're designed as humans to cognitively offload. So you can't blame kids when they take the shortcut to get that bit of homework done using an LLM. What we have to do is design those systems so that they are genuinely helping young people achieve the objective they've got, which is learning. So that might not answer your question about which things do you delegate at work, but I think you've got to be really careful when you're delegating tasks to an AI, which you need to do in order to develop knowledge and expertise and mastery.

Paula, what are your thoughts?

Yeah, I mean, I, To be honest, I think first we need to start by asking that question more, right? It's not again the the instinct has been this experimental phase with AI. It's just like try it, try it. But I think where we're emerging now is to be able to ask what are the parts of the process that actually are we really want to reserve for people and why? So for example, in customer service, I have a friend who works at a healthcare company, and she says to me. Well, it's not really that complicated, for example, to have AI schedule a medical appointment for someone. But when someone has just been diagnosed with a serious disease, they don't have AI do that. They have a person reach out. Why? Because there's more going on than just the mechanics of the scheduling, right? There's showing up with empathy and so on. And there's more examples just from customer service, right? Research tells us that when people are angry, they generally want to talk to a person. When they're embarrassed, often actually they want to talk to AI. Why? They don't want to be judged, and so on. And the point being that paying attention to those patterns is not just, you know, it's not just an academic exercise. It's how you're going to retain and grow these customer relationships. And I could give you the analog in probably every single domain, but one of the arguments in the book is like, this is actually the rational— it's not just the right thing to do, it's the rational thing to do. to think about and incorporate the human side of the interaction and take it seriously.

What do you think about interviewing candidates?

Yeah, well, it's so— so there's a chapter in the book that's about hiring an AI. And of course, there's, again, there's no shortage of disaster stories there. But then there's also a lot of really positive stories. And again, it comes down to intentionality. Done right, AI can actually help us focus on more objective criteria, on more objective skills that are required for roles, and not necessarily they had to have worked at this company or have this pedigree. It can be a democratizer, but it comes down to how it's managed. It comes down to humans actually intentionally managing AI.

Yeah. And so another thing that I want to talk about is There's a kind of paradox of the better that AI gets, the easier it is to actually hand more to it. But the more that we hand to it, the fewer chances there are for people to build the expertise themselves to be able to catch potential mistakes. So how do we use AI to kind of grow human capability as opposed eroding it? And Philip, I'll come to you on that one first.

Yeah. And, you know, I've been at lots of conferences where people are talking about AI, and one of the most dangerous ideas I hear is that humans don't need to develop knowledge anymore, right? And that young people in particular don't need to develop a detailed understanding of a subject, a domain. And I just think it's not true. There's no evidence at all that that's in young people's best interests. And in fact, all the evidence we see is that people who get the most out of working with AI systems are those who've got really deep, rich domain expertise, or who are using the AI systems to help them develop that expertise. You know, we hear lots, we've heard a lot, haven't we, about how kids don't need to learn to code anymore. And what I would say to that is, I mean, they do first, but it's not so that they can spend their lives writing lines of code. That's changed forever, right? The way that humans give instructions to computer has now changed. But the act of coding helps you develop a mental model and an understanding of computer systems that when you are then confronted with an AI system, you have a much richer understanding that you can interact with that system and build what you want to. And so I'm a big advocate for kids still investing the time and effort to develop deep subject knowledge in the areas that they're interested in, because that will help them work with AI systems more effectively.

Yeah.

Yeah. Well, it's interesting. There used to be a phrase that was very common in business. They worked their way up from the mailroom, right?

Yeah.

Mailrooms don't really exist very frequently anymore, but yet somehow we found new ways of creating these kind of entry-level roles where people learn the chops, get the mental model of what's going on in the business, and then are able to grow and add more and more value. So we're seeing experiments in that regard. I will tell you, Salesforce this year hired 1,000 new grads and interns because we thought it was a real opportunity to invest in entry-level work. We had 2 interns on our team this summer and we were 3. Actually, and we sat down and looked at their projects and they blew me away because they were teaching us how to use AI in new ways and we use AI in pretty sophisticated ways. So I think there's a lot of opportunity, but it takes intentionality. I will add one more thing, which is that there's a part of this that I'm very, very excited about, which is about how we design the technology interfaces in the first place so that, for example, When there's a really consequential decision to be made or a place that really does require judgment, there's a little bit of what we call positive or mindful friction that actually induces someone to pay attention. Or places where we say, actually, the right thing to do— for example, there's a story in the book about IKEA trying to have an innovative prototype of a couch, right? And they kept asking AI to help them innovate. And it kept reverting back to the mean. And then they realized the problem was they hadn't forced their own thinking first. They had to do their own thinking first about what was the difference. They started using metaphors like campfire, gathering site, and then they came up with this idea of couch in a box, which was lightweight and modular, and you could put it in a carrying container, got exhibited all over the world. The idea there was though, they had to preserve the place where people were thinking first. before they actually ask the AI to get involved. And so I think it's those types of patterns. It's early days, but we're gonna start to get familiar with where does the person weigh in, where does the AI weigh in, and how do we combine those things together?

Can I add one more point on that?

Yeah.

Because I think, you know, you're asking about humans and knowledge, and I think it's really important that everybody develops a Foundational understanding of how AI systems are built. Like, you know, how important it is that they're trained on data, the fact that they're probabilistic, what does that mean in terms of making errors and so on? Because otherwise, I think there's a risk that we're just using these black magical black boxes.

Yeah.

And I think, you know, the example you just cited with IKEA, people understanding more about the systems and the way that they're built really helps them get value out of them.

100%.

Yeah, yeah. And Paula, there was also another example in your book of social workers, I believe, who were offered the opportunity to use AI to eliminate certain parts, what seemed like inefficient time, and they didn't want that. So I guess that's the other kind of side of the coin. What did you learn from that, and what do you think leaders could take away from that? In terms of understanding, you know, where AI does belong and where it doesn't in social work?

Yeah, so the story that you're talking about, this was, I encountered this on a trip to the UK actually. I was talking with a technology executive at a town council, local government, and he was telling me about this experiment that they tried to run where they said to their social workers, hey, we're gonna give you AI note-taking capability. You're gonna go on these field visits, it's gonna take notes for you. It's going to document next steps and it's going to be amazing. You're going to save several hours. You don't have to go back to the office. You can go home. And he was telling me this. I was nodding along. That sounds like a good AI use case. Then he said, no, the social workers rejected it. Why? Why did they reject it? Why did they reject it? And he said, well, it's because they actually wanted to go back to the office because they had this ritual talking to their colleagues. Thinking about the hard cases that they had seen, trying to sort of diagnose what the best path forward was for those harder cases. And they didn't want to burden their spouses with every day talking through the hard cases at work. There was the value of this kind of social ritual that they had, and they had failed to take it into account. And for me, this was a wide— this was an eye-opening conversation because I think this probably happens more than we talk about. where we're so focused on just the efficiency or what the technology can do that we fail to ask the bigger question, what is the actual work that's happening here and how do we map it out? And what is the human side of it and how do we bring this all together? That is also how we get the best results and how we avoid wasting money.

Yeah. And I guess that's something that people, it's one of those things that maybe you would never have— the people making those decisions wouldn't have thought about otherwise.

Yeah.

And Philip, I mean, when you hear that story, what does it tell you about how much of a voice people should have in deciding where AI enters kind of work and learning?

Well, we're seeing it right now, right? The debate about technology in schools and young people's lives is taking a new turn. I mean, and we see it in the US, you know, in countries like the UK and Australia, we're seeing social media bans for under-16s. We're seeing parents campaigning for screen-free time or a reduction in the amount of time their kids Uh, are spending in front of computers. And you add into that the introduction of AI technologies, and we've now got, you know, decisions like we saw in New York just a week or so ago, um, with a, a thoughtful approach to, um, limiting the, uh, deployment of AI in school systems. So I think, you know, the first thing is we've got to involve teachers, students, and parents in this debate about technology and education more broadly. And, but specifically in relation to AI, I spend a lot of time talking to teachers all over the world, and there's wildly different views. You know, some are really embracing it and thinking about it in terms of their own productivity. Others are really, um, uh, doing some great experiments on how they can use AI systems to help their students learn. And others don't want anything to do with it, right? There's a really wide spectrum. And so I think we've gotta have that conversation pretty openly. And we've also gotta bring some evidence and research to it. You know, one of the examples, um, we think about, uh, a lot, and there's some work happening in the UK on, is the role of AI systems in providing feedback to students. So we know the research shows it's very strong that one of the most effective classroom interventions is high-quality, timely feedback, right? It just makes it, and it works at work too, by the way. It's not just for kids. The problem is teachers have got a limited amount of time that they can spend giving high-quality feedback to each student. And so, you know, very strong use case there around introducing AI tutoring systems that help kids get feedback. But we also know that learning is inherently a social activity. And, you know, we've learned this from the learning sciences over 100 years, that the act of debate, discussion, disagreement, Kind of reaching consensus maybe is one of the ways that we all as humans learn. And so again, when we're having that debate about what's the role of a technology like AI tutoring systems in schools, we've also got to help parents, teachers, students understand the evidence around it. So if you just went in and said, this is going to increase how much feedback your child gets, most people would be enthusiastically supportive of it. But then if you explain to them how important it is that kids are also talking to each other and developing ideas together, then you might change the way that you implement that technology. So I think it's hard and, you know, it's hard to get nuance into these discussions, but that's really what we need is we need to be discussing with parents and students and teachers, but we need to bring nuance and evidence into that debate too.

Yeah, and we were sort of talking earlier about the some of the attitudes from students. Can you talk a little bit more about that? I thought it was really interesting.

Yeah, look, we've seen it in college commencement speeches, right? You know, kids booing when they hear reference to AI. And there's lots of survey evidence now that teenagers are, you know, there's a sentiment emerging there of anti-AI from them. They don't like it when their teachers are using it in relation to their homework or marking or anything like that. It's those same kids who are the ones who are using a chatbot, you know, whichever one they've got access to, to talk about personal and social issues at home. And this duality is real. I think all of us have it, right? We have concerns about AI in one domain. We're using it without thinking about it in another domain. And I think, you know, the fact that AI now is both anthropomorphized, so it's meant to be like a human-like interaction because that lowers friction. Yeah. And it's showing up in systems where it wasn't there before without any big announcement, you know, you do a Google search now and you don't get somebody's website, you get an AI synthesis of what the system thinks is the answer to your question. And so I think that there's sort of like the, the way that AI is creeping into people's lived experience, I think is one of the reasons you've got this duality and people are feeling, you know, in one domain, um, they might be anti it, in another domain they're using it without questioning it.

And, um, I mean, you, you touched on there the kind of use of, um, of AI maybe as a bit of a therapist or to work through a problem like that. Is that one of the uses that AI is good for or bad for in your opinions?

Well, I mean, interesting point. I have opinions on this. I mean, my main concern is around regulation and safety. So, you know, if you go and talk to a therapist in most countries in the world, that person has a professional accreditation. that in the worst-case scenario can be removed from them if they are guilty of malpractice, if they treat patients poorly, right? And over a long time, humans have built these social technologies that give us assurance that the professionals we're relying on can be trusted. We have not built those systems for AI.

Yeah.

Like, who's accountable if AI gives my 15-year-old daughter Really bad advice?

Yeah.

Like, that's not clear to me at the moment. Whereas if a professional that she's interacting with gives her really bad advice, there is some form of social structure that allows that to be addressed. And I think that whether or not, um, interacting with AI gives you, you know, if you increases the amount of therapy available to everyone is one question, but the other is where are the accountability structures for it?

Right.

Yeah, I feel like that's a great— that's a good take. And I think you could apply that kind of framework to almost any use case, right? Is there accountability structure? And then how do you use it to democratize access? And can you find the right balance of those things? It's interesting because again, we've been through this early phase with agentic AI and businesses where a lot of it has been experimentation. Let's have a token budget, etc. And now we're kind of narrowing in on these specific use cases. And one of the things that I'm starting to hear emerge is this question of accountability. And because AI, as you said really aptly, part of the challenge of even getting a mental model of AI is it's everywhere, it's in every domain. And so it's hard to just have a concentrated conversation about it. But I think where I'm starting to see businesses evolve is, you know, they're having matrixed accountability where they're pushing accountability down to the places where like, I'm not going to tell you what an effective marketing campaign is with AI and what our brand guidelines should be. But there's going to be some sort of central governance that is about, you know, specific risks that you want to avoid. And then you're devolving accountability down, but making sure that there are named owners for specific things. And I think that's really important. But it's funny because I feel like almost everything you said applies to the world of work. Work is also inherently social. Managing people is inherently social. AI can help us be better managers. It can help us Practice having hard conversations. It can bring evidence to what we do, but it has to be used intentionally and it should be used in service of the relationship, not as a detriment or a replacement for the relationship.

And I, again, it's one of the reasons why we think AI literacy is relevant for everybody, even if they're not gonna be, you know, using it to build their own apps or whatever, because understanding the system gives you a sort of power and agency and ability to challenge it.

Yeah.

And ask questions like, Well, who's the human that I can raise this with?

Exactly, right.

And I say this all the time when I'm talking to kids, it's like, today your most likely experience of AI is gonna be a recommendation engine for what movie or music you listen to next. Within a very short space of time, finance decisions, healthcare decisions, HR decisions, Criminal justice decisions are increasingly gonna be delegated to machines and doing that within a framework where we are not disempowered as humans, I think is where the big challenge lies.

Yeah. Well, and that's why I called the book Manage the Machine, right? 'Cause it's easy to get overbowed by the technology itself, but that's not where the— it's, we all are actually now managing AI agents at work and we need to know how to do that, right? It's funny because I think when people become people managers, traditionally that is done without a lot of training. They're good at their job. They then are, you know, asked to manage a team. And, and I don't think we can make the same mistake with AI. Like, we need to make sure that people understand what it's good at and what maybe it is not good at. And we need to— people need to know how to, how to actually manage it. And I just think it's really, really critical to get the right results.

Yeah, and, uh, just, just in our last few minutes, I'd love, uh, to get some advice from, from each of you for, um, all of us who are kind of thinking about the many things that potentially could be solved, but, uh, or helped, um, by using AI, uh, in our jobs or at school. Um, what's the one question that, uh, we should be asking ourselves before deciding whether or not AI should actually be used for those tasks? Or— I'll come to you first.

I would go back to one thing I said before, which is what is the actual goal we're trying to achieve? Not just what is the efficiency we're trying to drive, but like, what is the actual vision for this work? What are the parts of it that should remain human? Or we may actually have new roles for humans to actually add more value. Asking, starting with that question flips it on its head. And again, I think this is how we actually get the right results from AI.

Yeah, well, look, I'm just gonna agree with Paula here. I think, you know, cognitive offloading is very good for the most part, right? I am useless at finding my way around places. I am so glad that Google Maps or, you know, any other app exists that enables me to get from here back to my hotel. So I can get to the airport later, right? But I'm not trying to develop a mental model of San Francisco. I don't need to. So if you are trying to learn something, think really carefully about, is this cognitive offloading that the system's offering me actually helping me achieve my objective? And again, it's like, I think you mentioned this earlier, friction is not always a bad thing.

Yeah.

Like putting a little bit of friction back in every now and again, forces us to reflect. And so be deliberate about the purpose for which you're using the AI and ask yourself, is this cognitive offloading, is this giving the task to this thing really helping me long-term? Or should I actually go away and read a book and figure it out for myself?

And the answer is yes, you should go read a book. And also, I think I'm going to take that. I love what you said.

I think that's great. Yes. That's, well, I think that's a really great place to wrap things up. Thank you both Thank you so much for your time. Please join me in thanking Philip and Paula, and thank you all for being here.

Thank you.