Skills every student needs for the AI economy

Hi everyone. Thank you so much for joining. I'm so excited to welcome Dr. Ben Shapiro and Dr. Amy J. Ko, both Microsoft AIEI Senior Fellows and professors at the University of Washington for this session in our Reshaping the Future of Work series. As always, we have our amazing moderator, Dr. Mark Esposito, who will lead the conversation on how we prepare the next generation for an AI-driven world. From education to building inclusive pathways that connect the classroom to real careers. Feel free to use the Q&A in the bottom navigation bar to ask questions throughout the session. Mark will do his best to address those. I'll pass it off to Mark and let's get started.

Thank you very much, Sydney. Good afternoon if you're connecting on the, on the West Coast, and good evening if you're on this coast, or definitely night if you are either in Europe or the Middle East, and probably morning in Asia. Very nice to welcome Amy and Ben to this conversation. First of all, I appreciate that we kind of conclude our Microsoft series with you two, and we've been prepping on this event for a few weeks now. So glad that we could bring you on this series. Maybe a starting question for both of you, and maybe I can start with you, Amy, and then same question for you, Ben. So the Microsoft Fellowship Both of you are scholars. You're professors at the University of Washington. So for some people, why would the scholars start having a fellowship for a technology company? And not everybody necessarily know about the work that Microsoft does in advancing research in kind of societal imperatives that are important for many of us. Love to have your thoughts on this. And it's also to know more how your research kind of fit into that experience. So Amy, over to you.

Yeah, thanks a lot, Mark, and thanks for having me here. I think that when I first saw the call from Microsoft, there were a couple of things that excited me about it. One of them was it's pretty rare to hear industry talk about education, and so it was exciting to see a new voice in the space really thinking about how does education have to transform, what are the impacts on education, and really taking that part of our broader of society seriously. So that was exciting. The other part of it was it's pretty rare to find contexts where a whole bunch of different parts of academia come together to talk about some broader phenomenon. So I have a background in computer science and design and a little bit in behavioral sciences too, like psychology. But I was excited about talking to people in economics, in history, in all of the other parts of academia that have something to contribute to this conversation. And making those contexts myself is super hard And it was awesome to see Microsoft make that context for us and then for us to be able to contribute all to it together.

That's great. Well, I mean, indeed, education is a conversation we'll try to have today. It's kind of our denominator for what has inspired the setup of this series. But before we jump into that, Ben, I'd love to hear your thoughts on this.

Sure, yeah, I mean, first of all, I agree with what Amy said, and to that I would add, you know, I think we're at a time right now where the interplay, both actual and potential, between innovation that's happening in industry and innovation that is happening in academia is, you know, more mutually influencing than ever. You know, a lot of the advances that we've seen in AI, particularly GenAI over the past decade, have been catalyzed by basic research that's happened in academia. And in turn, a lot of innovation that's happening in industry is being very, very rapidly adopted by students, faculty, staff, both to do research, to learn, to teach. To conduct various business processes. You know, universities are very large enterprises. We're— University of Washington is the largest agency of the Washington state government, for example. And so I think this coming together also offers opportunity for mutual learning, for us to influence one another and to better appreciate the places that each other is coming from and perhaps to push each other in ways that might not happen through our usual ways of interfacing. And all the better that Microsoft did a really wise thing of bringing together people with such a wide variety of backgrounds, both intellectually and then in the second cohort. I really love the way they've sort of expanded the geographic diversity to include more of the developing world.

That's so true, Ruben. I was noticing the first cohort, it was predominantly US with some cases in the UK, and we have Fabian from Oxford. And I noticed the second cohort has a wider geographical distribution. So when we'll eventually bring some of those folks to our series, it will be great, right, to see how much of the very same conversation that we're having somehow they might change as they change kind of the location in which they are experiencing. So thanks so much for the intro and to both of you. A little bit deeper into kind of your specific work, I mean, I'm going to maybe address this over to you. So a lot of the work that I think in your scholarship you argue for, Emmy, is you like to see more justice, more ethics, and social responsibility in computing education, right? And so imagine that you want to help us imagine what does an AI-ready student, right, looks like when this dimension are fully integrated into their learning journey, right? What would that be like? And I think it's fascinating you equally introduce yourself by talking about your computer science background, But you also mentioned other fields that, that range all the way down to behavioral science, right? So imagine this now converted to what could be a kind of a blueprint of a journey for a student that wants to integrate this more societal impactful subjects, right? How would you envision that?

Yeah, that's a great question. I think I want to step back a little bit to start answering it by looking at just the history of how people in general have learned computer science since we started thinking about it as a technology that the broader public would engage in, right? All the way back in the 1960s, the people who were using computation to do things of value, there was a very tiny group of people at universities, at NASA. A lot of women, for example, were doing the, the early programming of mainframes to make things work for the federal government and for science. When it came into universities, that shifted a lot. That culture shifted a lot to being particular subgroups of mostly wealthy men. And so that was a big transformation that happened. And so if you zoom all the way forward to now, who is in the classroom now? Who is doing that learning? What does it mean to successfully make sure that people can harness all of the powers and capabilities of computation? That hasn't really changed since those early days in the 1960s and '70s. There's little shifts here and there, but it's mostly the same group of people. So when I think about things like generative AI, And what it could hypothetically mean for some students is a radical reopening of who can create with computation and meet the needs that they have for their communities, their organizations, whatever they happen to be doing. And then there's the reality of it, right? Like there's the, there's the hype and promise of vibecoding making this something possible for everyone. And then there's the on-the-ground reality of it, which is it's actually really hard with current technologies to make that work in practice. You have to know a lot about computer science. You have to be able to read code that it's generating to diagnose why it's not giving you what you want. And so there's some bridging to do between that dream and the reality of it. And that's where I think things fall apart. So we're seeing a future-ready student really being one that both can harness these emerging technologies to envision and create the things that they want to create for the world, but also have some of those core fundamentals to be able to steer it in a meaningful and informed direction without having to sort of rely on others or rely on yet another prompt to figure out exactly what they're trying to create. And so that's a big shift in education. That's a big shift in what it means to program. It's a big shift in what it means to create software in the world. And I think there's a lot of exciting both research and innovation in industry that's going to be part of realizing some of that future.

Yeah, it's fascinating that I think we are at the very beginning, right? I mean, what could really become a full-fledged redesign that will be intersectional in comparison to what probably was before. And I'm not surprised to hear how computer science did not necessarily change that much in comparison to where it probably could have gone if we consider the factor that you just mentioned. And I appreciate that you are so open, right, about the advocacy required for this, which kind of bridges over to the question I wanted to ask Ben. Ben, very, very conversely to how Emmy is advocating for this, your work explores almost like the playful and embodied learning environment for computing. And it equally has inspired a bit of our prep script for today when we were looking at what we want to talk about. So when you're looking at these design principles, right, how do you think it's going to shape young people, especially when we are trying to put together AI, intuition, creativity. These are area where I find many people ask me question directly, but it's not easy for them to navigate that systematically, right? It's always about maybe what could they do, accidentally meeting the right mentor. I think you want to make this more systematic so that it's becoming something that more people can have access. So share with me and with everyone watching a little bit more about this, please.

Sure, I'm happy to, although I would just first say that I think playfulness is also a big part of the work that Amy and her students do. So I just, you know, I appreciate the contrast, but I also think that's a way in which we're actually kind of similar. I think for me, a lot of the way I approach research in the space is by not prioritizing computation or AI per se, but rather starting with— I prefer to start with what are the problems or goals that people already have, the hobbies that they already have, the passions they already have, the curiosities they already have, the ways they want to express themselves, and what are the ways in which we can invent new kinds of technologies or learning experiences that enable them to take those passions further. So to give an example, we had an NSF-funded project that wrapped up a couple years ago. This was co-led by Kayla Desports at NYU, who's absolutely terrific, where we worked in collaboration with a nonprofit called STEM From Dance, whose focus is working with teen women of color who are dancers and passionate about dancing and finding ways to integrate STEM learning into their dance practice. And so the vision there was to bring machine learning and AI and data science into their dance practices in really authentic ways. As an example, if you have dance moves that you're trying to do and you're trying to do really synchronized with one another, you could use microphones that can, you know, help you measure the sort of percussive impact of feet on a floor. And you could use visualizations because to see sort of where are those impacts happening over time. And if people are really out of sync, you're going to see this distribution that's really wide. If people are really in sync, you're going to see these very tight, you know, distributions. And so starting with ideas like that and then progressing into ideas like And this partly coincided with, one, the start of the COVID-19 pandemic and the explosion of TikTok. So we'd already been planning to do some of this but had to make a pivot in 2020, where originally our idea was that the young women could be dancing on a stage together, and then they would be writing programs that are using machine learning models to recognize where different parts of their bodies are and project visuals on top of them. You know, think different kinds of animations or even wearables that they would make that would react to the ways in which they're moving. But with the pandemic starting, instead of everyone being on the same stage, everybody was like in their teeny tiny New York apartments. But at the same time, TikTok and the genre of making and sharing online dance videos exploded. And so we were able to do a little bit of a pivot there And it became about being able to create these dynamic animated overlays that would play, that you could then mix into recordings of yourself dancing. And in doing that, you have to wrestle with all kinds of problems. So for example, we were using PoseNet, which is an open, pretty lightweight vision model for recognizing different, where different parts of the body are. But a thing that happens a lot when you dance is you end up turning your back to the camera. And so if you're a dancer who's written rules that are anchored on like, where are my eyes and things are in relation to my eyes, for example, now I've turned my back to the camera, the model is going to start returning nulls for that aspect of the body geometry. And suddenly your code is going to start breaking in all kinds of interesting, weird ways. And those might be ways that are interesting artistically, or they might be ways that don't align with your artistic vision. And so from there, you could either go deeper down the rabbit hole of, well, how do we program in cases where we're going to have missing data? Or you might just change your choreography so, you know, you're always facing the camera and working around this limitation. So the idea there really is to, as much as possible, keep front and center the sort of expressive intentions or the curious intentions if we're talking about, for example, science. And let that drive our design research around AI and computing education.

Wow.

You know, I have to tell you, I have never heard about this before. And so it is— I can't imagine anyone listening to you feeling fascinated and equally curious on how this eventually will look like and having that as an experience. But I think it's a great way for us to understand where playfulness and creativity, they become actually actionable, right? In the perfect example of dancing and how different that experience then shaped, I think, the inquiry around the AI models that have to be eventually investigated later on, rather than thinking that the AI model is self-centered itself, right? And consuming some importance by simply being about maybe just coding. So this is fascinating. And I think it's maybe connecting to the next question I have for Emily before before we maybe try to tap on the question about the employment part of our conversation. So kind of expanding on this conversation that the band has shaped for us, I mean, I think a lot of your work equally tries to help learners, right, to kind of integrate the hidden complexity of software systems, where basically AI is not just about the technical part. There is the probabilistic nature of AI that are naturally going to create a space of conversation about failure, about biases, about uncertainty, right? So how do you bring that into your conversation with your research and with your teaching? So imagine you having people that are expecting to engage with software programs. But now we have to give them this understanding that, yeah, you know, there will be buffers and limits in the way we use it. There will be uncertainties, there will be mistakes. So how do you kind of bridge that side so that they are becoming aware of the probabilistic nature of this technology? I think you're muted.

Yeah, I unmuted.

Yeah.

I love that question. I guess I would set up this contrast that we live with, right? With the kind of procedural deterministic code that has built the computational world that we live in for the most part, and the increasingly probabilistic programs, the machine learning programs that more of our world is now being built with, right? Those are two parts of computer science and computing in general that we could all learn about. And I think that in a lot of my past work, I've done a lot of research to understand how do people learn to reason about that procedural kind of code? How do they think about it? How do they build models in their head about it? What's hard for people to learn about that skill and hard to learn about programming languages, for example? And the things that are hard about that discrete logic and reasoning about discrete logic and thinking about all of the the complex interactions that can happen when you have a program that is no longer just one function, but doing potentially hundreds of functions or thousands of things that are interacting with each other. There's a lot of hard skills to learn in that space. And then you shift over to the probabilistic side. And that set of stuff is very, very different, right? You start having to build reasoning about statistics, about distributions, about not only lower-level statistical ideas like false positives, for example, but things like bias that aren't really about the statistics alone, but they're about the relationship between the model, a statistical model that's been learned, and the world. So we've done things like, let's go and study how children reason about these things and how their reasoning might differ from, from older groups. We've done a lot of studies with sibling pairs, for example. Let's say there's a 7-year-old and a 9-year-old, and we'll think about how they reason about algorithms and think about how they reason about probability realistic types of programs as well. And a lot of what we see in how they do that reasoning shows that they draw heavily upon what they know already. And usually what they know already are the notions of fairness that exist in their family or their sibling relationships, for example. They will decide that an algorithm is fair or that some AI's output is fair if they can map it on to some of the moral reasoning that they think about in their family structures too. And then that shifts into deeper and more rigorous understandings over time. So while we've spent decades figuring out how to teach people programming languages, we probably also will have to spend decades figuring out how to teach them to reason about these probabilistic and ethical kinds of frameworks as well. And we've only just started in that, in that work.

Yeah. Wow. So, I mean, you inspire me to maybe extend this level of the conversation to then with one more question before maybe we cross the bridge to the, to the employment part. Then one, one thing that I, I think I found when I was trying preparing for this is your work on equity in computing. And so one of the questions I have, there must be students that are underrepresented by technology that have been prevented from engaging in a meaningful way, especially with AI. So how do you help them? How do you overcome these gaps that might feel or that might be structural? And I think that the question in the overarching sense is like we do We know there is a problem with equity in general, the equity in technology, equity of access. These are not necessarily technology that have reached out everybody in the same way, right? And we feel it in many different ways. But I'd love to hear your thoughts on this. I think it's where I'm still exploring this nexus between computing or computer sciences and sort of like the social dimension that sometimes gets forgotten or gets to some degree obliterated by the, I would say, the dominant economic narrative about technology, right, which is mostly on the efficiency of it. So share your, your, your sense, two cents on this. Love to hear your thoughts.

Sure. So I think this is a, this is a many-layered thing. One of those layers being that Learning is what my colleagues here at the University of Washington— and there used to be a thing here called the Life Center that they made this beautiful diagram and they sort of talk about learning as lifelong and life-wide, right? So it's not like we stop learning as soon as we're no longer students, right? We learn our whole lives. Often that happens around major life transitions, right? You start a new job, you might have to learn a bunch of things. Maybe they're about a corporate culture. Or maybe they're about particular technologies used at that company. You have children. Suddenly there's a whole lot you have to learn about how to take care of them, how to raise them, how to not panic when their temperature reaches up to a certain level and then when to actually panic. Same happens when we start caring for aging parents. All across our lifespan, we're learning. And we're doing that learning at home, at school, at work. In hobbies, in relationships. And so I think that's one layer we need to think about is if people are learning across all these spaces and across all these time spans, including at work, what are the ways in which there are barriers to tech equity in terms of how well we support that learning? So an example, that, you know, I sometimes hear about from my students is sometimes like their families will buy smart devices for their homes or their parents will try to use, you know, the speech-to-text input mode on their phones. And even though they speak English fluently, they might have accents. And so the tech works way less well for them. And that's a case where it's actually affecting older people because they're more likely to have have heavier accents than young people who are growing up here. And right now there's very little agency that they have to do something about that, to actually change the tech to make it better. And I think it speaks to some limitations in the industrial practices of software production. And so one of the things that we're trying to do in some of our research is to develop both technologies as well as educational programs where young people could work together in, for example, high immigrant percentage communities to make customized versions of the various ML models that are embedded in the AI technologies that are used every day, but actually work better for the older people in their lives. And that's important both because it gives them agency to address some of the equity challenges that they observe affecting their elders, but it also helps them to start learning some of the machine learning skills that are actually going to be increasingly important in the workplace and in the workforce, including techniques for measuring and evaluating model performance, techniques for collecting and annotating data, techniques for modifying machine learning models via techniques like fine-tuning, and then techniques for evaluating model quality so they can actually measure if these things are getting better over time. So what we're really trying to do in a lot of our work is use the equity challenges that people observe and that they observe being harmful to people that they care about as both motivation for learning and then as an opportunity to actually make an impact on their communities.

Great. You know, I, I I was doing a short in-residence fellowship for literally a week in Kansas. And I found out, Ben, that many of the students, about 50% of the students at the university were first-generation students. And when I was asking the question, I said, what do you mean by that? Like, yes, it's the first kids in their family who go to college, right? And I think, yeah, there's a reality sometimes doesn't necessarily get captured by the headlines, but I'm really— really, really—

but I think there's also, there's also a political dimension there, which is, you know, a thing that I've heard, heard a lot about from young people is the self-blame that they see or hear from their parents or grandparents. Like, it doesn't work because I have a bad accent. It's like, no, it doesn't work because it wasn't made to work with your accent. There's a difference there. There's a— there's an ability to actually recognize who's responsible for this problem. It's not the parent who learned English fluently as an adult. No, it's the maker of this technology. And by recognizing that, that actually is a way to think about their own future responsibility as technology designers, makers, marketers, salespeople, adopters, you know, purchasers, to consider these things and then make them part of their decision-making criteria. Yeah.

Yeah. Yeah, which I think it, it, it echoes well with what you're saying before about the second cohort of fellows with a wider geographical representation, also to have a better mapping of what it means, right, to be in an institute for AI economy that doesn't necessarily, uh, honors a US-centric view. Um, and, and I, I appreciate this a lot, Ben, and, and the way you usually represent it is. Um, so folks, if you allow me to start kind of moving the conversation towards the employment. Like, as an educator myself, many students, they do come, they talk to me. I guess in the last couple of years I have noticed increase in anxiety about the labor market. And, and I think it's just not just in the US, around the world. It is a much more globally distributed, diffuse, I think, degree of uncertainty about what happens. And so I'd like you, I mean, I'm gonna start with you to help me think about an AI-ready role that is not just ready in terms of coding and algorithmic knowledge. So imagine if an employer is asking you, well, I'd like to hire somebody that can really work with AI, but not just about coding and algorithmic knowledge, right? I like to think about them much more well-rounded profile. How do you think we should represent that opportunity to employers so that maybe we can equally anticipate what the market will start appreciating more or how we can educate the market at the same time to appreciate more?

Yeah, I think, I think that's a great question because I think there's a lot of emerging complexity around the labor markets that we see that are not just due to uncertainty, but technological change like generative AI, but also just as always, new opportunities that emerge too. So So I kind of want to think about these two different scenarios. One is where it's an employer trying to think about what kinds of things can students learn for them to be better prepared for what those futures are. And that's the case for most students, that they're going to go to an employer and try to meet those needs, and employer's going to try to articulate them. And that relationship is really messy right now, right? Employers actually don't know what they need. They don't quite know what kinds of literacies are important. And students don't really know what those needs are because employers can't really articulate them. And then we have higher education in the middle trying to sort out and clarify some of that. So, and as an example, we're preparing a new AI certificate here in the Information School at the University of Washington that's trying to solve that exact problem where we prepare students that are the experts on generative AI that organizations need to clarify those questions and to help transform whatever their HR priorities are, their AI investments are, their governance questions, their risk assessments around AI. They don't have those people often in their organizations to help them figure out who to look for and what skills to get. So that's one way that we might unlock it, right, is to try to, like Ben said, bring together industry and academia to come and build partnerships around what is that learning and how can it unlock all of the potential that organizations can't yet name. Yeah, they can't really organize it yet. The other side of it, and this is, you know, another thing that's happening in the Information School and the Allen School engages in this a lot as well, is just pure entrepreneurship. It is often the case that students who are passionate about creating new opportunity, that they might be better served by just going and creating their own enterprises that tackle those new opportunities that they see. And finding ways of creating value in ways that existing organizations can't yet see. And so we see students doing that as well as taking the knowledge that they've built around generative AI, the opportunities that they see that bigger or established orgs can't see, and just go directly compete with them. And sometimes that works and sometimes it doesn't. I have my own background being a chief technology officer and co-founder of a startup too. And so I know what that's like. Feels like to take that leap. And I think both of those have to happen at the same time, right? Really enriching the capacity of organizations and empowering students to go off and be entrepreneurial, and we'll figure it out through all of those systems.

Yeah, I appreciate that. I mean, I think you hit the nail on the head when you say about, you know, academia and industry. It is what has largely approached me to my career at the pretty I would say advanced stage of my academic career, I started to have this urge that I wanted to be closer to an industry player. And I say very often, sometimes I repeat it on the series, the biggest learning, well, the learning curve that I had in my latest part of the career has been when I joined the industry, right? And really being confronted with an entirely different logic that is happening and then cannot be unbundled from what we do in university. It has to be somehow integrated. So I appreciate that, the perspective a lot. But I have a question that is a question for which I have no answer myself, although I feel it is a question that is important to me, and I love to pick your brain on this. A lot of your work is about experimentation, right? When I'm trying to translate this in a conversation with an employer, I think it depends who I'm talking to. Some employers, they're happy to hear it because I think they have already crossed that space in their mind in which they understand in the experimentation, it is a form of innovation. Innovation, other employers that will immediately create a sense of resistance to that. How can you, can we help the industry to appreciate this presentation without feeling that it is so like a half-baked approach? It's not, it's not just an attempt that is only preliminary, but it is really a form of engaging with important questions. What would be your, your suggestion in this case?

Is that from, for me or for Amy or both?

Yeah, yeah, well, I was, I was thinking about you, Ben, but sure. Yeah, we are.

I mean, yeah, I think there's different kinds of experimentation that organizations do. Sometimes experimentation is about new product and service offerings. Sometimes experimentation is about new business processes for delivering existing offerings. Sometimes It's about new organizational structures. Sometimes it's about new types of partnerships and relationships that they want to enter into. And I think where these things can get really challenging is when you don't know what your goals are. Like, what does better mean? And how will you know it if you see it? You know, I've seen so many times especially in roles I've had in industry, people doing things like, you know, can you help? I'll get questions like, hey, can you help me how to figure out how to use AI in my work? And I say, okay, well, what problem do you want to solve? Well, you know, we were told we should try to use AI. And I'm like, okay, but what business problem are you trying to solve? And I think if you're letting the tail wag the dog in the sense of like using AI to use AI you're probably going about it wrong. If your experiment is about seeing an opportunity to work in a new way or develop, deliver a new kind of value to your customers or to overcome problems that you recognize, or maybe even some people in the organization believe exist in our organization and some don't, then I think you can start to define metrics or milestones or indicators of better or progress or at least interesting and use those to start working towards systematic processes for trying things out and deciding when to call it quits. You know, the thing about experimentation is if it's really experimentation, it's not always gonna work. And different organizations have very different sort of cultural tolerances for risk. Even different Even the same companies, you know, you talk to people from different divisions located in different countries and there might be different, you know, degrees of risk comfort or risk tolerance in those different pockets of the company located in different geos. And those things also are both barriers to, but also opportunities for change. You know, I tend to think it's probably an unhealthy sign of an organization if failure is sort of almost always punished because it means, well, then you have a very low risk tolerance and people are not going to experiment. They're not going to experiment meaningfully. Yeah.

Yeah.

Right. And so I think sometimes we might be able to use these kinds of, these sort of inflection point moments where the, hey, there's this disruptive technology. We're not really sure what to do. As also a way to think about like, well, is our organizational culture in the right place for us to be experimenting and making change? And if not, it's not a tech problem, it's a culture problem that you have to solve in order to be able to make change with technology.

Yeah, I think, Ben, you mentioned 3 levels, right? There is a technology side, which kind of takes care of itself once you're clear on, on the cultural side. And then you have a stipulation of the business problem trying to address, right? And I think that the priority is on the culture and the strategy before being the technology. And I'm happy that kind of is reinforcing what I think some of us has been thinking as well, right? In feeling that a bit of a disconnect between where we sometimes we prioritize and where we instead we should prioritize. So folks, if that's okay with you, I'm going to start shifting us towards some of the questions that have joined. And this is also the time where I'm inviting those who are watching live to put some of your questions in the Q&A so that we can have a look. But there is a question that Sydney has helped me to synthesize so I can read it out to you. It's from Hatef. And the question is about how might early AI education affect that long-run productivity in labor market outcomes. And I think the question, if I understood this correctly, Hatem, is like before basically university, right? So if you're thinking about what we should be doing, for example, at high schools or even before, middle schools, how do you see kind of the need to do, right, that are important not necessarily at the level where they are becoming university students, but even before? What would you recommend in this case? And any of you folks is fine for this question. I think we can, we can have both of you sharing maybe your thoughts on this.

Yeah, I mean, I think I'd love to build on what Ben just talked about too around risk, right? I work with a lot of middle school teachers and they are in lots of different schools and just those same exact cultural dynamics that Ben talked about. They show up in schools as well. There are some schools and school leaders who are ready to have their teachers dive into something new, try something, not know if it's gonna work, take the flak of parents being unhappy that they've deviated from whatever assumptions they have around what students should learn and tried something bold. And sometimes that's gonna be the thing that is, um, some really new approach to AI literacy, some engagement with emerging technologies where it's not even clear how to assess students' learning. Or to decide what they should learn about it, but they're trusting youth and teachers to go and explore and really understand the capabilities of things. It's that exact same kind of experimentation in learning that sometimes organizations need as well. And I think that fostering that culture of risk-taking in these times of change can be just as important in schools. And we often don't let schools do that I see that in the other schools that I was referring to, where there is huge amounts of not only risk aversion, but a sort of culture of scarcity too, especially in the United States, where public education sees huge disparities in the amount of funding that goes to individual schools. When a school has no resources to even do the basics, they cannot imagine sometimes creating a context for youth to learn about AI or computing more broadly and trusting teachers to try that, or even taking the bet on hiring teachers to make that space for learning possible. So unless we collectively across, not only in the United States where we are, but in all of the countries in the world say, we want schools to experiment, we want youth to be part of those experiments, we want teachers to be part of those experiments, and we want families to step aside and just trust that great things are going to come from all of those transformations that youth and teachers and potentially their families and communities might imagine, we're not going to see any of that long-term transformation happen. We're going to end up with the same kind of stagnation that might happen with a for-profit enterprise that's risk-averse and can't respond to change.

But maybe I'd love to have your thoughts on the same, right, in terms of what we should be doing at the maybe early years about this. Your thoughts here, where you see we have a space for— I'll call it intervention, but probably there's a better word for it.

Yeah, I mean, I think one of the things that Amy mentioned earlier is worth coming back to. So she mentioned how in the world of software, we're sort of transitioning from a a world where things were discrete, they were logical, they were deterministic, to a world in which we have a lot more probabilistic behavior in the kinds of systems that we're building, right? So, um, you know, machine learning models are probabilistic statistical objects, whereas traditional programs are not. They're sort of logical symbolic objects. And I I think learning to think about the world as a place full of probabilistic behavior is a really powerful turn that often we don't help enough young people to make. And moreover, you know, there's a lot of work in science education that, you know, over the last 20 years has gotten into What does it mean for young people to learn to think about phenomena in the world in systematic ways, to build models of things like how does a shadow work? What are shadows? How can we explain what makes shadows appear? How do you build a model of shadows, right? It seems so obvious, but actually to a child it's not. And You know, you have to have a model for light and diffraction and diffusion. And why are the edges of a shadow blurry? Well, now you're starting to get into some of these probabilistic Gaussian models, things like this. And I think, I think creating more opportunities for young people to analyze things in the world that they're curious about and to do it from this sort of lens of modeling, whether it's symbolic or whether it's statistical is a really powerful thing to do. And one of the things that we know from research in science education as well as math education is that children are capable of this kind of very sophisticated thinking, often much younger than we give them credit for. But to come back to something we talked about moments ago, if you're gonna, if you're gonna build a vision for education that is rooted in young people's curiosities about the world, you have to build a culture that supports curiosity, risk-taking, etc. Because the truth is, if the more you, you know, if you put everything on rails, it's like, great, we're all going to answer this one question. Well, 10 people in the room might be interested in that question, but 20 might be bored to tears by it. So you have to create room for, for differentiation for exploration, for passion, and for connection, including, you know, connection to family, connection to culture and things like this.

Yeah, really appreciate that. But I'm currently traveling in the Middle East and I'm currently in Saudi Arabia. One of the things that I always find fascinating is how much their love for the future gives them space to think that countries that have imagination have a future and countries who don't have imagination naturally can't have a future. I mean, at least it's a dramatic statement, but I think it's really how do we represent this at the school level, right? Where we're really helping kids to imagine, right, the world that they would like to have and use the tools that they have, including the technology to get there. Right. We become this grand vision, right, that I think is I mean, that's right.

And I think right now it's, it's becoming really stark how many countries' politics have started to diverge in terms of, is it about imaginations of futures, or is it about trying to recreate pasts? And I think by and large, the ones that are imagining hopeful futures are the societies that are flourishing the most.

Yeah, I have to say, sometimes the, the current geopolitical landscape feels that we are nostalgic about reviving the past, and that's extremely unsettling, to be very honest. But there's a question that Mesa is bringing up, and I think, I mean, that's a good question maybe to ask you in terms of— she's bringing us back into computer science curriculums, right? So how— the question she's at— he or she asking is that is how do you do you as educators see computer science curriculum at the college level shifting? And I guess not necessarily just only in terms of what we discussed so far, but I think there's been a change in computer science curriculum with the advent of technology like AI to the point that this could no longer be ignored. It had to become important. But again, I come from an entirely different department. I did my training in economics, so I'm extremely ignorant about this. So help me understand this question: how did you see the change of the curriculum, the computer science curriculums, considering what we discussed so far today?

Yeah, I think I'm going to borrow just some, some ideas from Ben and I's colleague Dan Grossman over in the Allen School, the Paul G. Allen School of Computer Science and Engineering. One of the things that he likes to talk about is that computer science teaching and curricula have always changed in response to innovations in computer science. And how we program systems today isn't like how we programmed them 10 years ago, 20 years ago, 30 years ago. And so there's kind of always a reinvention happening in thinking about what that teaching is. Maybe in recent memory it's been a little bit more stable. There's just been a lot more change in industry around how software gets created, and higher ed hasn't always caught up with that in the way that we teach what programming and programming languages are. Mm-hmm. Now I think with things like generative AI, what's really shifting is just this question around when you delegate to a machine to synthesize code for you. And all of this synthesis work around generating programs, this is work that's been going on for 20 years in research. So we've been grappling with that question for a couple of decades now, and we've learned a lot about when that's a good idea, when it's not, when it's easy to do, when it's hard to do. And it's very clear with large language models, that it's sometimes good at it, sometimes bad at it, depends a lot on the training data. And so that kind of skill of deciding when to delegate, that's really the thing that's sort of this emerging new skill that students have to learn and a new skill that senior devs have to learn in industry as well. And so I think that's what we're all sorting through in curricula is trying to figure out what is that skill? What does it mean to be good at it? How do you teach it to people? How do you know that they've learned it well? How do you differentiate between doing it at an excellent level versus doing it at a novice level? And I don't think we have that full knowledge yet about that spectrum of skills. And I certainly don't think that industry has that knowledge yet either. In higher ed and research, we spend a lot more time studying those questions than industry does. And we've seen, for example, some really clear studies that show that not only is there a spectrum of skill in deciding when to delegate and evaluating the results of, of an LLM, but it sort of bifurcates success as well. If you're really good about thinking about when to delegate and regulating your thinking about that, you'll do even better than you were before at constructing programs. If you're not good about thinking about your thinking, that metacognition that's really central to learning, but just problem solving in general, you're actually worse than you were before. And so that split in students' learning success, that's something that's the sort of grand challenge of trying to figure out how to teach programming these days is how do you bring everybody in to that higher level of excellence as opposed to splitting and making the underperformers even worse?

Yeah. Well, you know, I, I think Ben, you were using the, the term agency before, right? That's a great way of understanding how do you basically build agency by understanding agency as almost a responsibility, right, in that degree of delegation.

Yeah, I think that's right. I mean, I was just thinking of a study I read recently where right now even software engineers who are pretty experienced using agents and other sort of generative AI development tools are generally pretty bad at estimating whether the practices they're using they're doing with these tools are making them more effective or not. So often they think they're becoming more efficient, but actually they're doing, they're taking like 20% more time to complete the same software engineering task. So right now there's a real metacognitive challenge in being able to not only decide what to do, but even to reflect on how well things are going when you're doing it. One of the things that I'm finding quite interesting is my colleague Mike Ernst, this, this term is teaching a new course here that's basically how to program with AI. And this is a course that sort of takes a bunch of things that he's used to teaching in the context of sort of upper-level software engineering courses, like how do you do a good code review? You know, what are best practices for pair programming? Programming, all the sort of social aspects, all the places where sort of social interaction intersects with software engineering, sort of the production of programs. And many, many of those practices are pretty portable to interaction with agentic AI tools. So a lot of the things that you ought to do if you're reviewing, you know, a pull request or other contribution of code to a project from a colleague, are exactly the things you ought to be doing when working with a cursor or another agentic AI tool. But those are things that often students don't learn in universities and that senior devs are mentoring junior devs in how to do. But we're taking the approach of, well, let's bring these tools into the learning experience and let's teach students those skills as soon as we can so that then they're able to practice them in an educational setting, get feedback, and then when they're entering the job market, they have a bunch of those skills that they don't need to wait till they're senior devs to have. They can have them from the get-go. Yeah.

You know, I appreciate Ben and Emmy, and I'm aware we should kind of bring it over to a closer end. And even if sometimes we can't address all the questions coming, but I'll try maybe to ask you for the final one that is trying to integrate what came through the Q&A. So if you imagine that you are giving a suggestion not to one of your students in computer science, but somebody coming from an entirely different field that doesn't have necessarily the same implicit— in this case, well, Mahab was asking a good question. He's a medical doctor. He'd like to get involved with AI. It looks like that for them, the space to cover before you can eventually make the decision, of course, takes into account a different set of decisions in general because of the distance they might be from the exposure to technology. So where would you find for, for non-AI, let's say native students to some degree, that first step to be? And I know that this is not a scientific question. It's more about where you think that steps should be made, but I think it's a good way for us to finish on our series. Any of you, and I'd like to have both of your perspective on this. So Ben, Amy, please.

Sure. I mean, so yeah, so one place to start is to think about what aspects of the kind of work that you do. You know, if you're a doctor, doctors do a whole lot of stuff, right? They not only interact with patients and try to diagnose illness and explain it to patients in ways that make sense to them, but they, they're essentially entrepreneurs unless they're in really large hospitals. They have to handle billing and deal with insurance companies and, you know, all these kinds of things, at least in the US. Maybe it's easier elsewhere. So a place to start might be to think about what are aspects of your work where the risk is relatively low. But the potential benefits to you as a practitioner in terms of time saving or ability to do a better job might be there such that you could then take off-the-shelf tools that companies like Microsoft are putting out, things like Copilot Studio, where you could start to figure out are there particular aspects of your work that might make sense to experiment with AI workflows around. And as you do that, you're going to generate lots of questions along with answers. And those questions are then new things for you to learn, right? Each one sort of offers you— each step forward you take is going to offer you many things that you might then have to learn to take that next step forward. But I would recommend, especially if you're in a field with stakes as high as medicine, start with the lowest risk. Aspects of that. So you don't want to be making diagnosis decisions. You may want to be really careful about medical billing so you don't accidentally commit billing fraud because of some generated bill. But think about where you can have relatively low risk, but you know enough about the domain so that you can catch errors in the tools that you're working with and figure out what do I have to do to mitigate them to get the work to be at a high enough quality that I'd stand behind it. And I think by doing that, you pretty quickly have to contend with a lot of the things that we want people to learn about AI. That's great.

I mean, any final remarks?

Yeah, sure. I'll try to broaden it a little bit too as we go out. Ben mentioned earlier the Life Center here. Just this was a really great celebration of the idea that everybody is learning throughout their lives. And learning is a central part of living. And I think a lot of what Ben just described too around all of the kinds of learning that somebody might do in a domain that they want to bring AI into or that they want to gain expertise in, figuring out how to orchestrate that learning, figuring out how to organize and plan your learning, figuring out what resources to learn from, finding places to do that learning and people to do that learning with, those are all really hard problems And I just want to observe that we have had a solution to that problem in many cases, and that is teachers, right? And a teacher can be many different things. Sometimes we are teachers for ourselves. We can organize our own learning, figure out what questions to ask, what resources to consult. Sometimes we need somebody who is an expert in teaching. Sometimes all we need is somebody to learn with, and we can be teachers for each other. And so In this time where we don't quite have all of the formal systems of education to do this learning, I think that kind of broader view of learning is going to be really central for everyone. Sometimes we're going to have to teach ourselves. Sometimes we're going to teach each other by finding other people to learn with. Sometimes we're going to find informal teachers who have some expertise and some knowledge and they can help guide us. Sometimes you're going to find, maybe especially in the next couple of years in higher education, formal places to do that learning too, where we're building some of this learning into our academic programs and opportunities. And I think that's an exciting time for us to all wrestle with all of the changes happening in the world. That's going to mean a lot of learning. Let's all be smart about how we go about doing it.

I mean, and then look, I— you were asking me, I mean, before we started, what defines kind of a good series, right? And it's the fact that we had to stop because the time is up. Up, otherwise we could continue nonetheless, right? Because the curiosity and the interest in learning from each other is there. And I have definitely learned a lot from talking to both of you. I love the fact that you're sharing a lot of passion for what is ahead of us. And I know that you are committed daily to advance your own field by integrating, by making them more seamless conversation, by building what I was sharing before, in introducing your profile, this intersectionality that is so important these days, regardless of our designated department of affiliation. So truly, truly happy to have met you. It was virtual, and, you know, I always hope that eventually life will give us a chance to meet in person, but I really appreciate the time you took, busy schedule end of the semester. I guess you are also finishing grading and anything that comes with the job that we have. But I am happy that we are concluding our Microsoft Series 2025 with you. We resume again in 2026, but it's been great to learn from, from the fellows of the Microsoft program, exceptional scholars that are bringing exceptional research to the forefront. And, and I hope that they inspire everyone who watch it. This will be shared later on so that more people can learn from the conversation we had. Since it's a propitious time, I wish you happy holidays since I can do it now. I couldn't do it in November. It was too early, right? But I can do it now that we're in December. And then again, thanks for being with me today. Really, really nice to meet you both, guys.

Thanks for the conversation.

All the best. Bye-bye.

Cheers.

Bye-bye.

Thanks for having us.

Thanks. Bye-bye.