Scaling Responsible AI: From Enthusiasm to Execution with Noelle Russell

Exciting. We have to start with a deepfake, right? Like, that's the game we are playing right now. I always invite all of you to, to kind of think about a world where you can right now, really by the end of today, build a deepfake of yourself with zero technical skills. That was— I think it took me 2 minutes recorded on my phone, uh, and about 7 minutes of patiently waiting. And then at the end of it, not only did it deepfake my likeness and my hair, mind you, but also my voice. Yay! All right, like, isn't that exciting that we can deepfake our voice with no skills? Oh right, and it's free. Yay! So anyone in the world can deepfake themselves and put it out on the internet and, and never tell anyone that it's not real. And like, great way to start. Let's, let's get this party started. If you are interested though, a lot of the things I'll talk to you about today are real tools that you can use. HeyGen is actually powered by ElevenLabs, both pretty big labs in the, uh, synthesized media space. But both of them have the opportunity for you to go in and record yourself and create what's called an instant avatar. And that is not terrible for an instant avatar. It's a little scary, but it's not terrible. So I'm going to encourage you to think about, think about, uh, using that for For those of you— I know I see some friendly, friendly faces in the audience. Thank you so much for being here. Um, this is my QR code. It's dangerous. QR codes are dangerous. However, this should take you to LinkedIn. So if you want to step on the edge of danger and become my friend on LinkedIn, you could also just Google me, Noelle Russell, and I'll come right up. So either way, whatever you feel more safe, uh, doing, it is a chance for us to be friends. The most important thing I learned, probably coming out of COVID is the value of human connection, especially in the world of AI. My network is pretty rich, and my network is your network. There are many of you in here who know how sincere I am about that, but because some of you don't know me, um, I'm sincere about that. So this is me. You can also find me, Noelle Russell, on basically all social media platforms, um, and It's just my chance to say I really hope we get to be friends. A lot of things, a lot of people that I am friends with today, I look at this time last year and go, oh my gosh, we met at South by Southwest, the very first one. So I'm really honored to be here for a second year. They normally don't ask people to come back. So I'm very excited that I got a chance to come back and talk to you all. Things have changed for those of you who, you know, have been around the last couple of years. I think I've always been very vocal about my beliefs in responsible AI, and my family is one reason. Yes, nearly Noelle, affectionately called, did mention my 6 children. That was not like a hallucination. That was real. I do have 6 kiddos. That's them. We just took a Disney cruise and we do studio photos. It's the only time I can get everyone in one room at one time. And that is all of us. My husband, who is here, thank you very much for traveling with me. We both— yes. Left our children with my mom. So thanks, Mom, uh, for doing all of that. But the reason I got into this world, into technology, into AI, even started actually quite a long time ago. My son there, his name's Max, he's now 19 years old, but he was my first baby, my very first baby. And I was— when he was born, I remember, I mean, he was a week late. So any of you have gone through this journey, you're like, okay, let's get this party started. Uh, let's go. And I was a first-time mom, so I had no idea what to expect. And, uh, then he ended up, you know, we had a delivery. Delivery was great. He started nursing. And then 2 days after he, um, was born, I was like, we go home now, right? I think. Uh, and no one came. And then eventually a doctor did show up and ended up whisking him away to the, what I now understand to be an intensive care unit for babies. But I didn't know anything about this at the time. They whisked him away to the NICU and another doctor came in who I would eventually understand that role very well. But he came in and he decided to maybe share with me some news that would change my life forever, really. He said, "Hey, you know, it looks like— it took us a couple days, but it looks like your son was born with a condition known as Down syndrome." And as a brand new mom, I didn't entirely know what that meant. I had taken no prenatal testing. I was a young mom. They didn't— say anything about this at all. And he must have seen the look of confusion on my face because he said, don't worry, you don't have to do this. Like, you don't have to go out into the motherhood realm with this situation. There are institutions that are ready and willing to take this baby. He's going to need lots of things that you probably aren't prepared to do, so it's really probably better for both of you. Um, and the world is not very kind to people with this condition, so it's really better that you kind of hide him from the cruelty of this world. And I 100% believe that this— I'm sure they're no longer with us, but this 80-year-old person meant everything in a very nice way. But I also, you know, and after, of course, he must have noticed that I was like, institution? Did you just say that like out loud? It was still— it was 20 years ago, but still, it still just didn't seem real. He was like, don't worry, there are actually families that take babies too. So there are families waiting in the wings to adopt 'Your baby.' And I remember right then two things popped into my mind. One, I'm a Latina, and in my culture, uh, my grandmother would throw a shoe at me if I did anything terrible. And this would be constituted as a bad thing if I left my baby with someone else and did not come home. The second thing though is that I was a technologist. I learned all about the world of tech the first time the world was going to end. Y2K, do you remember? Yeah, I, I affectionately think about it because we saved the world. I mean, You're welcome. Like, we're here. Many of you participated. I could tell by the looks on your faces. We did a thing. People don't realize it, but like, the world was saved. And so as a result of that, I was like, if I could save the world, I feel like I could use tech to save my son. I wanted to make the world better for him, to make the world available to him. And so if you look at my LinkedIn, now that we're friends, you can go check out my career. I handcrafted my career. Like handcrafted it to work in the companies that were pushing the envelope on what technology could do. However, I never was on an accessibility team. I was never like— that wasn't my skill set. I was a builder. But what I did do was become very interested in making sure that accessibility was a thread that I wove into everything I did. And so that turned into me climbing kind of through the, the big tech companies. I was at IBM for a long time. I went to VMware when virtualization was becoming a thing. Then I end up at Amazon as a cloud architect, a principal cloud architect, doing pretty good. My boss loved me. He was like, you're a top performer, everyone loves your stuff, you're doing great. And then I get an email from Jeff Bezos. Um, it was a great moment, right? Email from Jeff Bezos. Okay, I've now found out it wasn't actually from him, it was from like his team, but I thought it was from him. And that's the most important thing. But in this email it said, we're going to build something new, we're going to build the Star Trek computer, and we need new people, new ideas to come and play. Nowhere in this email did it say you need to be an AI person. It didn't even mention AI. It was like, we need divergent thinkers to join this team. And I was like, well, that's me, I want to do this. So I decide to jump, and the first thing I do is go to my boss, who unfortunately back then you had to get permission from your boss to do anything. So I went to my boss, I'm like, hey, And you all have probably had this experience where you have a great idea and you're all excited, and then enter the dream squasher, right? In this case, my boss at this time, but it's been lots of people in my life. My mom, my economics teacher, like all these people are like, "Oh, you wanna join an AI team? So cute." No. Like he literally told me, "No, you can't do that. You're not capable of that. You're not technical enough." Fun fact, he did not say to my face— probably good that he didn't— that I wasn't technical. But he told the people who worked for me, she won't be going anywhere, she's not technical enough to do that work. I didn't find that out either until later, and I was like, oh, here's what happens sometimes when someone tells you no. And, uh, you know, this little GPT-like thing happens in your brain where it translates it instantaneously from no to let's go. So I went around him, I talked to the Amazon Alexa team. I said, I am the person, you need me, this is going to be awesome, I'm doing this. And I was allowed to join that team. The number one reason I was allowed was not related to me at all. Jeff Bezos became desperate for talent on that team because it came on the heels of a failure. Anyone remember that? It was called like the Fire Phone. It was a bit of a disaster. Beautiful tech, amazing hardware, Just not a great— didn't work out. And so a lot of people in the company were like, ooh, I don't want to be part of another not working out thing in this box in a kitchen. Not feeling that. This was like 2013. Most people were like, this isn't gonna go well. And so I was like, I actually didn't care if it went well because I realized we were gonna build tech you could talk to. And my narrow vision of what I wanted was my son to be able to just talk to tech. To be able to create bank accounts, to be able to get help with his healthcare, to be able to use technology frictionlessly. And so it didn't matter whether it was going to be commercialized. I didn't even care about that. So I joined the team. I became, uh, one of— well, we'll, we'll just, you know, skip to the end. It went very well. Um, I was the only woman on the team and the only Latina. But here's the sad news: I am still usually that on a team. And if you are a woman in the tech space, if you are a woman in the AI space, if you are a woman in the responsible AI space, you often are in a room by yourself. However, what surprised me 10 years ago was that I was also the only caregiver. I was the only parent. I was the only cat owner. Like, I was in Amazon and I was like I, I was code-switching into a dog owner. I'm like, yeah, my cats code-switch too. They're very dog-like now, as a matter of fact. But this journey, you know, I, I realized I was a divergent thinker just because of my lived experience. That when Jeff Bezos— and this time it was him— he walked into a room with us and he said, here are the 20 apps I want you to build. And all these apps were like Workout App 1 and like Abs for Days and like Biceps 2 and 3. And I don't know if you remember this phase of Jeff Bezos, like he was like, "Grrr." I think he's still a little bit like this, but he was very interested at it. It was half the skills were like workout type skills. The other half were like kitchen skills, recipe skills, like cool stuff like that. It was interesting to me because I immediately was like, "Well, I don't wanna build any of that. I would like to build something different. Of course, I'm a cat owner. I'm gonna think different." And so I raised my hand in that meeting and I said, "Hey." Jeff, can we make something else? Can we build a different thing? And he said something that I, in my mind, think about all the time, every time I start something new. He's like, absolutely. On your own time, with your own money, you can build whatever you want for Alexa. And you can build as much as you want. Like infinitely. And this was a new thing, because when you were at Amazon, you were kind of like, you bleed orange and this is all you do. But he was like, go ahead. Side hustle building Alexa skills. Where everyone around me built 1, maybe 2 skills, I built over 100. There was a moment in time when I owned 10% of all the applications running on Alexa, which was, in my mind, I was like, I did not know that until my team presented me with the 10% t-shirt, and I was like, oh, and then it was gone. Like, then somebody else had built 5 more, but it's fine. It started really well, but my skills were were skills that most of the developers I worked with, they would say to me, "Mm, I don't think that's gonna work out." One of them, for example, was mindfulness. And I remember thinking to myself, "Oh my gosh, wouldn't it be cool if I didn't need my phone to trigger a meditation or mindfulness?" Like, I thought how ironic it was that we would have a phone to do a mindfulness thing. I'm like, "Aren't we trying to get away from the phone?" So I didn't really wanna build a mobile app to do it. I wanted to use my voice. And then hear sounds of sprinkling brooks or whatever they're called. So I built that skill, and the developers that I worked with, they all looked at me and they're like, "Who needs mindfulness in the kitchen?" And I remember being like, "You don't have any kids yet. It's okay. Good for you. But one day you might thank me." And they did, by the way. Many of them came back and went, "Okay, you were on to something there." Oftentimes, your good ideas will originally be met with this like, I don't think that's going to work out. And then later they come back and go, you know what, I'm so glad you listened to my advice about doing that thing. Didn't that— I'm so glad I helped you with that. I'm like, you're right. I'm glad you helped me too. But that is the world that I lived in. And that is what brought me to a bunch of these awards that I ended up getting. But none of these awards I got for doing a thing per se. They were always for asking a question. And my question to Jeff that day, "Can I build stuff that isn't on your list?" And he said, "Absolutely. You can build whatever you want in your spare time." The nuance to that was, if you build it and people like it, you'll keep building it. If they don't like it, you will stop building it. That was his premise. And that became incredibly important because it meant we needed to use feedback to decide whether or not what we were doing was a good idea. I then turned that into a mantra. All the teams I led at Amazon, and I hired over 2,000 people during the Alexa Heyday. Um, they're, they're probably no longer working there, sadly, with the latest layoff. But hopefully we are seeing, you know, when, when we build these teams, one of the things I always tell them is 95% of our roadmap should be driven by customer feedback. 95%, which leaves a little bit of that like old school IBM, I want that feature, so build it. Um, but everything else is a customer saying Why doesn't it do this? Why is this broken? I wish this would happen. And that muscle memory that I started to build there, well, it, it worked out for me. Um, I ended up moving on from Amazon to Microsoft, but before that, Jeff Bezos introduced me to a quote. Some of you know this quote, but it's so good that I'm going to share it with you anyway. And some of you may have forgotten it. Um, but if you've never heard of it, it's written by Ralph Waldo Emerson. He is a poet laureate, American poet, metaphysician. And so I'm going to read it to you, and then I'm going to tell you why it's so important. What is success? First off, it's good. Jeff Bezos put this on Twitter back in the day, um, and he said— I— he took a picture of his fridge with this quote in it in a frame. He said, I show it to my kids every day. And so that's like inspiring, right? Good old Jeff. No matter what they say about you, um, I feel like this is good. This is good. So what is success? To laugh often and much. To win the respect of intelligent people and the affection of children. To earn the appreciation of honest critics and endure the betrayal of false friends. To appreciate the beauty, to find the best in others, to leave the world a bit better, whether by a healthy child, a garden patch, or a redeemed social condition. To know even one life has breathed easier because you have lived. This is to have succeeded. I was like, good on you, Jeff. Like, I also measure my success this way. But then I get off stage, like 5 years ago, I get off stage and an older gentleman walks up to me and he's like, you research that, like, go look it up. I'm like, I'm coachable, I'll look it up. So I go look it up, and not on page 1 of Google, not on page 2 of Google, but on page 3. Really good stuff on page 3, by the way. Um, but on page 3, way at the bottom, I find out that it wasn't Ralph Waldo Emerson that wrote this poem, it was Bessie Anderson Stanley. I was like, what nefariousness is this? Like, I literally was like, they must have been like enemies. He stole it from her. This is— I am feminist movement, like, launched. And what I found out was, is actually they shared the stage quite a bit, that they were— I don't know if they were friends, but they certainly weren't enemies. There was no hard feelings. But what ended up happening is that over time, As Bessie and Ralph shared the stage, Bessie stopped speaking and Ralph kept speaking. And then eventually Ralph passed on, and someone was like, "Hey, I love that quote. Who wrote that?" And they're like, "Oh yeah, Ralph. Ralph was the guy who said that." And then from then on, we lost Bessie. And so I became very concerned about the Bessies of the world. As a matter of fact, I used to wear a shirt that said, I, you know, like, save the Bessies. But people got it a little bit wrong, so I had to change that to like, I love AI, which is what I do now. But, but save the Bessies, I still believe it every day. I have a little Post-it next to my thing that says save the Bessies. And here's why: because when you reread this quote as Bessie, it's different. Again, gentlemen, I'm not sure how y'all feel, but if you look And you read, endure the betrayal of false friends. As a woman, I'm like, who hurt you? Bessie, I know this pain. Right? Like, it's different for me. Like, everything. And maybe Ralph Waldo Emerson is a garden patch kind of guy. Maybe. But garden patches, healthy children, I just was like, Bessie got lost. So when I think to myself, save the Bessies, I'm really talking about in the world of data science, in the world where we are collecting data, our job first and foremost is to save the Bessies, to protect the lineage of where we got that data. And here's the good news: it only takes one very simple question that should become part of your vernacular. Where did you get that? It's a very simple question. Where did that come from? You know how many executive briefings I have been in and no one ever asked the question where they got the thing that they were gonna base the rest of their year on? I'm like, where did that data come from? You know, I always tell people, especially startups, I'm like, you're one acquisition away from the Death Star. I know that's inspiring, uh, but, but what that means is that like no matter what good you do, no matter how much you try to protect the besties, if you get acquired by a company that doesn't care about the same things you do, that story is going to change. So you need to do whatever you can. This is the beauty of open source. A lot of people think it's open source versus closed source, but the reality is open source protects the besties in a closed-source architecture. It is a mutually— this of course comes from my time at Red Hat and productizing open-source software, but I've come to realize that open source has a hugely valuable relationship to proprietary software. It allows us to be transparent, it allows us to share the lineage of where we get our data, and it is critical right now. So that's the first homework, is to Really, like, if it takes a Post-it, maybe save it as a picture on your, like, iPhone or Android. But ask where the— all the goodness. Any time you're like, well, that's pretty good, you should be like, wait, where'd that come from? Because one, it might have come from somewhere you didn't expect. Two, a machine might have just made it up. And we'll talk more about that in a few minutes. So what I want to share with you now is what I had an opportunity to do when I went to Microsoft. So Amazon, I launched Alexa, 4 language models, one of them Australian English and Indian English and Irish and 5 English languages. I didn't know that was a thing until then, but lots of English languages, which I was really proud of. But I also— I also launched Spanish, Italian, French. And by the end of this, I was pretty tired. I was like, it's the same thing, new language. And it's very ironic now because with the new advancements in large language models, all that work becomes obsolete, which is— I mean, we've worked really hard and now we're like, okay, well, that was fun. Off to the next. Great. Good life lesson, I suppose. But Microsoft came to me and they're like, hey, you were just at Amazon. You made, you know, you helped make Amazon Alexa a household name. We went from no one knowing it to being in 100 million homes. Not to mention how many of you unplugged it and put it in a closet. It's fine, it's fine. I've gotten over it a little bit. Not really. Um, but there is— there was a groundswell moment where a lot of people— where I used to say, do you know Alexa? And people were like, huh? And now I say, do you know Alexa? And only like 1 out of 100 people say, huh? Um, which gives me opportunity. But Microsoft came to me and said, we want you to do the same thing for democratized AI. What we're going to call Azure Cognitive Services or Azure AI. I was like, "That sounds fun, let's go." So this is the first campaign I launched when I was there, and I want to share it with you. Today, right now, you have more power at your fingertips than entire generations that came before you. Think about that. That's what technology really is. It's possibility. It's adaptability. It's capability. But in the end, it's only a tool. What's a hammer without a person who swings it? It's not about what technology can do. It's about what you can do with it. You're the voice and it's the microphone. When you're the artist, it's the paintbrush. We are living in the future we always dreamed of. We have mixed reality that changes how we see the world and AI empowering us to change the world we see. You have more power at your fingertips than entire generations that came before you. So here's the question: What will you do with it? Every time I see that, I'm like, "Let's go!" Like, "Let's go build something! This is gonna be awesome!" It was an incredible moment because it was the least AI technical campaign we ran. This ended up in the Super Bowl that year. We were extremely proud of it. And what it allowed us to do was really the most important words that I ever hear ringing in that video is, "AI is just a tool." And I've now learned the hard way that AI is just a tool. And as with most tools, people don't understand. If you try to pick up a hammer and throw it against the wall, like, nothing's gonna happen except maybe you've hurt the wall. And that's unfortunate. And we're seeing that at scale with AI solutions. Now, when I was at Microsoft, I started off with 17 baby models, little research models. These were like 2 to 3 people on these models. And my job was to help productize them, bring them into the new world of productized AI, which from the baby, little baby model world, that was terrifying to them. So here's another moment I want to just share with you is that At that time, of course, I'm a technologist. I learned Java in order to help with saving the world in Y2K. I became very proficient. I'm a software engineer at heart. But when they asked me to chaperone these small teams that were building research models that would eventually become big teams, it wasn't really about the tech. It was a little bit about the tech, but it was much more about them needing to see a happy smiling face, even if they wouldn't smile or barely looked up from their shoes most days, I would still walk down the hallway and be like, hey, how's it going? And they'd be like, yes, Noelle, it's fine, thank you, thank you. Um, but they needed someone who would show them that this was a good thing. They needed high energy. And I will tell you, the probably the biggest skill set that I had was building code pipelines that worked. That's probably the biggest one. But after that, it was doing it in a way that was positive and exciting and, dare I say, responsible. And I'll talk to you about that in just a second. But this is where Bruiser was born. So I got all of these baby models, right, at Microsoft, 17 of them in computer vision, in OCR, handwriting detection, and, uh, natural language speech recognition, like all of these models. And I began to refer to them as baby tigers. They were like cute little baby tigers. Everyone wanted to come in and be like, oh my gosh, you're so cute. So, "Can I take a picture? Like, it's so cute. I want to be on the team. It seems fun. Let's have fun." And I was like, "Research can be fun. We're not in research anymore." So I was the sad bearer of the truth that this baby tiger was going to eventually become bigger. And in baby tiger mode, everyone thinks what you're doing is fun and exciting, and almost always it's small. It's little. It is a single thing that you're trying to do with the model. However, over time, which I have learned at Microsoft, I mean, I went with Amazon Alexa. That model went from nothing to— I mean, we're now at over 500 million homes. I can guarantee you that in the days that we started in the apps that I built, we never considered 100 million homes. We never considered 500 million homes. We didn't really consider that you would turn them off because you were scared it was listening, because that was the whole idea. Was that it would listen to you. Um, like, it was very shocking to the data side. They're like, wait, what? Why would they turn it off? Because it's listening. But they signed the privacy agreement. They didn't understand. I don't understand. Um, so this process of baby tiger mode, everyone's in it. You're all in it now. Like, ChatGPT is a baby tiger. It's super cute. We can do cute things with it. The goal of a language model, large language model, its job is to give you an answer and you'd be like, "Well, that's not bad. It's kinda cute. Kinda like it. I wanna share it with people." It's not to evoke this challenging concern about what will happen tomorrow. So I began to characterize this as the difference between day 1, baby tiger mode, and day 2, which is, well, big tiger mode. And what I was shocked to find out is that no one in baby tiger mode ever looked at that tiger and went, Hey, I'm so cute. Um, look at those paws. How big are you going to be? Or razor-sharp teeth at birth. What do you eat? Where are you going to live? How long will you live? And what if I don't want you anymore? None of those things came up. I gained a skill during COVID adopting cats. I became a rescuer. I don't know if you know this, if you've ever rescued an animal, they like interview you. To rescue a homeless animal. Yeah, a homeless animal in a cardboard box. They're like, "Can I ask you a few questions first?" I'm like, "Sure." And they're like, "What are you gonna do with it? Where is it gonna live? What are you gonna feed it? Are you gonna declaw it?" Oh, and my favorite one, in the last 2 cats we adopted— last ones we adopted, um, I'm saying that for my husband in case he doesn't believe me, uh, but the last 2 that we adopted they asked to do a home visit. They wanted to come to our house for a homeless cat. And then they came in two, so they were like, "Oh no, see how they're hugging? They can't be separated." What? I didn't know they could do that. Anyway, so now we have— we had four. But it's all good though. I love them. But we were in like, we were in this moment and I was shocked to find out that when I Go help a company adopt a baby tiger, adopt a little new model. No one asks any questions at all. We don't do home visits. We don't say, "Hey, how big are you gonna be? Like, are you gonna hurt anybody?" One thing I always say— I'm gonna see if I can get this guy to— there we go. One thing I often will say is, you know, baby tigers become big tigers, and big tigers kill people, right? Right, they kill people. Like, these models that we're starting to use in these very small ways, almost insignificant use cases, become extremely unwieldy if you don't ask a few really important questions. So I created a framework. I ended up spending all of 2023 going all over the world. Many of you have seen the craziness that became my life over the last 2 years. Um, I spent all of 2023 and 2024 flying around and talking to brands, talking to executives. Um, I created a company actually 10 years ago called the AI Leadership Institute, and I did that because I was so worried executives were going to invest in AI and not know what they were doing. So I was like, let me just start this side hustle and I will just go pro bono and talk to these companies and make sure that they have a third party being like, Do you even know what you're doing with this? And that was before AI was cool. That, that was vegetables to kids. Like, I was in there being like, you really need to know how to do this. Now it's like candy. It's probably worse than candy. Everybody is jumping into the world of artificial intelligence and they're not asking any questions really at all. And so I, uh, got an opportunity to work on— I mean, I did probably 300 executive conversations. Those led to hundreds of, um, of whiteboard sessions with technical leadership. Those led to hundreds of thousands of employees getting trained on how to use AI. So I now kind of term it going from the boardroom to the whiteboard to the keyboard. I also care a lot about our education system, so I'll throw in the chalkboard. Like, this process of literacy, of understanding, of questioning, is an incredibly critical time. So what I did, of course, I went and talked to all these companies. I got them to say yes because I'm excited. I'm like, oh my gosh, this is so good for you, you should do this. They do it, and some of them were successful, and some of them were very successful. So, uh, IDC, a research company, a study— they did a study on about 170 different projects. 40 of them were projects that I was a part of. Now, I'm a Microsoft MVP, which means they treat me better than I ever got treated as an employee. Um, so if you, if you want to know more about that, come talk to me. Uh, but I was an MVP, which meant they introduced me to technology very early. They came to me and they said, uh, during COVID they were like, hey, we've got this new model, who wants to test it? These emails, they like come into your world, like, try to pay attention. Because I was like, I should do that. I like playing around with models. Well, that model ended up becoming ChatGPT a year later, and I'm now sitting with all this knowledge about ChatGPT. I'm like, this is great, we should sell this. And I went to my boss and was like, we should sell this. Anyone want to guess what he said? Yeah, no, you can't sell this stuff. Like, who sells generative AI? So I was like, oh, you're right. Can I try anyway? Like, what's the harm having a brown girl go try some stuff, right? So I go and I start talking to people and I start showing them what I had learned over the last year at Microsoft. And what we found out was that there were a certain number of use cases that were low risk, high value, and high return on investment. And I'm gonna share with you what those are. But the benefit was in the span of about 9 months, we generated $300 million. We did so well, in fact, that the CEO was like, what's happening over there? And then that faithful person came and was like, oh my gosh, I'm so glad I supported you in this business. I'm like, I'm so glad you did too. Okay, bye. So I, I left that organization and, and dove into building the AI Leadership Institute because it's time. Like, it's a great time. To be an educator, to be someone who's willing to advocate for proper use of this technology. But the good news is, is I've actually learned from 222 or 223 projects that I've done now in AI. I've learned exactly what it means to be successful. So I'm going to share these secrets with you. I framed it as Lead AI. This is not fancy. I have these principles. They've been my principles for about 10 years, but I was like, GPT, is there any way you can make this memorable? And it came up with this. I was like, awesome. So the— do that, right? How can you help people know what you know? Create little memorable things and make it easy for them to be reminded of the important things that took you a decade to come up with. So Lead AI is just a mechanism for framing what I've seen repeated over and over in the most successful AI projects in the world. The first one is having good leadership, right? So what this means— I have a, a post on Instagram, if you follow me, noelle.ai, you'll see my very first pinned post is Virtues of a Mindful Leader. And what it means is that when you are leading AI projects, you actually have to be very intentional. It's not I mean, I don't know. I've been led by lots of people. I can count on, well, a finger how many leaders were really good at it. And not because they didn't— I don't know why. I mean, I do know why. I don't think they really understood. I don't think anyone taught them. I think being a mom of 6, I've learned a lot about being a leader and saying things in different ways to get different kids to respond in different ways, but all of them doing what I'm asking them to. Like, I've learned that skill, but leadership is a lot about influence. It's a lot about holding an ethical bar. It's a lot about holding an operational excellence bar. It's, it's really about holding the bar and encouraging, inspiring your people to rise to that bar. And what I found is that in my teams, people were like playing to the lowest common denominator, and that's super frustrating for someone who's ambitious. So leadership became so important that I even created a podcast called The Lamplighter's Leadership. I don't know if you know lamplighters. Back in the day, 1800s, we used to have like actual lampposts with gas in them on the street, and there would be people, men probably, that would climb up and light them. And what I loved about the concept of a lamplighter is that they would go up and light a lamp, and if you were watching them from your window, you'd be like, oh, there's Bob doing his lamp lighting. Thanks, Bob. But as Bob would light the lamps, you would still see the light turn on on the lamps he was lighting. But eventually you couldn't see Bob at all. And I loved that idea of leadership, that our job, especially in the world of AI, when we're one acquisition away from the Death Star, like that we let our light be so consistent, so bright, that even if you can't see me, you still see the thing that I am trying to shine light on. And that, as a leader, becomes a philosophy that I think many of us have to really just take a little bit of time to understand. Most of us will unders— again, I found out this interesting fact: caregivers, parents, um, and pet owners are actually some of the best prompt engineers in the world. And what the reason for that It's a study. The reason for that is because you're patient and willing to think about things in more than one way to get the outcome that you desire. But you're patient. Talking to machines requires a lot of patience, just like talking to children and teenagers and well-grown adults, all of humans. And patience is something that I find we are growing less and less interested in cultivating as a virtue. But leadership really means how do we make sure that we identify that it's not a technical solution all the time. Sometimes, actually all the time, it's a people problem, and we'll talk more about that. The second thing that I found out: every successful project had a person on the team, not necessarily the leader, but someone who is championing ethics and ethical foundations. What this means is that you understand the value of building transparency into every application. If you build something and a stranger walks by, they should be able to understand what you did. They don't need to see your code. They don't need to see the bits and bytes, but they do need to understand why you did the thing so that when you are acquired, there's an instruction manual for how it's supposed to be used successfully. If you don't do that, people can just change it. They can do what Ralph Waldo Emerson did. And if you go back and you look at Bessie's original poem, it is different. The essence is the same, but it's different now because he was like, well, I can't say it like that, I'm Ralph Waldo Emerson, I'm going to change it up a little bit. And that's what happens over time. Unless you provide instructions, it's much harder for someone to go, that's not supposed to be that way, if, if you wrote how it's supposed to be. And that is part of the framework. So, so building ethical foundations means that as a thread that weaves through everything you do. You're transparent, you're explainable, you think about fairness and trust. These are core values that are instilled in every project. I know you probably think like, I don't know, intuitively I thought every project was that way. Turns out none of them are like that. None of them. Like, I go in and I ask them, hey, Baby Tiger, so like, what did they tell you as you grow up? Any rules at all? No, like, let's just have a good time. Um, it's cute and fun. We don't need rules right now. No one's going to use this thing. It's the number one thing I hear in models that grow very big. No one's going to use it. We don't have to do any of that stuff. Like, it's— we have 100 invocations right now. 17 research models at Microsoft all had less than 100 calls to them per month. Those same models today are getting 5 to 7 billion requests per month. Billion, like with a B. This is concerning because at the beginning everyone was like, we don't really have to worry about it. It's just this little research model and it's now serving us like multiple small countries, lots and lots of people. There is no such thing as personas in the world of AI. If you're a marketer, like personas, they're cute, they're baby tigers, but to grow you have to think of everyone. That kind of leads me to the next one. So I always like to talk about strategy. I'm not going to spend a lot of time here because I, I just want to let you know, if you have a strategy, that's great. Revisit it in the light of the things that we've talked about today. But if you don't have an AI strategy, you do have an AI strategy. It's to have no strategy. None. So you are— anyone remember Blockbuster? Yeah, yeah. Or, or Netflix? Like, one is not here, the other one is here. And so we know how the story goes, right? So you are now in a blockbuster moment. You are the, the executives at that table going, I'm sure it's fine, I'm sure we don't have to like change anything, I'm sure it'll be great. Oh, it wasn't great. And it's fine, like I said, maybe they wanted to retire, maybe they didn't want those people to work there. Like, who am I to judge? But strategy is That's the voice in the room that says we're going to think differently because the future is changing. It's about the future. It's forward-looking. Someone should be thinking about this. And again, you have to spend a lot of time there, but having no forward-looking strategy on how AI is going to impact your business, it's a baby tiger move. Like, you're gonna get hurt by that. Um, another one— so this is the D part, L-E-A-D. This is probably my favorite, design with inclusivity. Now this might surprise you, but in my experience looking at, I don't know, 200 very successful AI projects, many of them billions of inferences a month, when I look at those projects, the reason they were successful is because they were inclusive. Not at the beginning, sadly, but by the end. It's way harder to build in inclusivity at the end. I have a phrase that I use from the accessibility world Never build for someone without them, right? Never build for us without us. I have a son with Down syndrome. When I was younger and he was younger, I was like, I can build all this stuff for you. I didn't realize till much later that he didn't want any of that stuff. The things he wanted, he couldn't articulate yet, but I was way off as a mom just trying to help my son. I had no idea what he needed. I had no idea what would be valuable. So you have to really think about How are you going to embrace the people you're building for? And in a world where AI meets billions of people, you're going to have to include a lot of them. So the solution to this is a term known as AI red teaming, borrowed from cyber. We got some cyber friends in the house, um, borrowed from cyber but not cyber at all. It's a little bit— cyber has this concept of, uh, attacks or building a red team to do what's called adversarial attacks. And that's important. You want someone to look at your model and just ask it all these hard questions and see if it'll do something bad. You wanna make sure you try to provoke the model to do bad things. 90% of the errors that we found on Amazon Alexa were actually benign attacks. There were users that had no idea that they were actually doing something bad and that they uncovered like, oh, I didn't know that would happen. And then they're like, what else can I do? It is not adversarial. It's not a person in a hoodie in their garage trying to like knock you out. Those people do exist, sadly, but that's not our biggest concern in many of these models. Our biggest concern is accidental, is benign, is infringing typically on a model's— just something we didn't think about. Data scientists typically are the ones who build a model. Most of the time when we go to test that model, the data scientists test them too. And almost always, and I do a lot of these, they'll look and be like, "That's pretty good." And then their friend will be like, "Yeah, it is. Good job, Bob. Good job, Steve. Noelle, what are you doing here?" Right? Like, this is the mode everyone agrees with each other, and it seems good until you launch it into production. Fun fact about Amazon Alexa, We launch Alexa, 2014. We start getting users. We're like, "Huh, I wonder why we're getting these weird reviews, bad reviews. People are like, 'It doesn't work.'" Of course it works. We wouldn't have launched it if it didn't work. So we go look and we decide to do our product research on Reddit. Very good platform, by the way, for this type of research. I'd encourage you not to get your product feedback on Reddit. I'd encourage you to do that, I don't know, before you launch, but we decided to wait till after. And we'd watch people be like, wow, uh, do you— do you— did anyone see that Alexa doesn't know the height of this mountain, which it totally should have known? We were like, huh. And then now it knows it. Another fun fact we heard on Reddit, and this is now verifiable with data: how many of you may have noticed that women yell at Alexa at a much higher rate than men. It may be, maybe not for you, but definitely for you if you're a woman. Um, your tendency— some of you aren't yellers, and I get that, but I might be. And on occasion, I will be talking to Alexa, and I will be asking it to play a song from Hamilton, which I will be seeing in this beautiful city. Um, right, I can't wait. Um, so I'm, I'm, I'm asking it to play— we do ice baths There's a 4-minute song in Hamilton I play every single time to do an ice bath because it's hard and I need that song. And so I'm like, please play this song, and it won't play it. And then he walks by like this, Alexa, would you play the song? And it like instantly starts playing. And I'm like, what is happening? I made you, I created you. And what I found out later was that the model is actually quite distinctly slanted towards Middle America, male voices. And that's intentional because we asked our demographics available at Amazon Alexa in the first year of its birth. We said, hey, everybody in the company, please come and test this model so we can see how it's working. Yeah, and we asked them and they showed up. The problem was we did it in the morning, in the afternoon, and in the evening. We're like, anytime you have, except, you know, Jeff Bezos. Except working hours. Like, we're not going to pay you to test this thing, but if you want to come in your spare time, please come and test it. The only people that showed up— well, it wasn't women who showed up, and there weren't very many women anyway, but those women were busy and they did not decide to show up. We didn't know that that was going to impact our model until 2 years later when Reddit was like, why do I have to yell at Alexa? Why isn't Alexa listening to me? So I'd encourage you, like I said, like if you design for inclusivity, if we— I mean, that's pretty easy. Like, I don't know, gender, like do that. Um, another one, ethnicity. So I was, uh, I'm a bit of a nerd. So I'm at Amazon early, maybe year 2, and we get a message from an NFL player. We don't— Jeff Bezos gets this message. His name's Marshawn Lynch. If you don't know him, he's like Anyway, he's into Skittles and betting on sports now, but he was a really big NFL player in Seattle, so he had everyone's attention. So the leader of our team comes in and is like, "Who knows Marshawn Lynch?" And I, of course, do not, 'cause I don't watch football, or didn't. I do now, I guess, but I didn't at the time. So I'm just like hammering away at whatever I'm doing. They're like, "Noelle, you're going to Oakland." Oakland, California. In Oakland, California, I didn't know this, But they talk different there, apparently. So I get there, Marshawn Lynch, big football player. I walk in, and maybe you all would know this, but I didn't. He said, hey, welcome, would you like some Hennessy? And I didn't know what Hennessy was. Do you know what Hennessy is? I didn't know. But, uh, it was 10:00 in the morning, and I texted my husband. I'm like, so he's like, say no. I was like, okay, but I had no idea. Anyway, it was So that was the moment, and I'm like, okay, Marshawn, like, let's get this party started. Talk to your Alexa device. You say it's not working. What isn't working? He's like, oh, I call it, it doesn't answer. I'm like, great, call it. And he's like, oh, okay, Alexis. And I'm like, well, there you go. So I'm like, oh, no problem, Marshawn. It's Alexa, like with an A at the end. He's like, oh, okay, Alexis. And I was like, huh, okay, so it's with an A. Ah, and he's like, that's what I said. And I was like, absolutely, thank you. And I left. And I go back, and this is the first time I realized how real AI is made. We— I went back, I got 3 data scientists into a room, and we recoded the wake word for Alexis. And now, see, you hear me? Alexis, Alexa, uh, Alexi. Like, there's a whole bunch of variances we added to accommodate this, and he— it's just because he's from Oakland and they just— that's how they said it there. It turns out though, people over the age of 85 also say Alexis. So we accidentally, by fixing it for this random NFL player, served a whole bunch of people that were very grateful. As a matter of fact, there's a Saturday Night Live skit called Alexis Silver, all about our debacle. Um, but it's really funny. You got— you should definitely check it out. But designing with inclusivity, like, I just hope that we do this Intentionally, it's like Blockbuster and Netflix. We know how this works when you don't do it. The teams that are the most successful design with inclusivity from the start. And what that means is that you build a team of people, not technologists, not data scientists, but people that know the customer, that know the partner ecosystem, that know the people who are going to use it, and they empathetically start asking questions as if they are that demographic, as many as possible for as long as possible until you absolutely have to go to production. Now, LEAD AI is an acronym. We just went through leadership, we went through AI strategy, right? We went through the ability for us to design with inclusivity, and the last two are AI and one is governance, because I can't— you know, you got to make some rules. Baby tiger, they, they really want a fence so that they don't run away and hurt themselves. And when they get bigger, they'd really prefer a much more luxurious area to play, but they still want to not hurt anyone, and they also would like to enjoy and roam. And models are very, very similar in this way. They want to be able to grow, but not haphazardly. They want to be grounded in the truth. They want to be grounded in what's called guardrails. The most successful AI projects in the world are grounded. They know what the truth is. They understand humanity. By the way, every single model when it's born has no idea about truth or humanity or bias or fairness or any of these things we say are important. The only one who's going to educate a model on those things, guess what, is you. You the human, me the developer, but most importantly, you the user. I don't know if you've seen a model. When you use ChatGPT or any of these models, there's always this thumbs up, thumbs down thing. Have you seen this before? So if you see that, two things have to happen. One, in order for a model to get better, you have to tell it what it did was wrong. Just like a kid, just like a human, really. You have to tell it it's wrong, because if you don't say it's wrong, the model's like, well, I guess that was pretty good, I'll keep doing that. And it's funny because at scale We don't realize the impact of non— like, not stating the problem ends up creating massive hallucinations at scale because the model thinks it's doing a good thing. It's making a good choice. You didn't tell me I wasn't making a good choice, so I'm going to keep doing that. And when we're— I mean, you saw ChatGPT's massive growth. No code, no limits. People do not need to know any special skills to start talking to this model. And their first inclination when it does something wrong is to turn it off. And it's the worst way to train an AI model. It'll actually lead to a disaster, which we're watching unfold on Twitter every day with the OpenAI leadership team. Um, but that leads to finally, right, so we've got all these ideas, we've got good leadership, they're doing things intentionally, we've got the ability for us to design with as many inclusive perspectives as possible, we've got a strategy that identifies our forward-thinking forward-moving motion, the most important thing to figure out is what's the first thing that you do now. So this is what I do in these boardrooms all over the world. They'll bring me in and I'm like, great, you're all excited, let's go. And they're like, well, well, now what do we do? What's the thing we build? And I created kind of a rubric for figuring that out that's been pretty successful. It starts, surprisingly maybe, with your core values as a company. Now you might be— I don't want to make you sad, but when I go into big companies, top 100 brands in the world, and I ask them like, what's your core values, they're not immediately accessible to these people. Like, they're not like, oh yeah, we live and breathe by integrity and blah blah blah. Like, they're like, oh, Mary from marketing, she just put that page up, so let's call her in here. I'm like, yeah, we should have Mary in here for sure. But these are These are like the things you're gonna keep— you're like, it's gonna keep you up at night if you don't do it. Like, I remember, um, OpenAI— it took 7 years for OpenAI to become usable by the public. 7 years of going into the office and it being broken and not working and people hitting their head against the wall, people quitting. We didn't know it then because it wasn't on social media, but there was a lot of turnover and frustration. And then it worked. I don't know if you know this, But Lin-Manuel Miranda, it took him 7 years to get Hamilton off the ground. Same thing, same thing. Frustrated. People were like, really? You're going to make a bunch of white guys be exciting on Broadway? That's not going to happen. Like nonstop, this isn't going to work. It's not going to work. It's failing. This is broken. You can't find the right talent. You're not going to be able to find a bunch of brown people to do this. Like all of this. No, no, no, no. All this. And this is again consistent with successful AI projects. You will be met with a bunch of No, no, no, no, no. So my secret to kind of getting around that is to pick a use case that drives to yes. And what that means is that first, it's aligned with the core values of the company, so everyone's like, ooh, that feels good, it's valuable, we love that, we all agree on that. The second thing is risk. How do I make sure that it's not going to hurt my brand? And then finally, build what I call a minimum remarkable product. Pick one use case very small that leads to more, but it starts small and it moves the needle. It generates revenue. Could save costs, but I like the revenue better. You can generate infinitely more revenue. There's only so much cost you can cut before you cut into the muscle of your company. And so I realized that once companies knew what to build, we had infinite number of tools and applications. New models get released every day. We have so much power at our fingertips, more than any generation that's come before us. Just like Common said, more than any generation, we can build. I literally could go and whiteboard out an application, just say, here's the browser, here's the backend database, here's how I want you to save it, here's the firewall rule I want you to create, take a picture of it, feed it to a generative model, and it will write the code for me. Right now, 45% of all code is being written by buy machines gratefully because it's the code humans don't want to write. So don't get it twisted and think like, oh my gosh, look at all the jobs. No, we don't have enough humans coding right now. We are down by 4 million developers. We are in the hole. Most people doing this work are like, please help me, I don't like this job very much. I used to love coding, right? Don't you? Oh, like when you used to love something and then it becomes so awful that you're like, now I hate it. Like peanut butter and jelly. Like, you see it all the time. Now I'm like, can't stand it. But this is the world we're in where AI now intentionally used— MRP, Minimum Remarkable Product. If we use a minimum remarkable product idea, we now get to build AI systems that free up the human to do the thing they love to do: ideate, create, establish, like build authentic expressions of whatever it is they're trying to build. Take away the nonsense. And let's face it, there's a lot of nonsense. But with all these tools, I just want to check to see if we're all friends and family. Let's see, let's see how it goes. Have you heard of this one? Let's see if you know it. With great power comes— yes, we are friends! Yes, Uncle Ben, or Voltaire for those of you in the back, um, either one provided us this really good guidance that anytime we are given a huge amount of capability, that it comes directly correlated to our responsibility to use that wisely. And in the world of baby tigers, may you remember it forever, that in a world where we are adopting little models going, oh, that's pretty cool, I didn't know I could do that, oh, that's neat, that we instantly think about the responsibility we have to make sure that as we build these cute little use cases, that those little things will become big things, and those big things, if not cared for, could actually hurt people. Could not serve them, could take away the very credit we were trying to give them, could keep them from getting the apartment that we actually built the application to help them get. These are real sad stories. I call them AI gone wrong. Like, these are real production challenges that I've watched happen in healthcare, in finance, and they impact— not— I always say, like, the rising tide of AI, it doesn't just affect the yachts. Like the Accentures of the world, the, the Deloits, the PwCs. It affects all the boats and not in a positive way unless we're very, very diligent about it. So I didn't want to leave without giving you my, my promised golden list. These are the use cases we found out over 2 years generate for every dollar you put into a use case built responsibly in these buckets. You are— can now rely on data to guarantee— I won't say guarantee, maybe to assure— if you do it correctly, you can get $3 back. A dollar in, $3 back. So people ask me, how did you make $300 million in 9 months? That's crazy. Well, I kind of cheated. I did a bunch of these projects and I watched the ones that generated the highest ROI. This slide identifies those use cases. For every dollar we put in, we got $8 back. $8. So it's easier, if you think about it, for me to generate massive revenue for an organization if I know the cheat code to— if I build these things, they will give me much higher ROI. This is not just me, this is actually based on a study done with the IDC. They've actually documented all of the use cases, so you can dive deep if some of you are like, well, that's pretty cool, I want to know more about that. That is not a secret, it's publicly available. Um, but in order to do it, I do have some things that every company has done with me. So AI Leadership Institute, these are the types of things that we do with people. Building policy and strategy. Again, if you don't have one, you have one, and it's probably really dangerous. So I'd encourage you to at least get this under your belt. The easiest policy to create is to build an internal company GPT and tell all your employees only use that one. Fun fact, I was just at a company yesterday and they did not, um, they did not— 33% of people were not using the internal GPT. And I was like, well, why is that? And they're like, well, it's not very good. And I was like, well, yeah, like, of course it's not good, you're not using it. Like, how is it going to get better? It's like a little kid. It's like saying, wow, little kid, you just, you know, cut all the heads off the roses of our neighbors, huh? You know, like, and just walking away. Like, you have to tell it, like, that's Not a good idea. Here's why. These models are meant to be trained, and they are not meant to be trained by data scientists. They're meant to be trained by users, by you. So now you are— you now have a new lease on life with the up and the down and the thumbs. And I'll do more of that because these models need it. The other side of that coin, though, is that companies actually need to build teams that accept that feedback, that do something with that. So that's one of the questions. Remember earlier I was like, you know, you got to ask a couple questions. That's the question I always ask, is like, what do you do with your reinforcement learning feedback? And they're like, our what? I'm like, you know, the thumbs up and thumbs down thing on the app I just tested in your company. And they're like, huh, we should, we should look into that. Yeah, we should, you know. So leadership essentials, we got to teach people that there is a— this is a science, it's not art. The art comes from the human side. This is a science. We know what works. We know what fails. The challenge we have is literacy, is that people just don't know, and there aren't enough people with experience in production AI telling everyone like, hey, there's, there's rules here, there's things we can do. AI solutions architecture— I believe all of you are now knighted AI solutions architects, right? You now understand some of the challenges and some of the, the concerns, but AI solutions architecture means that you think about every problem with a lens of AI as a tool in my toolbox that I can use to solve this. Not, I have AI in my hand and it's a hammer and everything's a nail. Whack, whack, whack, whack, whack. Like, it's just a tool and it might work and it might not be appropriate, but you should know it and you should use it and you should always be thinking, how do I solve this problem differently than I would have solved it 20 years ago? Because the answers are different now. Even if AI is not part of it. I think it'd be hard to believe that it wouldn't be, but even if it's not part of it at all, the answer will be different than if you ask this question even 5 years ago. Finally, my belief is that all of us should learn AI by building AI. If you've never built an AI model yet, you should definitely do this. I run a workshop, it's called the Bot Builder Workshop— we— Bot Builder Boot Camp, if I want to be all alliterative. Um, but the benefit of it is that in 4 hours you learn to build a model end-to-end and a responsible AI model. At the end of the day, it's really prompt engineering at scale. How do you become a prompt engineer? Every one of you is a prompt engineer if you've ever typed words into a model before, like a ChatGPT, a Copilot, a Gemini. It's not an accident, like, it's intentional. It's a science. You do have to know how to talk to a machine. And the sooner you understand guardrails and safety systems and cyber attacks, the better you will be and the more comfortable I'll sleep at night knowing that I've pushed you out of the world with this information. So big launch, I just like 3 days ago found out that my book, it's not done yet. I mean, it's in— I did my part, someone's editing it, but it did go up on Amazon and I was on a webinar today. And I actually found out it's also at Target, which means I've arrived. Um, but this— thank you. But this is the book. And if you want to join me in my community, we do calls all the time. I do them in Australian-friendly time, so at 8 PM Eastern, which means this around 11 AM, uh, during the week and on the weekends. My goal is that you just know what I know, that my network becomes your network, that I'm so grateful that you listened to this amazing session. It's been very fun. The last thing I'm going to do with you all today is, of course, invite you to be friends. Noelle Russell, very simple to remember, but also take a selfie. Are you up for a selfie? All right. It's like my thing. Some of you know, if you want, it's your chance. You can like scooch in because I won't be able to— maybe I'll do a video, but I'm going to take a selfie from this stage. So that I can remember, because one of you is going to do really cool things. I'm going to be like, I taught them everything I know. Look, I took, I took the picture. All right, let's get this party started. All right, are you ready? Say cheese. Oh wait, say Mickey. No, I'm just kidding. Thanks, everybody.