Aaron Gustafson from Microsoft discusses AI applications for accessibility, including custom neural voices for people with speech loss like ALS, AI tools integrated into software development, and grant-funded projects like AI-driven prosthetic design and AAC communication apps. The conversation examines responsible AI considerations around energy, bias in training data, and the need for inclusive datasets.
Chapters
Good morning, afternoon, evening, whatever it might be for folks joining us. Thank you so much. My name is Jeremy Osborne. I am the Director of Learning for Aquent Gymnasium. I'm here with my colleague, Andrew Miller, who is the Program Director of Gymnasium. And we are joined by a guest, Aaron Gustafson, and we'll get into a little bit more about what he does. In just a moment. But we're really, really grateful to have everyone join us. We're going to be talking a little bit about the role of AI in accessibility. And this is one of our— how many live streams have we done, Andrew, exactly?
Oh, wow, that's a really good question. More than a few.
Yep, that's the correct answer. We have done more than a few.
Yeah, we're data-driven.
Yeah, and a lot of the stuff that we have talked about in the past has been— I mean, we've covered a lot of subjects, but of course AI has been on a lot of people's mind and radar. And I think we have the pleasure today of talking to Aaron about some of the aspects of AI that maybe we don't think about on a day-to-day basis when we're seeing like cool goofy images populating our Today, we're going to discuss some of the ways that AI is making a meaningful difference for people with disabilities or impairments. And that's going to range across the board from things such as generating alt text for images, voice preservation and recognition, text transformation, all sorts of things. Aaron, first of all, welcome to our, our little livestream here. Thank you.
Cozy little livestream.
Yep, cozy.
Thank you so much. Thanks for having me. Yeah, it's like a, uh, it's like a return to port for me because, uh, let's see, in 2000 I think I worked through Aquent for a little while.
So yeah, that's right. And not only that, we should mention— that's, that's true, I remember we discussed that. Um, that's almost a quarter century ago, just saying.
Isn't that wild?
I've, I've earned all this gray.
And we should also mention that, uh, you know, we run a free online course platform called Gymnasium. That's kind of what we do in addition to these live streams. Aaron also is one of our first instructors, and we can talk a little bit about your background, but you kind of started like in the web development and web design back in the day, and you did our course Modern Web Design, which is—
It's one of our most popular classes. That is still one of our most popular classes.
Almost, almost 10 years old now. I think we're probably due for an update soon.
Yeah, we are. We, we did— we tried to make that as evergreen as possible, but, uh, I don't know, I think you did a pretty good job of—
Yeah, it's— I intended for it to be pretty timeless, uh, more philosophical than, uh, than like, here's the cutting-edge things. Yeah.
And so I don't know if I mentioned, uh, at the beginning, again, you're, you're official role is— make sure I'm going to get this right— Principal Accessibility Innovation Strategist at Microsoft. Is that correct?
Yep, yep, that's correct. I'm, I'm, I think officially in terms of career track, I'm a Technical Program Manager, uh, but yeah, principal level, and I, I get to focus on accessibility innovation both internally and externally to Microsoft, which is a lot of fun.
Yeah, and I'm kind of curious if you can expand on a little bit. Like, I know that, um, again, so like Your core skills kind of started off as like web development, I guess it's fair to say. But, um, yeah, we're gonna see that if captions can— we had a question from our LinkedIn users.
They should be.
They should be, but Roman maybe behind the scenes can track that down. Uh, it would be good to get captions enabled here. So yeah, but so your day-to-day, I think when we had a call a couple of weeks ago, I was kind of curious, like, what in general do you end up doing and some of the things that you are responsible for having to do with all things AI and accessible?
I seem to remember when we asked you, like, so you're involved in a lot of things. So what is your day-to-day? And I think that was literally the phrase. And when you were done with that inventory of things, it sounded like more things that would fit into 24 hours. So, you might be the busiest person. You probably are, but you do have— you're involved in a lot of very important work that's going on in and around the Microsoft sphere. So, yeah, definitely tell us all the things.
All the things. So, my role's kind of a unique one in that I get to work on a lot of just really amazing and impactful projects. Projects. I have been working a lot over the last year and a half with Team Gleason, which is an ALS support organization, to investigate creation of synthetic voices. So a lot of times when we think about AI-created voices, we think about deepfakes. There's been a lot of obviously news about that. But what we're looking at is actually how can people consensually create a copy of their voice for using when they no longer have use of their natural voice. So, you know, with ALS, eventually a lot of people are robbed of their voice as they lose control of their vocal cords. They get dysarthria. It becomes harder to understand them. And so they need to rely on AAC software, which stands for augmentative and alternative communication. So if you see symbol boards that people can, you know, compose sentences on and then it gets read out for them. So how can we bring a a natural-sounding voice clone to that experience so you can continue to show up as yourself and to really continue to have your personality, to have your loved ones continue to be able to hear you as opposed to hearing you sound like a computerized voice. We all were kind of used to hearing Stephen Hawking and stuff like that. Eventually that became his voice, right? But for most of us, our family kind of grows up around us. Our kids learn to hear us how we are. And if all of a sudden something happens, like you get ALS or throat cancer, we've We've seen that happen with Val Kilmer. Lots of things can happen that can rob you of the use of your voice, even in the short term. I mean, if all of a sudden you have laryngitis and all of a sudden you can't speak, how can we enable you to continue to communicate as the authentic you in that way? So that's been something I've been exploring with Team Gleason. We ran a pilot with them to provide high-quality synthetic voices using some of the Microsoft technology for this. And the reaction to that was tremendous. You know, it was really impactful for folks. I think you've got a video that you want to show in a minute. But the work was really interesting. It was a lot of fun, just amazing to see, you know, how people respond to it. Also to think through the implications. You know, what does this mean in terms of, you know, risk to people's voices being stolen?
Mm-hmm.
Or, you know, how do we ensure that, you know, we do have you know, the actual consent for somebody that they're using their actual voice recordings, all these sort of thorny issues. So that's, that's one, one area that I'm working in. I'm also spending a lot of time thinking about sort of the, the end-to-end picture of accessibility tooling. So, you know, I come from a web design background, as you mentioned. So I've been working on the web since '96, and I've had a lot of, a lot of experiences across the board in projects. from planning stages all the way through to backend development and stuff like that. And the, you know, all along the line, the, the truism about the cost of, of, you know, making changes the further you go into the product holds true, and that holds true for accessibility as well. So it's much more expensive to fix an accessibility bug in production than it would have been to catch it in planning or design or when you were coding. And so I'm looking at, you know, what is the, the landscape of, you know, the software development lifecycle, what are the, the touch points in there, and how can we as Microsoft improve on the experience our developers are having internally? What could we potentially bring external? How do Copilots like what we see in VS Code, the GitHub Copilot and stuff like that, how do those factor in? What is the quality of the code in terms of accessibility that they're producing? How can we improve that? That's another aspect of what I'm doing. A third bucket is I work I'm on the advisory board for the Speech Accessibility Project, which is a consortium of different companies who were interested in gathering affected speech samples from a variety of— from people with a variety of different disabilities. So the initial push was to work with people with ALS, with cerebral palsy, Down syndrome, stroke, and Parkinson's and to gather speech data from them in order to train our speech recognition models to be better. There have already been published results of improvements in that space, both from Microsoft and from Apple as well. Those are going to find their way into your daily products that you interact with. There's already been improvements to the Azure Speech Recognition, There will be improvements coming to other products as well. As I mentioned, Apple's been working on that. Amazon and Google and Meta are also part of that consortium. They're doing a lot of interesting stuff as well. That's another piece. There's a bunch of other things too, but those are 3 tentpole pieces that I'm working on currently.
Yeah, that seems like a lot for everyone. Actually, interestingly enough, I mean, we're going to show the video about the Team Gleason stuff. But before we do that, again, just from a high level, because we have folks who, again, they join us, we've talked about some of the processes behind AI and so forth. But so it sounds like there's like a large training component behind some of these things. And so for example, for the Team Gleason, is it? Fair to say that there's a training process that takes time where you have the ability to record an individual's voice and then, like, I'm just curious, like, how does that work?
Yeah. So in terms of the synthetic voice creation, the process varies based on the type of model you want to work with. Microsoft has several different models and every company's models are a little bit different in terms of how much data they need to work with. But the 3 models that we have, there's the personal voice, which is almost a— some people refer to it as a zero-shot. I think of it more as a one-shot. You record a statement giving permission to create a synthetic version of your voice and then it creates a pretty good quality but somewhat lower quality voice, and that's all it needed was that one statement.
Right.
Some only need— the Custom Neural Voice is the umbrella that a lot of the Microsoft voices fall under. The Custom Neural Voice Lite model takes around 3 minutes of audio to create a pretty good quality voice. Then there's CNV Pro, which can take anywhere from 20 minutes to several hours. That's the one that tends to get used for at least traditionally tends to get used for things like professional voiceovers and stuff like that, where let's say you need to get high-quality samples, you want to generate more emotionality in the voice, you need to be able to re-record some dialogue or something like that, or some overdub, or you're using, you know, you're creating a brand voice and you're hiring a voice actor to create a brand voice and you need to be able to use that without having to call that person into the studio. That's where CNV Pro tends to come in. But that was actually— the CNV Pro is what we used for the folks at Team Gleason when we were working with them. But it's pretty amazing. I mean, I've seen— we had a gentleman that we worked with who has ALS, and we were able to take some recordings that he gave us that were really, really poor quality recordings from WhatsApp voice notes that he would send to his colleagues.
Okay.
I think we only had somewhere around like 70 or 90 seconds of audio and we were able— he's in Argentina, so in Argentine Spanish, and we were able to create a voice for him. We didn't even have an Argentine Spanish base model, so we had to go with the closest geographically. We had to go with the Mexican Spanish one, which was not ideal. Some of the words are mispronounced or sound like he comes from a different region in Argentina. But overall, he was really impressed with how accurate the voice was to his actual voice pre-losing the ability to speak.
Well, I think I have that queued up as a screen share, but I need our administrator, Roman, if that's possible for you to share that with our audience. And so again, this is on YouTube here, and an example of how Team Gleason helped Use the Azure Custom Neural Voice. Let's—
Yeah, this is Brian Johnson, is his name.
Brian Johnson. Great, thank you. I'm going to just go ahead and play. This is about 3 minutes and 53 seconds. We'll go from here. If someone can just confirm that the sound is coming through, that would be great too. Here we go.
Brian is somebody everybody should know. He's just witty and funny and likes to tell a lot of jokes, a lot of dad jokes.
I used to bite my nails, but thanks to ALS, I don't anymore. My name is Brian Jensen. I was born and raised on the outskirts of New Orleans. I married my amazing wife in 2003. We have created a beautiful family of 7.
Around 2019, Brian started having some issues with his ankle. I was able to get him an appointment with a doctor, and he was like, I hate to tell you, but this is ALS. We were just— we were definitely not ready. Like, my heart hurt for him being so caught off guard. For those people who haven't experienced ALS, you don't wish it on your worst enemy. It's a neurodegenerative disease that essentially stops sending signals from your brain to the rest of your body to the point where You lose your ability to use your arms or your legs, your voice.
My voice was my superpower. I am a conversationalist, so my voice weakened over a year's time, and the whole year was filled with sorrow.
We might joke like, oh, I hate the sound of my own voice, but in reality, your voice is your personality. It creates an actual emotional auditory event in the brain of your loved one.
My voice was the scariest thing to lose, more than eating and even breathing, because I knew there were fixes for those things. But my voice? How would I live in a world where my kids wouldn't be able to hear me say, I love you?
Over the course of somebody's progression, one thing that they can do to be proactive is preserving their voice.
One of the challenges with voice banking in and of itself, because it creates a computerized version of their voice, is that it doesn't have a lot of emotion and tone. So with what Microsoft is doing, their voice banking technology really gives someone back that intonation and a little bit more of their personality.
There's just something intimate about hearing his voice, his little quirks. I'm just excited to really just hear him.
Any technology that can get my voice as close to normal Well, that is a gift not just to me, but to my whole family.
So, Brian, today we're gonna play like a newer version of your voice, and I'm hoping when you hear this and when you hear this, it's gonna remind you a little bit more of what he sounded like beforehand. So you just choose any phrase and then I'll play it on here as well.
Hey, beautiful people, I am so happy to be with all of you.
All right, let me type it out.
Hey, beautiful people. I am so happy to be with all of you.
Dang!
Hey, Christy, I love you.
Oh my goodness. I can't believe that, really. Like, that's really crazy. The biggest opportunities when it comes to emerging technologies is The gap that it can fill, not just here for people with ALS, but you think about anybody whose speech is affected. AI has the ability to increase communication, increase independence.
It's the next wave for accessibility.
Here at Team Gleason, we have a phrase, no white flags. From my experience working with individuals living with this disease, that really means that they will not give up who they are.
I am completely in awe and amazed by voice. I think this sounds just like me, and I am overwhelmed with joy that my family gets to experience this part of me. In many ways, I feel like I am back.
I can't tell you how many times I've watched that video, and it still like gives me like tingles and chills.
Oh yeah, uh, amazing. I mean, yeah, I'm agreeing with all of these comments we're seeing in the LinkedIn one.
Exactly. No, it's phenomenal. I mean, some of the people that we've worked with, there is a— so Brian used to be a preacher, and so his voice, you know, was his superpower, like, absolutely. And so thankfully we had recordings of him before he lost use of his voice. But a lot of people, they don't have recordings from before, or they don't have that many. And, you know, when you're diagnosed with something as major as ALS, there is so much to deal with. And historically, to create an honestly fairly poor quality synthetic version of your voice using kind of last generation of technologies, you would have to spend hours recording. It's exhausting, especially when, you know, you're— you know, if you are— let's say you've already lost some of your mobility, your ability to use a mouse or something like that. So having to use the computer with your eyes is very, very tiring. And having to go through the process of recording your voice, it would just— it would wear you down. And a lot of people, you know, not to be like a downer, but like there were a lot of people who would be in the process of trying to record their voice and they would, transition, right, without ever having done that. And so the fact that we can help people to create their synthetic voices and high-quality synthetic voices at that in just a few minutes of their speech that they might be able to take from, you know, a voicemail that they left their loved one or something like that is just incredibly powerful. So it's been amazing to get to work on that project. And I've been in, you know, in addition to the pilot, I'm continuing to work on some other stuff with Team Gleason to try and really democratize access to this sort of technology for people who have ALS and then eventually people who have other sorts of progressive speech disorders.
So what are some of the ways that you're doing that to get access to the accessibility?
One of the things that I'm doing with them, and we showed this off a little bit at the ASHA conference, which is a conference for speech-language pathologists and folks who kind of work in the speech space, is Team Gleason will be releasing a dashboard initially for just their users, but they have some goals of turning this into an open-source project that other organizations would be able to stand up on their own. But If I were to be a new client of Team Gleason, I could totally self-serve making my recordings. All of my recordings get stored on a— in a location that I control, so Team Gleason does not have your recordings. You keep those, and then you decide what voice vendors you want to use, and we've already talked to a couple of different AI voice vendors about participating in the dashboard. And so you'll create the recordings on your own time, You'll have complete control over them, and then you'll be able to decide which vendors you want to try. And if it doesn't work out with a particular vendor, you don't like the voice you get, you could try a different one, or you could try several at once. But ultimately, the individual is in charge, and they control their privacy, and they control, you know, who has access to what and when they allow somebody to access their— essentially their biometric data, right?
Mm-hmm.
Which is what their voice is. And then, you know, simultaneously, we're trying to figure out, you know, how can we work with the— you know, some of the voices already have playback in some of the common AAC software, but how can we work to expand that ecosystem as well? As I'm sure some of you know, a lot of the voice models that are out there, these high-quality AI ones, are cloud-based, and that is problematic for people who might have limited connectivity or people who might have limited connectivity throughout the day. So really making sure that we can find ways to securely enable them to have an offline voice model that can continue to operate for them but still be, you know, secure and private. So some thorny challenges, but definitely important work.
And just for some of our less, maybe not, you know, less tech-savvy folks, so when you say some of the this, uh, the voice data is in the cloud, does that mean it's having to request the or process the, the voice translation over the web?
It's not like a remote server. Yeah, yeah, a lot of them, um, a lot of the, the, um, voices, even some of the ones that we offer, like I mentioned the personal voice one, that is a purely online model. That one is not available offline. There are some of the, the custom neural voice models are able to be run offline. So that's an option. And then I know other vendors like Acapella have offline voice models as well that can be stored and run locally on your machine.
Okay.
Yeah, I guess I'm kind of curious. I mean, some of the— you know, as Andrew and I have been kind of tracking the— you know, this last few years and trying to understand, you know, where things are going, I think one of the things That's kind of obvious to anyone who just follows the general field of AI, is that these rapid advancements that we're seeing, these like huge quantum leaps, and like, wow, this thing can now do this that only 6 months ago could do that. And so I'm curious if— and maybe this will apply— this is a broader question that applies to some of the other examples we'll look at. Do you feel like this is the same trajectory that we're seeing in, for lack of a better word, this accessibility space? Like, are we seeing these kind of—
Yeah.
And honestly, so I would say on the whole, I am, or at least kind of when I came into this role, I was very much an AI skeptic and I still remain very skeptical about AI. And I think a lot of that is due to the demos that are out there are just not super compelling from a societal standpoint. You know, I see so much risk to what's being done out there in terms of, you know, spreading misinformation or, you know, it's sort of like I look at AI now and sort of like the ubiquitous access to it or near ubiquitous access to it a lot the same way I look at how the web kind of came about. And the web was, kind of really good in some ways and really not so good in other ways. So the low barrier to entry to creating a website just gave voice to so many people to share their, you know, share their experiences and for us to all learn from them and really just democratized access. But at the same time, it provided those same sort of tools to people who wanted to act in bad faith, right? Or who wanted to spread propaganda or misinformation or biased opinions about groups of people and that sort of stuff and enable them to find each other and mobilize as well, right? So like it's— and I feel like the amplification that the web provided and search engines on top of the web provided is sort of even more built up in AI because so many of the language models and such are trained on information that they found on the web with zero—
Right.
zero vetting of the quality of the content that was going in. You know, we've got that age-old concept of garbage in, garbage out. And so, you know, without proper tuning and guardrails and such, we see a lot of the biases that exist in the datasets that drive these models being amplified, right, by the models themselves. And so I think it doesn't mean the models are worthless. It doesn't mean that there's not value there, that there's not stuff that these models can do really well, but we have to be really thoughtful about how we approach them. So, you know, if you've sucked in a bunch of data that is, say, biased against people who are deaf, for instance, then, you know, the model is going to mirror that, right? Because it— and amplify that. You know, I've heard examples from people where, you know, they start off a conversation with a chatbot like, I'm blind and I need to know how to do X, Y, or Z. And the chatbot would respond with, I'm sorry to hear that, and then try and help them solve their problem. And that's, that's really diminishing, right? Like, that's not how— like, right, it's just a fact. You don't need to give an opinion about that. And you can just move on and help them solve their problem, what they actually came to you for. And we see these same sorts of biases show up in in image creation models where there's not a lot of training data around disability objects. So if you asked it to, you know, show you what, you know, a Braille touch feedback device looked like or, you know, something like that, it's gonna be all over the place or it may not do it. Like one of the examples that I show folks a lot is if you ask for an image of a blind person, almost invariably that person is blindfolded. And they, you know, if they are shown with a cane, I've seen them shown with like hooked canes that come up to their hip as opposed to what we would, you know, traditionally see as a white cane, those sorts of things. Just, and it's not that these models are bad, but they just don't have the data to be able to do these things right, you know. And so it's really important that we think about the inputs that go into these models, make sure that they're representative of people with with disabilities, which is why the Speech Accessibility Project exists. But, you know, we need those same sorts of things for other kinds of data as well. We need to make sure that disability data is included in the datasets that are training these models, whether they're language models, vision models, computer vision models, being able to go around and recognize objects. So, you know, we often think about, you know, the computer vision models, the early ones that we would see on the web were like, is this a hot dog or is it not a hot dog? You know, like those sorts of things, right?
Yeah.
But a lot of those vision models were taught to identify a hot dog. You know, it maybe, maybe it's slightly skewed, but it's in isolation against a white background as opposed to, you know, a hot dog that maybe somebody who is blind is holding their mobile device and they, they don't have it fully in focus. They don't have it, you know, their device might be moving and so it might be blurry. It's not going to be centered. You know, it's— there's going to be a lot more going on. But if you're helping somebody, you know, find their food, find their keys, find their purse or their backpack, Which are all things that people who are blind and rely on these devices do, we need to make sure that they can recognize those devices and tell them where they are. So we need to train them how to recognize blurry images, images that are only a portion of an object or something of that nature. So we need to be really thoughtful about this and, you know, honestly cast as wide a net as possible for the types of data that we're bringing into these models to make sure that they're, they're able to actually Realize their full potential.
The one thing that you mentioned, Aaron, was the bias that's inherent in the training, that's inherent in the models. And a very sort of an everyday sort of experiment you can do is just have any one of these image generation platforms produce a Show me a successful executive, right? And we'll just show me a lawyer, show me any role, right? And just take a look at like, what does Firefly produce versus, you know, ChatGPT versus Gemini, et cetera. And the biases are there, they're right there. You can see it's like, oh, oh, okay. Only people of a particular, who look a certain way. could possibly be successful executives, right? And that's the—
Yep.
I mean, eventually that'll get watered down, hopefully, and so that it'll be more fair. One thing that you mentioned though, that, you know, you had told us about this application that we raced off to install on our phones and checked it out, and it was absolutely fantastic, right? Seeing AI, and we'll share some links At the end, but the application is incredible. And so here I am holding the phone and without any difficulties, without any challenges, and I'm sort of pointing it at things and it's telling me what I'm seeing and how many people and how close. And it's just, it's mind-boggling. And I didn't even think about how much work it must have required to be able to do that in, shall we say, atypical— circumstances where maybe someone has a tremor, you know, and the photograph is going to be blurry or the camera is not going to be held straight. And one of the features that, you know, I was thinking about, not— but I'll call this out as my own ignorance. I don't really know. One of the things it'll do is it'll read a label, which I think would be really cool. You're in a store and you need help To figure out what this product is and what's in it. And what— how are the UPCs in the same place on every product? How would you know? So you would basically just hold the object up and move the phone around until it identified the place for that?
Yeah. And some of the models, I mean, depending on the model that you're using, they may read the label itself. They might read the UPC code or a QR code that's on it. There are some that have been trained to recognize like the packaging as well, so that you don't even need to pull something down from the shelf. You could scan along the shelf and see what certain things are. So there's a, you know, and some folks might be familiar with that from some online shopping experiences and stuff where you could take a picture of something and get the price of it.
Mm-hmm.
So the, you know, the technology is definitely advancing in terms of that. To be able to help folks figure out, you know, what is the product I want to pull off the shelf? You know, is my hand next to it? That sort of thing.
Yeah. Yeah.
That application made me realize just how liberating this technology is for people, right? Just, I can't imagine being shut off from one of my senses and to be able to regain some aspect of that through technology would be incredibly empowering.
Yeah, and that's the Seeing AI app.
Yep, and I actually have, I have an example of that loaded up. Um, we can actually watch that again. So this is actually one I found. This is from an organization, the Foundation for, for Blind Children. And so this, uh, gentleman— and I'm sorry, I don't know his name, I kind of skipped ahead— um, he is about to kind of demo, uh, this one example. So let's just quickly do that. And you're seeing on the left The application itself, and then on the right, obviously someone is filming him as he's doing this. Much more clearly.
Channel take picture button. And for handwriting mode to work, we just get ready with our take picture button.
We double tap.
Processing a sunset pictorial. It actually said a sunset pictorial. Good information. I've also been successful in using handwriting mode to read very low contrast items like credit cards. That wraps up our 6 minutes.
So that was just obviously not a full demo, but that was kind of just a quick little taste of, of the app.
Um, and again, yeah, there's, there's a, um, there's a really great video with, uh, Saqib, uh, who is one of the, the creators of Seeing AI Um, and Trevor Noah, um, that we can probably grab the link for and drop in. But, um, I think it's— I think it's The Question with Trevor Noah or something like that. I forget what the name of his show is. But, um, but yeah, he has a really good, um, interview with Saqib, and they kind of use it on the set that they're on, and Saqib shows off some of the functionality of Seeing AI. It's, it's really impressive. Um, I mean, being able to pick out matching clothes from your closet, being able to do you know, all sorts of things like that. The amount of independence that it enables people to have is pretty substantial and just keeps getting better. And to me, that's ultimately the really— I think the really interesting thing and the thing that really drives me every day is not so much thinking like, how can we empower people, but how can we give people the tools to empower themselves? Because ultimately they should be in control of their own lives. Like, we you know, they don't, they don't need us. And that's— I feel like that really does align with, with Microsoft's mission, which is why I feel, you know, like this is the right place for me to be, because we are about, you know, helping people to achieve more and to, to do what they, um, what they want and need to do in their lives. Uh, and so that really aligns with the work that I'm doing.
I feel like I should have mentioned that at the beginning too, just some housekeeping. Like, we, uh, we're gonna save like 10 minutes Ish towards the end for audience questions. And right now, the way this— we're set up here, we can only— we are broadcasting this in a few places. We can only see the ones that are coming in from LinkedIn. So we do have a couple questions that came in, and we'll try to get to those towards the end. But please, just kind of, if you have running thoughts, comments, just throw them into the chat. Again, only on LinkedIn at this point, and then we'll try to cover those as best as we can. Yeah, so another— yeah, I mean, in a way, so some of the questions that we're seeing here just related to what you were talking about earlier about this idea of bias, and we know that these are, as Andrew mentioned, like these are things that we can see in generative imaging. And I had an opportunity to kind of sit on a— we had a chat about the impact of LLMs and AI on things like the environment and electricity and all of that. I'm kind of curious, like, what to me It seems clear to me— the views and opinions of this are mine only at this point— but that the benefits of, you know, some of the costs that we're seeing that maybe are not so obvious when you're creating this stuff seem obvious when you're creating things that improve in general. You know, the society, but kind of, I'm curious, like, I know that you had sent us some link to some of the responsible AI principles. In general, like, what's— what are your thoughts on things like that, some of the hidden costs? To me, it seems worth it. But—
Yeah, I mean, I think a lot about this. And I think, you know, especially as I'm I'm looking into this space of accessibility tooling and how can we improve the accessibility of our products in a better way so that we're basically instead of— trying to use an analogy here, like, or not use an analogy, an example. But let's say you have an accessibility issue in your website. The contrast is too low, for instance. you know, somebody might have a plugin or, you know, eventually since people are AI-fying everything, you know, there might be some AI that's adjusting the contrast. That feels like very, very expensive to me because then each individual person is having to, you know, hit that AI endpoint or whatever to be able to adjust the contrast to something that is appropriate for them. Could that be like found earlier and become less expensive to do. Because if you're changing it at the beginning of the process in the, you know, the design phase of the software development lifecycle, you're catching it there. You're— not only is it less expensive to change because it's cheaper to fix things earlier on in the project than it is later on in the project, but you're only running that AI query once as opposed to running it for each person in the world who, right, might be visiting that site. So in the same way, I think we need to think about, you know, where— when we're using AI technologies, where are we using them in the process, and is that the most efficient place to be using those, and how frequently are we jumping to that AI solution? So, you know, some of the things that I'm thinking about are, especially with this, like, computer operation world that we're starting to see, where you're able to have an AI driving driving a computer. Phenomenally powerful, really interesting concept, especially for accessibility testing, which is just really challenging to scale. When you've got, you know, maybe 100 different websites that you're managing, or 1,000 different websites, or 4,000 different websites that you're managing with, you know, hundreds to thousands of pages each, you cannot scale accessibility— like human accessibility testing. Across all of those places, right? So could models allow us to do that, you know, at scale?
Yeah.
You know, certainly less expensively, but what are the costs of that? So the costs of that are all the processing power, all the energy we're having to use. Those models are not only interacting with the computer, they're also doing computer vision, which is expensive. And there's like a— there's an orchestration of different things, and there's real— like, there's money costs, but then there's environmental costs, the energy use, all that sort of stuff that just kind of goes out from that. And we may decide that, yes, that is valuable, but is there a way to translate that into— or to produce an artifact out of the process of doing that that is then repeatable, that does not involve having to go back to the AI? So, you know, just to throw out an example, If you have an AI go through an interface to accomplish a key task like booking a flight, right, on a website, you know, could it leave effectively a paper trail of sorts that could be automated in Selenium or Playwright or one of those sorts of like WebDriver-based tools that could then run it at much less expense, you know, on a local machine or something like that? You know, that, that would be something that would would ultimately be better. Can we use a smaller model? Can we use a model that is local to the device? So like the— both the Chrome team and the Edge team are working a lot on local models embedded in your browser, making those available not just to websites but also to plugins. So, you know, having the overhead of doing that cut down dramatically by, you know, the work is not happening in the cloud, so there's not the communication latency, there's not the energy use of the communication, there's not the energy use of the big model sitting somewhere in the cloud. It's all able to happen locally on your machine on a very small and efficient model. Like, that seems like a really good direction to go. You know, are we able to do the really expensive things once, like the of the synthetic voice one time and then move it to an embedded model that can be run, you know, much more cheaply on a local device so that we're not having to do that communication with the cloud. Not only is it going to make a better user experience, but it's going to be much more thoughtful in terms of how it is using the limited resources we have on this planet.
So I guess the key is in your intention, right? And to be to being thoughtful of these things, right? And because that's the starting point. If it never occurs to you that that's not a concern, then it won't be a concern. And if the lack of concern early on in your work then becomes— it cascades, and then it's like it's built in, just like bias in models, just like, you know, oh, we'll get to the alt descriptions later.
Yeah, right?
Right.
And then it's like, oh, well, maybe when there's a lawsuit, is that when you— is that a good time for you?
Even then, you know, people pay to make lawsuits go away, right? Like they'll settle out of court or whatever. But yeah, and I think it really comes down to the experiences you've had that have shaped your perspective, right? I happen to have a very different experience of the web the first time I used it than most people, where when I was in school, we didn't have the ability to use a graphical browser to access the web. So I had to use a text-based browser the first time.
Nice.
And I went to a site that was all image maps. And I had no idea what that was at the time, but all I saw was literally in square brackets, image, image, image, image, image. And I was like, this, this web thing is BS. And, you know, I didn't, didn't want anything to do with it. You know, a couple years later, ended up that being, you know, what I ended up doing for my career. But that, that experience of being shut out, um, really shaped my, my perception of how it is that we should be building for the world. You know, I want to make sure that anybody who needs to use a particular tool, whether it's to access their bank account, to reach somebody in their local government, or what have you, that they have the ability to do so. And there shouldn't be any barriers to them doing that. And there are lots of people who love to put up blinders and say, oh, that's not a problem. Like, I've had somebody, you know, say— I forget what it was— like, oh, we have no users who don't have JavaScript turned on, for for example, to use the web. And I was like, how do you know that? And he's like, well, our analytics tell us. And I was like, okay, tell me about your analytics. What are you using for analytics? Oh, we're using Google Analytics.
Okay.
Google Analytics is JavaScript-based. So if the JavaScript is not running, you're not going to get that information. And it was just like this long pause.
Yeah.
People just don't realize that, you know, they're creating these artificial bubbles to live in that are not the real world. And I think the more we expose ourselves to the experiences that other people are having, the more we can consider their experiences and bring them into the process and to make sure that they're involved in the design of new tools and new softwares that we're building and stuff like that. And I think, you know, the more we bring those diverse perspectives, I think the more we find really good compelling use cases for the technologies that we're creating. So there's a— one of the questions in the chat is about like, when is AI worth it? You know, I think, you know, having AI draft your term paper is probably not a great use of the technology. But if you're dyslexic and you use a language model to help you take a really complicated document and distill it down into something that you can more easily understand, you know, bullet points for you, that sort of thing. Like, that's huge. That's a game changer for somebody who is, you know, struggling to read really dense text in, you know, an academic report or something like that. You know, same thing with being able to have image descriptions for complicated charts or maps and those sorts of things. Like, there's a lot there that is hugely empowering and really beneficial for a wide variety of people. Not just people who have disabilities, but people who want to or who benefit from consuming things in different ways. You know, I look at captions, for instance. I do have a, you know, a slight hearing disability, but I started turning captions on when my son was born and I wanted to be able to watch TV while he was sleeping in my arms.
Yep.
And, you know, he's 9 now and we always watch everything with subtitles on. And, you know, he even hates it when there's not subtitles on now.
Yep.
And, you know, that's one of the things I don't like about going to the movie theater now is I don't have subtitles.
Amazing.
Because frankly, you know, the background noise is always too much and the audio, the vocals are always too far down. So, you know, I think, you know, as we start to kind of broaden our perspectives, we create more opportunities for more people to have better experiences in the world. Same thing with, like, if we go back to the classic example of accessibility of the curb cut, right? You know, the curb cuts that were originally created in Berkeley were created by sledgehammers as people were trying to take back their city and try and make it accessible to people in wheelchairs. And they would go out in the middle of the night and smash the curbs in order to create ramps.
Yeah.
And now they're a fixture in most cities. And, you know, it's not just people in wheelchairs who benefit. kids on bikes, it's the people pushing strollers, it's the delivery people who need to, you know, get a ton of packages into a large, a large high-rise building in downtown. There's so many folks who benefit from that now. And, you know, at the time it was, it was vandalism, right?
Right.
Like, this was, this was, this was an act of defiance. But now, you know, it's, it's, it's an accommodation that helps so many people. And I think that, that we can look at technologies like AI and find use cases like that that can be super transformative for, you know, everybody. Because, you know, people who have, you know, who are dealing with cognitive issues, cognitive disabilities, you know, they have the same things that, you know, some of us may have after a really exhausting day, right? Like, we are have cognitive disabilities as well that might be temporary, and we can benefit from the same technology that benefits, you know, them every day. And so I think as we look to that kind of expansive opportunity space, we create the real use cases for a lot of these technologies rather than just these like sizzle reels of, oh, look, I can have, you know, I can have a witty conversation with this chat agent.
Right. So I guess that kind of leads into my next question, um, and I'll just again pause for a second. And folks, anyone who, you know, if you want to start throwing questions in LinkedIn, cool. We've got a couple already, but feel free. This is the time just in general because we've got about 10 minutes left. Um, but to that point, I guess, um, I, I feel like all, you know, all 3 of us were there at kind of the, the onset of the web and we saw Again, it go in several directions and then kind of take a step back and be like, yeah, that didn't really work out, did it? And I'm curious, like, I do feel like we're in a little bit of that stage now where, again, a common, I guess, common complaint is it's being pushed on us in a lot of places where we didn't ask for it, didn't expect it, etc., etc. And so I guess, you know, is that just the nature of adopting a new technology? It just seems like this seems to be so much faster. It's like happening so much more quickly.
I mean, it's the classic Gartner hype cycle. If folks are familiar with that, you know, I forget all the things. I just always remember the trough of disillusionment.
But that's the one that we all remember, isn't it?
I remember that.
Yeah.
Yeah. So I think there are some— yeah, there's a lot of people just throwing things at the wall and seeing what sticks. And, you know, there are certainly some compelling examples. But to me, a lot of the places where I've found just real, I don't know, inspiration behind the work they're doing is in the disability space and figuring out how these tools can be used to do more. Chad was asking about the AI for Accessibility Grant Program, which is another program that I've run where I get to spend Microsoft's money investing in outside projects and folks who are doing some really interesting things. There is an organization based out of Radboud University which is in the Netherlands, and they've been working with Masanga Hospital in Sierra Leone, and they are using AI to help them to customize prosthetics for people who have limb loss. They're specifically focused on below and above elbow and below and above knee amputations, and what they're doing is they are using 3D scans of the folks' stump, and then they bring that model into their software and they use AI to help them figure out how to create better socket fit with less rubbing. And that is tremendous. And it completely— and then they're 3D printing the sockets. And so it is completely revolutionizing the way that prosthetics are being created. Because the traditional sort of Red Cross model was, you know, making a plaster cast and then doing, you know, a— I forget what it's called— like a mold of that. And then like basically creating it and it taking like multiple steps, having to shave it down manually. All of that sort of stuff is very heavily manual process and can be done much more quickly using AI to help them figure that out. Just remarkable stuff. We've worked with Cboard, which it's the letter C board, B-O-A-R-D, .io, and they are one of the AAC software I mentioned earlier. They are an open-source AAC software company, and they have been doing tremendous work on their AAC software itself, but they've also been doing work on a new product that we actually showed at Ability Summit, Microsoft's Ability Summit last week, called FluentMind AI. And it is, to the person's question about aphasia, it could be something really useful for that where it's actually, it's an AAC software, but it's listening to the conversation that you're having with people in real time. And then it is listening to that conversation and proposing responses that you might want to give based on what it knows about you. So it's really helping you to compose sentences quicker because that's one of the things that's very challenging for people who are using AI for access or using AAC software is that, you know, it can take 60 to 90 seconds to draft a reply, especially if you're using eye gaze. And so this dramatically speeds that up by using speech. Speech-to-text, so catching that, uh, and combining the, the speech-to-text with a language model to propose, uh, potential responses. And then you can choose from those responses or give it a little bit of a hint of a direction you want to go in with your response and then propose a response to you. That's pretty amazing. Um, there's a team at Global Symbols, um, that are based in the UK, and they are using image generation to create symbol sets for AAC software that remain thematically, like visually consistent, but are culturally relevant. So being able to, you know, if you were doing a symbol for a police officer, you know, that police officer will probably look different in Britain versus Indonesia versus Thailand versus Japan, China, etc. So Being able to do that, they shared a story with us early on about creating symbols, like using generative AI to create symbols so that a young man could participate in the Muslim call to prayer, which he had not been able to do with his brothers in years. All of a sudden he had the symbols to be able to do that, and that's amazing. Being able to channel the image generation capabilities of AI to really expand the capacity of AAC software and similar things is really— it creates so much more opportunity for people to live the lives that they want to live and do the things that they need to do.
Out of curiosity, this is— so these AI for accessibility grants, are these things that people, like if there were people in our audience who might have things that they were curious in doing? Is this something that's like a public submission, or is it— how does that work?
We have traditionally put out calls, and they've usually been thematic. So we've had some that have been focused on mental health. We've had some that have been focused on generative AI. We've had some that have been focused in other areas. I'm not sure what the future holds for the program right now because I focus is in a couple of different areas.
Yeah.
But yeah, that's traditionally how it's gone. We put out an RFP and, you know, sometimes we'll also make strategic investments if we're made aware of something and we think it's really interesting. We might have a couple of conversations with a team and then decide that we want to invest in it outside of a funding round. That has certainly happened. But yeah, there's some just amazing things. I know we're coming up on time. One more I just want to share from the mental health standpoint. Really interesting project. We worked with an organization in India to create a product called iWill. And what, what it is is cognitive behavioral therapy via chatbot, which to me, like, when I initially heard the pitch, I was like, ooh, that's, that's kind of scary, right? Like, this, this is a real, in some cases, a life-or-death thing for people when you're talking about folks' mental health. But what I came to realize is that The— there is just a tremendous lack of mental health support for people in India. I don't remember the exact numbers, but I think it's like 0.6 mental health professionals per 100,000 people in India. So there's just— it's— the capacity is just not there to help people. And so what iWill was doing is they, they had psychologists on staff who were practicing, who were interested in taking with permission, the transcripts of cognitive behavioral therapy sessions in Hindi end to end and take those and use those to train a model so that it could do a better job of having conversations with people in Hindi via apps. And so they worked on that and it's been extremely successful. They had a plan to be able to deliver that to people who didn't have devices themselves, because there are a lot of people who don't have the money for a device. So being able to go to, you know, a private room within a community center to be able to engage in CBT, pretty amazing. So they're, you know, another phenomenal organization doing great work, and they're continuing to take that even further, doing some voice-related stuff for people who are low literacy as well. So there's some really interesting applications in that space as well.
Cool. We need to figure out how to— I believe that for the folks who have registered for this event, we can go back and put some of these resources. I think, Andrew, we can do this.
I think we just add it to the chat. I think we can just copy and paste them into the chat. Exactly.
I can see people who want to respect their time. Some people have to take off. So I guess we'll just, we'll kind of wrap it up officially. Thank you so much, Aaron. This is like really eye-opening.
Yeah.
I think it's super encouraging to see that, you know, that people are kind of thinking about the responsible uses of AI and also kind of, again, some of the less heralded but meaningful applications of that. So super encouraging. Again, for folks who— and I know that one last one, we'll do another link of resources. You've written a little bit about this on our favorite web magazine, A List Apart. And there's a great article that you did there about opportunities for AI. And people should check that out as well. Andrew, any parting words that you have or?
No, just thank you, Aaron, for sharing this with us. It's so easy to get caught up in the mire of all the negative press and concerns and, you know, like nothing is free, right? Nothing is free, but this is absolutely payoff, right? So the, you know, there was a point like, when is AI worth it? When is AI worth the extra cycles at a data center? When is it worth the potential for misuse? And it's like, these cases are where it's justified, right? Like it's, it's a tool that changes lives in such positive ways. Denying people that would be absurd. We use electricity more frivolously making stupid pictures. With AI, right? So I think it's really important that this, that this, that this work gets showcased because it's, it's, it's critical. So thank you.
Yes.
Seconded. And yeah, so we, we will end now. Again, thanks everyone for joining us. We look forward to doing some more of these in the future. And, uh, again, Aaron, thank you so much.
Yeah, thank you for the conversation. I really enjoyed it.
All right.
More from Aaron
Falling in Love With Forms - Required Field (Narrator + Internet Explorer 11)
