From research to product at Azure AI

Mixed reality or virtual reality or augmented reality is really good example of applied AI. So when you're doing hand tracking, for example, that we do in HoloLens, you can imagine that it's literally looking at your hands and saying, okay, this finger is pointing one particular way, it's pointing that way, or I'm grasping my hand here. And to even create a data set, of course you can capture a lot of hand angles, but you can also create different 3D orientations of a hand and have the label data set so you can train your your hand detection models and so forth. And even all these devices and HoloLens, you can speak to it. So there's lot of onboard speech recognition and so forth. So there is one is, of course it's a very good example of applied AI for us, but the second one is for our customers and our developers. It is actually the simplifying and accelerating the development experience and really at the end of the day, helping customers realize more value out of these technologies.

Bandwidth for changelog is provided by Fastly. Learn more at fastly.com Our feature flags are powered by LaunchDarkly. Check them out at launchdarkly.com and we're hosted on Lino Cloud servers. Get $100 in hosting credit@lino.com changelog what up friends? You might not be aware, but we've been partnered with Linode since 2016. That's a long time ago. Way back when we first launched our open source platform that you now see@changelog.com Linode was there to help us and we are so grateful. Fast forward several years now and Linode is still in our corner behind the scenes, helping us to ensure we're running on the best cloud infrastructure out there. We trust Linode. They keep it fast and they keep it simple. Get $100 in free credit at Lindo.com changelog Again, $100 in free credit at Linode.com changelog.

Welcome to AI, a weekly podcast that makes artificial intelligence practical, productive and accessible to everyone. This is where conversations around AI, machine learning and data science happen. Join the community and slack with us around various topics of the show@changeog.com community and follow us on Twitter. We're racticalaifm.

Welcome to another edition of the Practical AI Podcast. My name is Chris Benson. I'm a principal Emerging Tech Strategist at Lockheed Martin and with me, as always, is Daniel Whitenack, a data scientist with SIL International. How's it going today, Daniel?

It's going great. I Think we discussed when we recorded last week, which was pre us Thanksgiving, our plans for Tofurkey for Thanksgiving, which I had. I'm not sure if you, if you partook as well.

I did. So anyone who listens to us regularly hears Daniel and I talk about our plant based meals. And I'll take a two second divergence. I had an interesting entry into Thanksgiving dinner that has nothing to do with AIML whatsoever. We were just about 5, 10 minutes from sitting down and I got a call from a neighbor saying I have a large black snake in my house. Can you come help me? And so anyone that's listening to us knows I'm the animal.

Yeah.

And you're on call for that sort of thing.

I think I am. So I like totally separate from all this, what we're talking about today. I do a bunch of animal stuff and I actually run a snake hotline. How weird is that? Do you know anyone else that runs a snake hotline? So I get the call, I go and I have a neighbor with a large black king snake in their basement. And they were very upset because they were not very snake savvy people. And so I calmed them down. We got the screams to stop and I picked the kingsnake up and carried it delicately outside and released it. Educated them along the way. As the screaming stopped.

You made an unexpected house visit on Thanksgiving.

It was, it was. And then I dashed home just in time to sit down for Thanksgiving dinner. So it was a memorable Thanksgiving dinner in that way. Yes. A little odd story that goes with it.

Yeah. That's awesome. Yeah. Well, we have some interesting things to discuss today including, you know, if you were to have taken a picture of that experience, you might have wanted to come up with a caption for it and that might have been somewhat difficult. That's one of the tasks that we're going to discuss today.

Indeed.

Yeah. And we're really excited to have with us Bharat Sandhu who is the director of Azure, AI and mixed reality at Microsoft. Welcome Bharat.

Thank you. Thank you for having me.

Yeah. If you could just give us a little bit of an idea about your background and how you got into AI and what you're doing now.

Yeah, sure. By the way, just for Thanksgiving even we had a plant based Thanksgiving this last week, so we joined you.

Ah, good.

No snakes though.

You have any, any recommendations on that front for our listeners? We always try to throw in an occasional recommendation.

You know, I, we just tried like. So we try to reduce the amount of meat we eat. My wife and I. So we tried it this Thanksgiving and we just cooked up some new recipes. The green beans recipe was. It turned out extremely well. It was spicy and, you know, it sounds normally the different kind of Thanksgiving food, a little bit of spice in it, but yeah, I don't know, we just copied some recipe offline on the web so I can share that with you.

It's good. I don't know that mine ever look quite as tasty as the picture on the recipe, but sometimes just finding those recipes online, it works out pretty good before long.

Because this is not a new topic. We have had other folks on the show talking about this on the side before we got in the main topic. Pretty soon we're gonna need our own conference. It'll be like the Plant based AI ML and Data Science Practitioner conference. Yeah, just saying.

Yeah. It'll describe to you what you cooked and what you could have cooked. And actually cooking is kind of like machine learning. Right. It's like not just the ingredients, it's how you ship them and all the work that goes in it.

I'm glad you bring that up. Actually. It's one of my favorite examples that I give when I'm teaching workshops is a lot of industrial AI applications are really more like using a recipe and bringing your ingredients to it rather than sort of designing the recipe itself. Right. And you know, maybe that gets into some of how people use Azure AI and other things which we can talk about. But yeah, before we get into that, I'd love to hear a little bit about your background and. Yeah. How you ended up where you're at right now.

Yeah. So, you know, I've been at Microsoft for a bit over 10 years, close to 11 years now. But, you know, I'm a computer engineer by training. My first job was a computer engineer writing code. And you know, as before we got on the call, we were talking with, you know, Chris was Lockheed Martin. So my first job was this company at National Instruments. And you know, I started as an engineer but then I moved out to work with customers in our field engineering and sales division back then.

Are they the ones that make labview? Is that the.

Yes, that's right.

Okay. Yeah, that's my experience with National Instruments. For those that don't know, LabVIEW is like software that helps you interface with all sorts of different instruments and circuits and design various things and. Yeah. Anyway.

Yeah. And some of the work, you know, like, so if you're an engineer by training, you know, and you. And you start working with customers like Sikorsky, making the Blackhawk helicopter, United Technology, the fuel cell and Pratt Whitney making the, you know, the engines for I think a lot of fighters, including the jet F35 now.

Correct.

It was quite fascinating because back in the day, not that many companies were collecting a lot of data, but these companies were collecting a ton of data and kind of building hardware in the loop testing and those kind of things. So it was a fascinating time. And so that was my kind of like early career learnings, working with folks building literally jet engines and working with tons and tons of data and trying to make sense of them. We actually focus a lot on hardware in the loop testing, which is before you actually build something and you want to instrument it all up, you want to maybe do some simulation. So Math Lab and Simulink were common players we worked with, competed with also back in the day. And nevertheless I did that and got my mba. Then I joined Microsoft. And Microsoft, my career has mostly been commercializing incubation businesses. So first was Unified Communications, which it's funny we talk about over zoom right now, but 11 years ago, at least for Microsoft, it was a new entry back then. And then I joined our IoT Azure IoT team. We were launching our Azure IoT offering and then the big data space and the last three years, really the AI space. And very recently the mixed reality and the common trend has been commercializing these very emerging new technologies and making it a viable business value proposition for companies, for our partners and for our customers, making it easy for them to actually benefit from these technologies.

It's pretty interesting. I think a lot of companies that are getting into AI like whatever AI based system they're creating might be one of those things that they're incubating within their own organization. So do you have any insights from your experience kind of productizing these different, you know, incubated ideas into actual offerings? Do you have any insights for those out there that are maybe their AI based system is their internal incubation project. Any suggestions for them or advice?

Yeah, a comment. First of all, technologies come and go. Deep neural networks are fancy today will be quite passe in a few years time. Just like reinforcement learning is kind of coming into work in some areas. These are just technology. So never please get fascinated by technology. And at least for those of us who work in technology, companies have to be very careful of that because we get super excited about new techniques. It really is actually at the end of the day, business has not changed. You know, we have customers and we have to serve them better and we have competition and we have to provide better offerings than the competition. So being super grounded, whether a project is going to help dramatically reduce cost, improve operations or increase revenue is the most fundamental starting point. And you know, I have seen, you know, I think we've all have a lot of organizations have incubation or innovation offices, which is great. But as we work with customers, it's always like, okay, what is the core business need? And more importantly, have the key stakeholders bought into it up front? And then they've signed up for maybe a year long project or maybe sometimes longer to prove out whether this kind of venture, in this case AI for this particular application is actually going to provide value for the business or not. So just setting up the project in a very methodical way with key stakeholders bought in all the way up front, and then having regular meetings and making sure we have the KPIs identified or improving them as we go through it is super important. So that'll be the one thing we'll say like at Microsoft we kind of make sure we stay true to that. At the same time, for us, we have the luxury, like with some of the companies like Google and Apple and Amazon to do a lot of just pure research. But a lot of companies in the world do not have the, you know, the luxury sometimes. So especially if their core competency is not technology or AI machine learning. So for these companies, it's our, I think, duty to help them kind of walk through how they can evaluate projects, not just technologies.

Gotcha.

So I'll just go back to kind of going back to business basics.

When I was introducing you and I was mentioning your title, it's one, one thing I was wanting to follow up on is, is as part of your title is mixed reality. And we hear about all sorts of different types of, you know, something reality, vir reality, augmented reality, augmented. But what is mixed reality? Or at least how does Microsoft define that?

Yeah, sure. So mixed reality for us, you know, and I think X reality is a new term that's also started kicking around. But basically like if you look at our, the way we interface with our phones or laptops, it's a 2D screen and then you have things like Oculus Rift or Windows virtual reality headsets. And I got an Oculus a few weeks back, I love it. And Quest, sorry, Oculus, it's a virtual reality. So there you're in a virtual reality environment, but you're in a separate environment. You're totally removed from the physical space. And then you've augmented reality, which hololens is an example, Magic Leap is another example where you can actually see through and you can see holograms. So you can have presence of the world around you as well as virtual elements in it. Now, mixed reality is a term that spans augmented and virtual reality because what you'll see happening more and more, and even with Quest, I was quite impressed to see you can actually see through and see the augmented reality. So from a development platform perspective, you should not have two separate set of tools and development environments and Unity for this and unreal for that. You really want to have a consistent development platform, a consistent ecosystem. And really for our customers who use these technologies, some will of course have a better experience in virtual reality, like gaming. But for a lot of like industrial commercial use cases, augmented reality is much more effective. But they also have a seamless experience through these different devices. So mixed reality as a term really is meant to span the world of virtual reality and augmented reality and really say it's a common development platform and a common user experience. It's really more of a term at this point, but really it's meant to bridge the two different worlds.

Right now it sounds like kind of that element of common has a lot to do with it in terms of spanning. So would it be fair to almost say it's kind of taking augmented reality and applying tooling, common tooling across so that you're, your productivity is improved without some of the natural distinctions on the two sides of it?

Absolutely, yes. If you're a developer, you don't have to learn two separate types of development stacks and so so forth. Right. There of course be some nuances, right. But more and more, as much as we can make it easier for customers to go from an iOS or a phone based augmented reality system to really like holographic, like holographic HoloLens kind of experience to even a virtual reality experience, and the developers don't have to learn a different stack every single time, different set of tools every single time. I think that's additive value to the ecosystem. So that's one thing as an end customer. Also we're not there yet, but we have a consistent experience and they'll have a consistent experience through the different devices and so so forth. But yeah, mixed reality is basically trying to say is a bridging of these virtual reality and augmented reality worlds.

Is that where the kind of common thread with the other things you're involved with in terms of the Azure AI platform? I know there's just from my own experience, so vastly different number of environments that people Use for developing AI are part of the goals with the Azure AI platform. Similar in the sense of bridging some of those and creating a unified development experience.

Yeah, also like AI is, I would say, a much more mature place than mixed reality or AR so far. But yes, I think there's a common, like, you know, basically just simplifying the developer experience is a common thread for sure. The other thing is also, by the way, mixed reality or virtual reality or augmented reality is really good example of applied AI. So when you're doing hand tracking, for example, that we do in HoloLens, like, you can imagine that it's literally looking at your hands and saying, okay, this finger is pointing one particular way, it's pointing that way, or grasping my hand here. And to even create a data set, of course you can capture a lot of hand angles, but you can Also, in a 3D studio like Maya or something, create different 3D orientations of a hand and have the label data set so you can train your hand detection models and so forth. And even all these devices and HoloLens, you can speak to it. So there's lot of onboard speech recognition and so forth. So there is one is, of course, it's a very good example of applied AI for us, but the second one is for our customers and our developers. It is actually the simplifying and accelerating the development experience and really, at the end of the day, helping customers realize more value out of these technologies.

So as you describe it that way, and would it be fair to say, if you take applied AI, meaning deep learning, reinforcement learning, things like that, and enabling the mixed reality environment or experience with these technologies, and that becomes kind of the use case, if you will, for the applied AI as an enablement technology. Is that a fair way, way of talking about the relationship between them?

Yeah, yeah, absolutely. And of course, not every single application needs to have the customer, the partner, build an AI element to it too. But like, you know, we had this example in, I believe it was Disney, where they actually had a depth sensor sensing people walk up to a screen and I think they were doing the Stranger Things. Yes, they were doing the Stranger thing in Atman. And then the monster, the Stranger thing will come out and say, who are you? You'll say your name. They will use speech to text or speech recognition to understand your name, but then do modulation of the speech and speak to text your speech to speak back to you and to scare you and so forth. So there you actually have blending of AR&AI from a customer perspective too. So right now these are two distinct businesses for us. Some places they overlap when it comes to customer use cases, but a lot of the way we take these technologies to market has a lot of commonalities of kind of going from super early incubation into commercial businesses or else. So there's a business connectivity issue. Also.

Changelog is the best way for you to directly support practical AI. Join today and unlock access to a private feed that makes the ads disappear, gets you closer to the metal and help sustain our production of practical AI into the future. Simply follow the changelog link in your show notes or point your favorite Web browser to changelog.com Once again, that's changelog.com.

Changelog.

It's better.

So Bharat, I'm kind of interested, you know, while we're talking about the Azure AI platform, to dig in a bit to what developing AI in the cloud looks like now. So in its very sort of raw form, I imagine people might think, oh, I could spin up a cloud instance or a virtual machine, maybe attach a GPU to that and, and do some AI things. But of course, each cloud provider has developed a whole suite of different things that aid AI development. What does that picture look like in Azure AI right now?

Yeah. So you know our AI stack. Do you mind if I just explain how we take a look at what Azure is?

Absolutely.

So first of all, there's a ton of innovation in AI in the open source world, which is awesomeness. So our goal is to make sure we can provide that to our customers in a more packaged fashion. We also have a ton of research centers around the world. We have eight resource centers and the work they do is really fine tuning and developing algorithms. And their charter has really been to say, okay, let's go and build the best text to speech or speech to text or image captioning that we'll cover later on. Techniques. Now these are researchers and PhDs in machine learning. We have more of those now than we used to have maybe many years back, but they specialize in really deep machine learning space. We take all the work they do and then we bring it to Azure AI. And our goal on Azure AI is to use all the open source technologies available, all the work we're doing in that space too, and taking all the research milestones and baking them on a platform that we can give to developers and data scientists so then they can use it from a simple no cost free API call to kind of doing highly high scale reinforcement learning techniques. We also, by the way, test this stuff in our own products and we have the. I think we're just lucky to have many different types of endpoints from teams which if you were to join a team session now, you can do transcription of all the calls, they can do recording and transcription and translation. Right. So all that's powered by Azure AI within PowerPoint. If you've used it recently, there's something called PowerPoint Designer that looks at your PowerPoint bullets and says okay, hey, maybe you want to actually have a better representation of the slide. And it's really using Azure Machine Learning under the covers to do all the inferencing and give you those recommendations. And even Xbox, you know, the holiday season is here, but if you have an Xbox, if you log in, you're given a personalized screen. So if you log in Daniel or Chris, you'll get a different dashboard and that's powered by Azure AI and a service called Personalizer that uses reinforcement learning behind the scenes to do very personal recommendations for folks. And the same technology is available to developers. So our approach to bringing AI to developers and data scientists is we take things in the open source world, we of course do our own research and then we package it and really goal is to make it usable for customers, whether they're very professional data scientists. So give them access to the best collaborated notebook environments we can with highly scalable compute. So they can do the really highest end training work they need to do all the way to simple APIs that any developer can call using any other language of the choice and add AI to the applications in a very simple fashion. And then what we have also done recently is we started creating what we call scenario specific app services and we can get into that. A good example would be a metrics advisor where we have an anomaly detection models running behind the scenes, but they also have a bunch of other logic around them to ingest data from multiple time series sources of data, to do root cause analysis and to have composable UX elements that a developer can embed in the application. So if they want to do more of metrics monitoring, we don't just give them a model, but we give them a higher level still a PaaS service that does a lot of the other work and the other business logic for that thing so that really becomes a platform. So at the base we have Azure Machine Learning which is used to train machine learning models. On top of that we have customizable AI models, we call them cognitive services, those are the speech, vision language and so forth. Then on top of that we have a set of Services that are really tuned for separate kind of scenarios like form recognition to metrics, monitoring and so forth. And that's really a stack for Azure AI. We do go one step beyond that and we have a platform called Power Platform. I'm not sure if you are aware of that. It is basically a platform for business users. It's a no code, low code environment and really allows citizen developers really to go ahead and design applications. So all the Azure AI tech that I described is provided in a low code, no code way in this environment for anybody with a citizen developer to, to build AI into their applications without having to write code.

That's fantastic. So first of all, I want to say regarding my next question, you've actually answered it in a fantastic format in kind of going through a lot of the benefits. But I also want to kind of distill it a little bit because so much of our audience are practitioners. That's right. That are engaged in AI and it is very, very typical for them to be in an environment where they're going to engage in a project and they may have access not only to Microsoft, but competitors as well. And they're trying to justify that. If a practitioner has just listened to this episode and we have more to go, obviously. But they're, they're like, I want to do Microsoft AI on Azure. How would you differentiate it or how would you recommend they differentiate it for their, their manager or whoever they are having to report back to to say, I definitely don't want to go with the other guys, I want to go with this one. How do you position that?

Yeah, first of all, I must just say it's better for the AI field if the customers go with anybody and they do an amazing AI project. Right. So I think.

Understood.

I just want to acknowledge that like, look, we have great competition and that's great for the industry and we're excited to play a role in this thing. But what guides us is our core three made investments which we've had the fortune of working with multiple enterprises over many, many decades as a company. So we really honed into kind of making things enterprise grade. So really breaks down into three main things. First is making sure we can give the tools to developers and bring all the heritage of IDE tools and so so forth on their terms, which means they can use the tools and frameworks of their choice, whether the professional data scientists that love Jupyter notebooks. Right. And they want to use TensorFlow or PyTorch. Great. But then have it optimized for the in High scale training, the kind of work we do with OpenAI for example, and give them really fleets of very high powerful compute infrastructure, which is just not GPUs but also the networking layer for them to be able to do training very cost effectively at the end of the day and highly reliably. Right, but that's for what I would call the more professional practitioners who are really looking for tooling to do ML in a scalable, reproducible fashion. And that's really the Azure machine learning layer, where our investments on making that really robust machine learning platform on the cloud at the PLAS layer is really resonating with the customers. But the second one would be for customers who are not deep machine learning experts. And for them it's easy to say, hey, you can have an API call and you can have an AI model in your application, but the AI model has to be really good, otherwise it will work, but it'll work in a bad way. So our commitment to quality, whether it's speech to text or text to speech, or you're getting Turing models, which are very large scale models but available through an API call, so your language or text analytics is very high fidelity. And then also the high end services. So it's just not having models, but ability to kind of get to market faster for form recognition and other scenarios like window indexing and so forth. We just have the services that a lot of times our competitors sometimes don't have it at this level. We do and be able to kind of provide this level of different, various level of access points for developers to work on, but work on one environment together. So that's one thing. Other two things that I'll just say, which I think we sometimes take for granted but is super valuable for our customers, is really baking them and making them enterprise grade. Like it might sound boring, but things like vnet and having ability to run these in environments that are highly regulated and having the certifications, having all the services and models be certified and so forth goes a long way. And last one, which has surprised us too is our commitment to responsible AI is proving us to allow customers to take AI and use it in their applications much more effectively. So if you had not just principles, but even tooling that we provide to our customers and guidance that we provide to apply AI in a responsible fashion. So really those are the three things. One is like allowing customers to kind of get access to AI on their terms, whether they're professional data scientists or just developers who don't have machine learning expertise at all, but want to get access to AI. Second, giving them an enterprise grade fashion so they can deploy with confidence. And third one, also giving them responsible AI, not just guidance, but also tools and capabilities to do it. And having it all built into our products. Not just you can go to this doc here, you can go to this GitHub resource repo there. It's all available as part of our platform.

That was a great explanation you mentioned along the way. OpenAI and I know that Microsoft has an exclusive relationship on GPT3, which obviously the whole industry is excited about with OpenAI. I'm assuming that at some point that becomes pretty normally accessible via Azure. Anything you can share on that along where that's going to go or what we should expect.

Yeah, we're still working with them on a lot of these things right now. Like right now what we wanted to make sure is we can enable OpenAI to really break through AI research and to give them amazing cloud resources. The things we spoke about. I think we have now the fifth largest supercomputer, maybe probably the first one in the cloud that they access to build these models. These are about 300,000 CPU cores, 10,000 GPUs and the networking layer that goes with it. And then also it allows us to kind of develop our optimization. So you might have heard of something of onnx runtime, which is really used to be for inferencing, high speed inferencing, but we've also tuned it to do high speed training. But all those optimizations kind of came in also with the work that we've been doing with OpenAI and even internally and all that stuff. So more to come on where some of this work shows up. But we also, I think our goal is to allow OpenAI to do amazing work like GPT3 and give them the ability to do it from tooling and all that. And all that also shows up for our customers, even if they don't get, let's say, access to GPT3 model today, which they can get from OpenAI. But all the work that went into enabling OpenAI to do the work is available to our customers also.

Yeah, just before we leave the subject of the Azure AI platform, where are some good resources for people to dig in and get started? Maybe specifically for data scientists or software engineers who are, are wanting to get their hands dirty, you know, working a little bit with the Azure AI platform, where can they get started? Any recommendations for them as they're sort of onboarding?

Yeah, you know, so I'll sit there like depending on where they are three main ways I'll recommend them if they're already using they already know machine learning. You know the machine learning practitioners, you know just jump into Azure Machine Learning so you know documentation. We have a a bunch of courses on our Microsoft Learn. We have them on Udacity too now and other courses you can take courses. But if you know machine learning the best way is to kind of just dive into the product. And again the product's really designed for ML as a team sport. So especially for folks that have AI teams or data science teams are data engineers, data scientists and they want to work in a collaborative environment. They'll really benefit from AML and having it to do it with really modular ML pipeline. So it'll be great to jump into the product and learn about what an ML platform in the cloud provides like Azure Machine learning. So just go to Azure machine learning or ML.azure.com and if you don't have Azure subscription you'll be able to go sign it up. It's free and you'll get access to it and that's a great place you can learn. Documentation is always a really good place and easy place for people to access. If folks who are wanting to learn and same with other rest of the portfolio. You can also go to www.azure.comai that will take you to all the different offerings we have. But if folks who are interested in learning machine learning who might not be experts. We recently partnered with Udacity to launch a couple of courses and they're really good in depth courses that have been developed by us in partnership with them. That'll be the other second place. I'll recommend them. We also have great training on our Microsoft Learning website and there can go deep in different areas. So those will be the three main areas, the product itself, you know, please give us feedback on what you don't like so we can improve it and the learning courses that we just mentioned.

So we're at and in our conversations prior to the show we had talked a little bit about some of the work that Microsoft is doing research wise and productizing research to enable computer vision to be used to help people who are blind or have low vision. And we talked a little bit about the Seeing AI app and other things. But I was wondering if you could kind of just give us a little bit of a briefing on Microsoft's work in this area and I know that they had a pretty interesting milestone in terms of image capturing. So maybe you can go into that a little bit.

Yeah, as I mentioned earlier, big Focus is kind of doing good research but then putting into products and things we've done in the responsible AI work from model interpretability, fairness detection and all that, which is available in Azure Machine learning. We did the same thing very recently for the Cognitive Services family of products, which is the customizable AI models. In particular, we did the image captioning milestone. So you might know this basically novel image captioning benchmark. And basically if you look at image captioning, it is a technique where you show the computer an image and it says it's an image of blah. Normally the way these things have been trained is by actually giving them images and complete sentences of what these images are. So what the research team did was something different. So they actually kind of went about how children will learn vocabulary. So when we're learning, we basically say, hey, there's an apple and there's a picture of an apple and this is an apple. This picture of an orange. Here's this is an orange. We never say this is an apple sitting on a table. So that's kind of like the first process the team went through which is building a visual vocabulary by really training object and what the object was. Then that was how they did the pre training of the model. Then for the fine tuning, they actually then started giving full sentences to the image to the model and so doing the inference time. What it first does is it does a visual vocabulary. So it's an image, it contains A, B, C. Then based on this fine tuning step, it is able to put together sentences together. So we've had image captions for a while but. But we really dramatically improved it. So some couple of examples was it was a picture of, and I'm happy to share these before and after. It's easier to see it than explain it. But imagine a picture of a doctor with a stethoscope. Before it said a woman looking at a camera, now it says a woman wearing a stethoscope around her neck. So very much more accurate. Before this picture of a satellite looking up in the sky, it said a yellow sunset because yeah, there was a sunset, but now it says a satellite dish on a dirt road. So basically it was because of the ability to kind of the way they trained the model and really it was actually also done there was a benchmark that they were able to beat and really achieve human parity, which really means the computer is describing images in the same way that you or I would describe images or sometimes even better. So that was kind of the research milestone. But these research Milestones are very fascinating. Organizations have them around the world all the time. What we want to do was bring them to our developers. So in record time we brought them to Cognitive Services. It's a feature of Computer Vision API that's available as part of Cognitive Services. There's a free tier, people can try it. We've also, by the way, embedded it into our applications. So in PowerPoint and Word, when you insert an image, you can do alt text to describe the image and that model is there now. Seeing AI as the application that was developed to really help people with blindness to see the world around them. And you can only imagine the kind of impact this kind of technology can have in their world. So that's a super quick overview of the research that went in and then really how we bring it to developers and customers.

I think this is amazing. Oftentimes on the show we talk about various things that, like object recognition or computer vision or other things that are either seen by a lot of people as, you know, competitions to be a part of or benchmarks to break. But really seeing how that can actually connect to benefit people in a really positive way. So impacting people in terms of the seeing AI app or helping accessibility in other ways, that's a really amazing place where AI can have a contribution. So I really appreciate you sharing that and Microsoft for really emphasizing that because it is really amazing work. How long has the seeing AI app been around and have there been impressions of the app from the blind or low vision community?

Yeah, it's been there for many years. I love to go see how many exact years, but at least the last five years, I believe it's been out there five, four years. I believe it runs from war too in the industry for disabilities, but those things are great. But really at the end of the day, I think it's our ability to provide something to the world and improve people's lives. It really goes to our mission of empowering people, all kind of people, to do more with what they have. So it's been there for a while. It has good adoption. Shakib, who created it, is blind himself. He works at Microsoft. And we're just super honored to have this as one example of us kind of helping customers do more. The other example you might not know is something we have called Immersive Reader, which is also by the way available as a cognitive service that developers can put in their applications. But if you download Edge browser, you can right click on the and it says read aloud or Immersive reading. So it, it does A lot of different things. It allows you to blow up the text, allow you to focus on separate parts of the text, of course, read the text to you and so forth. But that's another thing we've been doing to use AI to help people with disabilities just do their job really well.

Love hearing that. We talk about these kind of use cases pretty regularly on the show. The general classification of AI for good in different ways and making the world a better place. So those, those are pretty fantastic. As you were going through that a moment ago, it made me wonder. You were talking about research and stuff and within Microsoft, as new research is coming out either internally or external to the organization and everyone's consuming it and trying to understand the latest advancements that are going on. How do you get from that point where you read a scientific paper and at some point somebody realizes that there is a service in Azure that you can use, you know, utilize that and it becomes something that people like us will then use through Azure Cloud. What does that process at a high level kind of look like to get from cool new thing that we're reading about to something that you're now offering the public?

Yeah, look, some of this work is what I call loosely coupled because if you try to like orchestrate it too finely, then things break down because a lot of research is exploratory and it is kind of like should not be tied to commercial goals directly. Right. And the area of just pure research and pure learning. So we have a dedicated research arm there, but this totally connected to our product side. So an AI is a great example and quantum computing is another example. And there's some spaces where really like cutting edge research has a direct line into our engineering teams where we actually always looking to see, especially when we look at vision, speech, language and what we started called decision category, which is where the computer is making a decision or AI algorithms making decision, where something is not just classifying it, but actually making a decision and recommended something on behalf automatically, not just classifying it. Again, these are kind of four, let's say domains of research also that we really invest in. So our research teams stay super plugged in with our engineering teams or rather our engineering team stay super plugged in with their research teams. And as these things happen, we're able to kind of, we have established mechanism to take kind of literally raw research and have pipelines built to kind of put them into production and processes to do that. And one other thing we've done a lot is have a responsible AI thread throughout this area because you know, it's easy to say we have principles of responsible AI and we of course do. It's also like, hey, was the data collected with consent? Was when the data was collected for this kind of research breakthrough or before we productize it? What other ways we need to augment the data set to make sure it's not biased or does have inherent biases and all that stuff. And there's a pretty very big ethics committee that works through from research into kind of releasing it as a product to the world where we augment anything that might have been missed in the research phase. Because that's really not the focus oftentimes, as you can imagine, is loosely coupled, but it's connected. And then we have set of processes that walk us through that enables us to kind of bring these things to market really quickly but do it in a responsible fashion too.

I think it's pretty interesting that you've had the chance to work for many years on these incubated projects within Microsoft. And it strikes me that this might give you a unique perspective in terms of the future trajectory of some AI related technologies and maybe things on the horizon. So I'm curious to know a little bit about what excites you about the future of practical AI or AI platform or the mixed reality space and some crossover there. What are some of the things like when you're falling to sleep at night, what are those things that are kind of running through your mind that you're dreaming about?

This is my favorite question when we ask people this because we get some really cool answers coming back. So no pressure.

But yeah, we've never had anyone pull out their dream journal and read something off.

But since you said let me pull it out, look with AI is a fascinating space because I think it's not just an appearing incubation space now. It's getting more and more mainstream, many other technologies and even when it compare other technologies as well, gaining mainstream adoption faster. And it's really happening. I just believe because they're what we call digital native companies especially that have been built on the foundation, they've been built in the cloud and have used AI and ML as a way to really separate themselves. So the things that came excited to be more precise is we're seeing companies that you would not imagine using machine learning and doing it not just for descriptive analysis, but actually running ML pipelines, building thousands of models. We have a company called AJ and Australia, they make windmills and now they're building a machine learning model per windmill, right? Not just one generic Machine learning model one Nestle is another very commonly known name. Right. But it's not a born in the cloud company. It's a very old company that has really adopted technology to, to really help them out innovate other their competition. And they're also by the way using Azure machine learning in this case to do pitching the email phishing attacks to detect them because they have a huge employee base and they're using very sophisticated ML pipelines to not only train the model once but to actually look at all the emails that come in and to kind of then stack rank them in the risk factor and then to run more sophisticated machine learning models to really reduce the amount of things that are happening. Scandinavian Aliens is another one. You know, they use machine learning to do fraudulent fraud detection activities and when they want to say hey Daniel Chris, you're a loyalty member but you did something bad, they need to be very sure before they ask you that question. And they're using again machine learning and responsible machine learning capabilities to make sure the models are built. I have super high confidence they're using model interpretability and all that thing. So what really gets me excited is a we have many more mainstream companies using AI, not just in the lab, but literally doing thousands of models in production and you know, really revolutionizing the businesses now, you know, so it's really not in the hands of let's say Google, Microsoft, Apple only. It's really, you know, what I would call, you know, I'm sure Lockheed Martin, you know, also does a bunch of machine learning, although I don't work with them anymore a little bit. Right. But this mainstream company is benefiting from this a lot. That gets me excited every single day. The second thing is the space is innovating at a super fast pace. Right. So we have gans, we have all these new techniques coming in and it almost seems like what was cool two years ago is quite passe now we all use Scikit Learn, but nobody really talks about it that much in the sense right now we're all about putting deep learning models in production at high scale and just pace of getting new AI research into the world. It's just super fascinating. And what keeps me excited is I think everybody in this field is AI should never be in the hands of a few and increasing it's not now as it does get to reach more people. You have to do it in a very responsible fashion. And that word cannot be used loosely in the sense that anybody can build a model using automated machine learning but unless you give them the ability to look under the hood to understand did the data have any issues? Were there any biases built into the data? Would they have high cardinality issues just because the model is 99% accurate? If it's classifying the wrong thing and it's missing nine of the 10 cancer detection, it's a pretty damn bad model. But being able to now bringing the sophistication of being a data scientist to people who know data scientists, not easy. And we should never trivialize a using API or automated machine learning. Have a model and you're lucky you're happy and go deploy it around the world. Now the fun part is, okay, now that we simplified some aspects of machine learning, how do we make sure it's applied in a way that people are applying it, fully understand the implications? And that's a super exciting space for me and what the industry is doing, what Microsoft's doing, not just simplifying it, but making it practical at the end of the day so more people can benefit from it. And you know, it's a never ending cycle. New research comes in, taking to market and doing in a way that most people can benefit from it. Controlling the hype around this topic, but really driving the, you know, benefits for our customers. So it's a long winded answer, but it is kind of making sure AI is applicable to a large set of customers but in a practical fashion, not in a buzzwordy fashion. Not just saying one click and boom, you have a big model and the second one is just the pace of innovation and new techniques coming in and the opportunity it offers to customers even who don't have data to do machine learning. Like things like reinforcement learning techniques when applied correctly can solve some of these kind of issues. So it's really making this not a special topic but really widely useful topic is what excites me and keeps me up.

Awesome. Well, I don't know about you Chris, but I'm inspired.

I'm psyched right now.

If you can't see a really great way to end our conversation here, I think our listeners will be equally as inspired. I hope that they get out and try some of these things in Azure Cloud. We'll make sure and include links in our show notes to all of those things that we discuss. So make sure you get out there and try some some things get hands on and yeah. Thank you so much for joining us Brod.

Thank you for having me and thanks for having a great podcast.

Thank you.

Come hang out with Daniel, Chris and hundreds of other AI practitioners in our community. Slack. It's a cool place to be. Not a lot of noise, some great signal, and best of all, it's totally free. Check it out@changeaw.com community and don't forget to follow the show on Twitter for AI news and links, highlights from past episodes and more. We are PracticalAIFM. We'd love to have you following along. Thanks to Daniel and Chris for hosting Practical AI week in and week out to the mysterious Breakmaster Cylinder for the excellent beats you hear on all changelog podcasts. To our sponsors who have our back Fastly, Linode and launchdarkly, and to you for listening. We appreciate your time and attention. That's all for now. On the next episode of Practical AI, we are running Daniel's panel at the R Conference talking AI for good, so stay tuned for that one hitting your feed next week.