UBER and Intel’s Machine Learning platforms

Bandwidth for changelog is provided by Fastly. Learn more at fastly.com we move fast and fix things here at Changelog because of Rollbar. Check them out@rollbar.com and we're hosted on Linode servers. Head to linode.com changelog this episode is brought to you by DigitalOcean. They now have CPU optimized droplets with dedicated hyper threads from Best in class Intel CPUs for all your machine learning and batch processing needs. You can easily spin up their one click machine learning and AI application. This gives you immediate access to Python 3R, Jupyter Notebook, TensorFlow, SciKit and PyTorch. Use our special link to get a 100 credit for DigitalOcean and try today for free at the.co changelog once again.co changelog. Welcome to Practical AI, a weekly podcast about making artificial intelligence practical, productive and accessible to everyone. This is where conversations around AI, machine learning and data science happen. Join the community and slack with us around various topics of the show@changelog.com community. Follow us on Twitter. We're PracticalAI FM and now onto the show.

Cormac, thanks for joining me here at O'Reilly AI it's great to have the chance to talk to you. I know you just got out of your talk a little bit earlier. You talked about portability and performance in embedded deep learning. Can we have both? So I want to dig into that a little bit more later. But first I'd love to hear. I know you work and help lead the Movidius group at intel and I'd love for you to just kind of let the audience know what Movidius is if they haven't heard about it, what you're doing and what you're working on now.

Yeah. Hey, thanks Daniel. And yeah, good to talk I guess. Yeah, my name is Cormac Brick. I lead kind of VPU architecture at Atmovidius as part of Intel. VPU for us is a kind of visual processing and that's the key engine we have in our product line. So yeah, I kind of lead that architecture. And at Nvidius we're very passionate about machine learning and computer vision at the edge. This is something we've been at for a long time, going back five, six years even before we were part of intel and we have multiple products now in the field. And yeah, we've learned a lot as of course as a result of all of that interaction with customers over the years. And yeah, the goal of the talk this morning was to really kind of reflect back some of that knowledge, as in what have we learned about tuning neural networks for embedded silicon and then also tuning embedded silicon for neural networks to kind of just reflect back what some of the realities are when you go to take a network to the edge, what's kind of really required to make that run really, really well.

Awesome. Yeah. So just to kind of dig into that a little bit deeper, when you're talking about customers that are tuning neural networks for the edge on things like VPUs, which you mentioned, what are some of the kind of customer use cases around this and people that have found a lot of value in going down that road?

Yeah, sure. So we at Nvidia, we have customers who are engaged heavily in things like digital security and kind of smart city type use cases. We're really making more intelligent cameras. That's one big use case. We've also shipped a lot of products on drones. That's another use case, as well as a lot of things around robotics and smart devices and camera devices as well. So there's things like the Google Clips product that's on the market now that uses our Myriad 2 silicon. A lot of the DJI drones have used the Myriad 2 silicon as well. And they have things like you can wave at the drone, using your hands to control it, and then put out your palm and the drone can land in the palm of your hand. So really, really compelling use cases that have been enabled through our silicon through the use of, I guess, both vision and AI kind of working hand in hand.

Awesome. Yeah. And just to kind of confirm that I was actually at GopherCon last week and one of the keynotes, I think on the second day or something, they used a drone with a Myriad chip in it to do some facial recognition and all of that. It was some cool stuff. So let's kind of dive into a little bit more about what you talked about. Is there in these types of use cases where you're wanting to run your neural network in a drone or in a camera or whatever it is, explain a little bit the tension between kind of portability and performance that we've seen in the past and the state of it now?

Yeah, sure. So I guess what we've seen is a lot of, you know, if you go on Arxiv or if you go to NIPS or ICLR or these sort of conferences or CVP or like leading academic vision conferences, we'll kind of find there that people are. There's a lot of work being done to kind of optimize neural networks for things like kind of ImageNet or MSCOCO or kind of academic data sets. And that's awesome in terms of pushing the envelope of the field and advancing the science and it's moving super fast. So then typically when embedded engineers will start off a problem, they have access to that sort of research and these sort of models and then they kind of want to do something that's going to work for them and their device. Right. And one of the things they would find is a lot of the models that are available out there were tuned on imagenet, which is great at recognizing a thousand classes of images and it can differentiate one sort of whale from a different type of porpoise and this sort of stuff. Right. Very, very fine grained classification. Yeah, not so much in the real world. Right. We have different problems to solve. So then in the real world we may care about, hey, my robot wants to be able to recognize 100 common objects found in the home or with this sort of, in this security camera. We want to be able to recognize these different types of objects that are happening. Yeah. So different problems. And often those problems are simpler than the thousand class problem from imagenet. So one of the things we were talking about this morning is using techniques like model pruning and sparsification to, you know, if you're doing what we would call domain transfer. So you go from your 1000 class problem, you know, let's say if you were taking Resnet50 and you're now retraining that for your home robot, which wants to recognize 100 images, you'll find that you can get away with a much simpler network with less representational capacity to solve that 100 image problem than the one you started off in the 1000 image problem. So we were sharing some results and some techniques specifically around channel pruning, which is very, very powerful technique when you are doing domain transfer to a simpler problem domain. And also looking at techniques like sparsification, which is introducing more zeros into a neural network, because that's great in terms of, on platforms that support memory compression of neural network models, it'll enable those models to run much more faster in bandwidth limited devices such as those typically found on the edge.

Awesome. So in terms of like, let's say that I'm working on, I'm working on one of these robotics problems or whatever it is, and I'm using a neural network and I want to pursue some of these methods to kind of prune it down or optimize it for that setting or for that architecture. What's kind of the process and the barriers that I would face as of now going into that. And what's kind of the state of the usability of these tools and that sort of thing?

That's a great question because for sure we were presenting a lot of work this morning saying hey, we're able to take a network and do this sort of pruning and quantization and sparsification and then go from 8 bit weights to 4 bit weights and this sort of stuff. But you know, straight up today, pretty non trivial to repeat the results that we were kind of showing this morning. Right. To bridge that gap, you know, working. We now work in intel as part of the AI Products Group. As part of the Intel AI Products Group, there is an open source project called Distiller. It's one of the resources listed in my slides, I think on the final slide. And I believe they'll get posted to O'Reilly at some point.

Yeah, we'll put them in the show links here as well.

So yeah, there's a link to something on GitHub called Distiller. And there. One of the things we're doing is if you went back maybe kind of 12 months ago, you'd have found, oh, this is an awesome quantization technique that somebody published, some grad student published a Pytorch fork or something with this. And then here's something else that was available in TensorFlow for quantization and here is something else that was available in a different framework. What we were doing is really kind of taking all the of those techniques that are available in a fairly fragmented way across the Internet and trying to put them under one roof in a way that's kind of a little bit easier to access. And that was the goal of the Disturber project is really to show that. And it's an ongoing project at intel within AIPG to have this kind of set of tools. So they were available in Pytorch and that's great because Pytorch can export to OnNX, which is then widely available. But in addition to the work we're doing, it's entirely appropriate though to give a shout out to the work the TensorFlow team are doing. So there's under TensorFlow contrib. Yeah, there's a bunch of useful tools there on both quantization and on pruning as well. Right. And there's pretty strong ecosystem there also showing a variety of techniques.

Okay, yeah, so it is at least to a point where I could get a model off of some repository maybe in Pytorch or wherever and have some tooling that's publicly available to prune that data down for certain architectures?

Yeah, yeah.

What about prepping the model for certain, maybe specialized hardware you mentioned, like VPUs. And I know there's a lot of other people pursuing things around, of course, GPUs, but also FPGAs and other things. What is kind of the state of the art? Are these kind of pruning methods and all of that tied into that world or is that something totally separate?

Yeah, and that's also a good question. And it was one of the kind of the goals of the talk today was to show that, hey, you know, here's kind of four key techniques that you can use that will work well on any hardware and on some hardware will work extra well. But if you employ these techniques, you're not going to hurt your model's ability to run across a broad range of silicon. Right. So and those techniques specifically are kind of model pruning, sparsification, using fewer, using, you know, quantizing a network to eight bits and then doing further quantization on weights to use kind of a lower bit depth. So if you employ those kind of four techniques, you will still have a model. If you take a model and you represent it in onnx or in tensorflow, you'll still have a model that can work well on a wide variety of devices. But on some devices it's going to work extra well. Right, because different silicon will have different abilities to run quantized models at varying degrees of acceleration. And also different silicon will of varying degrees of, let's say, weight compression and technology. And even in extreme cases for sparsity, there's some silicon out there that can process sparse networks directly and in an accelerated fashion. So against a variety of silicon, you can employ these four techniques and get really, really good results across a range of silicon and even better results in some silicon. So that was the core point. But to answer the second part of your question in the final slide, we're making the point as well that hey, if you set out to have with a single network and you know the piece of silicon you're running on, absolutely. There's other techniques you can employ to really fit that piece of silicon as best as you can to really make this one network shine on this combination of this network and this silicon. And there's been some very interesting work published on that in the last couple of months and it's a pretty, pretty hot research topic now is, is showing how to like using, you may be Familiar with kind of automl. Right. So being able to use that type of techniques to kind of. To refine a model or to learn a model that works really, really well on a particular version of silicon with these types of performance cliffs and trade offs. Yeah, so it's a. That's a pretty active area of research. That's pretty interesting.

Awesome, awesome. And I know that one of the things that I've appreciated as kind of like as I'm hacking on things at home is that a lot of the stuff that you've come out with through Movidius makes it really easy to experiment with neural networks on a lot of different types of devices through the Neural Compute Stick and other things. I was wondering if you had any interesting stories or customer experiences that you've heard about of people enabling new sorts of things with these devices.

Yeah, we really enjoy the experience of launching the first version of the Neural Compute stick based on Myriad 2. And it was great to get out there and meet lots of developers. And also when we launched that, I guess it was kind of. We announced it some time before and we really launched it then at CVPR last year. Yeah, it was great to see what everybody was doing, but also to kind of show them, hey, AI at the edge is possible. Right. If you go back 15 months, people or two years ago, people really associated AI with the cloud. So our first goal was to kind of break down the perceived barriers.

And.

For more people to be able to use AI and to see, hey, AI at the edge is possible.

Right.

So that was our initial goal and it was great experience. Very enjoyable talking to all the developers. A couple of things we've seen, we've seen people use this. One of the software ambassadors for intel use this to do a prototype kind of water filter. So kind of taking the guts of a microscope, putting that up to a camera into a Raspberry PI with Nvidia's Neural Compute stick connected and being able to show that you could actually use this to detect water impurities. So to have an entirely offline water impurity detection device that could be used effectively like on premises at the edge with no cloud connection or anything like this. Super cool idea. Right. And be able to show that that's possible. Equally, we have people putting them on a drone to detect sharks and the water. Also doing kind of prototype medical imaging to detect melanoma on the skin. Also kind of driven by image classification. Yeah, so they're just a few things, but there's been a lot of other fun projects posted on GitHub and I don't have a link to our ModelZoo site and example site, but I can provide you with them for the blog page also.

Awesome. Yeah, we'll make sure that gets in our show notes for sure. Yeah. Well, I appreciate you taking time again kind of to wrap things up here. I was wondering, you know, from your perspective, since you've been working in this space for a while, what can we look forward to, you know, over the next couple of years with performing AI at the edge? What are you excited about and what do you think we'll see for the next couple years?

Yeah, I think. I think we're definitely going to see a lot more silicon become available, both from, both from Nvidia's intel, also from a bunch of competitors. And I think that's going to be really interesting as inference silicon. You know, there's kind of metrics business people would track, like the number of like, ops per watt we can deliver or the number of ops per dollar we can deliver. And we'll expect both of those metrics to progress at a really, really fast pace over the next number of years. And if I look at what people are able to do with the first version of the neural compute, stick with the capabilities that has and while I can't disclose product roadmaps with some visibility of the type of things, things we're going to see in terms of the volume of compute that various people can bring to market at much lower price points and much lower power points, I'm really excited to see how that's going to play out and the type of things people will do with that. I think it's going to be very exciting space to watch in the next few years.

Awesome. Well, thank you again for taking time and enjoy the rest of the conference.

Thanks, Tyler.

I'm Tim Smith and my show Away from Keyboard explores the human side of creative work. You'll hear stories, sometimes deeply personal, about the triumphs and struggles of doing what you love.

Jumping off into the abyss is kind of my skill. And so I'm not saying that it's not scary. I'm saying that perhaps my skill is just not being able to estimate how scary it will be.

New episodes premiere every other Wednesday. Find the show@changelog.com afk or wherever you listen to podcasts.

Well, thanks for joining us, Mike. It's great to chat with you and meet you here at O'Reilly AI. I've heard about Michelangelo a bit little, this ML platform that you guys have developed at Uber, and I'd love to Hear a little bit more about it, but first give us a little background of who you are, how you ended up where you are.

Yeah, thanks. Happy to be here. Yeah. So I currently am the product lead for ML infrastructure at Uber and that encompasses a lot of things, most notably the Michelangelo platform. A little bit of background on me is I'm an electrical engineer by training and I out of sight school. I worked at Google and one of the kind of places I got my ML chops, so to speak, which is weird to say, is I worked on the ads team at Google, specifically the ads auction group. And I was the product manager for all of the ML signals that go into the ads auction there. So these really like real time, high scale, super productionized ML systems that predict if you're going to click an ad and if you, if you're gonna, this ad's gonna be relevant and stuff like that. So that's kind of like where I learned how to do ML. Right. And probably best in industry in terms of productionized machine learning. And then about three years ago I joined Uber where we started the Michelangelo, which is not named after me in any way.

That's a shame.

Yeah, I get that question all the time. We started the Michelangelo platform, which helps people, which helps data scientists and engineers across the company build ML systems, kind of prototype, explore ML systems, build them and then deploy them into production and serve predictions at scale.

Yeah. So why, if you're in a company that's trying to build up their AI presence within the company, why would they need an ML platform? Why isn't like jupyter notebooks everywhere just fine for people?

One of the things so kind of like the state of Uber's ML stuff about three years ago was that a lot of people were trying to do that. Right? So there was a lot of people, you know, grad students learned how to build their ML models in their grad school classes and whatever. And they have their own ways to do it. Everybody has their own, I use R, I use Python. And what we saw was that people were trying to either trying to productionize like an R model and run an R runtime in production at high low latency, which is just like very challenging and kind of people will cringe when they hear that today. Secondly, you would see like teams that did have data scientists that did have engineer support. They would build up these bespoke towers of infrastructure at a per use case basis that would tend to be less well built just because they had lower resources but like duplicative of different pieces of Infrastructure that people would build to serve these models in production across all the different ML use cases of the company has. And then kind of the scariest is people just wouldn't get started at all because they wouldn't have a way to. Some people wouldn't have a way to get their models into production. So we saw the opportunity to build a common platform to help people have a unified way to build models and to, and this is the trickiest part, put those same models that they prototyped on into production to make those predictions and along the way bring a lot of data science best practices built into the system. Reproducibility, common analyses and all of that kind of like versioning and all that kind of good stuff that is kind of like these data science best practices that aren't yet really well established. We have a lot of really well established software engineering best practices that everybody knows CICD and version control and stuff like that. That stuff's not as well well appreciated in the data science community. And it's just because a lot of this work is new and it's not like these guys don't understand the importance of it, but it's just like the best processes and the best patterns for building this stuff have not yet, we've not really converged on those yet. So kind of spent a lot of effort to focus on where we think this stuff is going to go and to help build the tools to enable to like empower data scientists to kind of do the right thing from the beginning.

Awesome. So how many people are using Michelangelo at Uber?

That's really, that's really hard to say. I would say we probably have more than. So this platform supports machine learning use cases across the company. So everything from like fraud related things to predicting how long it's going to take a car to get to you, to even like Uber Eats, like ranking dishes in the Uber Eats app. All of the main ML stuff runs through this platform now. But and this is just like an interesting kind of platform development challenge is we have a lot of people who kind of use it. They're like, hey, I kind of want to build an ML thing. And they dabble and explore a couple little models they want to make, but maybe they never end up fully deploying that model to production. And so it's kind of tricky to say like how many actual use cases do you have on this system? We know it's well over 100, but it's hard for us in the platform to say is this something that this team is just using this as an experiment or is it fully productionized and deployed across the whole company? And that's just an area that we've just under invested in a little bit, but we think there's a lot more to do there.

As you've seen people start to use the system, are there features of it that you thought that kind of surprised you in the sense of how people rely on, relied on them or things that people needed that you didn't expect that they would need or other things?

Yeah, that's a really good question. And I've been reflecting on this a lot recently and I'm the product manager so it's kind of my job. But the thing that I would say that kind of has gotten disproportionate adoption given our maybe even like under investment into this, where we still could do a lot more in this space, but our users just adopted this overwhelmingly and they love it, is our feature store which is part of the platform and what that allows. So you know, common problems for managing features related for ML workflows are that you have to clean your data and transform your data and combine it all and also historically for into a training data set so you can train your model. But then once your model's created, how do you do all of those same transforms in the same way, the same process pre processing to that data in real time when you deploy your model? So there's kind of this dual type of ETL that happens in different compute environments that's really tricky and possibly on.

A variety of resources.

Yeah, and I mean we see a lot of like vendor solutions here, but I feel like we don't see anybody really tackling that kind of stuff. And I think it's partially because it's not sexy at all to work on that stuff and also because it's just super hard to do properly. And we've provided some nice ways for people to define their feature transforms to the platform and then be confident that those transforms will happen consistently across both computer environments, real time and offline. But I think the other interesting thing is we saw, let's take the Uber Eats world for example. They probably have more than 10 different models that they use to predict to rank dishes and whatever they do. A lot of those models use the same kind of features. And before this feature store, data scientists didn't have any insight into hey other people that were working on similar problems, what kind of feature pipelines had they built. And then when this feature store came along. Now when a data scientist wants to start a new model, they can Just look and see what features exist that are relevant for me. Let me just like start including or start off warm, start my model exploration process with the X features that are most relevant to this problem from the beginning. So there's a whole new element of collaboration, visibility, feature sharing that was previously not there. I really don't see much solutions in that space in industry today either. So I think that's a really promising area.

Cool. Yeah, I look forward to hearing more about that and definitely if you publish anything about that, we'll be happy to post that on the show links here. Yeah, the other thing I was curious about, just from the fact that you mentioned before that the incentives for data scientists are kind of different and not always aligned with producing production ready models and all of those things. How do you build up a team to build a ML platform where really you kind of need software engineering experience to be able to build something that's production ready but you need the knowledge and the expertise around machine learning to be able to understand what to build. So it's going to be relevant to the people you're building it for.

Yeah, so I think one of the nice things is that we've had a little bit of the leadership in our organization has been a relatively forward thinking to be willing to fund the development of an ML platform much earlier than I think is common in industry. And that's allowed us to get it wrong a couple times before we got it right, but we feel like we really got it really right now. And there's like a tension between data scientists want this nimbleness and flexibility throughout their exploration and prototyping stages. And you know, if you think of any productionized system, it's super stable. And so how do you kind of accomplish both of those constraints? It's a challenge. And so what we, some of the design philosophy that we're taking and this is, is always developing is we're trying to allow data scientists to work within our system using the tools that are most relevant for them. So we'd love for them to work in jupyter notebooks and write all their models the way they normally would. We can provide some helpful APIs for them, for example the feature store stuff to pull in their data so they don't have to re implement a whole bunch of work that already exists in terms of like enterprise intelligence that's already been done. But after a certain point when the prototyping stage is complete through. If you think of this machine learning life cycle where it's like now I want to actually Use this in production and maybe it doesn't mean you're going to launch it to the whole company and it's going to be and you're done with the project. Could just be like, I want to experiment with this on live traffic. We focus on making it relatively low activation energy to take your prototype and transform it into something that can go into these productionized, well engineered hardened systems that we can be confident will be stable from a systems perspective. And we still want to give data scientists the ability to monitor these models that are in production for not just systems issues like whatever applies to typical microservices, but also the data science monitoring. How accurate is this model over time? Are there any model drifts, stuff like that? And so there's a story for data scientists throughout the life cycle and a story for engineers throughout the life cycle and the balances and the challenge is like how do you balance between those at the different stages, taking into account all of the priorities for both stakeholders throughout.

Awesome. Yeah, that gives some great perspective. Well, to kind of end things out here, are there places online where people can find out more about what you guys have done and maybe also some things that you put out there that, that you might want to share?

Yeah, that's a good question. We've published a blog post about Michelangelo and I think October 2017 and it's pretty easy if you just search Michelangelo ML platform on Uber on Google rather you can find that. And we've published a lot of other pieces about related ML work we've done and I think we're likely to in the near future open up the kimono a little bit more on Michelangelo. So stay tuned.

Cool. Awesome. We'll look forward to that. We'll take. Thanks for thanks for joining and enjoy the rest of the conference.

Appreciate it.

All right, thank you for tuning into this episode of Practical AI. If you enjoyed this show, do us a favor, go on itunes, give us a rating, go in your podcast app and favorite it. If you are on Twitter or a social network, share a link with a friend, whatever you got to do. Share the show with a friend if you enjoyed it. And bandwidth for Changelog is provided by fastly. Learn more. Com and we catch our airs before our users do here at Changelog because of rollbar. Check them out@robot.com changelog and we're hosted on Linode Cloud Servers. Head to lino.com changelog check them out. Support this show. This episode is hosted by Daniel Whitenack and Chris Benson. Editing is done by Tim Smith. The music is by Breakmaster Cylinder, and you can find more shows just like this@change law.com when you go there, pop in your email address. Get our weekly email keeping you up to date with the news and podcasts for developers in your inbox every single week. Thanks for tuning in. We'll see you next week.