A short YouTube course announcement (May 21, 2024) introducing a new Qualcomm-led course on on-device AI. Andrew and Krishna Sveider (Senior Director of Engineering, Qualcomm) summarize why on-device inference matters, outline four core technical lessons — model capture/conversion and compilation, hardware acceleration and numeric validation, quantization for ~4x speed/size gains, and Android integration — and invite learners to enroll. The transcript emphasizes privacy, latency reduction, heterogeneity of mobile hardware (hundreds of device types), and Krishna’s decade of experience deploying many models and apps.
Chapters
Our server does inference in the cloud and it returns results to you. For example, the recommended system responsible for ordering content on your social media feed, or the shopping or movie apps that recommend products and movies most likely are doing inference up in the cloud. But there are many use cases where you probably want your AI app to run locally on your mobile device. For example, when you take a picture detecting faces to focus on stabilizing the image and so on are all done locally on your device. Some of your phone's voice assistant capabilities are also done locally. Let's say you want to build this mobile app that does real time semantic image segmentation on your mobile device. In this course you learn all the concepts needed to make such an AI model run on your phone. In fact, modern smartphones today can have hundreds of AI models running on device, including real time semantic segmentation, skin tone mapping and scene understanding while just capturing a picture. Between smartphones, security cameras, laptops, cars, drones, other robots, ar, VR headsets and so on, there are billions of devices where one can deploy AI models locally. I'm delighted that Krishna Sveider, who is Senior Director of Engineering at Qualcomm, will be teaching on all this. Krishna has been doing on device AI work for about a decade and has helped deploy over a thousand models directly on many devices, and over 100,000 applications have used the tech that he and his team have built.
Thanks Andrew. I'm really excited to teach this course here making your AI app. And this also allows us to create some excellent personalized experiences because your data remains there, your model remains there, and that is the power of pound device AI.
So in this course you learn how to take a previously trained model and get it to run on device. One key step is model conversion where you convert a trained model from say Pytorch or Tensorflow to device runtime. You take the model and you would freeze it in the form of a neural network graph. Then you can compile that graph to run on the device. Mobile and edge devices have heterogeneous processing units of CPUs, GPUs, as well as Neural Processing Units or NPUs. NPUs are much more power efficient than CPUs, but have less software flexibility and can carry out only certain operations. So if you know which devices you are deploying to, there are optimizations you can perform to tune to that device. But because of the newest brands and models of mobile phones out there right now, a mobile app might end up running on more than 300 different device types. Model compression and variations in hardware can cause variability in Model compression I used to carry four smartphones with me everywhere because I was testing my team's apps. I'm now down to carrying two smartphones. But with cloud based testing you can now automatically test on hundreds of devices.
In this course you will learn how to deploy models on device in a way in which you can reduce latency, improve efficiency and preserve privacy. You will deploy your first model on device with just a few lines of code. The model will do real time segmentation on the device and there are four main concepts you will learn in this course while deploying a model. The first is how to capture the neural network as a graph that's portable to another device, how you can compile it for a device. You understand how to deploy it with hardware acceleration and then you will learn how to validate the numerics of the hardware acceleration with that of the cloud. Finally, you will learn how to quantize a model and improve its throughput by nearly 4x while also reducing the size of the model by 4x. And you will learn how to integrate this model into an Android application that runs entirely on device.
Up until now a lot of AI inference has been done in the cloud. I think edge AI or on device AI deployment is also also rapidly taking off. So please sign up for this course and learn how to make on device AI work for you.
