AI in the browser

Bandwidth for Changelog is provided by Fastly. Learn more@fastly.com we move fast and fix things here at Changelog because of Rollbar. Check them out@rollbar.com and we're hosted on Linode Cloud Servers. Head to linode.com changelog this episode is brought to you by DigitalOcean, the simplest cloud platform out there and we're excited to share. They now offer dedicated virtual droplets and unlike standard droplets which use shared virtual CPU threads, their two performance plans general purp CPU optimized They have dedicated virtual CPU threads. This translates to higher performance and increased consistency during CPU intensive processes. So if you have build boxes, cicd, video encoding, machine learning, ad serving, game servers, databases, batch processing, data mining, application servers or active front end web servers that need to be full duty CPU all day every day, then check out Digital Ocean's dedicated virtual CPU droplets. Pricing is very competitive starting at 40 bucks a month. Learn more and get started for free with a 100 of credit at.co changelog again.co changelog. Welcome to Practical AI a weekly podcast about making artificial intelligence practical, productive and accessible to everyone. This is where conversations around AI, machine learning and data science happen. Join the community and slack with us around various topics of the show@changelog.com community. Follow us on Twitter. We're @practicalai FM and now onto the show.

Welcome to another episode of Practical AI where we try to make artificial intelligence practical, productive and accessible to everybody. My name is Chris Benson, I'm Principal AI Strategist at Lockheed Martin and with me today is my co host Daniel who is a Data scientist at SIL International. How's it going today Daniel?

It's it's going going great Chris. It's a beautiful fall day here in in Indiana and can't complain at all.

Cool. As we are looking into the fall here, I'm getting excited I guess. A couple of weeks after this episode comes out I will be at Nvidia GTC DC in Washington D.C. and I just wanted to say if any of our listeners are there, I'll be walking around a good bit of the time and a Practical AI T shirt with a jacket and if you happen to see me I hope you'll come up and introduce yourself and say hi.

Sounds good. I can't wait to hear what you learned there and hear about some of the content that's presented. I'm sure it'll be good.

Sounds good. Well, we have a pretty good episode I think lined up today. We are going to be talking about artificial intelligence in the browser.

Oh, exciting.

I know, I'm pretty excited about it and I know both of us have done over time a fair amount of webinar development separate from the AI stuff. So this is the episode where we get to start combining them together, hopefully. With us today we have Victor Divia and he is a research engineer at Cloudera's Fast Forward Labs. Welcome to the show, Victor.

Well, thanks a lot. It's really good to be here and I totally look forward to discussing machine learning the browser.

Fantastic. Well, I guess if you would start off by telling us a little bit about yourself, kind of how you, you know, how you got into this area and kind of how you found yourself arriving at Cloudera Fast Forward Labs so that you could start this interesting line of work.

Sure, absolutely. And so I could talk about it in terms of my educational background and then ease into the whole professional track.

Sure, whatever works.

Yeah. And so my, my background is a mix of computer science, that's software engineering and a bit of human computer interaction and more recently applied artificial intelligence. I have a master's degree from Carnegie Mellon University that's focused on software engineering and some management courses. And as part of that I did this thesis on building a web based public key cryptosystem that was introduced by my advisor and that was a nice experience. It's just that there was a lot of time spent implementing cryptographic functions in Java. And after that I felt it will be nice to move away from just the technical aspects of computer science and also look at some of the human aspects of computer science. And right after that I made an interesting choice. I moved to Africa, Lagos, Nigeria to be specific. And I started a company focused on making software focused on the African market. And I did that for about a year. And as part of that I also taught at a university in Lagos, Nigeria through some project that was co sponsored by MIT and Google. And as at that time I figured out, you know, I'm really interested in human aspects of computer science. And I had all this experience building software tools within the framework of a startup and also teach at a university. And then I decided to do a PhD and so my PhD was in information systems. I did at City University of Hong Kong and it was the main focus was quantitative user behavior studies. And at some point during my PhD I had the opportunity to do an internship at IBM Research and that's kind of where I got into AI. And so I interned with a group called the Cognitive Cognitive Environment Lab at IBM. And most of what that group did was trying to figure out good user experiences for applied machine learning. So we spent time taking APIs built by other research groups. So APIs around speech to text, text to speech and computer vision. And our goal was to use these tools and build them into interactive, in some cases room scale experiences. And that's kind of where all of my interests with AI kind of started. I started out applying models and after a while I spent time implementing some of These models in TensorFlow and Keras and essentially made the transition to start applying some of these custom built models to new problem spaces. And so I was at IBM Research for, as a postdoc and then a research scientist. And earlier this year I joined Cloudera Fast Forward Lab as a research scientist. And so it's a bit of a journey, but that's kind of how it all went down.

Yeah. So when, when you're talking about like user interaction or interfacing with AI, are you mostly talking about the sort of experiences like in Gmail like Autocomplete or Voice to Text and these sorts of things or, or even deeper in terms of helping a user kind of understand AI or use AI more effectively? How would you classify that sort of field of the interaction between humans and AI?

Right. So my work kind of cut across the two areas that you mentioned. And so one specific line of work has to do with using AI to make the user interaction easier. And so that might be in terms of reducing the cognitive loads associated with specific tasks. And good examples of that. The kind of thing you see in Gmail Autocomplete, you start typing out an email and some LSTM model recommends a few completions. And so an interesting project that I worked on in a similar line, something called Data to vs. And so with that project we designed a neural network, a sequence sequence model that could take user data. And based on user data, it will propose about 10 to 15 visualizations that made sense for that sort of data. And so the value here is that an analyst who perhaps has limited experience with authoring visualizations or writing code for visualizations, could take a tool like that, upload your data, and the model will actually generate code for about 10 to 15 visualizations that they could either accept or they could modify to get their task done. So that's one line of work. And so another interesting line of work at this intersection of HCI and AI has to do with tools that make AI more accessible and more essentially easier to use for software engineers or other type of technical or non technical users who strictly don't have a background in machine learning. And AI.

I know some people talk about democratizing AI. Is that sort of what they're meaning?

I guess yes, that's a good umbrella term to kind of describe that sort of work democratizing AI. I'm always not in a hurry to use that because that term can be reloaded and people have used it in all sorts of ways. But yeah, it's a good related term. The whole idea of democratizing AI and it has its advantages and its disadvantages, but the goal here is that if we make AI more accessible, then there are a lot of benefits that can come up. And so, for example, we want people with various backgrounds, various interests, various domain expertise actually coming and using AI tools. And that way we can kind of increase the diversity that kind of comes into this space. And one example of a project I worked on in that area was something called tjbot that was while I was at IBM. It's actually a maker kit made out of cardboard. And so it's a Raspberry PI on the inside, it's a cardboard piece that you could fold up into a humanoid looking robot. And we had actually gone ahead and written a JavaScript library that made it very easy to take a bunch of IBM pre trained machine learning models, integrate that into the bot and essentially prototype things like you would see on Siri or any other like AI enabled hardware device. And so that had a lot of success with schools, teachers, designers actually using it to actually start to use AI tools in different ways.

So got a question for you. You were talking a few minutes ago on UI about with Gmail and things and You've talked about JavaScript and I was kind of curious to even take a step back a little for a moment and ask, we're talking about the browser here. Why do you think people would want to run machine learning and AI models in the browser versus other environments? What is it that kind of drove that interest into the browser?

Right. So machine learning the browser, it's a fairly new area and I guess most of the time when I talk about it or discuss it with people, there's always a form of healthy skepticism and I think it's for good reason. And so if I'm going to take a step back just to discuss the two interesting aspects of machine learning. And so there's training. So this is the part where you create a model which is essentially a function that learns mappings between your input data and some kind of target. And essentially you get all your data collected, you have it cleaned up and you go through the process of kind of learning this function that solves a specific task. And so that's the training phase. And then the second part of machine learning is inference, where at some test time you get this model that's trained and you get it to actually perform the task. And typically we've used languages like Python, Java, C, R, Scala, Julia, and these are typically backend languages and they have a lot of nice functionalities like hardware, direct hardware access, multi threading, and it makes them work really well for intensive computations like machine learning. However, typically, most of all this has been done on the back end. There's been a clear separation between what you can do in the front end in an environment like the browser, because the browser has a bunch of limitations. And so it's single threaded and it's a sandbox environment with very limited access to system wide features. However, it turns out that there are a few benefits that kind of make this proposition interesting. And some of my favorite reasons have to do with three specific benefits. The first would be privacy, and perhaps this is the most compelling and interesting benefit that I really care about. And so if you could take a model and you could deploy that in the browser, then you could create an environment where the user data actually doesn't get down to any backend server. And I could give some examples of that somewhere down the line as we continue the conversation. The second interesting benefit, why machine learning might be interested in the browser, has to do with the ease of distribution. And so a few years ago I had a couple of friends who really wanted to get into machine learning, but they did give up because they spent a couple of days just trying to install TensorFlow. And so while over the last two years the user experience has become a lot better, there's still a lot of challenges, especially if you want to get a machine learning model or an application that uses machine learning deployed on an end user system. However, if you go ahead and do that in the browser, this is a much straightforward and much, much easier developer and end user experience. And then finally, the last interesting feature has to do with interactivity and latency. So off the bat, the browser is the web is designed to be interactive and it's really valuable for crafting rich interactive experiences. And so if there are situations where you have a model and you want to easily tailor that around user data, make changes and personalize that for a user, then the browser is a really excellent environment to interactively do all of that. These are three interesting reasons why I think it makes sense to actually explore machine learning in the browser.

So yeah, that's a great explanation. I'm wondering if along with that explanation, since we're always trying to be practical here and like you said, there's probably a lot of lists, listeners who are familiar with Python or Java or whatever it is, these sort of backend languages. Could you just kind of describe the JavaScript ecosystem a little bit? So there's like JavaScript, but then there's probably things like Node JS and other things that people have heard of. Could you kind of describe, I guess in general what those things are and how like machine learning is kind of touching each of those, or maybe it's specifically touching one thing like vanilla JavaScript or whatever it is.

That's a great question. One way to think of what you could actually do in terms of machine learning and JavaScript is to think in terms of the tools that are available today. Right now, most of that is TensorFlow JS for listeners who are not familiar with TensorFlow JS. It's a JavaScript library designed to enable machine learning in the browser and any other environment that's built with JavaScript like Node JS. And so most of my conversations around, you know, when I talk about essentially implementing machine learning in JavaScript, most of the time I'm actually referring to implementing machine learning using the TensorFlow JS library. And so with regards to the environment and platforms that are supported, TensorFlow JS allows you to build, train and perform inference both in the browser, in the browser environment as part of a front end web application. And it also lets you build, train and perform inference as part of a backend Node JS application. And so the library, one way to think of it is to think of it as having a few different installation versions. And so there's a version that could be bundled into a web application. It could be a vanilla JavaScript application where you could just in your web page, you could include a minified version of the TensorFlow JS library, or you could install it using build tools as part of a React or Vue JS application. And so that's for the front end of the browser. Similarly, on the back end you could npm install TensorFlow JS. And essentially for your backend applications built in Node JS, you could integrate TensorFlow models and you could also integrate install the GPU version of that same backend library. So it will be NPM install tensorflow gpu. And one other thing that I guess people will be interested in learning about would be around the performance of TensorFlow JS in both the back end and the front end. And so in the browser, it turns out that TensorFlow offers a vanilla CPU backend and also something called the WebGL backend. So most of us might be familiar with the WebGL standard. It's used for accelerated graphics compute. And the value here is that if you do have a GPU available in the machine through the WebGL standard, you can actually accelerate your computations right there in the browser. And so underneath, TensorFlow JS in the browser will take advantage of optimizations already implemented in WebGL, and that's how it's able to accelerate computations in the browser. Foreign. Santo Managing Editor here at changelog, the fact that you're listening to this means you are actively investing in your future. In this industry, things move fast and keeping up is hard work. Help us help you stay relevant by subscribing to Changelog weekly. We track, log and contextualize what's happening in software throughout the week and deliver it directly to your inbox on Sunday mornings. Head to changelog.com weekly to browse the archives, subscribe, and push the easy button on your continuing education. That's all from me. Once again, that's changelog.com weekly.

So, Victor, I was kind of curious. I know that you are involved with TensorFlow JS and have been using that and I was really wanting to learn about what it is and kind of how it fits in to the world of regular TensorFlow. Or does it? What's the relationship between the 2?

And so TensorFlow JS is one of the libraries of frameworks in the broader TensorFlow ecosystem, and the primary benefit it offers is allowing a developer design, build, train and perform inference for machine learning models using JavaScript either in the browser on the front end or on the back end in Node js. And so with regards to how it plays with the rest of the TensorFlow ecosystem, I like to think of it the TensorFlow JS workflows using three main approaches. The first is what I'll describe as the online workflow. With this workflow you can structure out your machine learning models. Essentially, if you're building a convolutional neural network, you would specify input layers, the convolutional layers, and your pulling layers, all of that. Using TensorFlow JS, you could train your model directly in the browser user data and then perform inference. And so clearly there are caveats around this. You probably want to do this with small models or models or models that didn't have a lot of data. So that's the first approach, something I describe as the online approach, and in.

This case the data, there's no actual data that's leaving the client device, is that right?

Yes, that is correct.

And does that kind of fit into that privacy advantage that you mentioned before?

Yes, yes, absolutely. And in this case, imagine you had user data already available on their machine and your application, the browser could get access to that data, train a model, and then perform inference without any data being sent to any backend server. So that kind of fits in with the privacy benefit. So the second potentially more common flow is something called the offline flow, where you could train your model using a large amount of data and large GPU clusters or whatever hardware you have available. And for this process, you could use TensorFlow Python or Keras models. And so just how you would train your models, traditionally in TensorFlow Python, you could go ahead and build your models, train them, your GPUs or TPU clusters, and then you could export that model, that's the output of that process. And then you could use the TensorFlow JS converter to then convert that into a format that can be loaded in JavaScript or in a JavaScript application, and then perform inference on that. So that's what I would refer to as second or the offline flow. And then finally the hybrid flow would be super similar to the offline flow, where you train offline, you convert your model and you import it. And then in the browser you could go ahead and fine tune that model using user data right in the browser. And so these are three potential flows that a developer could take advantage of when using TensorFlow JS. And I guess the interesting thing to note here is that for models that you've trained using the traditional TensorFlow Python, TensorFlow JS offers a converter software, a tool that lets you convert those pre trained models into a format called the web format that can be loaded in a JavaScript application. And so there's that opportunity to integrate whatever work you've been doing with TensorFlow Python and then bring that into the JavaScript of the web application space.

So when a user, like if I'm a developer and I'm thinking about maybe like privacy is important to me, or maybe the latency issues are important to me. And I'm thinking about which of these scenarios I should pursue, whether I want to be fully online or offline, or the hybrid situation like you're talking about, I guess part of that could be driven by the privacy concerns. But in terms of performance, like how much data or how big of a model can you Train in the online scenario versus offline. And also are some models, maybe the latest ones, that are like however many billions of parameters, maybe you can't actually, or can you optimize those and fit them into the browser to run? What are the sort of constraints with those things?

Right. So experience wise, I think most of the time people would only train models, low parameter models, small models in the browser. And I guess the reason is pretty clear. The browser is not the multi threaded high performance environment and it's perhaps not designed to train large models using large data sets, images, thousands of images right in the browser. So what I've typically seen people do if they were going to train models from scratch, and so they would use train smaller models, perhaps using tabular data with maybe a couple thousand records and not much more than that with respect to actually deploying high performance models in the browser. I guess the good best practice would be to think of model optimization during your model construction phase. And there are a few ways to go about that. And the idea is train your model offline, apply a bunch of optimizations, and this could be model quantization, this could be model compression. And the goal would be to export a model that's small enough that it doesn't hinder the web or interactive experience, and then you will typically then import that, but mainly just for inference in the browser.

So I guess you know how you've kind of covered these different ways of using it and how they integrate. Do you have any insight into maybe with, you know, Fast Forward Labs clients or anyone else you come across within the industry about how people are typically using TensorFlow JS and how they're fitting it in in real life? Kind of. Aside from the options that you've laid out, do you know what people are actually uptaking on?

Right. So there are a few interesting use cases I've seen across the community and also been highlighted by the TensorFlow JS community. And I think there's a really interesting experience application by Airbnb where as part of their user onboarding process, the user has to upload a photograph, an image of themselves and writing as part of their onboarding experience. They have a TensorFlow JS model in the browser that could look through whatever image has been uploaded and could tell the user if this particular image contains sensitive content or not. And so in some cases, users might upload their driver's license or other type of images that has potentially sensitive content. And the value proposition here would be to tell the user, I'm able to offer you this service telling you that you have potentially privacy sensitive information here. However, this data does not get to my server and I never see it.

Yeah, so you don't even have to worry about storing that. You're just providing a service to the user on their side.

Yes. So I think this is a really good and strong notion of privacy. The alternative would be that companies would typically say, oh, we have your data, we don't store it, we see it, but even if we store it, we will only use it appropriately. And so I think it's a stronger notion to say we're able to offer you a service, but we never actually see your data because it never gets to a service. So I think that's a really interesting application. And so for developers who have similar privacy, similar scenarios where having a strong notion of privacy is valuable, I think TensorFlow JS is a really strong candidate. Another interesting area, which is part of some work I have done, has been around attempting to design interactive experiences in the browser using the camera. And so I have this library called Hand Track js. Essentially it's an object detection model that's able to run in real time. And what it does is that it's able to track the location of human hands in any video frame or image frame right there in the browser. And so the idea is you could use interactions like that to create more engaging experiences like using your hand to control the game. Or for artistic installations, you could, rather than using the mouse to interact with that art installation, you could just have user stand in front of the computer and based on the feed that's coming from the camera, they could perform things like selections, touch grab and all of that. And the good thing is before now to perform this sort of engaging interactions, you had to have some kind of hardware sensor or some really complex backend server. But now with TensorFlow JS and a well optimized model, you could actually do all of that in the browser with no additional hardware and just access the user camera. And so these are two interesting examples that I think are engaging. I think it's the space that's still growing.

Yeah. So I'm curious. So I definitely see the advantage on the privacy side. I see the advantage on the sort of interaction side like you're talking about with the hand track js, which I definitely want to get into here in a second. What I was wondering, I anticipate that maybe some companies that like their AI models or their machine learning models, maybe it's part of their sort of bread and butter and how they make money or it's their market advantage or something. Would it be a problem for them to kind of push that model out to the browser in the sense of like, can users that are using one of these applications just like open up a JavaScript console and grab the model and use it themselves? Are there certain concerns development wise around keeping your model in house or things that you should be aware of as you actually port this model out to the browser?

Right. Yeah, you definitely raise a concern. And so at the moment, your model is just like any other web assets on your web application. And so just the same way you, the developer, is responsible for securing elements like images and videos and every other content that's integrated into the web page, they would need to think carefully around the security of the models. And so you're absolutely right. It will be possible without any specific security implementations. I mean, beyond putting the model behind some kind of login or restricting access to base and IP address or any other security practice. If those are not implemented, it will be possible for the model files to be downloaded and perhaps used offline. And so I guess this is a concern that, that the users should, developers should think about while they consider putting models in the browser.

Hey, guess what? Brain Science is officially launched. Episode number one is on the feed right now, so head to changelog.com Brady Brain Science to listen, to subscribe and to join us on this journey of exploring the human mind once again. Change law.com brain science or search for Brain Science in your favorite podcast app.

So I'm curious, you know, as we've kind of talked about doing AI in the browser all this time, as we start seeing AI in the browser come about, you know, how is that going to change how we're interacting with web apps now that these models are available to kind of drive or provide services to the apps that we're engaging in? And do you expect, with, you know, you were talking about hand track js, do you expect us to start having the ability to use gestures, control things in a scroll, and where we're getting beyond just the mouse or trackpad and the keyboard into a more rich user experience on web apps.

Right. As a researcher interested in human computer interaction and how AI can kind of influence or make that space better? I definitely entirely think that AI models, they're well posed to kind of create that feature where we have a new set of interactions that are just purely enabled by AI. I think right now we have a lot of good examples. And these things, they work so well that now we all take them for granted and so examples like Smart Reply, gmail, Smart Reply, Smart Compose, even Autocomplete on our phones. And so I think in similar vein, I definitely see the opportunity to have gesture based interactions based on camera and maybe command, much better voice based interaction. So speech, voice, computer vision, pointing, gestures and hopefully as these models become much smaller, much well optimized, it should be just as easy as adding just a really light JavaScript file and getting that to run. And so we are still a bit far from that. The main challenges here is that it's mainly around the size of these models. And so for example hand Track js, the current version in terms of megabyte, the model itself is about 18 megabytes and in web standards 500 kilobytes is already a lot of data. And so I guess this is one limitation that kind of bars people from widespread adoption. But there's a lot of research that's kind of pointing towards a future where high performance models, we could actually find ways to compress them with very little loss and loss of accuracy. And I think that kind of research is really going to be key to getting more of these models in production easily integrated into a lot of web applications.

So how does this fit into the, and this may be something you're familiar with or not, but I was kind of wondering while you were talking about that. And in terms of the limitations of the size of models coming down to devices, does this fit into the whole like federated training side of things where like some data scattered like between different phones and that sort of thing, or maybe that's totally separate from the, the JavaScript side of things is the difference like, you know, in the federated sense you're utilizing data that's on people's different devices and training a larger model that you would still use on the back end. Whereas like in the TensorFlow JS case, you're really interested in just kind of a single user and their, and their data or are you aware of that, how that kind of fits into this picture?

Right, so it's definitely related to federated learning. Like you mentioned is the whole idea where we have a federated model and at each end user or client device, we could train client models and then send some kind of model updates back to the server to have a much better or higher performance federated model. And the value here is that data still stays on the client devices, but just these model updates that don't compromise privacy, data privacy gets sent back to the server. And so how is tensorflow JS kind of connected to all of this, I guess the value here is that TensorFlow JS could be a tool that lets you implement federated learning on a global scale. I mean, this choice is something that developers have to make. But with TensorFlow JS you could definitely construct local models on end user devices using local data. And depending on how you want to structure your system, you could send model updates to some federated model within your server. And I think there's some experimental implementation of a federated learning model I Think on the TensorFlow JS GitHub repository. And so if people are interested in exploring that more, that that's a great place to start.

So I got a follow up question as well as we were talking about, you know, kind of the rise of gestures and richer interactions and I'm just kind of curious what your thoughts are. You know, now that, you know, for on iPhone and other Apple devices you have CoreML and you know, on Android you have the Google ML kit and is having those available on these end devices making a substantial difference in the ability to get there faster in terms of that richer user interface? Are you anticipating that those are or are they already being used heavily with tensorflow JS to try to get every possible processing capability out of whatever device you're on?

Within the TensorFlow ecosystem, there's this tool called TensorFlow Lite. I don't know if you're familiar with that.

Yep, we are, yeah.

And so TensorFlow Lite is all about, you know, finding ways to compress or optimize models such that they run on resource constrained environment. And there's a bit of relationship with TensorFlow JS. I guess the main difference being that TensorFlow JS is all about managing the whole machine learning experience in JavaScript while TensorFlow Lite is all about making models smaller such that they run in resource constrained environments. I think there's some relationship between both because Tensor js, you know, it does have some focus. And so the TensorFlow JS converter has some applications in model quantization where you could actually explore ways to make your model smaller such that it runs faster in the browser. But I think these are slightly different efforts and I also think the ability to have compressed models that run well on smartphones and resource constrained devices like the Raspberry PI. I think research in that general area should also be impactful and useful for, and transferable, I guess, to work being done with TensorFlow JS.

So let's say that for example, I know almost nothing about JavaScript, which is actually the case and even Though I've worked with front end developers and developed APIs and that sort of thing, I don't really know anything about JavaScript other than like an occasional hacking into something. So for someone in my position that's maybe coming from Python, what would you recommend in terms of getting hands on with TensorFlow JS? Is it best to kind of start by looking at some JavaScript code tutorial online and going through that and then jumping into tensorflow JS or are there kind of combined tutorials or resources that would be helpful? What are your recommendations there?

Of course it's always valuable to get a refresher on the, on the JavaScript language. And for beginners, people who are interested in getting into TensorFlow JS, I always recommend the tutorials on the TensorFlow JS website. And so that is tensorflow.org js so they have a bunch of tutorials that walk you through the APIs that are available within the library. And so just to give an overview, TensorFlow JS supports two main types of APIs. So the first is a low level linear algebra API. And so if you're interested in designing your multiplications, your additions, you want to implement your own loss functions, this would be the API to use. Definitely do not recommend it, except you really know what you're doing. And the second API it provides is something called Layers API, which is really similar in spirit to the Keras API structure or Keras API design. And so it's a really great way to reason about neural networks. And so if you have used the Keras API previously, using the TensorFlow JS layers API should be something familiar and easy. So use find traditional building blocks like LSTMs 2D convolutions, transpose layers, batch normalization layers, and essentially they are implemented just like you would implement that with Keras. And so to have your model built, you could compile it and then also get your accuracy metrics very similar to how you would do that in Keras. And so for people just interested in making the switch from let's say regular Python or maybe machine learning with Python to Tensor js, I would recommend looking at tutorials on the Layers API. And the other interesting thing here is that if you have models that are already built using and exported using the TensorFlow saved models format or the Keras saved models format, you can actually use the TensorFlow JS converter to convert that directly into the TensorFlow JS web model format. And then all you have to do is just spend some time learning how to load those and Use that for inference in the web application. So these are kind of like the mental steps to go through. It's useful to get a refresh in JavaScript and then if you have some experience with Keras, the learning curve isn't that bad anymore. And the TensorFlow JS website has a bunch of tutorials and they have a lot really good sample code on GitHub to get started.

Yeah, so the, it sounds like the layers thing. Like if I'm wanting to experiment like with layers, I could build a fairly easily build like a simple, you know, maybe a fully connected neural net that would, you know, solve some kind of toy problem, let's say iris classification or. So I could do that fairly easily with the layers and kind of get a feel for, for it. But then I could also take like maybe a pre trained model for image detection that's existing and try just to, just to do the inferencing part by using the TensorFlow converter. So those, those would be two things that would be reasonable to try first maybe.

Right? That is correct.

Cool. You know, a lot of what you kind of have worked on personally as related to the hand tracking that's, that's related to image and video based techniques. And I had just seen before we started recording, I saw that you and the Fast Forward Labs team released this Covnet playground and I was wondering if you could just mention what that is. That might be another great learning resource kind of beyond tensorflow JS but also related because a lot of the stuff shows image detection examples, right?

Yeah, thanks for that. The tool you're referring to is ConvNet playground and essentially it's a tool that lets you experiment and learn about how convolutional neural networks can be applied to the task of semantic image search. And so within the framework of that application we have a very simple definition where semantic search is all about giving an image. Find all other images that are similar to this image but just by looking at their content. And so the implementation is really simple. We get a convolutional neural network and we use that as a feature extractor on all the images. And based on this feature extracted we can compute some measure similarities in let's say cosine distance and Minkowski distance. And essentially that's how similarity is implemented. However, in practice there's a bunch of decisions that the data scientist needs to make. And so what model do I use for feature extraction if I was going to use a pre trained model? And so there are dozens of pre trained models out there Inception, VG16, VG19. And even more recently, there are new architectures that are enabled by neural architecture search, like EfficientNet and NASNET and MNASNET. And so the data scientist needs to make decisions on which one of these pre trained models do I use. And even when they select an architecture, they need to also decide do I use the entire model or do I use a subset of this model constructed from the original model? And so once they've made these decisions, they finally to decide what similarity metrics might be best for these. And so we built ConvNet Playground to kind of create an environment where all of these decisions and all the computations for these decisions have been made and the user can essentially interactively explore what the results look like. And so there's a search interface where we have some data sets and you could, for each image in the data set, you could make a selection and then view how each model performs in terms of search accuracy or search quality for that particular search query. And then we have visualizations that lets you compare how do the different model architectures compare? How well does the semantics for each model perform? And so we have UMAP visualizations of the feature embedding space and we have a bunch of graphs that lets you perform all of these comparisons. And so I'd really encourage everyone that has a chance and wants to learn about pre trained models to kind of explore that these are some of the really interesting insights there. That one common thing that data scientists do right now is that to extract features, they would just select VGG16 and select its last layer and use that as a feature extraction. And so it turns out that this might be inefficient. The VG16 has about 130 million parameters, but a model like EfficientNet B2 has about 5 million parameters. And it actually works better than VG16 in terms of extracted features for natural images. And these are some of the insights you could actually extract by exploring that interface and depending on the type of your data. So if you have retail data, if you have natural images, these different performances will change and you can kind of explore the space using ConvNet Playground. Cool.

Well, this has been a fascinating conversation. Thank you, Victor so much for coming onto the show and telling us all about TensorFlow JS and these other projects and stuff. Really great conversation.

Thank you.

And I guess for our listeners, we want to remind everyone that we have a number of different communities where you can reach out to us and have conversation. A lot of the show is built on your feedback and your Comments. You can go to changelog.com community and opt in. If you're on LinkedIn, there is a practical AI podcast group that you can join. We have our Slack community which you can reach by the website. And also because we're talking about JavaScript on this episode, we wanted to point out that the Changelog also has the JS Party podcast, which is a fantastic podcast having to do with all things JavaScript and you can find that@changelog.com jsparty so we look forward to to seeing you guys next time. Thank you very much.

All right, thank you for tuning in to this episode of Practically I. If you enjoyed this show, do us a favor. Go on itunes, give us a rating, go in your podcast app and favorite it. If you are on Twitter or social network, share a link with a friend. Whatever you gotta do. Share the show with a friend if you enjoyed it. And bandwidth for Changelog is provided by Fastly. Learn more@fastly.com and we catch our errors before our users do here at Changelog because of Rollbar. Check them out@robo.com changelog and we're hosted on Linode cloud servers. Head to linode.com changelog check them out. Support this show this episode is hosted by Daniel Whitenack and Chris Benson. The music is by Breakmaster Cylinder and you can find more shows just like this@changelow.com when you go there, pop in your email address. Get our weekly email keeping you up to date with the news and podcasts for developers in your inbox every single week. Thanks for tuning in. We'll see you next week.