AI for Visual Data: Computer Vision in Business

Sam sa. Foreign.

And thank you for joining today's webinar. My name is Rhys and I'll be your moderator today. We're going to kick off today's session in about four or five minutes or so. We're just waiting so everyone has a chance to join. In the meanwhile, though, we'd love to hear from you. So let us know where you're joining from using the chat or the comments, depending on what platform you're watching from. And yeah, tell us something that you'd like to learn from today's webinar. If you have any questions at all throughout the session, then please let us know using the chat. And yeah, I will be saving them for our Q and A if you haven't already, then get registered for the event today and we will send you the recording as well as any other resources that I'll be sharing as well. I'll be posting a link to that shortly in the chat, so keep your eyes peeled for that. But if you don't catch it in the chat, then you can go to datacamp.comwebinars Brilliant. I'll be back to repeat these messages for any new joiner shortly, but until then, enjoy the background music.

Sa. Sam.

Hello everyone and thank you for joining today's webinar. My name is Rhys and I'll be your moderator today. We're going to kick off today's session in about a minute or so. We're just waiting so everyone has a chance to join. Let us know where you're joining from using the chat or the comments, depending on what platform you're watching on. And yeah, let us know that something that you'd like to learn from today's webinar. If you have any questions at any point for any of our panelists or for all of our panelists, then please let us know using the chat or the comments. And I'll be saving them for the Q and A. We'll be running a Q and A towards the end of the session. But yeah, if you have any questions at any point, then please let us know as soon as they spring to mind. If you haven't already, then please register for the event and we will get the recording and resources sent to you by email. The link to do that is in the chat. I'll be posting it again shortly. But if you can't find it in the chat, then head over to datacamp.com forward/webinars. Brilliant. I think that's everything from me. So now I'll hand you over to your Host for today's session, Richie. Richie, please take it away.

Welcome to the Webinar. This is Richie now. Until recently, doing data science with image and video data was incredibly difficult. Advances in tools have meant that visual data is becoming increasingly important for businesses as AI allows you to extract more value from it. Now there are still a few challenges and subtleties to this, so today I've rounded up three experts to help you navigate making use of this data type. Sanjay Pichai is a co founder of Acridata, an AI platform for visual data where he heads up the data exploration business product and go to market strategy. He's a veteran of several startups and was previously an investment banker at Bear Stearns. Now Vishakha Gupta Kledat is the co founder and CEO of Aperture Data, a visual data management platform and she previously worked at Intel Labs where she led the design and development of their visual data management system. And Nitam is the computer vision team lead at dese, a deep learning development platform. He leads the productization of DESI's AI technology. So between them they've got a tremendous amount of knowledge on computer vision and I'm getting very excited to learn from them. So welcome to all three of you. I'm actually really excited for this discussion. So I guess just to kick it off, can you tell me like what are the most common business use cases of visual data? Who wants to go first? Nathan, do you want to go first?

Yes. Hi everyone. So visual data of course is very very common for computer revision algorithms across a wide range of business use cases. Here at Desi we serve customers across across the world. Recent use cases are advanced driver assistance systems, adas for automotive companies, quality inspection for manufacturing, also shelf monitoring and automation checkout for retail. And these are just few examples that's been cool.

I like that there's a few different industries in there. Biseka, do you want to add to that? What use cases have you seen?

Yeah, thanks so much. Hi everyone. So in terms of retail for example other than inventory management, you have shopper insights, you have cash address checkout in case of visual inspection. It's also detecting defects or worker safety e commerce, we actually work on visual recommendations so recommending based on the features that you see of the product and automated attribute classification. There's smart agriculture, detect good crops, bad crops, pests, harvests and then there's medical imaging where you help to help experts diagnose patients faster. So in addition to what Nathan said, there's all these other use cases that we work to work on as well.

See, that agriculture example is really interesting. I always think of agriculture as being low tech, you know, people looking about fields. But it's absolutely not at all, is it? There is some very.

No, I mean, in fact, people end up getting data from satellite imagery, they end up getting data from drone, and then they have their own cameras and other sensors. So in fact they do a lot, I mean, depending on of course, the scale of the company. There's a lot of things that happen and especially the ones that walk around and harvest crops. So there they have to detect what's the good part of it, what's the bad part, or if they're spraying pesticide, then which is the area they should spray it on.

That's fascinating. And Sanjay, do you have any other examples to add?

Well, actually I'll flip it other way and say if you were to just pose the question as to how many cameras are there in this world, it's 45 billion, accounting. And yes, that includes your cell phones and laptops and things like that. But even if you strip them off, the sheer amount of cameras that are deployed up kind of transcends across industries and the volume produced there is also substantial. For example, if you take just the consumer produced content alone, which largely, let's say YouTube as a proxy, you're talking about a one exabyte of data and counting. Right. So besides the industries that they have covered, one additional piece is also the difference because surveillance in that context, as well as aerial inspired sort of satellite and drones, which I think covered heavy emphasis on manufacturing too, because historically the whole automation of manufacturing has been happening for the last 25, 30 years. But now there's a shift from the classical computer vision to a more deep learning approach technique. So I think pretty much that isn't an industry that is where visual data or computer vision does not apply.

Okay, that's very interesting. I just seem like it's completely ubiquitous. Can you expand on the last point about going from classical computer vision to a deep learning approach? What does that mean?

Yeah, so the classical computer vision is largely based upon a very templatized approach. Very bespoke models are built. Let's say you're producing a widget and you're inspecting in line, on assembly line. You essentially look and build a model that looks for the shape and structure of the widget. Very purpose built. Now such model works great when you're looking at a very stereotypical set of data. Compare that or contrast that with something that's mobile or autonomous, be it a robot walking around or Autonomous car. You have to deal with a diverse set of environment. Right. People place things, things are moving all around and you're making decisions. So for that, the deep learning techniques, which the underpinnings of that is your neural network, CNNs and things of that nature are the foundations for that. So it's a very different environment where in the latter case, the data becomes even more prominence because models are becoming more of a black box and you just beat it with enough good data and the models or the deep learning models yield better results.

Okay, so it's going from just can you deal with a few specific objects to can you recognize any kind of object? That's interesting development. All right, so I want to talk about how data people get involved with this. So can you tell me about what sort of problems data scientists or machine learning engineers might be trying to solve with visual data? So how do those people get involved? I guess, Natan, you've not spoken for a while.

Yeah, perhaps. If I may add to what Sanjay said, in the classic computer vision, you solve the problem by designing features, while in deep learning approach, you just solve it by learning from a lot of examples. And this is why you need vast amounts of data.

Okay, that makes sense. All right, so just to go back to the sort of question like, where do data scientists get involved in the sort of the visual data process?

So.

Sorry, go on, Vishaki. You want to sound like you were keen to dive in there.

Oh yeah, I was going to say like, in terms of like, you know, where the data scientists come in typically like you when we took examples of inspection, you got to start training models to detect defects on a manufacturing line or warn of some hazards in a factory floor, training the models, extracting and learning how it's performing, how accurate it is compared to the data it trained on. You try on some validation data. Then very often we encounter cases where data scientists are working with embedding how good the embeddings are results in how good the KNN or the similarity searches tend to be. And so tuning that, there's a lot of working together with understanding the data and then making use of it, like making sure it's distributed, it's representative of all the things that you want to detect and train on. That's typically where we've seen data scientists or machine learning scientists work on and they are directly trying to solve for either improving the customer experience for their companies or worker safety or just improving the way the resources get utilized.

You mentioned the idea of embeddings being really important There. Can you just explain what an embedding is?

Should I.

Or whoever wants to explain what an embedding is?

I mean, the very straightforward way I think about embeddings is, let's say I need to describe our faces. There are different features on our faces, like your nose, eyes, the distance between the eyes, the color, everything. If you start assigning numbers to them and then put that together in an array, that's that feature vector that can describe my face with numerical values. And it tends to be smaller than if I were to capture the entire all the pixels on. And that's, you know, that's like my simplest way of thinking about vector embeddings. You guys can make it more machine learning specific.

Sanjay, you look like you wanted to jump in there.

Well, I'll probably give a little bit a simpler way of thinking. Think there are three stars in the sky, right? Star A and B are what looks to us as visually close to each other. And star, see, the third star is further apart. In real world, the second star could be really several light years behind. Right. But our eyes perception is these two stars are close when reality is, it is the other ways. The first star and the third star are close to each other. Now what I'm explaining is three dimensions. And what we're saying is an image could be a high dimensions, something that we cannot see it. But how do you represent the salient features? Which is true. If I take a picture, you will have a false illusion of A and B being close. But in reality, I need to surface the fact that A and C are actually the dots that are close to each other and B the furthest apart. Right. Now extrapolate this example into 2000 different dimensions. And that is sort of the representation of feature vectors and the salient aspects of that presented in a 2D embeddings.

And I think you combine both our explanations to really start thinking about how they get used.

So if I wouldn't. It's about turning your visual data, your images or whatever into numbers. So then you can do like, yeah, your numeric computations to see which things are similar and which things are different. All right. Okay. So can we get to some examples of like, what. What success looks like? So have you got any examples of companies that use visual data successfully? Who wants to go first? No one's looking excited. Sanjay.

You look like, you know, let's start with Auto. Yes, Autonomous vehicles are the holy grail that every company is chasing for. However, Auto, you know, has been evolving these driver assist functionalities Right. Some, many of us who are buying in the recent years cars you're probably starting to see adaptive cruise controls, lane shifting and all those things. All those things really happens with some sort of a computer vision. Right. And of course you probably on the other spectrum you're seeing the Waymos and Cruise who have sort of, you know, pretty close to self driving cars, which is, you know, which is true representation of a transformation. Just rewind 20 years ago and see what the cars that you own today or see in the market. It's a big transformation that people can associate with. Of course there are several examples, but I'll let my friends speak to it. All right.

Yeah, self driving car is definitely a cool example. Nathan, you've not spoken for a while. Do you want to give us some examples of what constitutes a success with visual data?

Yes, of course. So I think one example from new angle is responsive PDF files. I guess we all know what PDFs are our cell phones and we know that it's not very convenient to read PDFs on cell phones. Right. But in recent years we have responsive websites that act responsibly but not PDFs and this is something that we did recently and it works great. And I think you can use it now by the way, with Adobe's Liquid mode on their Acrobat app. So this is one example that comes from the engineering part, the computer science part where you use visual data, although it's a bit synthetic. This is one example.

Another example, the effect is it makes it easier to read if you're on mobile versus if you're on web. Is that the idea?

Yeah. Especially if you read the scientific papers. They are like two column papers. When you have them responsibly arranged, it's much easier.

Yes. Multi column pages on mobile, that's horrible to read.

I agree. Okay.

You had one more example, I believe.

Yeah. We can also talk about applied materials which they inspect chips. So this is another example of using deep learning models to inspect chips where you don't know where the defects are. So this is yet another example of using visual data.

Okay. So you're looking at images or video of chips and seeing where the problems. And is that more broadly applicable to other kinds of manufacturing?

Yeah, yeah, it's starting to be.

Okay, that sounds cool. And Vishaka, do you have more examples of who's doing things well?

Yeah, so we. One of the companies I can think of is Savager Technologies. They're scaling inventory management and planogram compliance. So you can like, you know, in The Northeast particularly. You can see their robots walking down grocery store aisles every morning, kind of like figuring out are the shelves stocked, are they correctly, the items correctly placed. Then there are companies like Zipin that are growing their cashierless checkout stores so that people can quickly zoom in and out. So it's along the lines of Amazon Go, but in a lot more places like stadiums and things like that. And some of the large retailers are having good luck doing visual recommendations on the E commerce side and shopper insights on the smart retail side. You can see papers and talks out of Home Depot, Walmart, then companies like Merck, some in energy, aerospace and medical imaging, insurance, in fact, insurance analysis, like companies like CCC Intelligence Solutions. So there's quite a few of them that have managed to do very well despite whatever challenges we are going to discuss about. And yeah, and they've deployed this in production, I would say.

Okay, that's very cool.

I like that.

Well, I'm not sure whether I really like the idea or whether it's kind of creepy having robots walk through the supermarket with you, but yeah.

They'Ve made quite the fun experiences. You know, they call their robot Marty with the eyes on it and it even like goes around parking lots to say hi to people. You should check out the videos.

Nice. Okay, yeah, that sounds like something worth checking out. All right, so I don't really have a strong sense of like, what are the easy things to do? 1 difficult things to do with computer vision. So for organizations who want to get started with this, like, what's a good first project to do? Nathan, do you want to go again?

Yeah, sure. So generally image classification tasks can be relatively simple, while detection or segmentation tasks are considered a bit more advanced. If you also think about it, for image classification, the output is one scalar or the class name, which basically means that you consume the image and output something very small and specific. For object detection, you output bounding boxes and for segmentation you output a segmentation mask. So image classification generally is very simple task, relatively simple task, but nonetheless, every task will be difficult if your data is biased. If some classes are underrepresented. For example, for object detection, if you have very small objects, then this is also hard. And I think that one step before you do any task, you need to make sure that your data is fine because then your chances to fail are higher.

Okay, it does seem like we do need to have a longer discussion about getting the right data and make sure that you haven't got biased best results. I do find that interesting that you said that image classification is kind of the simplest thing because you know it's outputting text rather than doing something more complicated. All right, Sanjay, do you have any ideas for like a good first project?

Yeah, I concur that, you know, probably classification is the easiest one to get going. Probably. But today, you know, there's so much off the shelf models that one could get going. That's the beauty about it. Gone are the days you had to build something from scratch. You probably heard of yolo which is a very popular model. They're lightweight versions. You can just log in to hugging face and pull a model a lightweight like a YOLO X detect. You could potentially run it in your own laptop, put some images and see the response out of it. That's a great starting point for many place but really getting down to work it for your environment, your business case will drive the choice of models that you want to go with. Now that requires some level of knowledge of what each models that off the shelf models are were designed and trained for and then you choose the closest model that relates to your domain of data. There's one thing that's detecting a coffee mug and laptop and things of that nature in a table versus doing detection of the grocery store example that Bissaka was mentioning. They have wide range of sauces to jam bottles to toothpaste. So which one are you going to pick? But as I said earlier, there's an easy way to jumpstart your project and then choosing the right data from your set and being able to train and adjust the models is going to yield pretty good results. Off the shelf.

It sounds like we agreed that image classification is the way to start and the tricky part is deciding what you can do image classification on. So Vishaka, do you want to talk about like how you get from. I'm going to do an image classification project so I'm going to get to, I'm going to get some value out of this.

Yeah. And in fact that's what the code along session is going to be in a few weeks. But you essentially like, you know, I was even going to say a step further. So when Sanjay alluded to the fact that you're going to have to see your use case, in fact I would say you have to see what data you have that is accessible to you, that's usable, the quality of that data. That's like sometimes the challenges start from even collecting or gathering together because it's usually scattered across a lot of different sources. Especially in a large company just getting a Sense of what the data that you have is, how well is it represented, how much is it labeled? You would essentially have to start from there. And then like Sanjay was saying, there are now models that are very easy to access and run through. For a small example, just like literally step by step, okay, this is where you put your data set, this is where you unzip it. This is the training set, this is a validation set. And you take, install Pytorch, choose a model and get going. The tricky part starts to be when you want to do this at scale. Then you have to start first with storing the data someplace you can access, being able to search, creating data sets. Then you see how an off the shelf model would perform on this data. And then you start to see, well, this is not the desired accuracy. Now you got to train it on your data so that it starts to understand more of your whatever images or videos you have in your organization. And that kind of takes you back to, okay, do we need more label data? So like, can I get it labeled? If it's not a lot that you need label, then you can use some open source tools like, you know, we've been building demos with Label Studio, for example, you can label some certain things. It lets you label bounding boxes, polygons, and then you can create data sets that put data along all these things. So let's say you want to recognize jam bottles and cereal boxes and whatnot so you'll have enough images in each category labeled that way. And then you retrain the model and then you start seeing. So you have to be able to look at the data before and after, see if your model is how well it's doing. And it sounds like a lot of steps, but I think there's now enough tooling along this entire spectrum to kind of simplify and a lot of it is pretty easily available.

That is great that you can just get started with off the shelf tools and get some value pretty quickly. And then you sort of worry about, it sounds like you worry about performance a bit later on. Just trying to get like the best performance after you when you have to.

Deploy and scale or when you have to do it company wide and you actually go from you know, research project to more of a, okay, we are getting good results now can we see this on our entire data? And then you're talking in some cases like millions of images or millions of videos that you've collected over time. And that's when you start to get into like scale. And how can you launch things in parallel and the classic systems problems show up for ML infra then.

Okay, yeah. So deploying machine learning is the tricky part. Right, sorry.

I would also argue that while off the shelf models do give you a decent performance, you should care about production from day zero because otherwise you will end up with some legacy off the shelf model that you're stuck with it and you will get decent accuracy.

And.

This is only because you started with off the shelf and then you kind of rolled with it. Also, as Sanjay and Vishaka mentioned, data is very important, but you also need to suit the architecture, the neural network architecture to the data itself. And this is also a crucial part. Basically, if you think of it as a logical gate, it is an end gate. You need good data and good model.

All right, so since good data has been mentioned a few times, let's get into that. So where do you get this good data from? Like who should be responsible in your organization for collecting data, for ensuring data quality, things like that. I guess it's going to vary a lot depending on the use case of. Does anyone got any tips on this? Sanjay, do you want to go?

Sure. So actually what we have seen in companies big and small, the behavior is slightly different. But the indisputable fact is most often these data are produced in a remote area compared to a data center or a cloud where all the data is stored, pre processed things of that nature. But if you start from the source, be it a factory floor where you have a camera deployed or a camera in a retail store, they all produce way too much data. Not all data is actually relevant to you. In a larger organizations there are dedicated teams that are managing both from an infrastructure point of view. Where do I store this data? What portion of data can I move through the network to a central place? Then there's whole set of data engineering data curation teams that are charted with what data have I collected of which what is that I'm looking for? Because most of them are garbage. When I'm trying to teach a model to detect something X, I need to find enough examples of X and variations of X picking that and having it labeled. And labeling is typically done by outside vendors. But for sensitive projects, people do it in house. So there are separation between a labeling team and the curation teams. In some cases they're all bundled in one that is the source to data preparation phase before your training kicks in.

Okay, sorry, I can add. So you know, combining what Nathan and Sanjay said here, good data also depends on a lot of understanding. So you know, as you as he was alluding to it comes from somewhere else. In case of E commerce, maybe not an Edge collection, it's uploaded by merchants or, you know, you take pictures with. Yeah, I guess in some ways that's edge. But there are a lot of teams involved and an understanding of what you're collecting and what do the metadata descriptions mean about all these things. So, like, for any of these images of videos, there'll always be some descriptive metadata. Even something as simple as, we collected this at factory 5 from camera 10 at this time. That's like very fundamental example of metadata that goes. But we've seen cases where the data engineering team defines that one of the fields as a string and the data science team assumes it's integer. It's like the classic problem around the understanding of data getting messed up. Or literally when people are passing around cloud buckets across these stages, then did you name the bucket correctly? Or someone named the bucket, that person left. What happened? The, the quality of the data, how much do you collect? How much do you replicate it? Are you working on the right version of it? Those things start to become like, something you have to pay attention to. Especially, like Nathan was saying, you have to think about production from the beginning. Otherwise you have to plan a lot of time to actually, like, redo all the things that you did. But now in production, and then you wonder like, oh, it's not what we expected. So that's like, I would say, like, solving for the good data problems should kind of span across, not just like, is it distributed in all the classes that I want it to be represented? Do I have enough examples correctly labeled? There's also like, are my labelers doing a good job? So there's a lot of like, validation steps that should go in along the way.

It just have a lot of room for error there. And you do need to be very careful, I guess, in terms of just like, simple like data governance type stuff. Just make sure that all your processes, make sure that you can even find images that you've stored somewhere in a cloud bucket. So actually, earlier, Nathan, you mentioned the idea that you need to make sure there's no bias in your data set. Can you expand on that and tell us what you mean there?

Yes. So I think that the most common mistakes is when you split the data into training and validation sets. I assume that everyone knows what training and validation sets are. So when you split the data, you need to make sure that the distributions match and not only the pixel distribution, but also class distribution and also location distribution and also a bounding box size distribution. If we're talking about object detection, for example, when we do autonomous vehicles. So it's also important that the distributions of the colors and the brightness will be the same across training and validation sets. Because especially for autonomous vehicles, when you drive at night then there are no bright colors, of course, rather than lights. And when you drive a day then most of the image is bright. So you also need to carefully split these images correspondingly. And for videos I think that you care about the temporal continuing which is also something that people tend to miss.

That's kind of interesting. The idea that something like with the autonomous vehicles is going to work during day but not at night because all the colors are different and therefore yeah, the model breaks. That's fascinating actually. Sanjay, do you have any more examples of where bias can creep into data sets and affect your model performance?

Yes, actually one of our customer, which is a very large tech company, they are actually having a dedicated team to focus on a bias. In their particular case there is set of information about the user is posted so the individuals who are partaking as a customer in the website and implicitly the racial bias is there because the population distribution is of that nature. But when you have certain feedbacks from the market being, let's say I'm giving five stars to you after engaging with you or maybe two stars. And if there is an implicit pattern, as let's say mentioned, if you so happen to pick a training data set that inadvertently captures or has the skewed information, what's really going to happen is when you are dealing with the real world data, you're going to make certain adverse choices. And in this world with social media and other stuff, one mistake is enough and next thing you know, company A is you know, doing or taking certain actions that are ratio skewed which is incorrect, which was not their intent. But this also goes back to the fact that I mentioned. Models are becoming too complex for people to be able to pin it down what it's doing. So many transformations are happening thereby the only way for you to control is feed with the right set of data. It really goes down to selection of data will have a cascading effect on how your model is going to eventually behave. To highlight the fact that this particular company has a team of computer vision guys and the job is to look for bias.

That is cool that there's sort of a dedicated resource there just to figure out what's going on with bias. And I like the idea, the example you gave about when you deal with people then you can end up with things like racial bias and discrimination. Vishakha, do you have any more examples of where bias can cause problems with computer vision?

Yeah, I could think of a medical imaging example. Like, you end up training a lot in, like, you know, your specific lab samples, and you don't get to see every kind of, like, you know, maybe every kind of X ray or every kind of CT scan. And then suddenly all these corner cases start appearing and the model is not ready for it. So, like, you have to keep introducing, like, every time. And that's why at one point the general thing was, oh, you train your model and you're done. But now it's become more like, okay, well, we got all these cases where it didn't work correctly. And that's where, like, you know, you have to retrain them to introduce all these corner cases. And then I think Nathan mentioned something about the lighting. And, you know, earlier we were talking about the agriculture example. Like, most of the times, hopefully you only see healthy crops, which means to see crops that have pests or some problems, you have to make sure you're putting in that extra effort to find. One of the questions I got asked at one point is everyone talks about finding similar data. We want to find dissimilar data because we want to make sure it's represented well enough so that we're training on a wide set of non similar data and things like that. So.

So it seems like the sort of common thread from all this is that when you have rare cases of things, then that seems to be the hard thing to train on. You get worse. Multiple performance. Is that sort of a good summary?

One of the. One of the cases, sometimes you're just not collecting in one section. Right. Like what Sanjay was alluding to, the face thing. If you just don't have representations of people, even though they exist, you just don't have that shown in the data that has been collected so far. Then it's not because it's a rare case, it's just because you don't have it.

It's just rare in your dataset.

Okay, Rare in your dataset. Yeah. So rare in that sense.

All right. Actually, for anyone in the audience who is interested in this idea of bias and computer vision, the next episode of the Databrain podcast coming out on Monday does deal with this topic. So we've got Joy Buolamwimi coming on the show, so please do look out for that on Monday. But back to the task in hand. So we talked a bit about some of the Challenges of actually getting these models to work. Are there any other challenges that we've not discussed yet? Like, what else can go wrong? What mistakes do people make? All right, yeah, we've already got like, what, 20 minutes left.

I'll narrow it down and then I'll stop speaking on that. I think, like, you know, I think we already referred to the whole think about going into production from the very beginning. So you need to be able to manage, like, define your data sets and manage them and not in a very ad hoc manner, but like, you know, easy to replicate manner. Like oftentimes when you're training, let's say, you apply certain augmentation to the, the data to get like, you know, to improve the quality of the model. You want to be. You want to make it repeatable. It's not like only one data scientist can reproduce it. You should, like, it should be representative across and, you know, how you do that is one of the challenges. We already talked about consistency across. But the two big things I'll mention is when you go from a scale of, let's say, training with a thousand videos, do you want to start training with 100,000? That's when you start running into systems challenges. That's when you're talking about how do you access these in parallel, how do you deal with your network, how do you deal with the costs associated with those training? Those are some of the other challenges. And then keeping this data secure and privacy of this data, all these things, I choose those out of my list of 10.

Yeah. I can certainly see how once you start getting to 100,000 of images, you are going to have real big data.

Problems to deal with, particularly for videos. And you have to split them and then you have to combine because your events run through across the splits that you did. And then you get into dealing with all these computer vision libraries, which sometimes can be a pain to install and deal with.

All right, fantastic. Sanjay, have you got any more problems that people need to deal with?

Yes, actually, just giving an analogy from my own customer. One of our top customers produce a petabyte of data a month and run it through our system. Right. That's a petabyte that goes to our system. I don't know how much more they actually generate a petabyte. For people not too familiar. It's kind of what, 10 to the power of 18 bytes? Right. And today a terabyte of data would. You can pack about 500 movies in a terabyte. Thousand terabytes better. And you can go from there. So that sheer volume they went from a fraction of the volume they were generating to a petabyte within a year and now they're producing a petabyte a month. That really tests everything that you had designed for which is where do I collect this data? What is the most efficient one petabyte even in the cloud would be exorbitant cost. So the choices one has to make or made will be put to test from scalability of data store, data ingestion and data processing side of things. You will be forced to pick a very small fraction of that data for your training purpose. Because unless you're Googles and Apples who can afford to spend multi millions of dollars in training, most companies don't have that liberty. Eventually you'll have to reduce down to a small set of data. And this data depending upon the organization you could be collecting it from different geos could be coming from a different factories and GEOs or other use cases. You have to worry about data residency because there are regulatory requirements as to the data shall not be moved from Europe to your central location in US privacy because often visual data, whether it's a storefront camera or a surveillance camera, you're registering people's faces and if you just go to Google Maps and look for the street view, they obviously get the license plate. All those things we take it for granted, but somebody has thought through that, made sure that privacy is installed. There's no bias, there is data residencies. All of those things have to be factored in.

Yeah, just now there's a lot of issues once you're really at scale. I can certainly imagine how trimming down your data set a bit unless you're running your own cloud platform like Amazon or Google or Microsoft. Nathan, have you got any more examples of things that can go wrong?

Yeah, of course, like everyone else here, I will add another layer of complexity. So you have a lot of data. Eventually you want to train models with this data. And training models with a lot of data is also very complicated. You don't train your models only once. If that is the case then you are a very lucky person. Most of the times you train tens or hundreds of models per day. Every model has different hyper parameter or architecture considerations. And you also need to version control all of your experiments to visualize them to quickly understand which ones are more promising, which ones are less promising. And one very important thing is I think that Bishaka mentioned it earlier, is that you need to have these experiments reproducible otherwise you will end up with one employee who is responsible for this model and another employee who is responsible for that model. And you somehow end up with having a model that you don't know on which data it was trained on, with what the parameters it was trained, and you can't change it, you can't modify it, you can't do anything. So this is one thing. Another thing is that people should, I think, think several times before trying to throw more data or to throw more parameters. In terms of model size, one image is worth a thousand words and I think that a good image is also worth a thousand of bad images. So you should also consider that before you start training on petabytes of data with, I don't know, 7 billion parameter model. So a good approach will save you a lot of data and a lot of compute.

Absolutely. I mean certainly the idea of just tracking what you've done and trying to realize that you need to manage everything just sounds like great advice. All right, so we've got a ton of great questions from the audience which we'll get to in a minute. I do want to have a couple of last questions though before we get there. So quick answers on these, I think. So for people who are interested in getting into this field, like they want to be a confused vision professional, what sort of skills and tools do you need? Vishaka, do you want to go first?

I would say, I mean just, you know there is a lot of good, there's a lot of good courses on deep learning that you can get started with especially hands on. Anytime you can do some hands on trial of these things because there's all these collab notebooks that you can launch and start trying. But in general I think it just like everything, if you're going to fundamental what the tools that you should be good at, just get really good at Python. That's not, that's something that it's going to need for everything and, and then just start trying like you know, go to like there's a lot of models that you can try from Hugging Face, there's a lot of data you can get from Kaggle. Just combine those things together and start working and start building the examples. And of course when you get to, you know, all these challenges we talked about, that's when you're doing like a company wide, like you know, you're doing something for the company. But if you're learning, I think just getting started from depends on how deep you want to go. So, so doing some of these classes and I think Data Camp has a lot of resources to learn as well, so just learning through that and as much as you can try hands on workshops or like some of your code along sessions and stuff.

Absolutely, yes, please, everyone please do come back to the code along. Yeah. And if you want to learn Python, you're definitely in the right place here on Data Camp. All right, Nathan, any other skills or tools you think people need to know about?

No, actually I think that Vishaka summarized it perfectly. You need to know Python, you need to have basically a will to learn new stuff. Everything is online, everything is very accessible. Just go ahead and try.

Absolutely, Sanjay. Anything to add on skills or tools or is that it? It's just a load of Python I guess. I know Pytorch just mentioned earlier, is that an important thing in this context?

Yes, it is one of the most popular frameworks that you go with. But if all of these things are a bit overwhelming, where do I start? If I don't know Python, if I don't know the foundational mathematics behind it, all those things you can take it from a business first approach or let's say I want to just kick the tires on what is this all models and things of that nature. You would be surprised. Just go to ChatGPT and can actually be a great assistant to just say walk me through the steps of how to install a model, for example. It actually does a pretty good job and it'll point you to some of the popular models. And the next step could be, well, you can go to the as extent of I don't even know how to install a model. If you're not familiar with Python then you need to install packages and things, things of that nature. Every one of these steps today we have that information, not to mention gazillion publications and medium. In fact, when I googled up about yolo, I recall seeing about a good public depth of material in Data Camp itself. So too much information. But if you want to start off small, I would just like I mentioned, sign up for a Colab account and if you start off with some basic instructions that are given in a jumpstart Python code, if you don't know that, just keep a GPT on the side button phrase. It actually gives you, you can learn probably Python in a faster way. Having a GPTs analysis and I do that with my daughter, so that's how I'm able to relate to these things.

I was just thinking with the hackathon we did with the team, you're trying to use the segment anything model and get polygons and then out of that ChatGPT was quite useful to get started.

Excellent. Yes. And for anyone interested, perhaps if I.

Can add something very short. So the mathematical background is important, but it is not the most important important thing. This is a second order thing that you should learn. So don't be overwhelmed of the amount of math there. This is only when you want to take your skills to the next level and actually improve yourself, improve your models and improve your understanding.

Absolutely. So my background was originally mathematics and I like people to know things about mathematics but yeah, on a day to day basis it's slightly less important than understanding the code and the concept. So just on your point Sanjay, about using COLAB and GPT on the side for people who are interested in a notebook, then please do use Data Camp Workspace which is a hosted Jupyter notebook and does have GPT built into it to help you write your code. All right, so with that we've got 10 minutes left for audience questions and we've got a lot of cool questions here. The first question comes from Mohammed saying, what's the best AI server? I guess. Do you want to pick? I'm not sure whether it's server or model. Tell me about models or platforms that you think are good for this. What should people know about. Does anyone want to go first? Nathan, what's your favorite computer vision model?

So we have our own DESI models that are by the way open source. So one of the recent models we have released is YOLO NAS which outperforms in terms of accuracy to latency trade off all the other yolos that are there around. Very easy to just take online use in production. It's also production oriented on day zero. So this is what I would suggest but of course I'm a bit biased.

All right, Yolandas maybe worth checking things out I guess. Don't be shy about selling things. Sanjay, do you have any recommendations?

Yeah, so that, I mean if you want to just go from our. The foundational models, I would say Clip is a Great one. Dyno V2 which was recently open source, Owl Vit, these are visual transformer models. All of these things are state of the art models. But I do want to call out while these are all great models, if you're really trying to pin it down to your use case, then you need to think through what Nathan was mentioning or what DESE offers, which is how do I take these models to fit into my target resources? Because one of the common thing is eventually going to be resource constrained in terms of inferences that you have to make in, let's say in an inspection line, you don't have a massive server running behind that. While all of the other models that I mentioned are going to yield very good results, it may not be transferable to run on a small lightweight inference. So you like to factor that in.

You've really got to pick the right model for your use case. Prijaka, have you got any favorites you want to tell us about?

Yeah, I can think of two that we recently use in making our demos. One is the clip. It's made it really easy to combine the text. So the multimodal aspect of it like combine be able to search for pizza and get images. It's pretty easy. And now when we build our demos and the other one is segment anything that we can extract the regions of interest in an image.

Okay.

Those are the ones that come top of the mind.

I like the idea of searching for picture and getting good results back. All right, so next question comes from Alawa saying how do you get from classical image processing methods to the fancy deep learning ones you talk about? Yeah. So if you've got a background in classical image processing, like what do you need to know for the deep learning stuff? Is it back to what we talked about before Pytorch and hugging face or is there anything more to it?

I think that the bridge between classical computer vision and deep learning computer vision is in the. You can think of it as when you have a convolutional neural networks. This network consists of convolutional layers and every layer is basically a metrics and these layers parameters are learned. For example, if edges, horizontal edges are the best feature to solve your task, then eventually the network will learn that this is the best feature. Weights of the metrics of this convolutional layer will look more like a classic edge horizontal or vertical edge detection. Same goes to Fourier, which is basically sine and cosine basis and to other things as well.

Okay, so just there's a sort of mathematical link between the classical stuff and the new stuff. That's cool to know. All right, next question comes from hello. Elal asks what should we do to protect our data? I think maybe there's two different options for this, like whether it's you're an individual person or whether you're a business. That's sort of two different questions, I guess. Take your pick as to which one you want to answer. Anyone who wants to go here, I.

Can give an answer from the enterprise perspective because that's what we end up seeing most of the time. I think there are certain things in terms of, well, if just having some hygiene around where you store the data, who you give access to. And that's kind of like I sometimes get the question, why can't we just put things in cloud buckets and let it use the data from there? I think it gets unmanageable pretty fast because you're passing around buckets, which means you keep adding permissions to different users. And that's not a very scalable model of, of protecting the data. And this is where it ties into utilizing a database, something that we also offer with Apache DB and stuff you have to think about, first of all, not creating replicas. One of the common things we run into when we talk to users is, well, my data original is very high resolution. Most of my models require something low resolution or my videos are recorded at 60 frames per second, but I require one frame per second in order to saturate my models. I have to create a copy of these data types that essentially you're increasing the surface area of how often that data is replicated. And then when you're asked to delete in one place, are you making sure you delete everywhere else? There are certain. I mean, this is a lot deeper discussion, but I would say just that originally, what we talked about, practices around data, how do you keep it consistent and make sure it's stored and you're smart about the permissions that you add for access? You let it. You have a way of consistently deleting when you're asked to delete. I think some of those steps we can go into step by step detail, but I don't think we'll have time to do that.

Yeah, maybe that's a whole separate webinar. Sanjay, you understand? Do you have anything to add on data privacy?

Yeah, well, data privacy, you know, goes even outside of the computer vision world, right? I mean, whether it's a credit card, swiping information, Social Security, all those things. But becomes very specifically in the AI world, what really happens is what an inevitable step is labeling and labelers. Today you look at companies like Scale and Label Box and other companies labeling, what they're really doing is farming out this data to third world country workers. You're taking an enterprise data, somebody who's decided to say a subset of the data needs to be used for training. And I'm going to call one of these companies for labeling. And what they are really doing is fragmenting that data and doing it across the globe. Now you can very easily see what are the places where these things can go wrong. This is a very easy path for data leakage. Auditing is very important as well as all other data security in flight at rest. All of these things matter. Some of these things are addressable other enterprise data as well. But very specifically in this particular use case, as you can see, data a, it's coming from multitude of sources. Fragmented sources is your input and as a part of labeling it gets fragmented again and distributed to somebody else. And then once post label it's coming back. So too many places where things can go wrong and controls have to be in place and monitored continuously.

Absolutely. All right, so we are at time now, but there are a lot of really great audience questions I want to get to. So we'll perhaps run over by just a couple of minutes for anyone who has to jump, please do catch us on the recording. And before we go back to audience questions, I want to say we've got a code along session on computer vision. There's actually smells from Aperture data that's on Tuesday, November 7th. So please do go to datacamp.comwebinars to sign up for that. Before we get to that, we've got a session on Friday on code along on visualizing sales data with pivot charts Excel and next week is ChatGPT week. So we've got three sessions on how to use ChatGPT effectively. All right, so next question comes from Alejandra. Maybe just quickly like. So Alejandra's a student wanting to know like what sort of degree is useful to get into computer vision. Are there any particular qualifications that help in this field? For those of you who do hiring, like what on a resume looks good.

I think today almost most universities offering computer science program pretty much geared their coursework for minting AI specialties. So you cannot go wrong there. Of course a foundation to that is mathematics. So it's something you can easily extend applied mathematics as a degree into the field of computer vision. And these are two immediate things that comes into mind.

Okay, so it seems like basically any sort of STEM degree. So you have science, computer science, technology, mathematics. Are all of those good, good entry points?

I would say. So we do like to see like you know, especially if you've gone through computer science courses. So there are specializations around data science specifically or machine learning and AI. But I also do think like an understanding of fundamentals of computer science. You know, just getting operating systems, some understanding data structure, some understanding. That's what so all those challenges we talked about through this panel, that's when you know, how to tackle them, rather than just having to wait and rely on the tools to package some of these things.

Okay, I would, I would start from doing something that you find interesting and then looking for a job in that field and not the other way around. Because Masters is hard when you do something you don't like.

Absolutely. Yeah. Doing spending a whole year on something tedious, then that's, that's not a good way to live. All right, next question comes from Alex. So Alex asks, I think, questions like, how do you make a business case? Or how do you decide whether it's viable to take on some of these projects? Again, this is probably a whole webinar in itself on how to do good business cases, but yeah, what are the sort of financial trade offs in terms of investment versus return? How do you think about that? Quickly?

I can answer this one in terms of. Well, right now, a lot of these processes are very manual. So like when, when a truck comes with delivery to the warehouse, and from the time that those items are put into the stores, you know what got delivered, what got damaged, how much of it got damaged, what was the reason for damage? Like, there are all these things that, you know, if you're scanning through, you can start detecting, like scanning from a camera, you can start detecting the nature of the damage, you can start detecting how many items were damaged. So from what I've learned, talking to some of the users, it's more around making those efficient and also not staying blocked on how long it takes for people to do this. If you can have, there are different ways of doing this. You can have drones scanning your warehouse aisles, or you can have, when the items are delivered, scale, scan it around that time. There are different versions of it, but essentially all these are towards. And you know, the larger these examples get, the more money is involved, which is essentially like what drives a business case. So it's primarily around efficiency and then trying to improve the processes that cause damage and things like that.

Okay, yeah, it does sound like you should have a reasonably good idea of like, how much stuff is getting damaged and how much that costs. You've got a, a tangible figure to be like, oh, we can save this much money. Okay.

Yeah. You can also think of the other ways. Right. The way we box it in, sort of a two by two in terms of the impact. Right. Is it going to be a life or death? Is it going to be profitability, or is it just nice to have? All right, for example, let's talk about, let's say a Netflix movie recommendation that comes to you, that is underlying. That's an AI or machine learning that's there. If it predicts a slightly different movie that doesn't appeal to you, there's a little bit of annoyance and disappointment. But you will still consume and think. Compare that with an autonomous car. If it fails to detect something, it could essentially kill somebody and the repercussions are very steep in that case. And to Visakha's point, in a retail one of our customer was working on a pick and place robots where it really from a bin of things. It picks an item and puts it there based upon the customer's delivery order and that eventually gets downstream package and shipped to you. If I end up shipping a wrong piece because my robot didn't pick the right one in a bin of things, it's going to have a disappointment from a customer experience. I received the thing, now I have to return it. It's going to cost the company. So there are serious business cases from economics point of view that you can factor in. So it kind of spreads. Goes across the potentially killing a person to having a disappointment from. From an experience. As a Netflix experience. I mentioned to economic impact.

Yeah, there's certainly a very wide range in terms of what are the consequences of getting this wrong. So that's. That just seemed like an interesting thing to factor into your decisions. All right, so we're well past time now, so I'm sorry we didn't get to all the other questions. But yeah, we're gonna have to give up at this point. So thank you once again to Bishaka, to Sanjay, to Nathan. Really brilliant discussion. I thoroughly enjoyed all that. Thank you to Rhys for moderating. Thank you to everyone who asked a question. Thank you to everyone who showed up. Please do join us again. For future webinars, go to datacamp.comwebinars and register. I will see you all again soon. All right, thanks again. Bye.

Thank you all.