Trends in data labeling

We put a lot of effort into making sure that the actual interface inside Labels Studio is very intuitive and easy to follow because as a data scientist we always focus on ourselves but we also put specific focus on non tech savvy users because those users can actually have a lot of very specific domain interest in knowledge and you want to be capturing that. But if you build a tool that is very complicated they would spend a lot of time figuring out how to use that. So we are trying to make it as simple as possible but yet powerful.

Big thanks to our partners Linode, Fastly and launchdarkly. We love Linode. They keep it fast and simple. Check them out@linode.com changelog Our bandwidth is provided by Fastly. Learn more at fastly.com and get your feature flags powered by LaunchDarkly. Get a demo at launchdarkly.com this episode is brought to you by our friends at rudderstack and we're calling all data engineers to check out rudderstack cloud and start building smart customer data pipelines. Rudderstack is warehouse first. No more silos. Rudderstack builds your customer data lake on your data warehouse, not theirs, enabling all functionality of a CDP with more security and retaining full ownership of your data. It's open source and API first. Rudderstack can be easily integrated into your existing development processes and because they're open source you can see all their code so you don't have to worry about vendor lock in or black boxes. And best of all, they have transparent pricing. Stop paying your CDP a premium to store your data. Rudderstack is free up to 500,000 events and pricing scales transparently. From there, learn more and get started@rudderstack.com again. Rudderstack.com that's R U-D D E R S T A C K.com.

Welcome to Practical AI, a weekly podcast that makes artificial intelligence practical, productive and accessible to everyone. This is where conversations around AI, machine learning and data science happen. Join the community and slack with us around various topics of the show@Kingjob.com community and follow us on Twitter. We're racticalaifm.

Well, welcome to another episode of Practical AI. This is Daniel Whitenack. I am a data scientist with SIL International and I'm joined as always by my co host Chris Benson, who is a strategist at Lockheed Martin. How you doing Chris?

Doing very well. Glad to be here.

Yeah, you were just telling me before we started recording that you had an animal rescue emergency right before I did.

And so I'M just glad to show up. I'm glad to be here today. It's a good day for me.

Yeah. Yeah, for sure. And you know what? Even more good news. So I don't know if you remember 2019, Chris.

That's like pre pandemic, isn't it?

Pre pandemic.

That's a history thing now.

I mean, yes, but pre pandemic, 2019, we had this great conversation about data labeling with Michael Malyuk of Label Studio. And Michael's back with us. How you doing, Michael?

Hey, guys.

Doing great.

I mean, it seems like we're post pandemic now, so we somehow survived.

Yeah, it's crazy that it's been two years since we talked to you.

Right, right.

Different world, right?

Yeah, it's insane. Like, we first were on a podcast pre pandemic. Now we're on the podcast, post pandemic. So if ever we make the third podcast, I'm kind of wondering what would happen in that timeframe.

I don't know. I'm so envious of you because we're still having it here in the American South. I live in the Atlanta metro area, and it's right now as bad as it was in the worst of the previous thing. So we have some work to do here. But I am incredibly envious of you fellas being in areas where things are getting a little bit better.

Well, yeah, I mean, who knows what the future holds? Who knows what the next. Like you say, Michael, the next two years will hold. But I mean, the one thing that's true is people will need to label data. I think that that's something we can all agree on.

That was such a good transition.

You know, I tried.

Oh, my gosh, that was great. I love the way he did that.

So I'm curious, Michael, just from your perspective, as you're day in and day out in data labeling world, like and through the Pand, what has shifted or changed? Or do you see like maybe certain trends happening over the past two years in data labeling? What are some of those shifts that you're seeing?

Yeah, good question. It depends how you look at, like the whole markets. What I guess we are seeing from our side when we are talking to our customers and to potential customers is just previously, two years ago, five years ago, the market, they were thinking about the data labeling as something that they just need to get done as quickly as possible. Not paying too much attention into the. Some of the gritty details of the labeling process I think is now changing, is that companies are realizing that labeling is actually a way of making your raw data set into what I call a liquid asset. And because of that, they're starting to pay more and more attention around into the process and everything that goes into the process, all the reports, the analytics, the metrics inside the labeling process. I think previously it was kind of, hey, we just need to get our data labeled and go to the next step as quickly as possible. Now it's something where they are actively investing resources into building the internal process and practices around labeling. I mean, we probably knew that for quite some time now. The accuracy of the model kind of more or less directly correlates with the quality of your labeled data sets. And I think that now becomes more and more like this common knowledge for the whole market, for the whole ML market.

Do you think that that's a sign of. Of just kind of this whole industry that we're in, maturing and recognizing that people are moving from that? Like, because I remember, I'm totally guilty of that. Just get it done right. I want to get through my labeling so I get to my training. And that's the exciting part. You know, the way we were thinking back then and now people are recogniz labeling is kind of strategic to their business.

Yeah, I think it's. If you look at the whole ML kind of market, it's pretty new. And there was a lot of really good developments in the infrastructure world. And because of that, the infrastructure for ML kind of becomes a commodity more or less right now. And where your actual strategic investments may be laying in is the labeling, how you process the data and how you prepare the data. Especially if you're working with a data that requires some sort of expert knowledge to be labeled. So everything around how you capture those expert knowledge into the way of labeling data, that's a very liquid asset that you have on your hands after you're done. Right.

Back when we talked the first time and kind of the world was always about the GPUs and everybody was buying the latest GPU coming out. But there's no competitive advantage because that's available to everybody and the same thing. Whereas the way you label is something that you can create strategic advantage for your business. That's based on your expertise.

Right, exactly.

If you have a great tool to do that with. And so I think that's really changed mindset wise since we talked last.

Yeah, exactly, exactly. And that's a great point because when you start paying more attention into the process and how you set up the process, the software that enables you to do that becomes very important. And from the software perspective, it needs to provide you with all sorts of reports around the labeling process so you can understand all the metrics, where we are and where are we going. So you have very tight control over.

That and those metrics around the data labeling process. Do you see customers kind of putting the importance on the quality of the labels, like how consistent they are, how correlated they are with this or that, or is it more important sort of quantity? Does what I'm asking make sense? Like in terms of quantity and quality, how have you seen people thinking about balancing those two things recently?

Yeah, good question. I think it depends on where the company is in terms of their ML adoption. What is the product that we are building? Is it just out of the RD stage and it's something new and we need to test it out so the quantity is more important over the quality or is it something that we are using, for example, in trading on the markets? Because there the quality becomes very correlated with the dollar impacts that your model is going to make. Right. So it depends on the model and what is the company's like their actual business use case? I think over time quality will dominate over the quantity.

And you mentioned culture a couple of times, like the culture around data labeling and also experts within an organ participating in labeling. Something that I've sort of both struggled with and tried to move forward with, but it's been a struggle, is figuring out all of the best practices around how you present a labeling task to your labelers, how they view the task, how there can be so much variability between labelers, how different groups need sort of different types of instructions or onboarding and this sort of thing. How does a company approach that type of situation and maybe start to build up a bit of that culture and a bit of that knowledge internally around expertise in data labeling.

I guess it's a very, very good question and a very complicated one.

I would love to know the answer because I don't quite yet know the answer and I need to probably soon.

I think you asked a fantastic question there, Daniel. I myself really want to understand that. That's the hard part. When you're getting in.

It's a complicated set of answers maybe. But yeah, you're closer to this than we are, Michael.

You're very right. Because when it comes to the data that is subjective in its nature, that's where you have all those struggles with labeling that. Because there can be so many different scenarios. For example, one of the more or less obvious things, when you are dealing with subjective data you want to distribute the same sample that you're labeling to multiple people and look at their consensus. What can happen next? You distribute the same sample to three people, you look at their consensus. All three of them agree, but that does not necessarily mean that they label it correctly because they may label that based on their knowledge that all three of them have, but those knowledge are incorrect.

So there's a bias there.

Yeah, it is a bias. Yeah, they basically all three of them are biased and they're biased in the wrong way. So there needs to be a verification step after that. But that's kind of on the. When you actually start labeling data. Right. What comes before that is when you mentioned instructions for the data labeling, what comes even before that? What we have found out working with the large organizations is the teams inside large organizations. They can run multiple data labeling projects. They may use the same semantically named labels, but the actual name of the label is different. So you can think about all the varieties, how you can use the label name for the first name and last name. It can be F name, first name can be just name and things like that. So even on the company level, when you start and you launch the labeling project, it can be already inconsistent with how other teams inside the organization are thinking about it. Right.

That begs the follow up. If you're in a company that is wanting to move, you may have dabbled in deep learning, but there's so many organizations out there that are still not fully in, they're not at a mature level. They're exploring it still, and they're trying to understand how to be productive without wasting lots of money and folks on that. And so how should a company be thinking strategically? So not just the practitioner who's doing the labeling themselves, but if you're leadership and you're wanting to invest in workflow and an infrastructure to support it, how should you be thinking about that in terms of what your company is trying to achieve so that you get the best out of your practitioners when you have them go in and actually do the thing?

Yeah, great question. So what I think from the company perspective, from the leadership perspective, take very small steps towards the goal. I think validating the use case with a very little labeling as little as possible is the first step. And understand the process, basically the process that gets you to a consistent, high quality labeling in the end. And this process would be different for every organization. Right. And it depends on the. On a number of things. One of those is what type of data we are labeling. Is it subjective? Does it require subject matter experts, et cetera, et cetera. And another kind of dimension is what resources do we have to do that? Do you want to include people from the operations teams? Is it going to be a data scientist labeling for the most part? How are we going to be doing the verification step? Who is going to be doing that? So understanding this process to get you to a consistently labeled results. I think this is a critical part in thinking strategically about the data labeling.

And one of the things that I've sort of run into a few times recently is even just like the concept of data labeling is like it's so common to us as like data scientists or practitioners and we totally get right away why we need this and often the huge value that it provides. But then convincing other groups within your organization about an investment in this area and thinking about this area, it's actually more difficult than I expect in many cases.

That's a good point.

Yeah, I mean, I don't know if it's, they think, oh, we're investing in AI, not data labeling or you know, whatever the like thought process is or.

What'S this label thing?

Yeah, yeah, I don't know. As like the CEO of a data labeling company, I'm sure you have many of these conversations with like maybe non technical people about this process. Any tips for like explaining sort of the value of this, an investment in this area to people that are maybe not data scientists or aren't doing the sort of, aren't using the data themselves.

It's so central and yet it's kind of invisible to people who aren't actually doing the work.

It's more or less like that because when you think about the data labeling and you think about the budget that you would need to invest into that, you kind of start thinking, oh well, maybe we can figure something out. So the way I'm thinking about it and the way I usually talk about it with our potential customers is we are right now in a pretty unique stage when ML&AI is being integrated almost into every organization in the world. One way or another. Products are being built based on the ML models. And for the companies, one of the easiest way to improve their model performance is actually improve their data labeling processes. And improving your model's performance would mean that your product becomes more competitive on the market. If your product is more competitive, you can capture more data with your products and you can improve your model even further. I look at it from the way why Google dominates the search space because they were the one to figure out the algorithm that would make their engines stronger with more of web pages that they crawl. And the same thing with the data labeling. You can improve the models and make the models stronger, more competitive, capture more data and keep improving basically over time. I think those companies that are investing to the data labeling now would dominate their specific markets.

That makes a lot of sense when you think about it, and I think it's something that is that outside of our tiny community it's not really understood very well. The effect effective labeling can either make a data set really, really useful and really productive across multiple things and yet if you don't do it well, you can end up with very poor results with the same data.

Right?

I think going to Daniel's comment a few minutes ago, I don't think that's well understood in a lot of executive ranks.

SignalWire is real time video tech to help you create interactive video experiences previously not possible. It gives you access to broadcast quality ultra low latency video that's proven and trusted by Amazon, Ring Doorbell Zoom and others. See why the future of video communication is being built on SignalWire? They have easy to deploy APIs, SDKs for the most popular programming languages, and expert Support from the OGS of software defined telecom tech. Try it today at signalwire.com and use code AI for 25 in developer credit. Just visit signalwire.com that's signalwire.com and use code A.I. to receive that 25 bucks looks once again, that's signalwire.com code A.I.

So I think we've done a good job at really diving into the sort of value of data labeling and how people are thinking about it now, which is fascinating. But I'm curious. On the tooling side, I definitely want to get into Label Studio and what has progress since then, because I have some thoughts and questions there too. But in terms of this space in general, how are you seeing this space of like tools around data labeling kind of grow and shift over these past couple of years?

It's pretty crazy. It's very crowded market in some sense. There are a lot of tooling. Even two years ago with our previous podcast, it was already pretty crowded. Now there are kind of more players and more tooling around the data labeling tools, which I think is interesting. So there are all sorts of data exploration tools as well that are now available. On another hand, it's very exciting to see a lot of smart people putting a lot of effort into building all that ecosystem at the moment.

Yeah. So with that, why don't you tell us a little bit about Label Studio itself in case people haven't caught the previous episode. But also a lot of, you know, a lot has been updated since then and I've got a little bit of a secret to tell. You know, Chris, I'm not unbiased in this conversation because I am a Label Studio fan and user. So our organization uses Label Studio and I've used it on a bunch of different things over the past couple of years. So I am personally very happy that we had that conversation two years ago because it saved me a lot of work over the past couple of years, you know, looking at Google sheets where people have tried to label something or something like that. Anyway, tell us a bit about what Label Studio is and how people can use it.

Maybe. Yeah, sure. So Label Studio, it's an open source data labeling tool. One of the key features that was a key feature two years ago and it's still one of the key features today, is that it's the most flexible data labeling tool. So instead of us giving you the interface that we think would work for your use case, you use the configuration language that we have created to build the interface for your specific data set. It's multi data type, so it supports a variety of data types if you want to label for text, for audios, for images. And it's also multimodal so you can put the text and the audio and multiple images on the same screen and label them at the same time. To install. There is our website, you can Google Label Studio. As I said, it's open source software. So there is a PIP package, there is a Docker container. All of that you can get up and running in a couple of minutes.

Yeah, I love the fact that I can just launch the server. Right. So Chris, I'm going to brag on Michael's work here for a second. So I can launch the Label Studio server and then it's available and then I can build different data labeling projects within the one server. Right. So it's not like I have to like pre configure my labeling task and then like launch it. I can like launch it and then like build my labeling task in the interface, which is kind of fun because I'm not like a UI person by any means. So being able to do that and the sort of customization has been cool for us. I know because like for example, the last one I set up was question answering dataset labeling. And even if there's like a pre configured thing for question answering, it's like there's a question, maybe a context or a passage from which you're answering that question and you like, select the answer in a text or something like that. But for our task, we actually had nine different contexts and we had to select the answer in each one of the nine different contexts. So, like, in another tool that wasn't like this, I would sort of have to shove all of that together and maybe like the same text and then like create little, like markers to separate them and do pre and post processing. But here I could just add like nine different text blocks to like Label. So I think for me that customization is like a really key feature. Maybe you have stats on this in terms of like, what people are using, or maybe you don't, but you have templates. But do you see a lot of people, like in your Slack channel and other things? What are some of the creative ways that you see people customizing these tasks?

I think one of the powerful features of Label Studio is that you usually just need around 10 lines, well, depending on the number of labels that you have. But sometimes you just need 10 lines of this configuration language to build a pretty complicated interface. And that's what we usually see in people coming up with interfaces with like 10, 20, 30 lines of code. But at one point I was on this interesting call with one of our potential customers at the moment, and they just shared their screen. They used 400 lines of configuration language to basically almost be like a web portal inside Label Studio. It was crazy just to look at that, like, all the different things that they have put in there, like how.

Much time they've spent in that window making that thing.

Right, right, right.

That sounded sort of like almost another mea culpa from Dan right there. Like I've done that, you know, kind.

Of like, oh, yeah, we won't get into that. But it was interesting to see, like.

What are some of the, like, extreme use cases that people have done with the tool.

So I'm going to follow up. Since I got to tell you, Michael, I don't think I've ever seen him this just purely excited and one of our episodes about something. So I'm. I'm very impressed.

Oh yeah, I'm a total fanboy.

He totally. And the listeners can't see, but we can see each other in the thing and. Yeah, totally, man. I want to convert what the excitement that Daniel has over to our listeners that haven't used it before. Can you take that and kind of talk a bit about your workflow for someone who hasn't had a chance yet? And they're sitting there in their car, they're in traffic right now and they're going, when I get to the office, I'm going to open this up and go do it. What should they expect? What should they be thinking about? What's the workflow look like?

Yeah, so in a nutshell, it's a web app, right? So this is something that runs in your browser. This is something that you can launch on your EC2 instance or whatever the server you're running on your laptop, it doesn't need a connection to the Internet. You go to your browser and you can start creating the data labeling project. As I said, it's very flexible so you can configure it specifically for your data sets no matter what. Your data sets consists of images, dataset, if you're doing the computer vision, the NLP tasks named recognition, audio, segmentation, all a variety of the use cases. And then what's also interesting is you can start connecting the machine learning models to help you do the annotation. And the flow of that is very interesting because you do the annotation, a batch of annotations and you retrain your model and then you do another batch and you retrain your model again and you get the new predictions out and you keep improving the model predictions this way. It almost feels like teaching a child to do a certain specific task for you. And this is something that I personally find very exciting because for me it was like, okay, I can see the actual improvement of the model in its predictions in the real time. And this is just one of the setups how you can set up the Label Studio. Another one might be that if you're running some sort of human in the loop types of ML pipeline, where if the model is not sure in its prediction, you send that to a person and the person would be labeled inside Label Studio. As soon as they label, that gets back into their retraining phase. And it's another use case. But we put a lot of effort into making sure that the actual interface inside Labels Studio is very intuitive and easy to follow. Because we as a data scientist, we always focus on ourselves, but we also put specific focus on non tech savvy users because those users can actually have a lot of very specific, the main and interesting knowledge and you want to be capturing that. But if you build a tool that is very complicated, they would spend a lot of time figuring out how to use that. So we are trying to make it as simple as possible, but yet powerful. I hope that I got somebody interested.

I bet you did.

Yeah. Maybe that's part of Going back to the hard questions that we discussed earlier about building that culture of data labeling, getting instructions right, setting up the task right, doing verification, all of these sorts of very hard things. If you don't have to add to that, like an interface that people is very unfamiliar and they can't figure out how to use, then that's at least one less thing to complicate those muddy waters. Yeah, so like a web app and a browser where people can just sort of click things. That's a nice scenario, I think. I'm sure you've looked at sort of user experience, UI type things over time. Michael, what has that experience been as you've developed the front end of Label Studio and maybe things that you thought would work and didn't work, or maybe things that you integrated and people love but you didn't think would have been as big of a deal.

He's smiling before he answers.

Many, many. What I think is interesting. One other reason why we want to give you this flexibility of building the UI is because you can keep only the relevant parts that are relevant to your data set and that minimizes the error on the annotators so they don't have something that they actually don't need to use in terms of the UI elements. Another thing on the UX and UI design, I want to really praise our open source community. So we have had dozens of UX sessions with our open source users. We basically just invite them and we ask them to click around and show them their prototypes. They've been very, very helpful in designing the application we have right now more than, well, not more than, but close to 2,500 people in our slack and some of those community members are very, very helpful and we can do ourselves only as much. But then the other bigger part comes from all the contributions in terms of like how we may think about certain cases from our community. On the mistakes part, we have done a lot of things that didn't really work, but we were quickly identifying those things that are kind of not really user friendly and just removing them. So we have this tendency to really understand what doesn't work. Try not to clutter the application because again, we try to make the tool as simple as possible, but yet to have all those powerful features, which is more of an art than a science, I guess.

So I'm curious. You've got me excited all over again. I remember the excitement of this because I went, I haven't been doing as much hands on as Daniel has, but I went and started using it as well afterwards. I just don't know it as well. I'm wondering though, as you've done these improvements over the last couple years and you're looking at a world where this is just going to become more and more embedded in business and so it's not this off thing. And the amount of data as people are finally starting to learn how to collect their data and not have it transient and gone and they're storing it and the volume of that data goes up by orders of magnitude and to some degree for use cases, the quality of the data may vary tremendously on that. How do you envision the process of labeling going forward in the years to come as the problem, you know, it's solved in the sense of what you've done so far, but it's an ever changing problem and so you're constantly going to have to chase that down. How are you seeing that curve into the future?

Yeah, great question. I think in the future. Well, there are different vectors. First of all, with ML adoption itself, we're just going to have more and more use cases that ML can support. Meaning that from the labeling perspective we would need to do a lot of the multimodal labeling and just different varieties of labeling that we are not thinking about right now. Then from the perspective of the volume, we going to have a lot of the really well pre trained models that are going to help us label a vast amount of data automatically and then basically what people would be concentrating on other edge cases, domain specific cases and just the cases where there are not good pre trained common knowledge models available yet. So I think with more ML adoption in the business, we'll see that the need for the labeling is only going to grow just because from the fact that ML is going to support more and more use cases within the business.

Do you get any sort of weak signals off of what people are doing with Label Studio in terms of the different domains of machine learning AI in terms of like people's focus on NLP or people's focus on audio or like new things with computer vision? Do you happen to see like trends in that with like new people coming into Slack wanting to do all of the sudden or you know, something like that?

Yeah, it's a good question, but it's not that easy to answer that too because what we see for example, we are one of the very, very few tools that supports time series labeling and majority of the people that are doing time series labeling they use Label Studio. So we see a lot of those people. But then on the other hand we see a Lot of people who are doing labeling for computer vision and they're, well, not comparable. There is more people doing computer vision, but we have more competitors in terms of the computer vision data labeling tools. So I would not be able probably to say that there is a one dominant data type that we see people using the tool to label. Again, because of the flexibility of the tool. And over time, I don't think that we see a lot of the change because the data types are distributed more or less equally. Just because from the fact that people usually use a few of the data types at the same time on the same screen. You can think about audio and then there is a description for the audio and you kind of want to have the audio on the same screen just for the reference and you're labeling the transcript. So it's hard to pick something and identify the change. Just given the flexibility of the tool and what it provides to the end user.

Are there a lot of use cases for having kind of that multimodal approach? If you go back a couple of years and people would tend to be either really focused on NLP or really focused on labeling convolutional, but they tended to be one or the other in my experience. And what I've seen since then is a lot more integrated integration and less emphasis on what it is. Are you seeing more of that in the tool and how does the tool accommodate that workflow as it's evolved into this kind of mesh of different approaches together?

Yeah, totally. So I think there is a clear trend for multimodal being on the rise. And from the tooling perspective, because you're building the interface yourself, you can put as many data types on the same screen as you want. And because you have this ability, even if you don't do the multimodal label and you label, let's say, just the text, right? Having the audio for the context, if the text is not clear because the transcription was screwed, you still would want to do that. So the tool kind of naturally supports this type of use cases. But yeah, I agree that I think that we are definitely moving into more of the multimodal world.

So we've talked a lot about the tool itself. It is an open source tool. And I'm always curious too, like in terms of people that are building a business around open source, whether that be Label Studio or Hugging Face or, you know, these sort of Grid AI people with Apache TVM or, you know, all of these different cases, it seems like people of course are always playing with that model and figuring it out. How it works. So you've got the open source tool, but then also you're dealing with a lot of data and open data is very much a topic right now. So how do you think about building your business in this space around an open source tool? And has that been a struggle in terms of that balance for the open source company?

It's always a struggle because it's open source. You have to identify what you actually want to sell as a product. So that's always a complexity. But I think there are some really strong advantages to being the open source company. One of them is a community. You get a lot of insights from the community. They help you improve your software the way you would not be able to improve your paid product. Another one, because of the community, you have one of the largest community of testers of your software, meaning that your software becomes so well tested just because of the pure adoption of that, that you have one of the most stable tools out there, which I think is great. And those advantages, I think at least to myself, they compound over those disadvantages where you have to struggle sometimes to identify if this needs to go into the open source or that goes into the paid product. And then another one that I think is really important from the company perspective is all our developers, they have direct access to the users of their software and that makes them really happy when somebody is getting a lot of value out of it and they talk about you on Twitter or somewhere else and they just tell you how great is the tool that you have built. For the developers, it's basically the best they can get and they get very excited about that. So I think, at least for myself, the advantages of being the open source company clearly beats all those disadvantages that we may have.

We've seen a lot of the tooling in the larger kind of AIML space start as open source, which was very different. And I can say this because I'm on the older side. From the way the software world started off, where it was all commercial and open source kind of came in, and it was small for a while, but it grew and took over. We've had the benefit in ML of being able to kind of start from that. Do you think that that will continue to be the model just because of the benefits that you just now drew out, that it allows you to kind of accelerate and be totally connected with your user base?

I think there are going to be, I hate to hypothesize on what future holds for us. If two years ago you would ask me about the future, I would probably not be able to predict the pandemic.

Coming, nor did any of us.

But I think it's going to be a mixture of both. I think they're going to be a huge market for the commercial only closed source solution and they're going to be a market for the open source a different models and I think both of them have their advantages and disadvantages. So I'm kind of not thinking about commercial being kind of in the way of open source. I think they can just go in parallel.

Yeah. And data labeling I'm guessing will always remain fairly challenging. There'll always be an opportunity, I think for people to build a business out of those things. Just because it is so challenging.

Right.

Yeah. I like what you were mentioning earlier about like one of the things you're sort of keeping track of is people using sort of models to seed their labeling. So pre trained models to do that and that's definitely something that has benefited us. So it's cool to see that you're integrating some of that ML layer within Label Studio. I'm curious as a user, maybe not as futuristic of a question, but what does the roadmap look like for Label Studio? What are those things that you really like, you're getting a lot of requests for or you really sort of at the top of your mind in terms of enabling in the tool?

Yeah, I think first of all we do have a public roadmap on our GitHub. I'll share one that I think probably the most exciting one. In the next couple of months we're going to be releasing the support for video. Right now you already can do the video classification type of tasks, but we'll be releasing the one where you'll be able to do the object tracking.

That's cool.

Yeah, this is going to be huge with the introduction of this proper video labeling. We'll be covering all major data types and that was our kind of goal for the labeling tool. So that's pretty exciting.

Yeah, I mean I know that there's like cloud video editing like tools and stuff now. I'm sure that that does definitely like put a bit, maybe more of a strain on the tool in terms of its performance than other things. Is that like, is that coming last because that major category that you're talking about coming last because of some of those challenges or is it just sort of like the growing need for that and that's what sort of came next?

It's a mix of both. It's both challenging from the. It's just the complexity of lvidia labeling is very high. Then we see more need in terms of our users asking us about labeling. And then we just started with images, audios and text. Just because the founders of the company, we all were coming from exactly these backgrounds ourselves. So the video labeling, for some reason it comes to the last one, but I think it would be a good kind of thing to get all of them under the one hood. Yeah.

So it's interesting to me that you said that and they announced that because I think it puts pressure maybe on an area in software that wasn't really thinking of themselves as having to worry quite so much about ML. It's not central to the thing, it's ancillary in some. But there's a whole new arena that you're going to be moving into in that sense.

Yeah, totally. And I think what's also exciting is because again, you can put different data types and we try to make them talk to each other. There are very interesting use cases that can be uncovered by that. One of the use cases that was very much requested by our community is, for example, when you're doing the time series labeling and you want to have the reference with your video stream because when you think about labeling those plots, they may not provide you enough information by themselves to do the labeling. But when you have those video stream that is, this kind of acts as a reference for what, for example, the robot arm was doing. Then you can effectively label the time series. And I think this kind of merge of different data types on the same screen supporting each other just opens another box of the use cases that would be available to the community.

Awesome. In two years from now when we are having this conversation again, what do you think we'll be talking about?

No more pandemics though.

Yeah, I mean, hopefully no more pandemics. Maybe the data labeling world will be fully taken over by Label Studio. But yeah, I certainly hope that we have that conversation.

I think he's going to have Steven Spielberg sitting next to him at that point.

Maybe I already have him.

Well, it's very been super fun, Michael. I really appreciate. I'm looking forward to that conversation in a couple years. So thank you so much for joining.

Likewise.

And we'll make sure our show notes include all the links to great Label Studio stuff. So all of you who are listening, definitely check it out and we'll talk to you soon. Michael.

Thanks guys. Thanks for having me.

Thank you for listening to Practical AI. We have a of bunch a bundle of awesome podcasts for you@changelog.com including our brand new show Ship it with Gerhard Lazou, a podcast about getting your best ideas into the world and seeing what happens. It's about the code, the ops, the infra, and the people that make it happen. Yes, we focus on the people because everything else is an implementation detail. Subscribe now@changelog.com shipit or simply search for ShipIt in your favorite podcast app. You'll find it. Of course, the Galaxy Brain move is to subscribe to our Master Feed. It's all changelog podcasts, including Practical AI and shipit in one place. Search Changelog Master Feed or head to changelog.com master and subscribe today. Practical AI is hosted by Daniel Whitenack and Chris Benson with music by Brickmaster Cylinder. We're brought to you by Fastly launch Darkly and Linode. That's all for now. We'll talk to you again next week. SA.