Open source data labeling tools

Bandwidth for Changelog is provided by Fastly. Learn more@fastly.com we move fast and fix things here at Changelog because of Rollbar. Check them out@rollbar.com and we're hosted on Linode Cloud Servers. Head to linode.com changelog this episode is brought to you by Digital Ocean, the simplest cloud platform out there and we're excited to share. They now offer dedicated virtual droplets and unlike standard droplets which use shared virtual CPU threads, there are two performance plans. General PURP CPU Optimized they have dedicated virtual CPU threads. This translates to higher performance and increased consistency during CPU intensive processes. So if you have build boxes, cicd Video encoding, machine learning, ad serving, game servers, databases, batch processing, data mining, application servers or active front end web servers that need to be full duty CPU all day every day, then check out Digital Ocean's dedicated virtual CPU droplets. Pricing is very competitive starting at 40 bucks a month. Learn more and get started for free with a 100 credit at.co changelog again.co changelog. Welcome to Practical AI, a weekly podcast about making artificial intelligence practical, productive and accessible to everyone. This is where conversations around AI, machine learning and data science happen. Join the community and slack with us around various topics of the show@changelog.com community. Follow us on Twitter. We're @practicalai FM and now onto the show.

Well, welcome to another episode of Practical AI. I'm Daniel Whitenack, a data scientist with SIL International, and as always I'm joined by my co host Chris Benson, a principal AI strategist at Lockheed Martin. How you doing Chris?

Doing great, Daniel. How's it going today?

It is going good and I think when listeners will be listening to this in the future, if we're to imagine into the future, you will be at the Nvidia conference, is that right?

That is accurate. As listeners are downloading this, I'm probably in Washington D.C. i've just finished doing some commentary for the Alpha Pilot race and those of you who aren't familiar, we had a recent episode about Alpha Pilot which is really cool, autonomous drone racing. And as of the point where you're downloading this on Tuesday, which may potentially be tomorrow for you, I will be conducting a sort of a fireside chat slash panel about Alpha Pilot and AI at the edge at Nvidia's GTC DC event on Tuesday evening. So I hope if you're listening to this and you happen to be at gtc, I hope you'll come attend and whether you do or not find me, I'll Be there all week and find me and let me know you're a practical AI listener and let's connect. I can't wait to meet people there.

Yeah, that sounds great. I can't wait to hear about how it goes. Well, today, you know, if I think about practical AI, Chris and I asked you, what is the most practical of practical AI things, what would you say?

You already know what I'm going to say. You're setting me up for that. It's labeling.

I'm teeing you up.

Oh, my God. It's labeling data. The bane of my existence over the last few years in terms of doing AI. If we can get the data labeled, then I'm finally at a happy place where I can start doing training, training and have some fun.

So today is all about the data and all about labeling the data. And we're joined by Michael Malook, who is CEO and founder at Hardex and one of the contributors and maintainers of Label Studio. Welcome, Michael.

Hello. Hello. Thanks for inviting me.

And we were able to grab Michael for an interview while he is doing his world travels. He's, I think right now in Moscow. Is that right?

Yep.

Awesome. Yeah. Well, thank you for joining us even in the midst of your jet lag.

Oh, yeah, yeah.

So if you could just give us a little bit of an intro to yourself, your background, how you kind of got involved in AI ML things, some of the things you've done, and how you ended up with a focus on data labeling. Sure.

So I got into AI. It was quite, I think, interesting path for me because I have started with Lisp programming in Common Lisp and that at some point I got this book by Peter Norvik, who I think right now at Google that is called paradise of Artificial Intelligence. And it was kind of a mixture between Common Lisp and using Common Lisp to program AI. It was quite an old book. Even when I got it, I think it was released.

Yeah, it sounds intense.

Yeah, it was like 91 or 93, something like that. But it got me really interested into the whole concept about AI and how to program AI systems. And so from there I kind of started to pick up all the actually need to be built in production level AI systems. And I'm here talking about more like math background, statistics background and. Yeah, so that's what got me really, really like really interested into the whole AI concept. Commonly 50 years old programming language.

Well, my first programming language was Fortran and used it quite a bit. So I'm glad that I left it behind. But we all have at least some of us have those roots in older languages.

Yeah. It's like if we're talking about Common Lisp, it's like I still kind of big fan of the language and it's a pity that it's not kind of catching up with and not becoming more mainstream these days.

How common is it to find people these days working in AI or machine learning and using Common Lisp?

Not common at all. Not common at all.

Not common at all. To give a pun.

Yeah. The only one who I can actually remember is the person, Gabor Melish, who has won a number of AI competitions. I think those were organized by Google and he wrote all his programs in Common Lisp and those was like very interested to study and to explore how he would approach things.

Yeah, yeah, that's cool. So you started out there. How did you end up coming to the point where you started thinking about founding your own thing that would impact the AI community?

Yeah, sure. Yeah. So at the time, and it was something about a year ago, me and my then to be co founder, we went to this hiking trip in the high altitude mountains and I think the lack of oxygen kind of contributed into us starting the company.

Where was this?

Himalayas.

Oh, wow.

Awesome.

Yeah.

What was the.

The max elevation?

So I think I was giving up at almost 6,000.

Yeah, 6,000 meters.

Yeah, yeah, yeah, yeah. My co founder, he went further, but the weather was really, really bad, so he had to turn back.

So. Yeah. So as co founders, you've been through a lot together.

I'm so jealous.

Yeah.

I mean, it's probably as hard as it gets. Like all those conditions, you know.

Yeah. What is VC funding and fundraising rounds as compared to that?

It's the easy trip. 2,000 meters.

Yeah. Nice. So the company is Hardex and so, you know, not to. I think we've already spoiled the punchline here, but it has to do with data labeling. So how did you kind of get to a place where data labeling was something that you felt like you wanted to devote a lot of effort to?

So we kind of, we were working on some algorithms in our spare time, just trying things. And by the time we decided to start a company, we all realized that at our past or current companies, the biggest issue that we had to handle ourselves was the data labeling part. And we were talking to multiple data scientists, machine learning experts, and they have all agreed that it's kind of unsolved problem and more or less like a holy grail, if you're able to get your labels in the right moment. In the right place, then you basically end up with high quality models. And by the time we realized that it's not only us who had these problems, but also other developers and other companies, then we decided to proceed and start the company around the data labeling.

So I for one am, you know, you heard how I reacted to that in the very beginning about labeling that being the bane of my existence. I for one am thanking you very much for attending to that issue. Because, you know, everybody in the data science world, you know, wants to do the sexy AI training and such as that. And this is a problem that has needed serious attention for a long time. So as I ask you about it, I just want to start by saying thank you very much.

It's a problem I feel like, have wider impact than like creating a single state of the art model and publishing like a very hyped paper.

Oh yeah.

Like this is a sort of problem that has a much wider impact.

Absolutely. So I guess I want to start off by just kind of, if you could kind of tell us a little bit about where does data labeling fit into the kind of the larger workflow that we're all engaged in in the AI world? And you know, how does it relate to the AI problems that we are all working on? And you know, Daniel has been focusing on lots of stuff having to do with language stuff and I've worked in robotics and different types of image classification stuff. So how does labeling fit into these workflows?

Yeah, sure. So I think my personal take is that data labeling and annotation is basically the core of any AI based product. Because if you are building on the label data, of course, because if you are not able to go into your data sets and relabel some things that for example, might have been labeled incorrectly or inaccurately, then you just end up with the models that are not working well. So my take is that data labeling, it's the first step that comes after collecting the data and it's something from where everything basically begins. And I also like to think about it in a way that sometimes when you get the data set or you collect the data set, you're not sure what's inside of it. So it's not only that you need to label it, but while you're doing the labeling, you are also kind of exploring your data set, finding the edge cases or some examples that you have not thought about. So I think the data labeling is basically the core functionality, or shall be a core functionality of any data science team or the team that builds AI based products.

Yeah, so in terms of the types of data that people generally need to label or annotate, as Chris mentioned, there's of course a lot of different types of data that are relevant to AI. And in some cases AI models might work on multiple types of data. But maybe you could just give some common types of data that people need to label or annotate. Or maybe the most common ones that you run across.

Sure. So I would say that most common ones images, of course. So when you are placing a bounding box on the image, when you are doing semantic segmentation on the image, you can also think about even classifying images as a data labeling problem. So you basically assign in a class to the entire image what you see on the image. Is it, for example, an image of a fruit or image of a person, then text. So you can be classifying text, for example, for sentiment, you can be doing named entity recognition audio. Again, you can be doing speaker separation, you can be classifying audio for number of classes or doing the multi class classification. So those, I would say most common ones, then you of course have time series. Now you also have 3D spaces with all the LIDAR sensors that are coming. And this data is coming from autonomous vehicles and videos. So I would say these six data types are the most common ones. And inside each data type you kind of have different types of labeling or annotations that you can apply to this particular data type. And that usually depends on what you are looking to achieve in the end. Yeah. And as you have said, you can also mix those.

Yeah. So the type of annotation that you perform is really tied to the task or the objective that you want your model to perform.

Yeah, totally.

Right. So it's like if you want to pick out entities in text, which is what's done in named entity recognition, like you want to determine are there any places or people that are mentioned in this text, then you need to have data that you have manually labeled in some sort of gold standard way to, to help the model figure out, based on those examples, what the relationship should be between input data and the output of those entities. So there's a whole bunch of, I guess there's probably infinite different types of annotations that you come up with because there's infinite different tasks that you might want to perform. Is that accurate?

Yep, totally. Yeah, it depends on the data set that you have at your disposal. And then it depends on also what you're looking to achieve with your model. And you're absolutely right. You are labeling the data and then based on your labeling, the model kind of learns the patterns and try to identify the same patterns from the new data that comes in. And that's how you basically get a prediction from your model.

So it occurs to me that we've kind of thrown some terms around a little bit and we've talked about labeling and annotation and recognizing that not everybody has had a chance to do this that might be listening. What does it actually mean to annotate that data? You know, when you're annotating or labeling data, what is it you're specifically doing? What is required to achieve that, and what is the practical benefit of going through that process?

Sure. So the process itself, again, it's very dependent on the data set and what you're trying to achieve, like the problem that you're trying to solve. But in general, I would describe it as basically creating some metadata for every item in your data set. So this metadata, for example, for an image, this metadata might be a rectangle at a certain position with a certain class applied to this rectangle, which is basically called a bounding box labeling. And to do so you need to have the tool that enables you to put this bounding box at the right position and to assign the label the class that you want to this bounding box. And a lot actually depends on the tool, how accurately and how quickly you can do that in case your data set is super big and in case you're looking for very high quality labeling. Because in the results, the quality of your models in most cases is directly tied to how accurately your data is labeled.

Gotcha.

And one of the things that you mentioned, by the way, being bounding box a couple of times, just wanted to note, kind of define that for a second, a bounding box. If you're looking at an image and you're trying to define the value of different parts of that image for purposes of training subsequently, then a bounding box is really just like it sounds. It's a geometric shape that you're assigning to different parts of the image to define the different areas of the image that you want the model to either focus on or not focus on. Yep, I just wanted to note that.

Yep.

Foreign.

What is up? Practically I listeners, we're working with Infinite Red to promote their free AI mini course. It's called AI Demystified. Learn more enroll@learnai.infinite.red this free 5 day mini course is a great introduction to the most important concepts, types and business applications for AI and machine learning. Each day of the course includes a lesson A quiz and an assignment to cement your learning. And after you've completed the course, you'll also get a certificate of completion for your LinkedIn profile or for your portfolio. If you've been feeling lost in the world of AI and hearing lots of buzzwords, then by the end of this mini course, you'll be able to speak intelligently about AI and machine learning and their practical business applications. Again, this course is completely free. Learn more and enroll@learn AI.infinite.red again. Learn AI.infinite.read.

Okay Michael, so let's say that I'm convinced that I need to do data labeling and I'm convinced that I should put time into it. It is an important part of my AI workflow and one of the most important parts because it has this direct impact on the quality of my predictions. And let's say that I have 100,000 samples or more to label. There's obvious challenges around the like time consuming nature that it would take to label each of those hundred thousand samples. Are there other challenges? Like I'm thinking in terms of maybe bias or like crowdsourcing this or like what sorts of challenges do people face when they're labeling data? Maybe other than the obvious one, like the time consuming nature of it?

Oh, many, many. So time. Yeah, definitely. If you have big enough data sets, millions of items, it's going to take a lot of time to label it then quality. How do you verify that the results of the labeling and the actual labels are. For example, if we're talking about bounding boxes, that the bounding boxes are in correct positions. Biases. When different people label the same data set, you may end up with different results. So personal biases then even before that you actually need to have a tool to help you do that because data sets are different. Types of annotations and labeling are very different. So you have to invest time into either creating your own tool or using something from the open source. And there are basically many more. I would say that two major ones are time and quality.

Yeah. So quality being around the verification of the data. Right. So like if I crowdsource, let's say a million parallel translations between two languages, how do I know that those were actually good translations given that I don't already have the model? So it seems like it's a sort of chicken and egg sort of thing. How do you deal with something like that?

Yeah, to add to that, that works if you can crowdsource. Right. So for example, if you are dealing with the data that requires kind of like domain Specific knowledge, for example, medical images. Right. You can't crowdsource that.

Yeah. And it's expensive to hire doctors, I imagine.

Yeah. Then another one, if privacy is an issue, then you also can't crowdsource that you need to have in house data labeling team. So for the quality control, there are multiple ways how you can verify the results. One of them is you can kind of label fraction of your data set, verify it multiple times that it was labeled correctly. Then you can train a model on top of that and further use this model to verify the subsequent labels that are coming. Another one, you can distribute the same task to multiple annotators and verify if they're in consensus between each other or not.

So when you're talking about sort of bringing the model into this process, is that what people refer to as model in the loop versus out of the loop labeling, where you actually kind of have a model that's trained on some of your data? When you're trying to label more data and updating that, is that what that means?

I call it more or less like automatic labeling. And here the most important piece is how do you pick those items in the first place? Basically, if you have a very large data set, how do you pick those items that you want to label first? And using those labels, you kind of can build the model, but how do you pick the items? And so this field is called active learning. And active learning is basically a way to pick the items from your data set that provide you enough information about the data set as a whole. So you analyze in the data set and picking exactly those items that you want to label first in order to be able to label the rest of the data set for you.

So how are people currently approaching data labeling at this point? You know, what are the range of techniques and the tooling that you have that you would use for that, that you might have seen and also I guess what's lacking in that at this point.

Yeah, so I think right now there are two ways basically. First one is using the services. So you just send your data sets to the service company and you get back the label data sets the results. And second one is either building your in house team or just using your data science team and using the tools to help them do that. So the problem with the first one with the service companies is that you don't have control over the process. So you just get back the results and then it's your job to verify if results are of good quality or not. In most cases, you don't get good Quality results. And then again, if you're dealing with a data that requires domain specific knowledge, usually you can't outsource that easily. And privacy is another issue. With the second one. What we have found out that a lot of companies, they are starting with some sort of using some sort of open source solution just to get their data labeled and basically build the first version of their models. And what they find out is that they need to upgrade the tool and tweak the tool to their needs more and more before it becomes this monsterious tool that you kind of don't want and you don't have resources to support anymore. At this point they're looking for something that is more production ready and is ready to scale.

So in terms of the range of things, I imagine that there's a ton of different types of models and architectures that people use for these different tasks. Like you've mentioned sentiment analysis, image classification, I imagine that the burden in different of these model types is heavier in terms of data labeling and maybe Chris, I know you've worked in masking images for robot perception and that sort of thing.

Yeah, different types of CNNs.

Yeah, that's much harder than let's say sentiment analysis in text where you just kind of say is it positive or negative as a result of that, are there types of problems or the types of models that you might want to create where there's already a lot of good data out there that's publicly labeled that you can use like let's say for sentiment analysis versus other problems where just due to the nature of how difficult it is to label, you're kind of stuck with doing it on your own?

Yeah, I think, yeah. So for some problems definitely. And like for the easier problems, in most cases you can use transfer learning. So you basically start with a pre trained model and then you label just a small fraction and you train the model, the transfer learning model with that data and that works pretty well. But in most cases you get the data, like especially if you're getting the data from the real world, then it's not that easy to use existing models for that.

So I guess this might be a good turning point to kind of talk about. You could tell us a little bit about your company and what Label Studio does and kind of how does the company and the product relate to each other and what are they?

Sure. So yeah, the company name is Hardix and Hardix is a data labeling platform that makes entire data science teams more productive and helps build higher quality, safer and smarter models as a result and we have open source product that is called Label Studio. So the difference between those is that Label Studio is just a front end part. So you get the labeling interface where you can upload your data and go item by item and label it and the company, the commercial offering is basically where you can also use our pre trained models to help to go through the data set faster. You can invite your whole team to collaborate on the data labeling and exploring your data sets. And we have also extensive process for the quality control helping you to verify that the results that you're getting are actually what you are looking for.

Yeah, so this is like Label Studio, you can kind of think as the open front end that anyone could use. You know, you could just get off of GitHub to help aid you in your annotation if let's say you want to start from scratch. But as you mentioned, starting from scratch isn't always necessary and isn't always practical or efficient. Right. So the things that you mentioned, I see you mentioned like auto pre labeling and native active learning, you already mentioned those things a little bit. But those are the things that kind of augment the process. Is that right?

Yep. And what I like about the open source, the Label Studio, it's the first open source data labeling tool that you can not only download and run, you can also embed it into your own pipelines.

Ah, okay. So does that mean like you can run it non interactively somehow or how would that work out in practice?

So in progress? Many different ways. So you can use the tool to create the labels and you can also use this tool to look at what at predictions of your model. So you can embed this tool into your pipeline and you can verify what your model predictions are. You can ask your team members or for example the main knowledge experts to provide the labels for specific items in your data set. So there are multiple ways how you can embed and use it and it's really flexible in the way how you can define the different types of tasks it can handle.

Could you describe some of those tasks as well? Just to kind of give us a sense of what all it can do and what's required for input and output on those?

Sure. We originally started from the idea that we as a founders of the company, of the company were coming from different machine learning backgrounds. So I was more concentrated on images and visual data and my co founders they were constrained on audio and text. So we had this idea about building the data labeling tool that is configurable. And so what we have created is Basically very high level kind of components that you stick together. Think about it as you are building the web page using HTML in the same way you are building your data labeling interface. It usually takes from 5 to 25 lines of HTML like config language and as the result you can get data labeling tool that you can use to label audio, images and text right now. And we will be adding video before the end of the year and you can do many different things. So all that we have discussed, like basically bounding boxes named entity, you can do all of them at the same time if you want to. And it's basically depending on the task and depending on the data set that you have, you can configure it as a like Swiss army knife. You can configure it and tailor it for your particular needs.

Foreign.

This episode is brought to you by Kubecon Cloud Native Con and you are invited to attend this flagship conference from the Cloud Native Computing Foundation, Kubecon Cloud Native con North America 2019. It's happening November 18th through the 21st in San Diego, California. This conference gathers adopters and technologists from leading up a source and cloud native community. Use the code KCNAPRACTICALAI19 again KCNA practical AI19 to get 10 off registration or check the shows for a special link to register and a link to the convince your boss letter. Again, check the shows for links to learn more and register.

All right, so every once in a while on this podcast, the topic intersects very nicely with a problem that I'm trying to solve in my own work. And this is one case. So in those cases, as Chris knows, I like to selfishly try to get the guests to help solve my problem on the fly. So one of those problems is I have not found a tool that will let me easily label reading comprehension data. So this is the case where you have like a question and a piece, a passage of text and then the output that you want is answer that is drawn from that text, maybe a span within the article text or something like that. And there's no tool out there that at least that I found that does that. So with Label Studio, could you kind of walk me through, let's say I came to Label Studio and what would it take to set up that sort of interface with these Label Studio components? Would that be possible?

Possible, yes. So I would suggest, yeah, basically installing it, then looking at the templates that we provide, use the template as a starting point and then looking at the tags that you can use for your particular problem. So based on Your description, I think, yeah, it's very much doable. Yep.

Yeah. So because I'm a data scientist and slash backend person and I don't know that much front end, but maybe I've hacked on HTML before, would that be something I could tackle? Like how much front end experience do I need?

So it depends on how much you want. I would say if there are components that cover your use case and we right now have around 20 different components, then you don't need any, almost zero knowledge about the front end developments. If you know how to stack up the HTML like text, then you're good to go. But then while we were creating the tool, we made it extensible in a way that you can create your own components. So for example, if we don't have right now support for the video, you can create your own components to render video and connect it to, for example, checkboxes that you can use for classification. So in most cases we try to cover, I would say 80% of the most common cases with the companies that we develop ourselves. But then we also give you an ability to extend it for your particular needs.

So if you're, you know, kind of extending that a little bit, if you're a data scientist or an AI developer and you're trying to integrate Label Studio into your own data pipeline and pull data out for experimentation, how does that integration go? And maybe draw an example or something like you're using a notebook with TensorFlow or whatever, PyTorch. And how does that look from a practical standpoint? If I'm going to sit down and use the tool.

Yeah. So from the notebook example, we will be releasing the package specifically for the Python notebook. That will make it super easy to initialize Label Studio inside the Python notebook. Work with that. If you are looking into integrating it into your workflow, you would need to install the NPM package. You just need to initialize that with the data from your data set and then you create kind of a ui. How to visualize this data and how to label it. So basically we split up the components that you have in Label Studio into two major ways. Those that are used to visualize the data and those that are used to label it. So you can think about it that if you are looking at the text documents, that's visualizing text. And then if you want to put the spans on the text, doing named entity recognition, that's another tag that is doing just that action. And you are also able to create, to load your predictions from your current models. If you have Those and you can also complete labeling in different ways. So for example, if you want to have multiple people look at the same text and let them label it, you'll have two different results and then you can compare those results and that kind.

Of pushing of predictions into the tool. Or let's say like for my training script, I want to pull the latest annotations out. I saw you mention some things about maybe a REST interface or something like that, or like how does that interaction work and the sort of the plumbing between the two. Once you've got Label Studio up and running, what's the most useful way of plumbing between an annotation tool and your training and inference in your experience?

Again, two ways. First one, where we provide just the front end part, right? It's basically npm, it's JavaScript package and you initialize the package and you send the data into the package. So there is basically, there is no API, nothing. It's as simple as that. And there is second part where we provide you with the data manager and we initialize the Label Studio front end for you. So in that case, you just give us give Label Studio the JSON formatted file and we read the data from there.

Cool. Yeah. This is still reasonably new in that you were just in the Himalayas thinking about it a year ago, but actually it seems like if I'm looking at the GitHub on label studio, it seems like there's been some activity there and there seems to be a bit of a community developing. Have you been able to get contributions of components and start to interact with the community in that context?

Yeah, community response thus far was great. And what people like about the Label Studio is that you can basically, in 10 minutes, you can build your own data labeling tool. That's one. Second one is that the UI is very simple. So because you only use what you actually need for your particular data labeling needs, you only use those components. And the UI is super simple. So there is nothing that you actually don't need in the ui. And as I have mentioned, it's first data labeling tool that you can actually embed into your applications and you can easily extend. Right now we're getting more bug reports than contributions, so contributions are always welcome.

You know, for some reason that doesn't surprise me. Yeah, but it's a good thing.

But it's a good thing.

Yeah, well, it can be a good thing.

It's a track to make things better. So, you know, it sounds like a pretty great endeavor. I'm really, really looking forward to using it. Myself, and I'm pretty excited about it. So it's been a great conversation from my standpoint. I guess having brought it this far, all the way from the Himalayas to it is code that the rest of us out here can start utilizing. What do you see as some of the biggest open problems that are still out there in this data labeling space? And how do you see it being augmented in the future, either by your own team or by contributions from outside? What's next?

Sure. So I think there are many things happening right now. A few that I want to mention is that I personally think that moving forward, at least some part of the data labeling is going to be commoditized just because the models are getting better and better and you'll be able to reuse those models at some point, maybe not right now, but soon, then there is another trend with a weak supervision that you can also use to label your data set. And so I think what's coming next is we really need to start putting a lot of thinking into quality control. Because what a lot of companies that I talked to found out is that you outsource your data labeling, you get back the labels, they are of very low quality, and as the results, your models are failing in the real world. That's a very common and valid concern and happens actually a lot. So quality, how do we verify that the labels are of the high quality? Another one is understanding and finding edge cases in your data sets and trying to understand how to label those. That's also very interesting because if you have real world data sets that consists of millions of images, there is no way you can look into each image by hand. You need some ways to automate that and pick those items for you. That needs some attention. So I would say, at least from my standpoint, we are right now concentrating on the quality control. A lot of.

Yeah, that makes a lot of sense. It sounds like there's no shortage of things to explore there and improve upon. But it does sound like actually AI augmentation of the labeling process, especially in terms of quality control. And all of those things is going to be really important. So if people are listening and they want to try out Label Studio, they want to kind of follow you on this journey. And as things come out, where can they find out more about Label Studio and maybe get started and try out a few things? And also maybe what are some of the great ways that maybe people could contribute to LabelStudio because you are getting those bug reports and other things, what are some of the ways that the community can give back as well.

Sure. So I would say the easiest way that you can try out LabelStudio right now is npm install labels studio. Then LabelStude IO is our website and GitHub. Yeah, it's GitHub.com hardaclabelstudio. We have documentation there and some quick guides how you can start very quickly. It's basically a couple lines of command line and you're up and running with the contributions. Yeah, you can open up the list of issues on GitHub for and with whatever you want to help us. We ideally are looking at the. If people are doing some sort of labeling themselves and we have not covered that cases yet. Instead of building their own data labeling tool, they can contribute components to Label Studio. So those contributions are very welcome, particularly in the 3D spaces in videos. We have some work done on time series, but there is always more to be done. So yeah, that would be perfect ways.

Awesome. Well, thank you for taking some time during your travels to deep dive with us on data labeling and talk about a lot of these challenges and a lot of the great things you're doing. Hard X and Label Studio. We'll definitely put those links in our show notes so people can find them. And I know I'll be coming back to my team to think about how we can get up a reading comprehension data labeling tool. Yeah, I was just gonna say he's.

Given me hope for the future here.

There is hope in the midst of data labeling.

A long way to go. Long way to go.

Yeah. Yeah. Still a long ways to go. But it's encouraging to know that people are working in this space and also that there's kind of a modular approach approach where we can build up components.

That's.

That's really exciting. So thank you so much for joining us. It was really, really great to talk to you and hope to run into you sometime at a. At a conference or somewhere.

Thanks very much.

Thanks a lot.

All right, thank you for tuning into this episode of Practically I. If you enjoyed this show, do us a favor, go on itunes, give us a rating, go in your podcast app and favorite it. If you are on Twitter or social network, share like link with a friend. Whatever you gotta do. Share the show with a friend if you enjoyed it. And bandwidth for changelog is provided by Fastly. Learn more@fastly.com and we catch our errors before our users do here at Changelog because of rollbar. Check them out@robot.com changelog and we're hosted on Linode Cloud Servers. Head to linode.com changelog check them out Support this show this episode is hosted by Daniel Whitenack and Chris Benson. The music is by Breakmaster Cylinder and you can find more shows just like this@change law.com when you go there, pop in your email address. Get our weekly email keeping you up to date with the news and podcast for developers in your inbox every single week. Thanks for tuning in. We'll see you next week.