Production data labeling workflows

Data prep is so challenging. It's probably the most challenging part of a project and it's oftentimes because of the sheer volume of data that is required. Oftentimes really highly paid and talented data scientists are managing projects in a highly manual way where their time and talent just isn't being utilized. And I'd say that's probably one of the biggest challenges we hear data scientists describe is that they're spending too much time manually managing project minutiae. Oftentimes it's the use of automation tools and project management platforms that can help them to kind of refocus their energies on higher level priorities and allow the application, the software application or the platform to automate a lot of the workflow and allow other team members to manage a higher percentage of the workflow.

Welcome to Practical AI, a weekly podcast making artificial intelligence practical, productive and accessible to everyone. Subscribe now if you haven't already. Head to PracticalAI FM for all the ways. Special thanks to our partners at Fastly for delivering our shows super fast to wherever you listen. Check them out@fastly.com and to our friends at Fly IO. We deploy our app servers close to our users and you can too learn more@fly IO.

Well, welcome to another episode of Practical AI. This is Daniel Whitenack. I'm a data scientist with SIL International and I'm normally joined by Chris Benson, who's a tech strategist at Lockheed Martin. But he he's doing great tech strategy things and traveling as part of those things. So he won't be joining today. But I've got a really, really wonderful guest and topic to talk about today. We've sort of been diving deep into a number of modeling related things in terms of stable diffusion and various things coming out and I think it'd be good to kind of shift and talk about again. We're Practical AI, so talking about some practical data related things would be worthwhile. And I'm really pleased today to have the CEO of Zelix with me, Mark Christiansen. His expertise is all in the area of data labeling and workflows around that and bespoke data processes. So welcome to the show, Mark. It's great to have you.

Thanks Dan. Glad to be here.

Yeah. Well, could you give us a little bit of a background about how you kind of got interested in this space of data labeling and kind of producing custom training data sets and eventually kind of built a business around that? How did that happen?

We didn't come out of a data science discipline actually. We came out of healthcare. And we've spent the last 17 years managing healthcare data at scale, specifically in the area of dictation and transcription. So kind of an entirely different field, but the thing we had in common was managing one large amounts of data at scale in healthcare. What we would do is we would record audio, and for 17 years we recorded audio from healthcare providers and then move that audio through an enrichment workflow, which just essentially was transcription. So we'd have skilled medical transcriptionists in the States and around the world who would take the audio and transcribe it into the completed healthcare note. And a few years ago, we met with a friend of ours who was an owner of an NLP company and he really liked our platform. We happened to be working with him on a speech recognition project. And we really have a need for this in managing trading data for NLP workflows. And so that launched a discovery process that took about two years. And we investigated the use case and determined that there was really a really neat fit. And so we modified the application for the next two years and then launched our training data services workflow called ZELX AI. So that's how we got into it.

That's super interesting. And I know data specifically in the healthcare space, there's some very interesting restrictions and very specific processes that you have to make sure that you're following in that healthcare space. I'm wondering if you think that that perspective on data and the security, the compliance things around that data did that sort of shape maybe how you think about handling data for some of these use cases. Any thoughts there?

Yeah, man, that's a great question. And you're totally right. Data security is so paramount in healthcare. And my colleague at the NLP company cited that as one of the specific reasons why the workflow that we had in healthcare was a great overlay for data training. The data has to be audited. So data should have a couple of different audit trails on it. Data should be encrypted both in transit and in rest. Data shouldn't reside on the devices of people that are involved in data labeling. So all those things were just a perfect fit and carryover between our healthcare workflow and an AI workflow. Yeah, you're right.

Interesting. Yeah. I'm wondering, maybe it's as you talk to this NLP colleague or as you've worked with clients around the world working on data labeling projects, from your perspective, how are data scientists most often labeling their data these days, and where do they encounter challenges because of how they're approaching data labeling?

Yeah, I mean, the greatest challenge we always hear, and it's an obvious one, is about getting data that's accurate enough to improve the model, especially in specialty use cases, or let's say new language modeling where a click worker approach doesn't hold up, it just doesn't work as well. And for that reason, for smaller projects, maybe the size of a few hundred to a few thousand data objects, a lot of our clients try to do the work in house just for the sake of, I'd say primarily retaining for the sake of quality control. But for larger projects, it's just too hard to do everything in house. And so it winds up being a combination of in house team members and outside vendors doing the data labeling. For our purposes and the approach that we took, we decided that rather than commoditize the role of the editor or the annotator, we'd invest more in training and compensating our labelers as a means of building long term relationships. And for us, we found that's an essential part of maintaining the kind of the consistency of the data quality and making sure that the data quality remains at the accuracy levels our clients require. And that's to be able to have those relationships with annotators that we can trust and that aren't just commoditized relationships.

Interesting. Yeah. So have you encountered cases where maybe clients come to you and they say, hey, we tried to throw up a crowdsourced task and get a bunch of labels. We invested a lot in that and then it didn't really help us that much. Do you think that those cases are maybe due to unclear instructions to the labelers or a sort of variety in the motivations of those data labelers or what do you think kind of leads to some of those quality issues from your perspective?

Yeah, the commoditization of the annotation workforce, I think it can be a project killer and a very high percentage of projects that launch, stall and never complete. And that's one of the key reasons we've talked with companies that try that approach and they wind up iterating the data so many times to try to get an accurate set of data that they can use that they ultimately wind up going to a more bespoke approach where the teams are more hand picked and more highly trained, even though the costs are higher in order to finally wind up with a data set that is useful. So yeah, I think that is one of the key problems that plague data aggregation projects and that is to wind up with a clean set of data that can be done Kind of on time and on budget.

Yeah, I know. Mark that. So in our projects, and we've done some speech projects as well, we've struggled with this also in terms of the data quality. And I remember in one case really we were saying, well, we need five labels for each sample because the variability between labelers is such that we need either a majority vote or we need to analyze how much they agree, one labeler to the other or something. And of course that gets really expensive over time. Could you speak a little bit to. You mentioned this training, focusing on training and bringing in this like upskilling these data labelers. What does training annotators look like in your projects and what maybe have you learned about what's important as you are training data labelers? Yeah.

I recently did a paper called Improving Model Accuracy through Better Translation and it was really just an attempt to lay out some tips for translating source text for natural language processing models. And one of the items I mentioned, and it's something that we've seen as we worked with teams around the world doing language projects, is that it's important for the editors and those involved to understand the use case. And that might seem like it's perhaps too much information or an unnecessary amount of information to share with the editors or the annotators, but once they understand the project description, or I should say, the better they understand the project, oftentimes it really does translate into higher quality data. And so I encourage companies to share that information with annotators so that they are more vested in the work that they're doing. And as an example of how a project description might be written up as part of the the guidelines for the annotators, it might be something like this project involves, and this I'll just cite briefly a paragraph out of the document. This project involves training a software application to automatically assist call center agents with their tasks and increase their efficiency. For example, if the customer says I have a warranty issue, the agent's software application can respond by automatically opening the customer's warranty clause, reducing the time required for the agent to assist the customer. The translation project consists of a set of English language scripts that reflect some of the typical conversations that occur between call center agents and customers. The purpose of this project is to translate those scripts into a target language in order to add NLP driven process automation into the call center's workflow, thereby adding new efficiencies to the agents and company. And so by giving that kind of insight and detail to the translators and the editors, it enables them to have more buy into the project and have a better understanding of how their work is going to be used.

I think that there's a lot of content about hyped AI data science things, but in reality what people are really wanting more content around is this sort of practical concerns of like hey, more my data labeling isn't actually working right? Like how can I fix that problem? And so I think there's a, from my perspective at least there's an eagerness for this sort of conversation where people are actually they have a lot of the other but they don't have enough like practicality in their content. So I think that that bodes well for this sort of conversation from my perspective.

Okay, that's encouraging to know because we're the guys on the process side so the sexy work is being done by you and the data scientists. We're more the guys down in the boiler room. We're kind of the operations team that makes the process happen but doesn't necessarily know a whole lot about the data science side of it.

Yeah, I think that in reality though, the data side is what is driving things. So yeah, I think that's good. SA. Well Mark, we've talked a little bit about the importance of training annotators. We've talked a little bit about kind of specific data concerns around healthcare and other things. I'm wondering from your perspective, since you're really plugged into the area around around how people are managing their data workflows, how they're managing their data labeling, from your perspective, what does the current data labeling, annotation and tooling landscape look like? What choices do people have and what does that landscape look like?

Right now from my perspective, the landscape seems to be rapidly changing, but I would say that off the shelf models are being used more often. They continue to improve and they're used either I'd say as is or with in house tuning. The projects we see and the projects we're getting more involved in are kind of specialty applications where off the shelf models aren't accurate enough or they don't exist. And cases might be places like medical documentation, labeling, sentiment and intent projects that have a highly customer specific language or vocabulary that can't be picked up by off the shelf models. And in specialty models, I'd say training data is needed in cases where unique vocabularies warrant kind of highly specialized bespoke model tuning. An example might be gathering business intelligence from call center interactions, for example, where the client is seeking to obtain business intelligence through an NLP automation process. And they need a model to be custom tuned and they need the model to be custom tuned to meet their business objectives. Another area would be, I guess, new language modeling. And that's exciting to me and encouraging because we're starting to see an uptick in interest in other major world languages where models don't exist in a production environment. On the tooling side, I'd say we've seen companies both big and small relying on a hybrid of in house data labeling and in house plus clickworker driven labeling and fully external third party labeling. But what we're not seeing is AI companies that have systems in place to manage those different approaches in a cohesive way. So there's a lot of manual aggregation, there's a lot of one off coding that gets done to unify the results from those hybrid sources. So to answer your question, on the tooling side, this is one area where the tooling is broadly not keeping pace with the growth of the industry.

And I know it's like one thing, and this comes from personal experience, it's one thing to get data labeled, like gather a label, it's another thing to develop a workflow around that that's integrated into your systems, integrated into your backend. What do you think are the challenges facing data scientists around this workflow side of things and the bespoke sort of things that they have to do to integrate data labeling into the sort of wider set of things that they're doing?

Yeah, data prep is so challenging, it's probably the most challenging part of a project. And it's oftentimes because of the sheer volume of data that is required. And what we see, not just at small companies, but even at big companies, is that highly skilled and oftentimes really highly paid and talented data scientists are managing projects in a highly manual way where their time and talent just isn't being utilized as efficiently as it could. Senior data scientists are doing things like vetting samples from annotators and doing quality scoring on annotators. And I'd say that's probably one of the biggest challenges we hear data scientists describe is that they're spending too much time manually managing project minutiae. And oftentimes it's the use of automation tools and project management platforms that can help them to kind of refocus their energies on higher level priorities and allow the application, the software application or the platform to automate a lot of the workflow and allow other team members to manage a higher percentage of the workflow. So I think that's one of the things we're seeing.

And along with that, how does Zelix specifically approach the data labeling problems that you've described? We've talked about sort of workflows. We've talked about the custom setups that are needed for certain tasks. We've talked about a variety of things. How has that filtered down into your approach specifically and the approach that Zelix takes?

Yeah, workflow platforms are all about moving off of spreadsheets and manual processes into processes that scale better. That's what we do. We're focused on the production process. So everything from training and managing the skilled labor to meeting deliverables on time and at quality levels that clients expect, keeping projects on budget, those are all things that training data services companies like we do and that we bring to the table. Our focus is on making complex workflows easier. And another part, this was interesting that one of our clients said to us one time is that they needed all of the stakeholders at the company to be able to see what was going on with the project and the platform. Our platform enabled them, and other platforms too. It enables stakeholders to do that. And there's all kinds of stakeholders at the production level and the commercial level for projects, because projects on the commercial side, you know, they're typically done on the request of a client and in service to a client. And so all kinds of different people outside of the data science team are involved. You know, the sales team, the ops team, the procurement team, the quality assurance team. And everybody needs to know what's going on. They want to see if the project's on budget, they want to see if the project's on time. They want to see if the quality thresholds that the client has set are being met. And so a platform gives everybody that kind of visibility. And I really enjoy and appreciate being able to do that for a company, because it does then keep all the stakeholders in the loop at the same time. It allows the data science team to not get bogged down managing minutia manually. So that's one of the neat things that, that we like to deliver. And then on the services side, the approach that is always about managing the workforce successfully and success in data sciences and in projects like this is measured in being able to deliver a project on time and on budget and at the accuracy levels that have been determined or set by the client and by the service provider as being the goals or the project's objectives. An example of this kind of how this can backfire is when service providers like us enter, let's say, a new language or a new project area, and maybe their client has come to them and said, can you do this or can you do a project in data labeling in this language? And of course, the knee jerk reaction is always, sure, we can do that, but if saying we can do that involves hiring a third party vendor in that target country or target language to do the project and it's done in a scramble, it can really backfire. And so hiring a third party vendor in cases like that can result in kind of a black box approach where you're unable to adequately measure quality and where you're unable to adequately manage deliverables so that projects wind up running late or projects are delivered with poor quality data, and then you're left kind of scrambling to do those corrections on the data internally, or finding another source to do those corrections for you, and it's a recipe for disaster. So for us, kind of the way we mitigate that is when we move into a new language, for example, the initial step is to do the hires and do the training ourselves so that we have our own team and we're not dependent on a third party vendor source for that labeling effort. And that way, even though it's going to take us longer and the cost might be higher, and there are cost sensitivities that are realities, but the truth is, if you're using a third party vendor and working out of a black box, chances are you're not going to be able to deliver the project on time and at cost, and so your cost and timeline are going to be affected anyway. And so we've opted for taking an approach that's more expensive to our clients, but that ultimately delivers projects with a higher quality and consistently higher quality data that are on time, that meet the turnaround deliverables, even though the price might be a little higher.

Yeah, and I think that's a really good and practical advice for the whole community that's trying to do a variety of these data labeling projects is really kind of at the start of these data labeling projects, not only thinking about gathering samples, but thinking about how is your workflow going to be managed and how are your annotators going to be trained. Because thinking about that stuff up front and taking time or spending more money on getting that in place from the start might actually save money in the longer term if you're not doing as many iterations of labeling. Right. If you start and you do a bunch of labeling and then you don't get the quality that you need, or you get some sort of unexpected biases or other things in your data that could cause more problems down the line. And one of the things the maybe this isn't a specific, I guess it could lead to specific quality issues. But one of the things that is hard for me as a technical introvert person who's not maybe the most people oriented person in the world, is thinking about all of the team dynamics that happen on a data labeling project and setting up maybe a disparate and kind of distributed set of labelers and vendors for a data labeling project. How can the problems associated with those sort of dynamics be addressed in this kind of online distributed labeling environment?

Yeah, you're totally right. There are inherent challenges in managing an online workforce, but many of those can be mitigated through kind of a well developed robust workflow application. You know, things like centralized controls, giving managers total visibility to what's happening in the workflow at any given moment. You know, the status of data objects as they're moving through the workflow and how you're doing against your timeline for deliverables. You know, those are the kind of things that software is really good at managing. As I mentioned earlier, we've seen cases even with really large companies where pretty complex projects were still being managed on a spreadsheet. And when you're doing that, there's almost no ability to manage the workflow effectively.

Well, Mark, given the sort of team dynamics that can happen, that we've been talking about this sort of variety of tasks that Zelix is exploring and other people exploring in the space, from like standardized machine learning tasks to more custom ones, I'm wondering what sort of like would you say about kind of proper ways to set up maybe manual and or annotated QA type of workflows associated with your data labeling?

Well, I can tell you a little bit about what we do. The first is to establish the ground truth version of the data object and for all data objects as they're moving through the workflow. And once we establish the ground truth data object, then we're able to measure the distance between that and the work that the editors are doing. And that helps to generate a whole lot of different metrics for us. Who needs additional training, how pay might be affected, how our costs are effective. If data objects are moving through the QA workflow more for some editors than others. The second thing that we do is a multi level QA workflow so that work gets automatically routed and that could be in cases where we've got new hires or maybe editors are being flagged via our auto check process. For certain error types. And then thirdly, we run an error script that dynamically checks against the known error list so that those items are routinely kind of recycled through the workflow to be re edited and reqa'd. So those are kind of some of the typical things we do. Judgments, of course, and multiple judgments on data objects is really important to make sure that using multiple layers of judgments is also important. And we do that through the QA workflow process as well.

It's been extremely helpful for me to think through some of the dynamics and the workflows associated with data labeling. I think it's, it's extremely practical and very useful. I'm wondering, as you kind of continue to be more and more involved in this space of data labeling and interacting with clients in the data science and AI space, what excites you about the future of sort of data science and AI practice? And maybe within that, what could easier data labeling enable in the longer term?

Well, when you look at the number of data sets that have been developed and models that have been developed so far, it's overwhelmingly all English based and in that regard probably largely focused on the US market and we're overwhelmingly the largest economy in the world, so that makes sense that it would be that way. But what I'm excited about is seeing the tools and expertise that have been developed in English modeling to now be used in other major world languages and specifically in developing economies where, where AI can be used to help developing economies move forward. All of those nations are generating customer and employee experience data in the form of things like, you know, like customer behavior data and online reviews and, you know, sentiment and intent data, medical data, things that are in unstructured format where AI can be used in a beneficial way.

Well, Mark, I'm really happy that you brought up this side of the impact of data and NLP across the world's languages. As our listeners will know, I'm very passionate about this topic and I'm really excited anytime we get to talk about that. It's something that excites me for the future as well. I've really appreciated you taking time out of your work with Zelix to help us parse through some of these data labeling challenges and the workflows associated with them. And yeah, really, really appreciate you taking time and looking forward to continuing our conversations over the coming months as I hit my own data labeling issues.

Thanks Dan, very much. I really enjoyed it.

Alright, that is our show for this week. If you dig it. Don't forget to subscribe. Head to PracticalAI FM for all the ways and if Practical AI has benefited your life, pay it forward by sharing the show with a friend or a colleague. Word of mouth is the number one way people find shows like ours. Thanks again to Fastly for fronting our static assets to fly IO, for backing our dynamic requests, to Breakmaster Cylinder for the Beats, and to you for listening. We appreciate you. That's all for now. We'll talk to you again on the next one.