Why Your Model is Probably Using Bad Data

Do you think your company and your vision is ahead of time?

I think it's going to be the next step of how data pipelines are managed. Hallucinations cost a lot. You cannot make a mistake. So it's like, oh, but it does well enough. But whatever ChatGPT or Claude does well enough. Well enough is not good enough.

Can we eliminate human in the loop one day?

I think it's possible. Sometime in the reasonably distant future we can capture the human input without the human having to go into a specific place to give that input.

Some company want to use your data for training? Isn't like labeling with experts really expensive?

The short answer is yes.

And let's say I want to build an AI agentic system. When do I need to worry about data labeling?

The answer is it varies.

What does data annotation mean in your definition for me?

Data annotation. Let me actually show you. How about that?

Okay, awesome. Hello everyone. This is two Set AI and today I'm sitting down with Ahmet Rashad from CO AI. Right, Ahmet, welcome. Before we dive in, could you tell us a little bit about yourself and what you're building?

Yeah, absolutely. I had a very interesting career trajectory. I started my career as an offshore oil driller and taught myself how to code and built a product and became a worked and been in software for the last 20 years since then. And, and right now we're building a data platform that for specialized data collection and labeling for AI applications. And that's Perl AI.

That's really cool. How did you transition from the oil rigs to like data labeling?

Complete coincidence, absolute coincidence. It's always, it's actually a recurring theme. It's chase the problem. I had a problem while working on the rig and tried to solve it in many different ways and then realized, oh, this can be solved with software. So taught myself how to code and built a product and that's how it transitioned. So chasing the problems and I actually like this software thing gives you kind of superpowers. So let's get deeper, let's get deeper into it.

Yeah, absolutely. So you're solving a training data problem for AI. I'm curious, does AI actually have a data problem?

AI will always, I think have a data problem. I think so. I think as the applications get more and more sophisticated, as we progress, even the older applications, as we start refining them, the data needs evolve and get more sophisticated as we move forward. You look at the use cases we used to solve a few years ago or 10 years ago, they were, they used to be a lot simpler. They used to be. They didn't require specializations, they didn't require as much precision, accuracy as they do now. They still required precision, accuracy and specialization, but not as much as they do now. Like, we're talking. We're talking to a customer today, for example. This is literally today and we're talking to a customer today and we're talking about use case in dentistry, and we're talking about what is the best way to detect lesions in CT scans. Right. And we're actually having a conversation with the lead clinician on the client side, on the customer side about is this way better or is that way better? And that's the level of detail we've gone down to, is what is the best way to actually detect lesions? Right. It's come a very long way.

You're the expert in this area. For me, for the AI era of data tagging, I'm thinking of, let's give you two images. Is this a dog, is this a cat? And right. It's with a person with common sense, like myself.

Yeah.

I can do that by recognizing. I don't know. Yeah. What could that complexity in the same.

It varies depending on how advanced or how much progression has been made in the specific area. For example, in. In some languages where the AI hasn't developed as much. For example, Arabic, which is quite nuanced, right, Because Arabic is written in one way, which is standard Arabic, but It's spoken probably 50 different ways. I speak Arabic, I'm a native speaker, but I speak Egyptian Arabic. I can understand someone from Syria, I can understand someone from Saudi, but if someone from Morocco speaks Arabic, I can't understand. Even though they're speaking exactly the same language, if they write it down, I can understand it. If I speak, they understand me. If they speak, I can't understand them. It's quite interesting. The applications haven't yet caught up with the reality of the world. There are still some things like, oh, we need to actually transcribe this properly. We need to actually identify what does this word actually mean within this context. Because the exact same word, if spoken between two people who are Moroccan, could mean something. But if someone who's speaking Moroccan is speaking to someone who's speaking Lebanese, it could mean something completely different. So the context really matters here.

For sure.

It matters like, dramatically changes the context, dramatically changes the meaning. Now, in other areas where we've overcome a lot of the language and the language isn't as problematic, like English, for example, we've gone into other fields where we're looking at much more sophisticated applications that aren't necessarily dedicated to the language. Like English is pretty much in a very good space. But we're looking at cases like dental. Dental is big. We're looking at medical, we're looking at robotics, we're looking at a lot of applications in legal and so on and so forth. Because again, legal is another one where. What's the right answer? The answer is always. It depends.

Exactly. It's not a common sense person can tell what is right, what is wrong?

Absolutely. If I show you a medical scan now of someone's brain, can you tell me if this is okay or not?

See, you show me. Did you have.

I mean, I can. It's just. I know if it's going to be HIPAA compliant, I need to scrape it first.

Yeah, it's not a dog recognition. All right. You mentioned about the reality of the real world today, of AI use cases and the complexity comes in with translation underserved languages and then serving the right context. You also mentioned about the image part. Right. Image recognition of the dental lesions. Those are very nuanced and expert driven as well. Right. Are we running out of data? Is that a problem we're solving too?

The answer is it depends. So let's just take a step back in the macro. It's very difficult to tell because it depends on the specific application. In some areas we are running out of data. In other applications, in other areas we're not. So, for example, if you're looking for English language, speech or text, we're definitely not running out of data. We have plenty of that. But if you're looking for, let's say, hybrid speech between someone who switches back and forth between a specific dialect of this language and French, then that might be a little bit more problematic. If you're looking for medical scans, that's another issue because we have a lot of those, but they are not easy to get access to.

Right.

So the answer is, again, it depends on the specific area in a lot of areas, but they're broken down into three categories. One, we have a lot. Two, we have a lot that we don't have access to, and three, we just don't have enough that is available, accessible and structured in the right way. So it depends on the use case. Are we running out of data or not? It really depends on the use case. That's a short answer.

And all three categories are needed for training language models or multimodality models?

Yes, because you think about language models, and language models are evolving so fast. And they are learning to do new skills every couple of weeks. Like every couple of weeks you hear about a new breakthrough, right? And these breakthroughs aren't just happening. They aren't being developed in these two weeks. They are actually. There are multiple research threads that are going on in parallel, and they're taking months to work on a month of planning and collecting data and figuring out how do we make this work and how do we do the training and sometimes generating data and sometimes with no data, and so on and so forth, or working on data that already exists and trying to extract more out of it. So right now we need to extract the most out of what we have. But I think we're going to hit an area. We're going to hit a bottom. We're going to hit a wall in many areas where we simply don't have enough data. I think that's going to. It's already happening in some areas, and it's going to be coming in more and more areas as we progress more. The rate of progress is just so high that I think this is inevitable unless something dramatic happens.

I'm very tempted to ask you about the trend in AI, but let's take one step back. I think you're talking about a different type of data annotation or data labeling from my definition. What does data annotation mean in your definition?

For me, data annotation means. Actually, you know what? Let me actually show you. How about that?

Awesome. Yay. Love live demos.

This one's slightly old, but it is. It is cleaned and it's hippocleared. Okay, so this is a sample from one of the earlier projects that we worked on. So basically this is a conversation that happens between a patient and a doctor. And this is a very multicultural, cosmopolitan, urban centers. So there are five basic languages that are spoken there. French is one of them. So basically the whole thing happens. It gets transcribed and then it gets translated. So it starts the transcription, and in this case it starts with French, it gets translated to English. All the PII gets abstracted. We take two kinds of things. We either we abstract the PII or we corrupt it. I actually prefer corruption because that way it's easier. Yeah, it's better. And then we extract all the key things, which are clinical signs, pathologies, allergies, exams and treatments. And all of this is done, automated. So far, a human hasn't touched any of this yet. And then it goes to a human. And the human validates all the stuff that's happened, all the stuff that detect that the software has done so far, and they validate it and they add the ICD codes. They validate all of this. They go through the scans, they identify the areas where there are anomalies or there's something that needs a follow up, they summarize the whole thing. And. Yeah, that's is on the label. Yeah, that's the final output. That's the final output.

Ooh. So QA control approved. So they can approve the labeling from here?

Yep.

Oh, cool. And they can. Yeah, add to comments.

They can add comments. And at the end of the day, you could imagine that this actually the way it happens is it gets triaged first. The case gets triaged from a general practitioner is, oh, actually this is a head scan. We might need to route it to someone who understands a different type of medicine. And we need to send it to a neurosurgeon, for example. Okay. So it gets routed to that person, and that person takes a look at it, does their own, does their own, adds their own notes, and then it gets validated by qa. And that's our version of data labeling. That's what I mean when I say data labeling.

Is the brain image also labeled as well? Is that one also labeled?

Yeah, yeah, yeah. If you look at the brain image like you see these dots, these yellow and blue dots, that's labeling. That's the labeling.

What can we do with these labeled results?

Yeah. So this teaches the customers models how to detect these anomalies and how to detect these contrasts and how to detect. Do the triage and how to extract the key values of the named entities, like the clinical signs, the pathologies, et cetera, et cetera. You'll find that there are things that are quite tricky like that, that you need to. Things that you need to understand context to get to understand properly. Like penicillin, for example. You need to understand whether penicillin, like, it's allergic, like medication could be something that they're on, or it could be something that they're allergic to, or it could be something that they've taken in the past. So you need to understand from the context of the conversation, which one is it?

Right?

And sometimes the speech is just. It's an ongoing conversation and it's not very clear. The problem with medicine is that there's no room for error. No, hallucinations cost a lot. You cannot make a mistake. So it's like, oh, but it does well enough. But whatever. ChatGPT or Claude does well enough. Enough is not good enough. Zero tolerance, zero tolerance, zero error tolerance.

This data can be used for medical training and workflows in the medical practices and I'm assuming also for training models for later use, because the first round is AI and then if you have better data every time, and then the I'll be improved.

Yeah.

So I think you answered a lot of my question already. Visual is better than a thousand words, right? The data labeling is not as simple as the data itself. By the way, I read all your blog posts on your website. All of them, thank God, they're very consumable. And there was one that mentioned about this expert context in model training. Do you remember that?

Yeah.

What's the value for expert context? This example you showed me is the doctor's notes and the brain imaging. Those are data. Is there more data that you're generating from this labeling process?

Yeah. So expert context, let's take a step. So when you think about expert context, when you learn Pythagoras theorem, when you first learn it, it's A squared plus B. A squared plus B squared equals C squared. Right. And then later on when you advance a little bit, you learn that there's more to it than that. Actually it's not just that, there's actually a little bit more. Now if you just know that and you are solving most use cases that most of us come across like it's actually fine. But then when you have a triangle that's not a right angle, but most people don't come across these use cases, don't have to use that type of geometry in their day to day lives. So that's when you need an expert to come in and actually figure out the nuance of things and actually give you feedback on what you're trying to solve, even though the original use case is your problem. So let's say you're a founder and you are in the legal space. Yes, legal space is very nuanced and you're trying to solve a broad use case, you might not be an expert in every single thing. So a lot of times what we've learned is the best way is not to give the experts full instructions, but part of the instructions is to be give us your feedback as well. How would you solve it? What do you think? It's actually good to encourage some dissent. Right. Because a lot of times specifically on the edge cases, it's usually the odd person out who's. Right.

Interesting.

So how do you encourage that balance? Like a lot of times consensus actually isn't the right answer.

Give you less instruction.

So it's a very tricky Balance, because in most use cases it's okay, but then on the edges you actually need the smaller population of people who are more creative, who are more the edges themselves to actually stand out.

How do you do that?

So as you engage more and more with your experts, you start getting to know them quite intimately and you understand who operates better under what conditions and you start understanding who is better at edge use cases and who's better at day to day production. So usually the way it ends up, I've been doing this for many years. The way it ends up is for regular day to day production you have the, let's call them the people in the middle of the curve and the people on the edges of the curve, they usually end up being the people that you escalate things to. So they're the people who watch things as after they got produced, they're like, actually this doesn't look right. Something's a little off here.

That is very different from if I say we just prompt engineered the workflow or like hire AI agents. Right. A lot of companies are saying we have AI employees. Hire your first AI employee. It doesn't work the same way.

It doesn't work the same way. You're. Don't get me wrong, we still use a lot of AI inputs and we still use a lot of prompt engineering. We still use a lot of those. On this specific case that you asked about, we're actually trying to extract that human wisdom and that variability, that inherent variability in humans that everyone complains about. It's actually a beautiful thing. It is, especially in subjective use cases. It's amazing, it's beautiful. And several of our customers actually want that, those, that subjectivity.

That's why you put that on your website. Extracting this wisdom or give me your data wisdom. So it's actually from experts, from humans?

Yes. And how do you create the right conditions for people to thrive and actually give you their wisdom?

How?

Well, first of all, the bottom line is treat people right. It's your secret. Treat people right. It's not so secret. Well, it's four things. You got to find the right people. You gotta give them the right tooling, you gotta give them the right incentives, you gotta give them the right training. You do those four things right and basically it works out. Now it's easier said than done. Of course it's incredibly nuanced and of course it goes without saying, don't treat them like it's a sweatshop, which is very tempting to do, but don't do it. Believe me. It doesn't work. I've seen people do it over and over again. They think they're saving money or they're reducing costs. You're not actually even that. You're actually increasing your cost quite significantly by doing that. The best way to solve the. Like everyone's trying to solve this triangle of cost, quality and speed. And the way to solve it is actually not to pay people less. It's. It's actually not.

That sounds like the wrong flywheel to get on.

It's a very wrong flywheel to get on. It's a very bad flywheel to get on. You want to get on a high performance, high motivation, high loyalty flywheel and.

You want to build a premium product that actually works. Right. Not the offshore teams.

Yes, yes. And a premium product with low total cost of ownership. Like you're genuinely delivering low total cost of ownership for the customer. What benefit is it if I sell you this piece of Data for a dollar, but you have to spend $10 to actually fix it, or I send it to you versus if I sell it to you for a dollar and 25 and you have to spend nothing to fix it?

I'll say best things about you on Reddit.

Yeah, sometimes. Yeah, it's inevitable.

What's the commonly expected error rates in handling these data? Quality. In your use cases you've seen, is there a tolerance level that you've seen in the industry or verticals or it's purely, you're off?

It is a very tricky question because in a lot of the industries that we handle, there are three main types of error rates. There is the, oh, we got some tolerance. We, it's okay. It's not a life or death situation. It's a drive through. It's. It's a customer support thing. It's fine. Like 99% is fine. It's okay. It's not life or death. Then there's the second case where it is life or death. This is a patient's life cannot mess up. And then there's the third where we simply don't know because the whole thing is very subjective. So we actually, we would love it to be at something, but we actually don't know what good is. So we got to try and experiment. So we just got to keep throwing things against the wall and see what sticks and then keep trying and trying over and over again. So these are the three main use cases that we come across for. The trick is to handle, of course, is the last one, because usually we work with the customer and we split the Pipeline into multiple different and they tried a different path for the zero error. Of course, it's impossible to maintain zero error forever. Like for many different reasons. The simplest of them is you cannot account for every exception and at some point you will disagree on how to handle an exception. So the customer might count it as an error. You thought that this is the right answer. You have a disagreement, so that counts as an error anyways, but you need to resolve it. So that's actually not a bad error. That's something that popped up and you need to resolve it because it's an exception that you resolve it.

Right.

The problem is in these use cases that, and we've seen this over and over again, customers are used to handling high error rates and they're just frustrated and they're just used to frankly not very good quality. So by the time we get in there and we tell them we can deliver this good quality, they're like, oh, yeah, sure, whatever. Let's talk about cost before we talk about cost. Let's just try the quality first. And by the time we're done and we finish the POC and they see the quality, it's oh, okay, let's talk now. So to answer your question, in the zero error realm, especially in fields like medical, dental, legal, the zero error quality is not very popular. There isn't a lot of people who can do that.

You're saying actually the, in the industry the quality is actually not very high at the moment.

It's very variable. Like sometimes it's high, sometimes it's very variable. The variability is the problem is that.

Benchmarked against the human error rate. Because if a doctor makes a mistake, the worst thing is they get sued and somebody is going to be responsible for it. If AI makes a mistake, it's not the same thing. Our goal is to have an AI that can at least do as good as a good average doctor. Right?

So yes and no. I think in medicine the objective isn't to replace the doctors per se. I think the objective is to help the doctors be more productive. At least the most immediate objective is to help the doctors be more productive and be able to serve more patients. When you look at the doctor workflows today, like if you break down their time into value add, incidental and non value add, they spend a lot of time on non value add and incidental stuff. Which is the value add is them actually seeing patients, them doing paperwork, them walking from building to building, them doing work that a nurse or a physician assistant or an AI could do that's all incidental or known value add. Right. So, absolutely. So I'm being very generous here, but probably less than 50% of their work is value add. And AI can help a lot with that. So I think it's not outrageous to say that I think we can double the number of patients that doctors serve per capita. I think we can double that number with AI. And I think that's the first step that we should be focused on. We're trying to replace doctors because we don't have enough doctors. At least in the US we don't have enough doctors in a lot of specialties. We don't have enough doctors.

Is accuracy the most important metric?

Yes. In a lot of ways, yes. Because if you are. See, the answer is always. It depends.

You're becoming a lawyer so much.

I spent so much time with lawyers. So you need to drive adoption. You need doctors to adopt this and actually use it. You need hospitals to adopt this and actually use it. They're worried about being sued, just like you said. What if the AI makes a mistake? Who's going to get sued? Probably the hospital and the doctor. So they're not going to use it unless they're very confident that it's not going to get them sued. They have more confidence in themselves not making a mistake, but they don't want someone else making a mistake and they get sued for it. So to drive adoption, they need very high confidence that this AI thing is not going to make a mistake that's going to get them in trouble. So the very high accuracy is actually driven both by that. It actually needs to be very high accuracy, but also it needs to drive confidence for the doctors themselves and the hospitals to be comfortable to adopt. And then they will adopt it. And then when they adopt, it becomes part of their workflow. And then the flywheel starts going, starts moving. Then it gets better. And then when it gets better. So a big chunk, a big problem in that space, specifically on why it's not getting better is because adoption isn't very high. It's not very low, but it's not very high. And with that adoption, we don't know what the edge cases is. We don't have enough to see how it's operating in the wild. So the flywheel isn't flying as well.

That makes sense. Well, the AI adoption process is a whole different story. Is your labeling process, is it scalable? Let's say if some company want to use your data for training large language models or for fine tuning, it requires a large amount of data Right. Like labeling with experts. Really expensive.

The short answer is yes. And the way we do it is so when we built this whole thing, we built it with the intention of having it at the end so that the customer can do it fully self serve even without our intervention.

What does that mean?

So it means that the customer can go in and they can use our tooling and they can set up a data pipeline. They can collect and label data through a combination of experts, agents, AI, other software lynchers, et cetera, et cetera, and design their pipeline the way they want it without any intervention from a third party like us or anyone else. So that saves cost, time, money, and also gives them control over their data without having to go through any other third parties. Now that was the ultimate goal. Now one of the very, very pleasant things that came out of us designing the software to operate like that is, and designing the process itself to operate like that is when we take on a project, of course in the beginning we try to get the quality very high, very quickly and we're rapidly iterating. And in the beginning, of course it's like you do things that don't scale, but then you work very rapidly to turn it into production volumes or production capability and to standardize it so that it can become self serve without our own intervention, so that it can operate on its own. The beautiful thing about things that can run on their own is that they are usually very efficient, they're non complicated, they're very standard. Standard things are low cost.

Right?

So once you figure out what the optimal flow is and you standardize it for that type of use case, the cost actually dramatically drops. Like it's not uncommon for us to see the cost drop after we've run a project for six months to a tenth of the initial cost.

It sounds like if I'm a medical center and I work with you and you will set up my pipeline for me during the PoC and then you'll help me get to the self service state and then the cost will be lower.

In that sense, the cost will be significantly lower. The longer term the project, the more time I have to standardize it and to make it much more efficient.

Do you have any ROI stories that you tell about? Does this make financial sense? I feel like if I'm on the fence at the medical center or law firm, I really don't know what to expect. Maybe a lot of your customer have this objection or question as well. How do you answer that ROI question?

Yeah, absolutely. So when we talk to customers, we never talk about it always comes up the question of cost. And we never talk about cost, we always talk about tco. Total cost of ownership. Right. What does that mean? So instead of thinking about how much are you going to pay me for this individual piece of data to be labeled, we talk about how much is it going to cost you for this using us versus using someone else? How much is it going to cost you for this specific piece of data to be ready to be used by the model, which involves your internal cost and our cost. For example, we had this use case and this is a very real use case. Actually it happened three months ago and it's incredible, right, because we quoted, we quoted $18 apiece for per document. The competition quoted $3.

Okay, 6.

6. We did the POC and we ran a little bit of volume through the customer, came to the conclusion that with us they needed to basically, statistically speaking, they didn't have to QC or QA anything. They could just throw it straight into the model. But we require them to QA or QC at least 5% anyways. So they're going to queue a QC 5% anyways. They don't need to hire anyone to do this. They were going to do it through their existing project manager or program manager. Good stuff, right? Their AIML engineer was going to take a look as well. No problem, no need to hire anyone else. They were looking at the other option which was $3 per document and they had to QA everything and they had to touch up everything, which meant it was 20x the work just for the QA, which meant they had to hire four more people job to do the QA and they had to hire three to four more people to finish the touch ups, plus a lot of additional work. And think about all the additional work that they needed. And they are in office, think about it. Computers, office space, all of that additional cost very quickly. Even before we finished the math, their TCO was almost twice as ours. So that $18 didn't sound so bad after all.

What were they trying to build?

Something in medical.

Medical. Using AI something. Okay.

Yeah. Okay.

When do I need to worry about data labeling? Let's say I want to build an AI agentic system, say the medical center. If I want to build an AI agentic system. Right. The first thing I think about is architecture and the data that I have and the pipelines. And it's the reasoning that I'm thinking first. When should I worry about data labeling or the quality of the data?

So the answer is it varies. So the question Is your model doing or your agent doing well enough or not? If it's not doing well enough, you need to go through an issue. Truth. Now, what I see is a lot of researchers default to, oh, let's make the algorithms better.

Right, exactly right. Let's fine tune.

Let's fine tune. There's a cap on how much you can fine tune. And I think if we think about it objectively and realistically, everyone knows when the cap is hitting, but we just don't want to stop because it's just so much fun.

I would do prompt engineering first and then maybe consider fine tune last. But all of these tactics are more related to the algorithm part or the architecture part, less about the data.

Yes, yes. I think the, In a lot of cases. So I would say if I were doing. I'm not saying this is the right approach, I'm not saying this is the perfect approach, but if I were doing it, I, I would very quickly test ROI on multiple different approaches. So I have my model. I would try to do some prompt engineering and see what the impact would be. I would try to give it a little bit more good data and see what the impact would be. I would try to improve the quality of existing data and see what the impact would be. I would try to combine some of these inputs and see what the impact would be because that would inform my decision on what would have the best impact on the output of my model. And usually it's a combination of the above. Of course, my, my view is a little biased because when we start engaging with customers, it's usually because they've either come to the conclusion or they're close, or it's in use cases where the data has the biggest impact, obviously. Right. So my, my, my subset is quite limited, so I'm biased, obviously. Right?

Yeah.

But I would say if it was me doing it, I would quickly experiment to figure out where I would get the biggest bang for my buck. I wouldn't go all in on anything until I figure out where I would get the biggest returns.

I, maybe I represent more top of the funnel that you haven't talked too much with. If I know that I have a data issue, I will call you. But do you feel you're doing a lot of this, like trying to get people to pay attention to data quality? Do you have to do that action like, hey, you should look into your data quality or do you feel like it's more people just come to you and tell you that, hey, I have data quality issues.

A lot of people who have serious data quality issues. They come to us and it's usually they come to us in a panic.

Safety.

Yeah, a lot of our, a lot of our customers, like the first call is, hey, I got you a number from a friend. And this is usually like a 8, 9pm call on a Friday or a Saturday evening, hey, my thing is about to explode. We're supposed to release this thing on Monday morning. It's flowing up in our faces. We know it's the data. Can you help? This is a common occurrence. So this happens quite often, more often than you think. And in a lot of other cases, data is grueling. Data is intimidating in a lot of ways. And people got used to thinking that we'll solve it, we'll figure it out when we get to it. Just because it's such a big problem. And you know what, things are moving, so let's just keep moving. When it becomes a blocker, we'll sort it out.

Right?

So people got used to that. The problem is when it becomes an actual blocker, it becomes a really big blocker, becomes a real big problem. So a lot of people, we end up getting a lot of these 9pm calls on a Saturday evening. But a lot of people realize this problem earlier on and they're like, hey, listen, we know this is going to be a problem. We want to work on it earlier. And we are trying to collect this type of data. So our interactions with data and this is biased completely. I'm sure the market's much wider than that. This is just our experience. Our interactions with data folks is either, oh, we're screwed or we're so early. We need your help in actually collecting the data and figuring out the data architecture altogether and what's it going to look like and how much data we need help us figure out how much data we need and why and what structure and from what diversity and so on and so forth. So it's either people who are looking way ahead or people are like, now is go time.

I remember I saw a research from MIT saying that 95% of the AI systems in the past two years fail in production. The demos work, but work in production. If let's say that stats is true, how much do you think it's contribute to the reason of data issues?

So Lana, that's, that's quite significant. I'll tell you. Like I, I like playing around, I like breaking things, I like building things and testing things out. I'll tell you, we built demos, like demos that actually impressed customers just to prove a Point. And we build them in two or three hours by non engineers just using some of the tools available out there like proper AI agents. And it's like two, three hours and they're working and they're working pretty well. But deploy them in production and they will fail miserably.

Right.

They don't have them, they don't have the infrastructure, they don't have the depth. You can demo whatever you want. Like it's not a problem at all. And I think in the middle of the AI frenzy we have a lot of companies that are able to demo things that raised a lot of money and based on very beautiful demos. But the reality is either the products don't have a lot of demand for them or the products are actually quite complicated and require a lot more effort and time and money to build than initially anticipated. Because the demos like this is, this is a double edged sword. We can build demos so quickly and test things out, which is great. But it's also very tricky because we are severely underestimating how much effort does it actually take to build product. Like Vibe coding doesn't build products.

Vibe coding? Yeah, yeah. I'm using test data. So it works on test data. But a lot of people building these systems are not anticipating what can happen in real world. What's the real world data look like?

Yeah, yeah. And the other thing that I hear a lot is oh, we're going to use synthetic data. Okay, synthetic data, great. But like synthetic data doesn't deal with the real world. Right. Like synthetic data is not going to generate exceptions like you see in the real world. It's going to generate exceptions limited, based on whatever you trained it on. Right. Based on the limited perception of the world that has. The world is fascinating. The world is unbelievably complex and we do not understand all the variables of the equation that's actually running the math of this world.

Exactly.

There's, there's a chance that we are actually having this conversation and there is a non zero chance that I'm talking to myself against the wall. Like it's a non zero chance. Like literally this is a non zero chance. And there's a non zero chance that this whole thing is a software simulation. Right.

This is real right away.

Like how can you tell? You can't tell. It's impossible to tell. How can you tell? The only way to tell is to do something so ridiculous that you over overclock the processor is better. I mean.

I know what you mean. How the real world is functioning is not something we can Prompt engineer. There are two rules or principles that we all abide by. Like whether it's physical or temporal, whatever it is, we don't think about it or we cannot use language to write them down one by one and make it work.

Absolutely, absolutely, absolutely. And even our understanding of those rules is similar to the Pythagoras theorem. Going back. Like our understanding could be the basic understanding, but there's a lot more to it.

Right.

It's Newtonian physics all over again. Like we thought this was the world. Actually the world is a lot more sophisticated than that.

And there are, there's a whole lot more that we don't know yet. So we can't simulate that.

Yeah, exactly, exactly. I've heard this over and over again. Before we started it was like, oh, and I'm quoting, It's like, oh, LLMs are going to do all the data labeling. Synthetic data is going to produce all the world's, it's going to produce all the data moving forward. Like this is nonsense.

Expert opinions are dead.

What are you talking about?

Okay, great.

Yeah, that's a good question. I have very strong opinions about this.

Help me to think about a top up defender or potential customer, let's say me. I'm a company. I'm building AI agents. One issue with the data that I'm working with is if a human or myself were to do something like your doctors. Right. I have a specific way of doing things. I can abstract certain areas of the document out and let's call it an experience. I'm doing things based on experience. It's an abstraction of a reasoning layer of the human soft process. Is that experience possible to label?

That experience is possible to capture in quite a few different ways. So first of all we can, you can capture exactly how you do it. You can capture how different people do it, you can capture and then how you label it, you can describe it, you can abstract it further into definable buckets and then within those definable buckets there's a way you do it, there's a way I do it, there's a way Jack does it, there's a way Jill does it, and so on and so forth. There's a way I do it in the morning, there's a way they do it in the afternoon. Right, Right.

Maybe that's the wisdom. I want the wisdom to be labeled.

That's the wisdom. And then, then there's another layer on top of it. So you abstract it, you extract into a bucket, let's say the bucket. Oh, this person is. Let's take a. Something very simple. Let's say you're booking travel, right? Or I don't know, give me an example of something you would.

Maybe. Let's say I'm drafting patents out of my product, I have to draft patents and then I have all the documents, conversations with the team building this whole thing, how it works, evaluation, all the data points. The end result is some sort of patent application, let's say.

Yep. So you can approach this in multiple different ways. You are approaching it in a specific way. You have a bunch of inputs, you have a bunch of outputs, and the output is a document that's structured, that has very specific clauses that need to be filled in a specific way. And the inputs are whatever the inputs are. And the inputs are very specific. Finite number of things that could vary. You could have some of them, you could have all of them. You could have within those, it could be different instances of each. So that's the first part. We're defining the components, the players, then how you approach it. Maybe you prefer to go from the beginning and then you work your way forward. Someone else might prefer to work from the end and work their way backwards. They like to write the patent first and then fill in the blanks and so on and so forth. You have different approaches to doing it. Maybe someone likes to go very quick forward and then backwards or the other way around and so on. So there are different ways and approaches to doing it. And then there's the evaluation of the output from the different techniques and the different people, which is, this is good, this is bad, this is better than this. This is a six, this is a seven, this is an eight, this is a nine based on a rubric. And here's exactly why. So you could be evaluating the technique, you could be evaluating the output. And both are valid because your technique could be great, but the output is not great, or the other way around, or both could be great, or both could be not good. So what I'm trying to teach the agent here is what's a great technique and what's a great output, so that ultimately it will try to mesh the two to get to that great technique that yields a great output.

I imagine this is needed for a patent office, for instance. Patent office might be working with many of these products and then they have a way of doing things right. Do you think they use your service?

I think they could. I think they could. And you'd be surprised. Like high volume lawyers have something very similar, especially the ones that work in contingency because they're not billing by the hour. They just get a cut of whatever the client makes out of the case, like personal injury. A lot of employment lawyers, like, they just take a cut of the compensation that the client makes. But the process is fairly standard. Ish. Not exactly, but fairly standard ish. So these lawyers, they could use our services, frankly, because the output is a demand letter or it's like a fairly standard output or set of outputs.

What are you going to label?

We would take the inputs. We would take all the inputs, which is, let's say in case of personal injury, we take the inputs, which are the police report, the medical reports, expenses, testimonies, et cetera, et cetera, write them into a standardized demand that goes to the insurance company. A projected amount that should be lower than what the insurance company underwrote. And if it's lower than what the insurance company underwrote, they would just pay out. And if you manage to do that, the insurance company just pays out and it's over. That's what 98, 99% of cases end up going. So you can automate that. So if we can do the labeling, where we can take all these inputs and teach the AI to translate them into good outputs. But it takes a lot of human judgment because as you can imagine, a lot of these things are handwritten, a lot of them are messed up. And as you can imagine, testimonies are like jumbled. Like you need to make sense out of them. Sometimes things don't add up. Yeah.

So in your process, the difference will be you will have experts in personal injury law to be part of the team, at least review 5% of it. This way, your labeling process includes both AI and human expert, so that your judgment on the insurance payout amount will be more accurate than AI.

Yeah, it would be faster and cheaper.

Right. Okay. It's very different because I was trying to compare with how I'm going to solve that problem with through prompting and workflow, because I can talk with a lawyer and I can get your experience and prompt it through. And you're saying that you have this expert in a loop somewhere along the line?

Yeah, we can actually. We can combine the two. And that would be. Actually, that would be awesome. That would be fantastic.

Can we eliminate human in the loop one day?

I think it's possible. Sometime in the reasonably distant, in somewhat distant future, we can capture the human input without the human having to go into a specific place to give that input. Especially with a lot of devices and with wearables and a lot of devices around us, we can capture that input significantly higher without the human having to go and actually describe it. So I think it could be possible, but I don't think it's anytime soon because I'm looking at some of the machine outputs and like you said, like, the demos are very impressive, but you look in reality at models, for example, identifying scar tissue on brain scans or lesions on teeth and snug graft and so on and so forth. It's not good at all. It's not reliable. So I think it's going to take some time. I think it's. Yeah, I think it's going to take a long time, but I think it's not out of the realm of possibility.

Let's keep watching. Do you think your company and your vision is ahead of time? What is your vision?

The concept. The concept itself isn't necessarily new. The concept of having human in the loop existed for a very long time. There were companies that you could hire 50 years ago that would provide expert service to augment your existing processes. Right. This existed for a very long time. The difference here is that this is highly specialized labor, highly specialized, folks. And it's not exactly labor. The difference is that we're extracting wisdom versus extracting hard labor or physical labor, and we're extracting wisdom versus extracting time. I think that part of it is a little bit new. I think the efficiencies that it's driving, the quality, the cost, the speed, efficiencies that it's driving, and the combination of solving all three together is new and is highly effective. I think it's necessary. I think it's the next step in the evolution of the space, and I think it's going to be the next step of how data pipelines are managed.

Thank you, Ahmed. Really enjoyed a conversation with you. I feel like I learned a lot.

Thank you, Angelina.

I've talked with founders who are building, like second brain building pure AI employees. I feel you're on the opposite side of the. You know, it's a structural philosophy in how AI adoption is going to change our lives and improve productivity. And then you're like, we need the human. And they're like, we don't need a human. And I feel like I'm on the fence. I'm like. But I believe. Yeah, I believe it's a very interesting trend, and I don't want to be eliminated. Right.

We need both. We need both.

Yeah. So trying to get more wisdom from fellow founders who are actually doing the hard things. Thanks for sharing your opinion and your insights.

Yeah, no, of course.

Yeah. That's good. I'll see you next time.