MSP Dumpster Fire Season 1 Episode 3: Training AI: Garbage In, Garbage Out

Welcome to the MSP Dumpster Fire series. This is season 1, the impact AI will have on your MSP. Welcome to the MSP Dumpster Fire series. This is season 1, the impact that AI will have on your MSP. I'm Robert Chiaffi. I am the CTO and co-founder of an MSP called Progressive Computing based in Yonkers, New York. And I'm joined with or by my friend Alex.

Hey guys, I'm Alex Farling and I'm one of the co-founders over at Empath.

Well, Alex, today's episode is episode 3 on educating and training AI. I almost feel like I'm a pet owner or something that I've got to kind of water and feed and train this thing.

I go back to those, uh, what were the little, what were the little pets you had on a keychain, uh, when we were kids? Something like that, right?

Yeah, whatever.

I feel like I do, I feel like I'm, I'm trying to feed and, I don't know, and care for some little digital pet that, uh, that needs to not go awry. But, uh, you know, if you watch the behavior that AI really has, it obviously needs to be cared for like a child because otherwise it will behave like one.

Yes. Yeah. Well, and this kind of just reinforces the concept that I learned on my learning journey with AI, and I'm by no means an expert. I'm still a student. But, you know, I realized pretty quickly in my study of AI that AI is not just some fancy programming, right? It's also having a large dataset of information at its disposal or for its consumption. This is how it learns. This is how it understands, or this is how it knows, or this is how it's trained. So I think that's what we're going to be diving deeper today into, that dumpster that's on fire for MSPs.

Yeah, we talk a lot about artificial intelligence, but what we're really doing here, and I think it's important to kind of explain this, and you've got some really great examples, but I think it's important to explain that these are really more like educated guesses than true artificial intelligence, right?

Correct, correct. So it's, uh, you know, if you think about like as a child, you know, as a father of, uh, two girls, uh, they're much older now, but, um, uh, you know, I remember when they were little kids and teaching them things, right? You know, and like picking up a cup and like pointing to it and say yellow, right? And picking up another one and red, uh, and giving lots of examples like that. Well, that was the way we, you know, educate our young. But there's an odd parallel here with artificial intelligence by feeding it information and telling what a good result looks like and what a bad result looks like. Right. And we do have some examples that we can go through.

And there are actually jobs now in this industry of telling artificial intelligence whether its result is good or bad. Right. It's kind of an interesting— Yeah, I mean, kind of an interesting shift here. We're literally paying someone to sit in front of a computer all day and say, good job, AI, bad job, AI.

It's part of what a data scientist will do is by taking those datasets and then marking them up to say, this is what, you know, this is a positive result, this is a negative result, and then feeding that data into the AI and training. And that's how it's trained, right? By having those data or sample sets, right? Now we're gonna get into stat— like, we're not going to get into, uh, the subject of statistics, but this is where a lot of statistics come in, right? A lot of mathematical models come in to determine, uh, what statistically is the correct answer and what is not.

Yeah, we, we get very quickly down to that, uh, statistical educated guess. So, uh, with that, why don't we jump in and take a look at some of the examples that you've got, because I think some of them are really, really interesting.

Yeah. So I mean, on a, just at a very high level, right? You know, the internet cat videos is I guess the cliché, but when you train AI to discern what a picture of a cat, you know, what a cat is and what a cat isn't, right? You start to feed it many, many pictures, right? And those data scientists will take lots of different cat pictures and say, this is a cat, right? And then they'll take things like a lot of animals that may look like a cat, like a puppy, right? Which you and I can discern very quickly and say, that's not a cat, that's a puppy, right? Or a lemur, right? A lemur has very cat-like features. And there's a lot of animals in the animal kingdom that have features that look like cats, but aren't cats. And so those are then marked as not cats. So you have a dataset of pictures that are marked cats, and ones that aren't. And then what happens is the AI, through its programming, is able to look at the elements of that each of those photographs, and then know, oh, okay, so this one where I took measurements of eye placement, and ear shapes and what the fur looks like and coloring and a lot of other factors like that. This is a cat because I was told it's a cat. And this other one that's similar is not a cat. So, okay, so I have some— I have this large dataset that I could— I can reference. And then once it learns from that dataset, now it knows statistically what is a cat, what is not. When it is fed a brand new picture, which it has never seen before, it can make a very good guess, almost not 100%, but a very close to 100% guess that yes, this is a cat or no, this is not.

And we're talking about literally a highly educated guess, right? With datasets of thousands or millions of data points and the ability to reference each of those almost instantaneously. It feels less than instantaneous sometimes when we're waiting on our AI to give us an answer. But really what it's doing is going back and grinding that dataset, right?

Yes. And I, you know, and I think a good example of where, you know, you've got like edge cases or things that are hard to discern, you know, if you, if you ask a human, hey, is this color blue? Like if you're showing them clearly blue, you know, unless you've got eyesight problems or you're colorblind, you're going to know that that color is blue. But once you start skewing that blue towards the green, right, it starts to get harder and harder to discern. Is this a blue or is it a green? Right. Like that aquamarine, right. And some people will discern things differently. And I think that's where on those edge cases where AI might start to make some errors, but guess what? So do humans.

Yeah, yeah, totally. But I think we live in a world where AI can become a powerful tool for you, but we have to operate under trust but verify, right? It also will get some things wrong. And we've gotta make sure that we sanity check it.

So I have another example, Alex, if you don't mind jumping into, you know, because the cat example is an example of something that's kind of, you know, kind of silly, like everyone's probably thinking, well, what's the use? Like, why do I want AI to tell me what a cat is and what it isn't? It's a great example to start with, right? From a learning, from a, you know, us learning as humans about AI and how it works. But the practical use isn't really there. So I started to look for some, okay, give me some practical use cases. Let's talk about the real world, right? And I found this example of John Deere, which everybody knows is those green and yellow tractors. I'm from New York. I mean, you know, I've seen them in pictures and I haven't seen them in real life, but I'm from the Eastern Shore of Maryland.

Believe me, I know what they look like. I get stuck behind them all the time.

You're in the same boat. And now you live down in Florida. So it's like there's, you know, in a more metro area. So I know that there's probably not a lot of tractors. Nope. But John Deere acquires this company called Blue River Technology for $305 million. And Blue River Technology was started, from my understanding, by a couple of Stanford grads, grad students, who built this AI to attach to a camera attached to a back of a tractor to look at photographs of good crops versus weeds, and then programmed an herbicide sprayer to spray a very precise stream of herbicide at only weeds, right? And so how did they teach this to discern between good crop versus a weed? Well, it's the same concept of the cats. They fed it millions of pictures and painstakingly, you know, said this is a corn stalk, this is a weed, right? This is a, you know, a vegetable plant that we want to not spray. And these are all the plants that we do want to spray. So they built this robot that can identify in the field good plants versus not, right? And so this machinery is literally attached to the back of a tractor, and the tractor driver, the farmer, just, you know, rolls through the field, and this thing just discerns what's a weed and what's not. Amazing.

Yeah, well, and this enables some really cool things, right? It enables the farmer to use crops that don't have to be Roundup-proof, right? We can use more natural, more organic crops. They can very, very specifically target what they're looking at. But it gets cool in the AI perspective because if you and I go ask AI a question, it sits and grinds and thinks about it, right? Because it's got to get this massive dataset. Comparatively, this dataset on weeds, while it's a lot of pictures of weeds, it's way smaller. And now they can pack this into something that's small enough that they can literally drive through the field. Those good weed, bad weed decisions can be made at the snap of a finger as they're driving along. And they're literally spraying these weeds as they go. It's amazing what we can do with this already. And I don't think we've even cracked the surface or scratched the surface of what this is capable of.

Yep. So as I continued on my journey, looking for other examples, I happened upon another— I don't want to say strange example, but another like kind of very interesting examples. So there's this company called Symrise, S-Y-M-R-I-S-E, and they're a major global player in the fragrance space. So they turned to IBM Watson, right, which we all know is the, the machine that beat Jeopardy champions, right, to study whether AI could be used or act as a nose, right? There are people who are hired, and Alex, I know you're going to chime in here because I think you have somebody in your family or a friend that has experience with this. But you You know, there are people who are trained and hired to be able to sniff certain components that go into perfume manufacturing. And so they're starting to use AI to discern of the billions of combinations of things, right? What would likely be something that a fragrance that we can assemble that would yield a good result. And again, it was Hey, when you mix these fragrances, you get very good smelling fragrances for humans. And then when you mix these, you get bad ones, right? So again, it just is using that same sort of concept of this powerful statistical program, along with large datasets to be able to help in the fragrance manufacturing business.

Yeah, so I mean, the real world human example of this is a sommelier. And we have a family friend who's a sommelier, and they brought us over this this sniffing kit, and it's literally like 75 bottles or something, and they're all little tiny, like, 2-milliliter bottles, and you uncork them and it's the strongest lemon smell you've ever smelled, or just something weird that you have no idea what it is. And next thing you know, they're like, oh, well, that's the scent of leather. Um, who knew? But these folks who can literally smell a glass of wine and say, oh, hints of this, this, this, and this, those people train their nose. Um, and now we're talking about using AI to formulate something that is going to smell like something that these folks have said is a positive smell, is a good thing, is something that people like.

Yeah. So one of the interesting things, and I want to kind of bring this up because this is going to lead into— it's sort of a little bit of a foreshadow of an upcoming episode that we're going to talk about, about fears and dangers of AI. But, you know, this is what I started to learn when I got into the details of these stories that we just spoke about. Was that these were ways to enhance their company performance, to enable employees to be more productive, to be able to focus on the things that they enjoy doing more, right, than instead of working through the minutia of things, right? And this is where AI can be very, very, very powerful. And I think we mentioned at another time in a previous episode about how people feared the computer was going to take jobs away and how this is something. And, you know, and again, staying on the positive here, this is a way to enhance their ability to perform their primary job functions.

Well, and I think it's important to differentiate what AI is doing versus what a human does, right? A human can actually open a bottle and smell it and detect a scent. AI is not doing that, right? AI is doing something very different, right? Not to say that one day we can't teach a computer to do that, and everything's possible, but, um, but, but today AI is using, um, basically chemistry to say, hey, I know if I make this chemical experiment, people have said that's positive. It has no idea what it actually smells like. It just knows that these combinations statistically wind up in a good place. And if I start to see similarities in those, well, then maybe I can create one that's close but different that's going to yield a similar result. So it's again, educated guessing, not utilizing senses that a human has.

And by the way, the output of these AI systems, just like when you ask ChatGPT to do something for you, like, hey, write an email to Alex to let him know that X, Y, and Z, right? And use such and such tone, I may get some output back if I've prompted it very specifically and well enough, but I'm still going to read that as a human and say, wait a minute, why did it choose these words? Or it has this sentence backwards. Or I mean, it got me close, right? But you still need humans to discern the quality and the accuracy of the information that it's being presented. So in the example of perfumes— My wife uses perfumes, eh?

Oh, go ahead and finish that thought.

No, I was going to say, in the example of perfumes, I would say it's probably going to produce a lot of outputs that are probably close, but they still may not be acceptable. It's, this is okay, but it's not really the scent that we're looking for. And it may make mistakes. Oh my goodness, this was awful. Let's feed that back in for training. And then there might be others that are like kind of close, but it's just off a little bit. And now the humans can step in and start to tweak it to get it from, you know, because remember, the difference between amateur and professional is, you know, it's very small difference. Right. But it's those small differences.

And it's typically— it's typically consistency. Right. You know, the amateur can, can hit a home run. He can't do it every sixth at bat or whatever. You know, the, the— my old business partner used to say she used to use the phrase, the phrase defeating the blank page. And I think that's one of the functions that AI gives you, right? You're always challenged when you sit down with this writer's block to go, where do I start? And AI can start, it can defeat the blank page, give you something to work from. But I'll give you a good example of it in our real world. As we're building Empath, we're looking at employee training and how do we do knowledge checks, right? I assign Robert a course and say, Robert, go take our AI fundamentals course. And you go take our AI fundamentals course. Now I have to check and make sure you actually did it. Instead of just click the play button and leave the room. So when we come back to do this knowledge check, well, can we just feed a large language model the entire closed captions for the course and have it ask questions? And what we found was that the standard available open and easily accessible large language models have been so trained and so programmed to be nice to you that it wouldn't tell you you were wrong. If we ask you a question, the answer is obvious and simple. And you came back and said, well, the sky is purple because it's Tuesday. The AI, the large language model would come back and say, nice try, Robert, you're really getting there, but maybe you need to study a little more. Say nothing negative to you. It would never give you a failing grade. Just go, good job, attaboy, keep up the good work. Right. And so there are places where we just need to be aware that as we train, we need to go in and specifically train these AIs so that they will have the behavior that we desire as opposed to the behavior that is desired and expected of those open and, you know, free, free in the wild AIs that, that are designed to do no harm, if you will.

Yeah, I mean, I can totally see your point because I'm thinking about a lot of the outputs that I've received from and not just the generative AIs like the text-based ones like ChatGPT, but even some of the, you know, the picture-based ones. Now, you know, maybe I need to learn my— or maybe I need to hone my prompting skills better. But, but some of the, you know, I thought I was giving it very specific instructions and not very complicated. And it was getting like— I was getting like some really weird stuff that wasn't even close. Yeah. And then when I finally got it a little closer, it was still like it just was off. Right.

And we see there's, I think, more training that needs to be done. Yeah, there is. And we also just need to understand how they work. Right back to your is it a cat, is it not a cat? It doesn't know what makes a cat. It knows I've seen 10 million pictures and 5 million of them were cats and 5 million were not. So I can identify those. But then when you ask it to draw you a cat on some of these generative art models, your cats will have 6 tails, you know, 2 tails or something really weird like that because the AI does not really know what a cat is. It's just going, hey, in my pictures of cats, there were tails. I should put in a tail or two. And in my pictures of people, there were fingers. So I should throw in 10 or 12 of those and just see how they work. And so we see these weird things in the way that AI does art, and I'm sure it will get better. But that training has to continue until we can refine our way to, to really fix it, or else trust but verify is a thing that has to happen forever.

And that, I think, what you said towards the end is the key to all of this, is that the AI needs to continuously be trained. And I think right now our expectations with a lot of these tools is a little high, right? Gartner has that hype curve, right? And we're now heading for that trough of disillusionment, right? Which is like, hey, you know, really, really hyped up. And now we're seeing that it really kind of sucks. But, you know, as it gets better, we'll start to get to that plateau of enlightenment where it'll start stabilizing. So, you know, I think we're on that downward slope where people are going to like throw up their hands in the air and say, you know, this is garbage, right? Like, why? I could draw a picture of a cat in 10 seconds. This one can't get it right in 10,000 tries. You know, with time, it's going to get better and better and better.

Well, and I think that goes back to the thing we said in the beginning, which was this is kind of like raising a child, right? And in AI years, we're still in the toddler years. We're trying to get a 2-year-old to draw us a picture of a cat. We're trying to get a 2-year-old to draw us a picture of somebody with a hand, with a 5-fingered hand. And so we're just in a point where more data is needed, more refinement is needed, a little bit of growing up is needed before we can really trust AI to be sure that what it's giving us is exactly what we need to see.

Right. And again, it really comes down to bad data versus good data, right? And it's— that's the whole purpose of what we're talking about today is the training elements of AI. So Alex, do we have any more on this topic that we need to talk about or are we kind of at the end?

No, I think that kind of wraps it up. You know, AI is only as good, I think, as the educator education and training and the data that you feed it. I have an amazing t-shirt that one of our empath friendlies gave me at Write a Boom. He brought me a t-shirt that is— I should have worn it for this episode. It is a poop emoji plus the Microsoft Copilot logo and an equal sign and a rainbow-colored poop emoji because the Copilot logo is all rainbow and everything. So it is literally garbage, you know, poop in to rainbow AI, give me rainbow-colored poop at the end, right? And it's basically a cute way of saying garbage in, garbage out is really— and that is a tale as old as time for programmers and developers. It's where we are with AI, and it's where we're going to be for probably the foreseeable future until we kind of hit some of that enlightenment that you were talking about.

So that's the, you know, uh, you, you, you kind of hit the nail right on the head for me as well. That's how I was gonna, uh, just leave a final thought about garbage in, garbage out. But, you know, people will often ask me or say to me, hey, I never use anything that I learned in college in my career. I mean, and, you know, I graduated in 1990, you know, very long time ago where technology was not anywhere near what it is today. But there are some universal concepts. And I think that is one that is still very applicable. I learned giggo back in college, right? And then brought that into my development career and then even into IT support and MSP world. It's— I think it eloquently and perfectly helps those that are in the technology business trying to get our arms around AI to understand very clearly about training that you give it garbage, you're going to get garbage. So it's a perfect way to kind of round this out.

Well, thanks for hanging out for this episode, Robert. I think we did it. I think we wrapped this one up. So we'll be back next episode to talk about different kinds of AI.

I can't wait for that, Alex. Thanks for joining me again today.

Thank you. This podcast was created by me, Alex Farling, and my good friend Robert Jaffe. This podcast is produced by Francis Belland. The music and artwork for this series were produced by artificial intelligence and edited by Francis Belland.