Towards stability and robustness

Problems, data requirements, business process constraints are so specific, right? And most of the problems cannot be solved repetitively. Good enough. And many times I tell my team at Beyond Minds that state of the art and models are critical part of the solution. Right? But they're probably 5% and we need to do a lot of problems to build this system that can deliver in production.

Big thanks to our partners Linode, Fastly and launchdarkly. We love Linode. They keep it fast and simple. Check them out@lino.com changelog our bandwidth is provided by Fastly. Learn more at fastly.com and get your feature flags powered by LaunchDarkly. Get a demo at launchdarkly.com.

With advancements in AI and deep learning evolving at lightning pace, it's more important now than ever to research the best options suited to your unique needs. This is particularly true when building custom systems and those systems that are GPU heavy. Not only do the applications running on the system matter, but your AI infrastructure and budget constraints need to be front of mind as well. PSSC Labs, which is an HPC and AI custom solutions provider based in California, has been creating high performance computing systems to meet their clients unique enterprise computing challenges for more than 25 years. And with cloud computing costs growing at astronomical rates, plus companies increasingly losing control of their data security, it is no wonder that enterprises and government agencies need to continually look for ways to take back control of their data. Solutions from PSSC Labs provide a cost effective, highly secure and performance guarantee that organizations need to reach their AI and machine learning goals. For more information and a free consultation, please visit pssclabs.com PracticalAI Once again, that's pssclabs.com practicalai. Welcome to PracticalAI, a weekly podcast that makes artificial intelligence practical, productive and accessible to everyone. This is where conversations around AI, machine learning and data science happen. Join the community and slack with us around various topics of the show@changelog.com community and follow us on Twitter. We're PracticalAIFN Foreign.

Welcome to another episode of Practical AI. This is Daniel Whitenack. I am a data scientist with SIL International and I'm joined as always by my co host Chris Benson, who is a tech strategist at Lockheed Martin. How you doing Chris?

Doing great, Daniel. How's it going today?

It's going great. And you know, this is practical AI and I'm super excited because today's episode is super practical. We have with us Roy Mekres, who is CTO at Beyond Minds. I saw a talk by him at GTC about garbage in, garbage out, which was really cool. So welcome Roy. Really excited to chat about these things.

Yeah. Hi guys. Hi Daniel. Hi Chris. Happy to be here and talk to the audience about practical AI elements.

Yeah, definitely you're the right person for this podcast for sure. Your talk, which we'll link in our show notes for sure. It was really good. I really enjoyed the practicalities of it, but also the emphasis on, hey, what are some of these challenges and blockers that prevent AI from producing value when you're trying to deploy it? So I'm wondering if maybe before we jump into the specifics of that, if you could just give us a little bit of information about how you got particularly interested in that topic and ended up talking about it in that context.

Yeah, brilliant. So again, happy to be here and talk to you about this amazing topic. So I came from academic background, I think in the last 10 years probably I'm doing computer vision for my living. Started in biomedical engineering, working on some medical imaging problems. Then I work a little bit on consultancy, some computer vision project, shifted into a PhD in computer vision, wrote several papers, were jumping around between CVPR and ECCV and talking about some very.

Doing the circuit.

Yeah, exactly. Some very academic ideas and some all over the place. I published papers on a random set of topics and then I met my co founder and we detect this problem where the academic research is amazing. 3% of all papers in the world are about AI. That's just mind blowing numbers. Oh really? That's what I read I need to look for.

Is that like a recent statistic or maybe it's more now, I don't know. That's pretty amazing.

Yeah. On a broad topic, right? But then at least in enterprise AI, outside Facebook, Google and Amazon, the use of AI is very low and most severely they're trying. But there is 90% failure rate, there is a colossalical problem there, systematical failure and we establish beyond mind. That's the company that I'm its founder in the last three years and that's what I've been doing for a living in order to bridge that gap between academic research and practical AI and production and real value from AI. That's my passion, that's what I'm doing as the cto, as a co founder. That's the company that I'm trying to build. And we're coming from a very technical data science, machine learning AI background, both me and the CEO, which is also coming from an electrical engineering background. And we understand AI, we understand the concept, we build Many, many different models train lots of problems. But the reality, where it meets the reality, there is a gap there, there are challenges and if we want we can dive a bit more into these challenges. But they're not organizational, they're technical problems.

Yeah.

And I'm curious since you did that sort of circuit of the academic research and published many things but now you're sort of helping clients productionize and operationalize their models now because I'm assuming you probably still read some of that research and keep up with it. Now when you look at that research, do you read papers differently or look at what people are doing differently in the sense of understanding what might be able to create value in the enterprise.

Versus not that's a philosophical topic. What is the role of academia versus the role of the industry around AI? And I think it's good that the academia is dealing with holistic problems, theoretical problems. That's its role. Right. And I think in AI specifically we're shifting towards different areas around more practical AI. Academic research. Definitely when the number one affiliate or the tried AI is Google and probably the second one is DeepMind and then some other universities. But in general I think it's good to have people that are dealing with sota state of the art. And I had this discussion with another with my supervisor during my PhD yesterday and she told me that they're dealing with sort on one hand but very down to earth algorithmical way of thinking what is needed to solve that problem, not what is state of the art. And that's a completely different mindset. And it could be that there are people that like different things. And I think by the way, that's probably one of the problems because if you are an insurance company and you have built your center of excellence and you went to Stanford and MIT and brought the best data scientists and PhDs from there, it's probably not likely that these people will know how to build a product in production that can actually deliver. They can do amazing research and novelty and innovation, but not necessarily deal with some production elements.

Yeah. The last my day just to be transparent on this was like sure, I thought my day was going to involve doing some AI, but really I just tried to convert audio files off of an RTSP stream into like a usable format into it and not get garbage out. That was basically my day. I didn't, I didn't train anything, I didn't even pre process, you know, anything major. I just worked on like this file conversion.

Yeah. On the other hand, by the way, you mentioned papers. Right. My rule of thumb is paper a day. I read paper a day. Not, you know, not deeply. Maybe deeply, like every week, once, two weeks. But I want to read an abstract, look at the figures, understand the concept once a day. And you know, we all the time use open source new ideas. And many times I tell my, I tell my team at Beyond Minds, which is more than 50 engineers today, that state of the art and models are critical part of the solution. Right. But they're probably 5% and we need to do a lot of problems to build this system that can deliver in production. I think that's an essence. That's my belief. From a model to a system. The model is maybe the hub, but there are so many different components there.

You know, while we're talking about that, you've said several things that really, really grabbed me in the last couple of minutes. And we talked about the difference in academia and industry and stuff. And we also noted that there's that crossover you have, you know, you have Google going into Stanford and cleaning out the department and pulling all of those experts in so they can take advantage of it. And so we're seeing a little bit of a difference in this practical. When, you know, when we talk about garbage in, garbage out, and the incredibly practical nature of this as data science, we are seeing a divide in industry where you do have academia and you have a kind of a select group of organizations that understand this and that they are able to do it and that they can take an idea all the way through the work stream and produce that system that's highly productive down the road. But then you're also seeing a large number of organizations, probably well over 99% of them out there that are really struggling with that. And before we dive all the way into the details of your talk, which I also saw your video, so I.

Kind of know what's coming. I'm looking forward to this.

But there's this big chasm right there. How is that cross? Because the understanding of these, of the data science, of it and how to actually do it end to end and having all of the steps and skills.

Necessary, it's a lot to know.

Is a lot to know.

Yeah.

What are your thoughts about that? How does that get reconciled going forward?

Well, I had a conversation with one of the Tier one banks a few weeks ago as part of my role as a cto. I do a lot of sales meetings. I meet clients occasionally and definitely part of our go to market and vertical is around financial services. So Anyway, at Tier 1 Bank in the US they have 1,000 engineers in the center of Excellence around AI, right? Okay, so that's Tier 1 Bank. But even if you go to Tier 2 Bank, let's say Fortune 1000 Company, they will build a center of excellence with tens of engineers. They will give a good emphasis on bringing people and talent and organizational elements, strategy and educating the executives, et cetera, et cetera. But they're still struggling for some complex elements. And I think the big element is becoming a technology driven company because Google and Amazon are not on the same challenges. But that's extreme cases. But also not other technology companies. I think when it comes to technology driven companies, people that understand data, understand engineering in their essence, that the product is technology versus the product is bank or insurance or manufacturing or telco or mining or oil and gas or whatever, most of the world, right. Most of the economic, they're not technology companies. And building a technology company, that's super complex and it's not unique to AI in my point of view. By the way, if you look at, I don't know, CRM or ERPs in the late 90s or beginning of 2000, right. So many people spent $5 million on trying to build CRM and then they bought Solsk Salesforce. Right. And you know, that's a super classic buy versus build dilemma, which I think we are super not balanced in AI for sure. And for good reason. By the way. That's one of my key assumptions as a company. My key observation about AI can tell you about that?

Absolutely.

Could you go into that in a little bit more detail?

Yeah, for sure.

He's got us pulling it out of the way.

Yeah, I know you put out the carrot there.

Specificity problems, data requirements, business process constraints are so specific and most of the problems cannot be solved repetitively. Good enough. Even if you take something that sounds super repetitive. Defect detection in manufacturing, we need to collect data, annotate bounding boxes and then train yolo. And we have a defect detection. We have an object detector. Amazing. But the reality shows that that's so different from production line. One is defects that are glass for smartphones and the other one is PCB boards. And then you have anomalies type of data and you have very clear defined effects and transparency and light. And what is the dynamic of the data? Is it shifting or. It's very consistent how you deal with the monitoring of that problem. Do you have. If it's a. It's a. Is it bottles, shampoo bottles that you don't really care about accuracy? If it's more than 95, you're okay. Or it's PCB boards that go into Lockheed Martin engines and you are super sensitive, like zero tolerance to errors, right? You have human in the loop. You don't have human in the loop. Is that a Super 10 GPUs per production line and you have the most amazing model or It's a small IoT device that you need to shrink and quantize and optimize and deploy in C or whatever. So many different requirements and constraints and how the business is working, how many people and you need to deal with that, right? And how you deal with that. If it cannot be solved by an off the shelf product, I cannot go and buy something. So okay, let's build an internal team. They will solve my problem. But then you come to another problem which is how internal teams are becoming a technology company. How to deal with data, how to build processes, how to build repetition, scale. All these elements that are usually the mindset of startups, of technology companies. And that's what I'm trying to bridge in life. Helping enterprise, bridging AI into production. Right? Can give you an example. I have a company that build something regarding sales prediction and they have 30 different lines of sale, 30 different products and they build an amazing model for one product and scaling that to 30 require them to multiply the team because they need them. For example, someone that all his life has his responsibility as a data engineer or junior data scientist or whatever to retrain the model every week because there is new data, something is changing. The hyperparameters need to be tuned. How you do that now multiply by 30 scale.

This episode is brought to you by Snowplow Analytics. Snowplow is the behavioral data management platform for data teams. Maximize the value of your behavioral data using Snowplow Insights, a managed data platform that's built on leading open source tech leveraged by tens of thousands of users. Capture and process high quality behavioral data from all your platforms and your products and deliver that data data to your cloud destination of choice. When marketing needs to make data informed decisions, when product needs next level understanding, and when analytics needs rich and accurate data. Snowplow is a solution for data teams who want to manage the collection, processing and warehousing of data across all their platforms and products. Get started and experience no plow data for yourself@snowplow analytics.com again snowplow analytics.com.

Roy I really appreciate your perspective on that. I think that the way that you think about specificity and scaling a data team is definitely a good perspective. I mean we just talked on our conversation between Chris and I about GitHub Copilot and like, oh, you know, there's people like, oh, now AI is writing our code. And you know, people have been saying that like software engineers were going to automate themselves away for however many decades. It's been a long time now. But, you know, there's such a need for specific solutions to specific problems in even just software engineering. Right. And then if you bring AI into that, which AI is now sort of like you said, infiltrating all of these different companies at all different sizes and all different ways in very specific ways, then there's a lot to solve there. One of the things that I saw you talked about a little bit was this idea of stability. Could you give us a sense of what does stability mean? Maybe in the world of software engineering, but then moving into the world of AI, how is stability something that needs to be on our minds as we productionize AI?

Yeah, sure. So, you know, I think if you're coming from an electrical engineering background, so stability is something that is usually defined for, let's say, linear system or something like that. Right. Bounding input, bounding output or something like that. But definitely in deep learning system, which are far from linear and far from being well understand in sense of what is bounding, even what's going on with the data in terms of stability, what data is going to break my model? Right. And you clearly know as a data scientist that AI is usually, when it fails, it fails silently. Right. I gave in the video, if you watch it. Right. So I call it the giraffe problem. Right. The most naive classifier probably that you did when you got to tell that.

Joke from your video.

You have to tell it.

Okay, I will do my best. Yeah. So the giraffe problem, right. A giraffe enters a bar and the bartender is your classifier, basically. And it tells the giraffe you are 100% a dog. I'm sure of it. Right. And you know, that's the reality in deep learning because the model as the bartender, right. He saw only cats and dogs. That's what he knows. And he will classify everything as a cat or a dog, even if it's a giraffe. And let's say as an extreme toy example. But the reality is full of this problem. Data shifts noise, someone changed the light bulb. The bias in the image is slightly shift by two gray levels. Right. The text is one tab corrupted. Right?

Yeah.

Et cetera. And how you deal with that, There.

Is no practitioner of deep Learning, I think who has done any amount of work in the field, who's going to not recognize the reality that that joke represents?

Yeah, that is true.

We have all experienced that.

Yeah. I always use the example. It's like if you trained a self driving car to drive perfectly and but you did it in Sweden and then you take that car and you plop it down in Australia. The first time you run across a kangaroo running across the road, something very interesting is going to happen. I mean maybe something really bad or just something really interesting. But yeah, it's occurs also. I mean we're talking a lot about images, but also for text and other things as well.

Definitely. Yeah. And I think stability is well coupled with the ability of the model to generalize. Right. Hypothetically you have a data set that is the only distribution of whatever and you can generalize well, but that's clearly not practical. You don't have access to the entire distribution and you cannot generalize to the entire distribution. Distribution obviously. And you definitely don't have labels. But when you shift into practice AI, I don't care about generalization. I want to be good on what I need to be good at and make sure that nothing that out of that good will come into my model. And that's why I call it garbage in, garbage out. I want to make sure that I have a training set, I have its distribution and that's the data that I'm going to get. How are we going to achieve that? Right. And I think that's one of the most important problems that cause this failure rate, the 90% failure rate that I talked about in the beginning. Because AI researchers, data scientists, they know how to build models and they can achieve this 95% accuracy in the lab for sure with a stable static data set. Right. The production is not stable, is not static and it's definitely dynamic. And it's going to shift. Data drifts, data shifts, whatever, something is going to change and how you react to that changes and why it's important. By the way, I think while we're progressing with value around AI, we're also shifting from say statistical applications, which I define as something where the difference between 80 or 81 or 82% accuracy is okay, nice, better to the business, but there are mission critical problems. Claim assessment in insurance, anti money laundering fraud detection, defect detection in PCB board, whatever. These are mission critical. Right. The difference between 80 and 81 is, is critical. It could be massive to the business. And I'm not even talking about applications where their business requirements is 99. I need 99 the model can achieve 90 what you do now, and that can be breached. And there is a way.

Yeah, I think that now that AI is intersecting with all of those different areas, I mean, you don't have to go far in a business to encounter one of those problems where if you do behave massively different than you did in the lab, then you're going to make a major impact to the business for sure. I'm wondering, you mentioned this idea that you want to make sure that the input that you give to your model is something that you expect in your input distribution. I'm wondering how maybe that mindset compares to. People talk about making their models robust and you see people maybe perturbing their image data sets to flip them all around or enlarge certain things. Or Maybe there's that OpenAI example where the robot's moving the Rubik's Cube and they poke it with a stick or a stuffed giraffe. Giraffes are a major theme in this conversation, I guess. But one approach seems to be thinking about making your model able to withstand or be robust against perturbations. And maybe the other is concerned with making sure that your model never sees things that it shouldn't see. Is that a proper distinction? Or how would you sort of look at those two approaches?

Yeah, I think it's a good way of putting that. I'm sure all the audience, including you too, are following Andrew. And data is the important part here. And if I can control the data, I can control the model and better data, etc. Super important elements. And in the ideal world, I want to have access to all the data that I need in the lab. And I want the data in production to be static. Right. Ideally. And unfortunately that's not practical in many, many areas. Right. And we can do our best, by the way. I think that's one of the key considerations of tech companies that make them better, is because data as a strategy is so, so significant within what they're doing from the very first mindset. Right. How I going to collect the data, Is it going to be clean? How I going to train down the line? Amazing AI models of that and not, okay, let's look at the entire organization, what data I have, what I can do with AI now.

So what kind of strategies do you have to deal with that in the sense of. So, you know, we have this capability to produce these models in some sort of controlled environment. They're going out there in the world and they're being subjected to this. What are some of the strategies that you use to address that?

Yeah, so the two most critical elements that I think are very, very straightforward but not easy to build out of distribution detection. I want to make sure that the data that comes into the model is within the distribution. Now that's a quite. Let's say it's not super new research area but it's not super super advanced. So I think there is active research route out of distribution detection and how to apply it to some kind of areas. But if you can build this amazing filter that can detect data that is out of distribution and make sure that it's not going into the model, detect the giraffes with respect to the cat and dogs images, you will be better. Then you have the other part which is confidence or uncertainty. The research call it uncertainty estimation. I like the term confidence estimation because it's more straightforward and not the negative. So if you have good confidence of prediction, right. I got an image, went to the model and then the model give me the prediction, right? Softmax, classification, whatever, right. It's probably 95% sure about that in terms of probability and well that's not good. I'm putting in the air here that air quotes there. That is not really probability of course, but it's clearly not right. That's part of the problem. And I want to estimate if the model is sure about this prediction like this usually term uncertainty estimation or confidence and how to scale that and how to calibrate that and the temperature scaling, whatever these techniques enable you to say wait, the model is suddenly not that sure that it's a dog and I can really use that number. And going back to the mission critical elements now I can threshold something and make sure that if the confidence is below the threshold, let's pass that to a human. I have a human in the loop in many applications, but I not necessarily know to use it efficiently. So if I know how to use it efficiently because I have the out of distribution detection and I have confidence estimation. So I can now make sure that my model is somewhat now more stable. It has its defense mechanism, garbage won't come in and if semi garbage will come in, I will have a confidence score that will tell me suspicious and that will improve my stability 100% and it will improve my ability to combine this model with human which is super important in many applications.

Changelog is the best way for you to directly support practical AI. Join today and unlock access to a private feed that makes the ads disappear, gets you closer to the metal and helps Sustain our production of practical AI into the future. Simply follow the changelog link in your show notes or point your favorite Web browser to changelog.com Once again, that's changelog.com.

Changelog better.

So you talk about these sort of two filters, one on the data coming in out of distribution filter and one on the data coming out of the model in terms of quality estimation or confidence scoring or risk assessment of the output of your model and forwarding that onto humans. In your experience working with companies, one of the things that I'm always curious about is like how you start to judge what you should show to humans versus what you shouldn't show to humans and figuring out where that threshold is. Because if you go one way too far, then maybe you're letting garbage through, right? And that's causing actual problems in your systems and all of that. If you go too far the other way, then you sort of start eating away at the, maybe the good graces that you've built up in terms of making things more efficient. Well, now you're just sending all of this stuff to humans and maybe it was all right and they're not changing much about the output. How do you sort of dial in that sort of system?

Yeah, well, I don't have an amazing rule of thumb to answer your question, but what I can say that it's this mindset of AI solution in an organization is not a technical problem per se. It's a business technology problem that combines some business decisions, business process and technology understanding. And I think if we go back like three years to 2018, I think the first wave of AI challenges were characterized by understanding where to start, what AI can do, understanding of manager and executives, what. What about AI? But while we progressing that and we have better and better decision makers that understand AI and responsible AI and what AI can deliver. And they did some AI for managers course at whatever university or online course, and we can have this open discussion, right? And product managers that are understanding technology, but understanding also the business. And I think even today we have the AI product manager role that exists in some companies, which is amazing, and that enable us to take better decision to develop better products that are driven on AI. I think that's the main take. You need business people from stage one along the way to take decision to be involved in what technology can develop, develop.

So I love where you're going there and I actually want to go there even, even a little bit farther with you if you. Because as we've come through the conversation and you've kind of delineated these strategies, you know, both on the front end and on the back end to solve both sides of that. You know, let's bring it back to this practical aspect of you're in your business and you're trying to get some stuff done and you have a product manager who's trying to contend with this. What are some of the things that you have seen in the market to actually try to implement these strategies and thus mitigate the problems that we're talking about in real life? And I'm not saying it's holistic or a complete list, but what are some practical things that listeners who might be facing their own set of problems want to do? Whether it be data engineering on the front end of some sort or what have you seen?

Well, here I have a rule of thumb. Don't do POCs. Try to avoid as much as possible from building this model in the lab. Try to understand what are the production challenges from day one. For example, very classical problem is around data, right? You want to make sure that the data that you are training your model is exactly identical from the data in production, or even take the data from production to train on it and make sure that the acquisition in production of the data or the data that goes around to your model is exactly the same, right? It's the same distribution, it's the same problems, it's the same noise. Try to understand the hardware, the requirements, who is going to work with it. How it's going to work is the data is shifting over time. Do I need to sample differently? Do I have a monitoring element? That brings me actually to a super important factor about AI in production, which I think is interesting, which is highly related, is evergreen AI. This AI over time, right? It's not. I build a model in product in the lab, I got 95%, I put it in production, I'm happy. It's bringing me value. Oops. Three months pass. The data was changing. The model doesn't work anymore. I need to go back, recollect data, annotate, retrain, deploy again. Okay, it's working again, right? How you deal with this cycle efficiently, because none of the organization is talking about one AI model. That's not the goal anywhere. We're talking about adoption, acceleration, mass organization, change, transformation, et cetera. And I think Facebook published a few weeks ago that they. What was that? That they shift all the models in production to Pytorch. I think that was the topic. And the amount of models that they are running is just mind blowing. And everybody want to be not everybody want to be Facebook, but I mean in terms of the technology wise and AI wise, everybody want to be Facebook.

Wise wishing they could pull that off.

Yeah, exactly.

It's funny that you say that as you're talking about trying to get to where you're truly reflecting production. And in my experience working with several organizations, I've often seen an attempt by the data science team working on it to do almost the opposite because they perceive that the ability to get to production data and to move with that is somewhat overwhelming. It's a daunting thing for them because it's a moving target, it's incredibly dirty.

Often maybe they're not software engineers.

Absolutely. And so when you're saying that I'm thinner going that what you're saying makes perfect sense and yet I've literally seen teams trying to do the opposite of that repeatedly. So I mean, that's really good advice you're making there.

Yeah, yeah, I totally agree. And you know, you said about the software engineer as a joke, but you know, that's not totally a joke because to build an API that will sample data from production to your training environment and will deploy that efficiently, that's clearly a software problem, right? And how to do that consistently. And monitoring data scientists build models, right. And solve algorithmic problems. But to build a system and et cetera, you need engineers, you need software engineers, right? And I think that's at least my belief is that squads are probably the way of solving problems, right? You need engineers, you need data people, you need data scientists and you need them to work together on the problem and think about production and think about all the requirements and you need product managers, right? And we, you need the business people as much as combined with mutual communication, same language if possible, in order to tackle this problem. I think take for example something that is all the time coming again and again, security and governance and maybe like if we go into financial services regulation, right. I'm sure non data scientists will take these elements into consideration without the proper guidance and business people shouting a team, you need explainability. It doesn't make sense without explainability because I need to show the regulator how this model is taking actions, etc. You have risk departments and if the organization become large and complex like a bank, it's become more and more complex to bring AI to production and you need more and more elements. Now we need explainability and confidence and out of distribution detection and we need to monitor the model and we need to make sure that the data is feeding in same from the production and training. And we need to be able to work with the human in the loop and retrain the model and how to keep these models on the rails, how to deal with maintenance. So many elements besides the model itself, going back to what I told you in the beginning, shifting from a model to a system, we need so many different elements and then how you scale that, that's a problem.

It almost sounds like most of the work is after the initial model training. Who would have thought? The other thing I'm thinking about while you're discussing this is that it is necessary to bring all of those people together like you're talking about. In particular, I think the infrastructure and software engineering and data science people, because software engineers, in terms of the stability that they think of, they think of it like, oh, I'm going to write my unit test, right? If I change my code and it breaks the test, then I know that I broke my code. And I also have this table of calls that I'm testing against and my API expects this data and it either works or it doesn't. It gets that data or it doesn't. Whereas in AI, you're talking about just testing that your model can take in a certain type of data and output the expected type of data, which is sort of what we would normally think of in testing, isn't enough because the data and the distribution of data that you're feeding in could change the behavior wildly. So that monitoring piece, like the monitoring and testing piece, you sort of say those words to software and infrastructure people and they think of one thing. But for AI it is sort of that, but it's not exactly that. It's that and more, I guess, is what I'm after. Do you see it the same way?

Yeah, totally. I think that every one of the problems that I mentioned are both software engineering problem but also algorithmic problem. Take monitoring for example. Right. Every software engineer understands what monitoring is. That's something that they have been doing for 30 years, monitoring for any system. You need to make sure logs, etc. And understand when it fail, etc. But it's such a complex research problem to monitor, model how you make sure when your model is failing, if the data is changing. These are all research problems. If the statistic of the data is changing, if the model is suddenly not well suited, where I'm exactly on the distribution versus the data that comes in, how I'm looking at the vectors that come out of the models and understand what's going on of the model, or take that into the hot world Today, monitoring is observability. Right. Is bringing observation. What is happening, that's even more complex. To understand what is happening and why what is failing? Well, that's become even more complex. So people need to work together. There are many different elements to this problem. Software, research, data infrastructure, etc. And my takeaway, that's what I have been doing in the last three years is building a platform that is dealing with these elements in a unified way. I believe that you can generalize many of these problems in a way that will enable scale. It's required lots of work. The problems need to be generalized enough. You need to think about these technology elements across the pipeline of the system. But that can be done also in a small scale, in an enterprise. But of course my. I have my own way.

Well, I was going to say you've brought that up in your own way. Would you take a moment or two and tell us about the specific work that you've been doing in that area? I'd love to know how you are tackling these problems that we've been talking about for a while.

Yeah, sure. Of course I won't be able to dive into all the fine details, but the idea of what Beyond MINDS is doing. In the last three years we have been developing a platform and this platform is not a developer tool. It's something that we use in order to provide solutions to enterprises on scale, faster, better and crossing and bridging this failure rate. And the components, the main components of this platform is a system that is wrapping the model, trying to solve problems that are agnostic to the model as much as possible. Let's take out the distribution for example. We have a component that's all its work right out of distribution is one method out of many others that we're using there to measure statistics and data and garbage in to make sure that garbage is not coming in. So we build an entire component, we call it an input gate and it's an entire research arena that is dealing with stability and then monitoring and observability. That's another one that is as much as possible model agnostic. Not all of these components can be totally model agnostic. Some of them need to be tweaked, for example out of distribution detection that's highly coupled with the data itself. I need to retrain that, but that's again something that can be scaled. How to retrain, how to do that efficiently, that I can bring solution to production in matter of weeks, months, and I think today probably 12 to 14 months. I think that's the average time to production in an enterprise. Our goal as a company to take that into a scale of weeks, 12 weeks will be amazing outcome for us. And that's enabled because of the platform. Right. Building the model, that's we will have to do that is customizable. But then the monitoring can come easily. How to combine human in the loop can come easily. Explainability for most problems that are required, that can come easily. Stability can improve the model. I have a retraining in production. I have an entire monitoring and solution management toolbox that we are using. So I have this entire platform that is wrapping the model and bringing them to production.

It sounds very similar to the conversation that we had with William Falcon around Pytorch Lightning in that one of the things he was observing as he was working with different companies and models is that developers were over and over again fighting with the code that they were writing to deal with the hardware, like multi GPUs versus single GPUs versus this GPU versus that. And so in Pytorch Lightning, the philosophy was to sort of decouple the model piece from that hardware stuff that everybody was trying to write. And that provided the boost that Pytorch Lightning is giving people. And it sounds like here there are these other components and actually a very large number of them that data scientists are also spending their time writing code on over and over and over again. There's the stability, the monitoring, the human in the loop piece, all of these pieces that promote stability and robustness. And they're having to do this over and over. So it's not efficient at all. So I definitely think that that mindset that you have at Beyond Mind is really interesting in that sense, that it could provide that sort of similar boost in these other areas, which is really exciting.

He's bringing good. I mean, as I was listening to that, the thing that was going through my mind is he's bringing really good architectural decisions to kind of to everyone. And we have seen that often, you know, people in data science haven't necessarily had that exposure, you know, to the software side, where that as it matured, as the field matured, that became part of it. So I love what you're saying because I think that really contributes toward the maturing of this industry that we're all in at this point.

Yeah, yeah. And you know, I mentioned lots of research problems, but there are also many engineering problems there. Right. Kubernetes and Docker and APIs and so many different elements, how to wrap models and deploy them. Efficiently and fast. And this is becoming common practice more and more. But there are still some elements that are not that trivial.

Well, I really appreciate you helping our listeners think through some of these things. I know my mind has been racing the whole time thinking about like five different projects that I have going on and how these things map into those projects, like every single one of them, how these things map into that. So I know it's been really good, good in that sense for me and I'm sure it will be for our listeners as well. We'll include show notes in the show notes, links to the talk and other things that you've put out. But thank you so much for joining us, Roy. It's been a pleasure.

Thank you very much. It was pleasure talking with you.

Thank you for listening to Practical AI. We appreciate your time and your attention. If you enjoyed this episode, help us out by spreading the word. Think of a friend, think of a colleague, somebody who would benefit from listening to it and send them a link. We'd really appreciate it. Practical AI is hosted by Chris Benson and Daniel Whitenack. It's produced by Jared Santo with music by Breakmaster Cylinder. Thanks again to our sponsors, Fastly Linode and launchdarkly. That's our show. We hope you enjoyed it and we'll talk to you again next week.

Sam.