Podcast

In a land of testers far and widely journey seeking answers, seeking skills, seeking a better way. O the Test Illumination Testing Podcast Guiding testers with our nation awesomeness. From ancient realms to modern days, they lead the way. Oh the Testicle Ovation Testing Podcast.

Hey, do you want to learn more about ML Pipeline best practices and probably testing at scale? Then you don't want to miss this episode because we have an expert joining us. Lakshmi is a machine learning engineer who previously worked at big companies like Intuit Income and Poshmark and she has over 14 years of experience. She's really an expert in this area. She focuses on end to end machine learning, operations and test automation and she loves to talk about how to approach testing in the AI ML world that we all find ourselves into nowadays. So you don't want to miss this episode. Check it out. Hey, before we get into it, as you know we live in an era of applications and smartphones. Businesses are leaving no stone unturned to ensure a smooth user experience, but it is easier said than done. Businesses struggle to test comprehensively due to poor device coverage, device availability, blue low power testing, setups that are slow and unreliable and limited scale of the Automation setup. Introducing BrowserStack App Automate, this week's sponsor. It's a scalable cloud based automation testing solution for your native or hybrid mobile apps. It's easy to use and offers a plethora of features to ensure faster and confident releases. With App Automate, you'll be able to integrate your test suites in minutes using the Browser Stack SDK without any code changes. Leverage Real Device Clock to run thousands of tests in parallel across over 20,000 plus real devices. You can also integrate with leading automation frameworks, CICD, project management tools and many, many more. You also can test your apps on internal development and staging environments or behind firewalls. You can even test really advanced user cases like biometric authentication, payments networks, etc. On their network of both iOS and Android Real devices. App Automate is also powered by their all new automation Dashboard that delivers comprehensive insights into performance and issues along with AI driven failure analysis and advanced debugging tools. So what are you waiting for? Join over 50,000 customers, including most Fortune 500 companies that trust App Automate to take their mobile app testing to the next level. See it for yourself though. Check it out. Support the show using the link down below. Hey Lakshmi, welcome to the Guild.

Thank you so much for the warm welcome and the warm introduction.

Absolutely. So I'm always curious to know. I know AI and machine learning is the Hot topic nowadays, but I think you've been involved in it for a bit now. So how did you get into AI and machine learning?

Initially started my career as an Oracle developer and since then I took a roller coaster. I've slowly transitioned into machine learning engineering over the period of 14 years, I would say. I think in the middle of my career that's where my manager trusted in me and like, you know, okay, Tejaspi, I think you can do this. And, and at, at that point in time I was doing this automation frameworks for email marketing projects. And that was the phase where machine learning was actively in development as well. At the boom of predictive analytics models was in Reese in 2019 or 2018 I would say. And so I got the opportunity and I quickly jumped on it and slowly transitioned into DevOps world I would say, but into machine learning operations slowly. So worked on bunch of models, building automations around the models and testing models. That's how I transitioned into an actual machine learning engineer. So it was a happy journey, I would say.

Yeah. How did you do it? I mean, so you just, how did you learn then to get yourself up to speed? Was it something you were working on and just you were learning on the job then?

Yeah, so I'm actively learning and I think in the beginning of those I did not understand what a model is because the world is so different in the traditional software engineering versus the machine learning world. Right. Working on the automation pipelines for testing your software applications is very straightforward because your application is, you know, you would know the expectation around how your application works. But in machine learning world your inputs are different. You don't have a hands on, like, you know, you cannot see them and you cannot put an automation pipeline on them like you know, on the UI or like, you know, how you generally write automation frameworks, whether on the web, application, UI or mobile application on different devices. Right. So I was learning and your podcast was one of the like, you know, I'm an active listener and you were also actively talking about automating using like, you know, playwright and things like that. I was slowly learning new technologies and I was going through this machine learning mastery.com which actually talks about how you build a model, how to understand a model, how the inputs and outputs works. So there are like a lot of learning places that I am on and I actively listen to podcasts and I make sure I hit on like, you know, at least read two or three research papers every now and then and understand how the machine learning model works. I think that was really important for me to put a guardrails around how the model has to behave as it's supposed to. So I was trying to connect the world of automation world and also the machine learning world and the whole model observability evolved in my career. So model observability is more about. When I say it touches the model monitoring concept. Where you are looking at the model, hey, is it behaving as it's supposed to just like in the traditional software engineering world? Is your application working as it's expected to? Right. Another would be understanding whether the model is having any bias towards the regions or like, you know, demographics and things like that and also thinking about what are the checks behind different like you know, how you can enforce data integrity into your pipeline. So these are the, like I'm slowly trying to, how do I put my automation knowledge into machine learning pipelines? That's how I slowly, one by one I like, you know, try to implement in my day to day work.

Nice. So for someone that's just listening and they may be kind of new to this. What, what it's a model before we dive into like how we get into the pipeline and all that.

Yeah, good question. I would say that it's an entity. I would say like, you know, you can think this entity in different ways. Right. Either you're asking a question to it or you're asking it to define certain responses for you in such a way that it's more looking like a, a human given like you know, human generated response. So that's where it is trying to the like, you know, automated responses for you. But intuitively I would say more towards like approaching towards humanized responses that you would get but not exactly. The more towards you move towards the generative AI model then the accuracy would be like it's very closer to human responses and than the traditional predictive analytics is more about categorizing. For example, you say what is the, let's say you write a text and you say what is the tone of this model? Like I know in the email marketing world we try to send a lot of messages towards a particular targeted user. Let's say you're writing a humorous message or you're writing a, conveying a happy message or you're trying to convey a serious message like you know, a professional message. And the model actually here, what it does is it tries to take in all that input and it categorizes of a text. If it's a text based model. Right. If it's a summarization model. What it would do is it would concise whatever you have given into a smaller version so you don't have to explicitly, what do you say?

So once again, a lot of times when I talk to testers, they're like, I'm not a mathematical wizard. I need to understand AI inside and out, machine learning inside and out. But I don't think that's true. Maybe am I wrong? Like how much of AI does someone need to understand to really get involved, get involved with? Do they really need to get into the guts of it?

It totally depends on the role. I would say the roles are evolving these days. And also when you really get into an organization, you're like, let's say you are on the machine learning development side or AI development side, then you would need to know all the statistical, like, you know, the knowledge about statistics and the machine learning algorithms and things like that. I was fortunate to be on the operation side of it. So I'm not an active model, like, you know, development person, but I'm more on the operation side of it where I actually help with like, you know, deploying and creating machine learning pipelines and deploying them. So here you wouldn't need a lot of ML development knowledge, but to pass through the interviews, you should at least have the knowledge of like, you know, how the models are developed and how you play around with the inputs and outputs to at least understand a knowledge around how like, you know, there are different types of models and I think it's really important to understand the landscape of the models. For example, there could be reinforcement models and there could be time series models. There could be just predictive, simple, lightweight models. So traditionally when you're transitioning from a software engineering role to machine learning role or ML role or AI engineering role, you will need to understand the overall landscape of it so you can excel in your career. So but you could be focused on one just machine learning operations, but you still need to have the knowledge of what are these models, how, how am I playing with my data and in what way am I interacting with my data? And if it's a time series model, how should be my pipeline? So that actually is important for you to build the entire pipeline, I would say.

Nice. So we mentioned pipelines a lot. I think people are familiar with pipelines. For people that are not familiar though with the machine learning pipeline, can you maybe walk us through what a machine learning pipeline looks like? Maybe break down the different layers that makes up a pipeline for machine learning?

Yes, typically a machine learning pipeline would have at the high level, I would say like you know, categorized into three layers. One is the feature pipeline, the second is the training pipeline. The third is the inference pipeline. And whatever you do to take the model from building to the prediction, it typically falls goes to this feature training and inference. It goes through like from feature to inference. In the feature pipeline. What happens is here you are playing around with the data, data pre processing happens, experimentation happens of your model and you're actively developing your model and you're playing with the feature stores and making sure the data is in the right format for the model to intake. Right. And in the training pipeline what happens is your feature is extracted from the feature pipeline and now the model is trying to learn depending on what your model size is, your training pipeline compute times and your training pipeline infrastructure depends. Right. And once the model is trained on your training pipeline, now it transitions into storing the artifacts and the inference pipeline kicks in and it takes the artifacts and it actually serves the model on the production. You integrate that into the front end application and now the model is live. I would say like, you know, the journey of the model begins on the feature pipeline. And I would say like, you know, the life is actually, you know, it's very lively and it can take all the inputs and outputs and it's performing. Right? Yeah.

So I know what other kinds of testing, like performance testing and even testing features and requirements for a normal piece of software when it goes in production, you're like, ugh, I wasn't expecting that. So when we talk about models, are there any ways that you can experiment with it or use evaluation practices that help the model perform well in production? Like how do you know that what you're testing or building the pipeline for once it does get into production, it meets your expectations.

Yeah. So there are bunch of things that you could do to make sure your model is performing as expected in the production. First is trying to you understand the requirements of how your model should behave in the production. Right. And once you have those requirements, you go back and do the R and D of your like model that you think about what's my data is where should I fetch it from and what's my the time length of the data. So the first and foremost thing is once you have the requirements, you have to make sure that you do not overfit the model. So for example, you know that the model is going to have evolution of like you know, requests coming in and there is a change in the concepts for your model. Then you have to ensure that your training data is Such that it has like almost all the demographics, the like all the information that you could feed in so the model can learn. Imagine you're putting a child to a test in the future, right? You make sure that your child learns everything and like then you put in the child in the real world and because the child has already learned everything and now they can put it to use their knowledge. So I would consider that way. So there's a lot of offline R and D analytics is going on and that's the experimentation loop. I would say you're making sure you have all the data, you take care of all the data quality checks in there and the type of evaluation methods you. I would also put in where like if it is going through the CI CD pipeline, I would look at the data constraint checks. That's a great library that I have used. It's called Great Expectations. Where the great expectations does this. Like if your table is in the right format, like if it has any null value, the basic things and also flexibility of adding thresholds to your data. If the model has a lot of sparse data. I would say a lot of machine learning models do have sparse data where the inputs and outputs are just like zeros and ones. And what happens is the overfitting happens for the models if you have lot of zeros and ones. So I think it's really important to understand the type of data that you're playing with it. And you put the guardrails against the data first in your pipeline.

So how are you testing this? Do you create like regression test and test and sprint? So when the model is in your pipeline before it goes to production, do you run automated tests that check different things to experiment to make sure it meets the criteria you set?

Yeah, we should do a bunch of things. I would say unit testing is hard in machine learning systems, but the way I have performed it is like, you know, putting the data constraint checks in the pipelines itself. On like, you know, pre prod, I was not very successful doing it on the local because you really do not have the luxury of playing around with real data on your local missions. Because of the GDPR constraints and things like that. We are not supposed to store the data on our local systems for more than 30 days. Right. So pre prod is the best friend for me here playing around with the data checks. Some type of unit testing would help to understand if it's a SQL model, if it is running on the SQL queries, then it's possible to put unit and integration tests in there. But if it is a fairly text based text summarization models, things like that. Then I would put like, you know, post evaluation checks. That means the model evaluation is happening at the output layer. Like you know, once the model goes to the inference pipeline, then you're storing all that input and you have the feedback loop happening. So the constraint checks happens on the feedback loop as well. And also another thing that you could do is you could look for the concept drifts on the model by integrating some model monitoring tools and observability tools. Right. You can look for the data drifts happening and you can look for. This is a whole new evolution happening. On the interpretability side. What you could do is you understand the inputs and outputs of the model like you know, offline, you do not have to do it online, but you take the inputs and outputs of the model and you perform the sharp analysis on it and lime analysis on it. And that actually tells you how your model is performing. Right. And that's a really good tool to use about the model. So I would say in a nutshell the evaluation should happen at all the three layers. Training. Once the experimentation happens, you look at the performance of the model by looking at the metrics like precision, recall, auroc, AUROC scores and you look at the latency as well, like the performance of the model. It's really important to understand how the model is performing. Is it like for example, if it's a text based model you expect the response to be in fraction of seconds. Right. If it's a batch model and like, you know, if it's a heavy under computation, you still need a good latency. It all depends on how you have actually set up your ML pipeline. So latency is one big factor that I would put a check on. Looking at the traffic. Yeah, yeah.

I wasn't even thinking about performance. But that is probably huge because people expect answers quickly nowadays. So that's interesting for sure. So I wrote down, I don't know why I wrote this down because I think you mentioned it a few times. Guardrails.

Yeah.

What are guardrails? If people are listening, how do you put in guardrails? What are guardrails for when we're talking about machine learning?

The reason I say guardrails is because in traditional software engineering world, you know, the expectation of your software application in the machine learning world, it's really difficult to have that expectation to constrain to a box. Right. You never know. Like, you know, for this output it should. You can only know the threshold of it. You can only know, okay, 70% of my model behaves this way and 30% I do not know. And there could be a change in the, like, you know, the incoming data and which will make the model accuracy score go low and your expectations are no longer working for you. So that's why I say guardrails. When I say guardrails, it's making sure that I keep a tab on my model through the entire pipeline and making sure, okay, I am confident that my model have feeded with the right data. I am not falling into any GDPR issues and I have the right quality of data. I do not have any. Especially with generative AI models, it's really important to make sure you do not have any bias feeding into the models. Right. So your incoming data has to be screened in such a way that you have the best quality of the data that you're playing around with. And in the feature pipeline, that's a ground for you to understand if you want to. Like, these days we are thinking about automated retraining pipelines. So now you want to retrain your model when you are actually not looking at like, you know, you want to retrain your model so it can perform really well. You're. You want to increase the precision of your model. So when we are thinking about the retraining, we should be very careful about how the retraining is happening. So put the guardrails around there. And in the training pipeline, you want to make sure that I know the typical 8020 rule. Everybody knows about where you are training your Data model with 80% of the data and 20% you are testing. Right. So if the 8020 is working fine, everybody looks at the accuracy. But I also look at the AROC scores because the accuracy itself is not the right metric for us to listen. Sometimes your accuracy can be 90%, but still the model doesn't give the right responses. So the Auroch scores give you the right measures. So I would also check for Auroch scores to understand whether the model is having the right context and giving me the right response. Right. And in the evaluation pipeline, in the model observability pipeline, that's where I have the right handle of whether model is going through any drift or not. So I would have the check on the drift happening in the last one week or last two weeks. The performance of the model has changed.

All right, dumb question. What's drift?

Okay, the drift is. Let's say I am. Think me as a model now. You expect me to answer questions on the. Let's say I'M a like, you know, math tutorial model. You ask me all math questions and I'm performing very nicely and like, you know, I give the right answers for you. Suddenly you ask me very different question, something related to science and I might have some context of science in my training pipeline soon. Now I could hallucinate and also give you science related responses, but I am not trained on science. Science. But I'm still giving you responses on the science models. Right? Science requests I could hallucinate and I'm. My responses are changing. So now when I say drift there is a change in the requests like, you know, the requests of questions in the math format. Also now you're seeing also the science related. Tomorrow you will see social related questions and tomorrow you will see something totally different like, you know, art based questions. So there is a change in the concept of your requests. Right. So that's one type of a drift. And in the same way you will see the drift in the responses as well. So now it's really important to keep a tab on this and understand where the drift is happening.

Yeah, Hearts were drifting. I love it.

Yeah. So you understand that the drift is happening at the request and you quickly train your model with more training data and make sure that the model is having more knowledge for you to give the right responses.

So would a guardrail be like, hey, I'm a math model so you have a can response like I'm not trained on this so I'm not even going to answer it or is that a bad way?

So it's interesting that you asked this question. The guardrails in this case for me for a math model would be if it's an LLM based model or if it's a predictive analytics model. But the scenario seems to be more towards the LLM model. Right. I would put a check on like three to four ways. For example, one way could be you were an if condition. If the response, if the questions and requests are coming incoming which is not related to computational related questions, then say that. Not like, you know, I'm not. I don't have enough context to, you know, respond to your question. So I would need more context. So you would make sure that the model is responding in such a way. Right. Or you would keep a tab on your outputs where you're saying it depends on totally how you have set the model. If there is a scores response course that you are keeping a tab on. If the model is giving a response to science, the score of the model would come down totally. If it's a math based model if it is seeing requests coming in from arts related or science related. If it doesn't have the score response the accuracy score would come down. So you put a guardrail on the accuracy score there and if it is below 80% or 90% then it would alert and you would send an alert on how the alerts are set especially like you know totally depends on your pipeline so the alert would kick in. Or if it's a monitoring based tool you have an anomaly detected on your output layer. Then you quickly look at the details of the why the score has went down, what are the requests. So you do a reverse engineering and reverse lookup to see why the change has why the score has gone down and how you fix it.

All right, before we go, is there one piece of actual advice you can give to someone to help them with their ML AI DevOps or automation testing efforts and what's the best way to find or contact you?

Very active on LinkedIn. I also do TopMed meetings if anybody is interested in like you know, have want to transition their careers into machine learning and AI. My Topmed meetings are free. I do free 30 minute sessions. Yeah I have like good resources of like you know how you want to transition your career. I would be happy to send that link as well.

Thanks again for your automation awesomeness. The links to everything of value we covered in this episode head on over to test guild.com a538 and if the show has helped you in any way, why not rate it and review it in itunes? Reviews really help in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Automation Podcast. Joe My mission is to help you succeed with creating end to end full stack automation awesomeness. As always, test everything and keep the good. Cheers. Hey, thank you for tuning in. It's incredible to connect with close to 400,000 followers across all our platforms and over 40,000 email subscribers who are at the forefront of automation testing and DevOps. If you haven't yet, join our vibrant community@testguild.com where you become part of our elite circle driving innovation in software testing and automation. And if you're a tool provider or have a service looking to empower our guild with solutions that elevate skills and tackle real world challenges. We're excited to collaborate. Visit testguild.info to explore how we can create transformative experiences together. Let's push the boundaries of what we can achieve.

With lutes and lyres the bards began their song A tune of knowledge, a melody of code through the air it spread like wildfire through the land Guiding tester showing us secrets to behold.