What Automation Technology to Use with Sneha Viswalingam and Carlos Kidman

Foreign.

Get ready to discover the most actionable end to end automation advice for some of the smartest testers on the planet. Hey, I'm Joe Calantonio, host of the Test Guild Automation Podcast and my goal is to help you succeed with creating automation awesomeness.

Welcome everyone to another episode of the Test Guild Automation Podcast. Today I want to share with you a chat I actually had Last week at LambdaTest TestView conference all about how to decide what automation technology to use for you and your team. We had a really lively discussion around this. You don't want to miss it. Listen up. The Test Guild Automation Podcast is sponsored by the fantastic folks at Source Labs, the cloud based test platform that helps ensure you can develop with confidence at every step from code to to deployment for Every framework, browser, OS, mobile device and API. Get a free trial. Visit testguild.comSourceLabs and click on the exclusive sponsor section to try it for free today. Check it out.

Hello everyone.

Welcome to our current panel discussion. This is your host, Modit. Welcome to our panel discussion on how to decide what automation technology to use. So this would be more of a panel discussion between our panelists, Joe Colantino from Tezcale, Sneha, who is director of Quality Engineering and everfi, and Carlos Kidman who is director of engineering at a stealth startup that he has recently joined. I'll be your host mudit, but I'll be there more in the backend helping more as a tech support. I think Joe, you will be leading the session so I'll leave the stage over to you.

Awesome. Thank you so much. And I welcome everyone to the LambdaTest Test U panel discussion and how to decide what automation technology to use. I do a survey every year and one of the biggest questions people always have that they struggle with is how do I choose what technology or solution to use. There's so many technologies out there and frameworks and a lot of things around machine learning and AI that are kind of clouding the waters. So I thought we'd touch on a few things that maybe help people address this. This major pain point. I see. So the first question though is I guess before we get into that, maybe we can go around, you can add more to your bio. I just gave you a quick intro, but anything else you want to say about maybe your role or anything you want to share with everyone? So Sneha, let's start with you.

Sure. So as Joe mentioned, I'm Sneha. I work at a company called Level 5. It's a netch company, so I Oversee the course QA side of things. And we build internal tools in the QA team to help generate automation using model based testing. That's a little bit of what I do, but otherwise I also have a podcast called Coda Stories where, you know, I interview people about their passion and hobbies and what drives them, what motivates them. Because as much as I like the technical side of the things, I also love and have a lot of passion for the human side of things. So that's what kind of drives me in my job.

Love it. How about you, Carlos?

Yeah. So I am the head of engineering at this super secret place, but it's an AI company and so we're using AI to automate some software testing and quality things. Super exciting. But Sneha sounds way more interesting than I do.

Very nice. So Sneha, you mentioned being interested in the human side of things. I'm just curious to know with the first question is how important is it to have a strategy in place before choosing a framework for your team? And maybe the human side comes into play is not all teams are the same. I think sometimes people like, here's the tool everyone uses, therefore my team shall use it. So do you have a strategy you like to lay out everfi or any company that you work for that maybe? Yeah.

Yeah. So I've worked in a couple of many companies now where, you know, we've, we've had to use different strategies based on the skill level of the team, the cost, you know, in mind. So automation is definitely a very, very important step in DevOps. Now we can get by with just adding whatever bunch of tests, but without having a strategy in place, our automation is not going to be a lot of the benefits that test automation can offer completely remains undiscovered. Some of the things that I would start with is what is the application under test, what is the technology that's involved in it, what is the user experience like and what is the complexity of the workflow? What is the testing that we already have so far? And then, you know, start thinking about what does the automation journey look like? What kind of, what is the scope of the testing that we want to do? Like, do we want regression functional unit testing? And then we want to start also thinking about budgeting because that's, that's a very big, that's a very, very big thing in this. Right. So depending on the budgeting that we have, we can decide to go in whatever direction. And the skill level of the folks in the QA team also kind you know, decides what, what direction we want to go. Do we want to build a customized framework within the company, or do we want to buy a tool or do we want to just go with an open source option? So I think strategy definitely comes in place. We need to answer all these set of questions before we try to decide what tool we want to go for. And these are some questions that, you know, helps with the strategy.

Great questions. Carlos, anything else you want to add to that or how maybe your strategy?

No, I mean, I think that, like, as Sneha is pointing out, like, there's, there's a lot of variables that go into it that you. It's not, it's not as simple as it's something for your open source because that comes with its own costs as well. So I think asking those questions, being aware of those, I guess, parameters or constraints that you have, how much money are you able to put into it? How many people? What are their skill set? Everything I talked about are the things you have to consider. On top of that, I have a lot of people ask me similar questions around. Like, Carlos, you're an AI, so what AI tool should I use? And I think that's going about it the wrong way.

Right.

And that's why you want some sort of a strategy. You have to say, well, what problem are you trying to solve? Because otherwise you're not going to prioritize anything. You're not going to know what your parameters are. If your only question is, what tool do I use? It doesn't sound like a very good exercise to go through because my immediate answer is, I don't know. And that's not a great answer. So I think going through what Sneha mentioned is probably a very, very good way to go about it.

Great, great. You know, when I worked for a large company, one thing we failed to do was look at our roadmap. So we started at a greenfield application, we chose tools, and then a year out when some of these other features started rolling in, we're like, oh my gosh, we didn't plan for that. Is that something you take into consideration as well, or does that come into play when you're doing legacy type of kind of framework building as well?

Very much so. So like I said, you have to kind of define what problems you're trying to solve, because without that, like, prioritization, not just on the business side, but on the, like, internal quality side of things, you are kind of just trying to add tools for some reason or add automation for some reason. And so it's difficult to measure the value of what you're building if you don't know what problem you're solving with it. And so looking at a roadmap, if you know what it's going to be worked on in the next quarter, you can start planning for what you would like to target first. So I had one person ask me, carlos, we're going to start doing application security. So, like, what are the best tools to use for that? And again, it goes back to. Security testing is a huge term. That's a very large, very broad thing. So for me to be like, oh yeah, just use this one tool like that, it doesn't work that way. Right. So instead it's, well, what are the things that you're trying to solve security wise? Why is that a topic that you're asking me? Of all the questions you could have asked me, why did you ask me about security testing? And I have to, as almost like a consultant, ask questions to find out, well, it came from this stakeholder. Well, the CTO is looking for this. And then eventually we figure out, okay, maybe OWASP is a great place to start. And if that's where you're going to start, then let's look at these tools to help with that. But if I just say security testing and let's just do that.

That'S the wrong place to start.

Absolutely. Jose says you're dropping gems in here. Both here. So thank you. So far, going well.

So just for you, Jose.

I know these, these questions just pop in my head. In the pre show you talked about getting acquired. Does that ever affect strategy? Does all of a sudden a company come in and say, here's the tools we use and so you need to come on board and have the same thing across the different silos? Or is that something you ever have to deal with?

Not right now, but I foresee some things, like, I would assume some of these things may happen, you know, in companies while acquisitions take place. I have myself not experienced it so far, but I do see that these are some of the reasons why we need to have scalable and modularized. You know, when we build something, we need to think about, okay, what if we have to change one aspect of it? What we have to remove one tool and add another tool. Like, is our framework, like, easy enough for us to be able to remove things and add things accordingly? Like, this is why we need to have a skill set, like skilled people working on these things, so that we're making our frameworks more like, you know, simple and easy to use and maintainable, modular, scalable and things like that.

Yeah, Joe, I, I, and I do have experience doing this exact thing.

Okay.

I, I worked at a company called Workfront. You can think of it as like a competitor of Jira and I'm not gonna go into that, but yeah, competitor to Jira. And we got acquired by Adobe and that's why I got into Adobe was because my company got purchased. And so one of the big changes that happened was at Workfront we were aws.

And then once we went to Adobe now we had to support GCP or Google Cloud platform as well as Azure. And so the way that we had to handle that was very unique because again, we had to prioritize. Well, we can't just say migrate everything over to that. You have to say, well, what are we going to start with first? And create a roadmap as to what we want to tackle first, what piece is going to give us the most return for our time. And instead of us having a, like a testing team do that. We ended up forming a platform engineering team, which is also a pretty hot topic now by the way, but we ended up making a platform engineering team and a team of architects and each architect represented a specific domain. So we had a front end architect, we had a database architect, I was the test architect and we would all work together to prioritize how do we want to do things, which team are we going to start with? Because you can imagine companies of that size, there's little companies within it, so you can't just say, all right, all developers use Cypress, there's way too many teams. And so you have to make these templates, you have to create automation, you have to create standards, tech radar templates, like all these different things to help teams be able to migrate and then pick one team, say let's start with this team so they can champion this transition and work from there. So it's a pretty unique situation where there's a lot of things happening at one time and but you still have to prioritize and that's why I keep saying that word, because you can't do everything at one time.

Absolutely. There's a lot of good stuff there. So I'm trying to pull out some things if I remember. So Snow, you mentioned, I love this that you know, if you design your, your strategy up front and the way you design things, even if you got acquired, it would be kind of seamless because you may not plan for that specific kind of scenario, but because of you created with a strategy in place that it becomes less of an issue. And someone told me, I don't know who. One of the biggest compliments you could have is when you leave a company and the framework you created is still used years after you leave. And like you said, making it so you design it that way up front. So I thought that was interesting. Why did I go there? I forgot. But question here is, so you have the strategy in place, you've created it in such a way that it's easy to consume, it's readable. How do you know then when you do decide, okay, we're going to look at this particular tool that it's going to work for my team? Is there a process in place that you use to say, let's see if it works, or a proof of concept or some sort of thing that you say before we actually go all in, let's see if it's actually going to work for us?

Yeah. So how I would approach this is like, you know, first figuring out the requirements, what is the problem we want to solve, and then as you mentioned, proof of concept is definitely a good way to go about it because you want to first make sure that you're able to get a quick end to end without any issues. And if you really want to make sure that you're choosing the right reporting tools and making sure that you're having all the elements that you really want. For example, for me, some of the things that I really needed was visual testing and audio testing and video testing. Because I work in NET Tech and we have a number of courses. We want to make sure that we're able to integrate all these parts in our framework really well. Again, as I mentioned, I may use use one vendor right now, but then I would decide to use another vendor if, you know, depending on the budgeting and cost, if you want to switch. And I want to be able to make it reusable and flexible so that I'm able to like pick it up and move to other things. So those are some things I would, I would take a look at. And also having a common standard, you know, coding practices, if the entire organization has some kind of coding practices, we want to like put that in while we are building our framework. So that is something I'd keep in mind. And if the framework that I have will be able to provide all the coverage that I'm looking for for the different kind of testing. Because accessibility testing is also something that I kind of put in my framework. So yes, doing a POC to make sure that I'm Able to, you know, add the different things that's needed for my requirement for my testing scenarios.

Nice, Carlos.

Yeah, plus one to that. But we use, and you've heard me say TechRadar a couple times now, and TechRadar is, it's supposed to be a document to kind of enable a discussion around what are the things that. Exactly what Sneha is saying, like, what are the things you need to consider? What are the things that we want as a team, what are things we don't want? And I think a lot of the time people don't ask that question of what are the things we need it to have, what are the things that we want to have, would like to have what things do we not care about? And asking those questions in a, in a document that everyone can point to and add comments to, and everything becomes this living document that you can then refer back to. So that way, like for example, Selenium Cyprus were the hotness a few years ago, and it was both of those fighting against each other, right? And then all of a sudden Playwright appears. And now everyone's tech radars got slapped in the face because now you have to put Playwright through the tech radar again. But it was a great exercise to get everyone to go through that again and say, okay, well, let's evaluate these three tools more granularly because, yeah, all three of them have their advantages and disadvantages. And so what do we want? If we need accessibility testing, then put that as a point that we want to have covered. Well, when we say accessibility testing, what do you mean? Add those bullet points and details and questions as well. And you keep doing that. If we wanted to do pen testing, fantastic. What do you mean by pen testing? Add those details, add those questions. So when you are going through a pocket, you're able to then try to check off all those bullet points or all those checkboxes to say, can this tool do what we need or not?

So, Carlos, I guess the big question here is then, since it's been through your tech radar, which one is better? Selenium Cypress or Playwright?

The tool I'm working on. No, I'm just kidding. Gee, man, seriously. Each team has. That's another topic.

There is a talk based on just that.

I can feel the eyes on me right now.

Yeah, Manaj says he would say Pilenium, honestly. But it is so team dependent and the people side of things, as Neha has said already, it depends. At a company, the size of Adobe, we had all of them and so more so what we needed was for Our test architecture platform engineering team to say here is what the tech radar is. So if you, a team said Playwright came out, we want to use Playwright, then they need to go through that tech radar and say, well, can Playwright do all of these things? And if it can, then that's something that the platform engineering team is approving or the architecture team is approving and you can use that. And we would let the teams be autonomous in that way. But I will say that it's, it's easy to say the new shiny thing and put it in. But in our tech radar we had things like you have to make sure you're aware of what the infrastructure team needs. Because for example, if we already have templates and automation for Cypress and Selenium, you wanting to add Playwright now requires either you, the team to take on that responsibility or now that's another epic for my platform engineering team because now we have to take on that load to create the infrastructure and pipelines and everything. So we ended up just using Selenium and Playwright or Cypress specifically for UI automation. But the same topic happens with a lot of things, right? Should we use New Relic vs Datadog? Should use the same concept applies for everything. Just this was an example for UI automation.

Very cool. I guess getting older. One of the good things, not a lot, is that you have some insight into how things were For a long time frame. Before it was always hard to get people just to do just basic automation. But I thought, interesting in this discussion, both all of you, you've mentioned visual validation testing, accessibility testing, security testing, pen testing, things that are more outside of just this traditional functional testing. So it's almost like end to end testing. So you know, how do, how do you then expect someone to know all those individual aspects of testing and all the tools that go with it? Sneha, when you were evaluating accessibility testing, I assume not everyone knew like how to do accessibility testing or even what tools to use. So how do you add that on top of all the other type of testing work that people need to do?

Just a lot of research. We were kind of lucky that we had a director join in our engineering who actually came from the company, from one of the accessibility related companies. He was one of the persons who built one of those. We had a lot more insight to it. We used Axe Core plugin with Cypress for accessibility testing and it worked out really well for us. I think more of just research. And we also had an accessibility test team dedicated at evofi, you know, to help us with something like this. And we took over the automation part of it to add it to our framework. But yeah, so whenever we want something added to automation, just a lot of research and trying to figure out what is the best tool, you know, that'll go with our framework and just to plug it in and see how, how that works for us.

Perfect, Joe. On, on, for me, it, it also depended. It depended on the people that we had. So if we had the skill set, like for example, we had a front end developer that was huge accessibility advocate. And so they're the ones who led the whole design system, making sure that all of our components were accessible, light, dark, mode, like all of that. They were so passionate about it that I let them run it. Right. It wasn't a tester, it was a developer that loved it so much, which was fantastic. So when it came to things like security, I am not a security expert. So looking at my team and saying we don't have someone who's an expert here, we could either spend a lot of time doing research and trying to like level ourselves up or I could hire another expert and bring them onto the team. So it. But again that, that is another business decision in and of itself. Do we spend the time leveling up our current team or do we hire someone externally and bring them into the company? And for some things it was leveling up the team. For others, like SecOps in this case, we ended up hiring a whole team for that. So each one of those things will. Again, it depends because it depends where your team is at, where your company's at, your culture, your skill sets, what people want to take on right now. Are they, how many hats are they wearing, things like that.

So yeah, yeah, I totally agree, Carlos. And also to add to the point, one thing that we did was we have a very good understanding with the engineering, like with the development team. So if we ever have to like we have this concept of Devon loan, so the devs come work with us in testing and help us build some of the tests that we probably don't have a lot of exposure to. So that has really worked well, just like working hand in hand with the development team. So that way when they go back they totally understand how these tests work and they're able to like help maintain and all of that. So just like, you know, being one with the developers really helps a lot.

Be one with the developers. Love that. Great advice. So Carlos, you mentioned, you brought this up, you said build versus buy in one of your answers. And you know, I think a lot of people are so into open source, sometimes that they just lose sight of, of maybe there are other tools out there that you can maybe purchase. And I'm looking at the questions and some people are talking about budget. So maybe that's one of the, the factors that go into this. So which you thought on build versus buy, is it always one or the other? When, when does it make sense to buy rather than like building your own lab or something for all the devices you need to test it?

Yeah, I mean, it depends.

The guy who built his own framework.

Yeah, I mean, seriously, I would. I usually lean towards the buying side of things instead of building just because I think that people underestimate almost always how much work it takes to build and maintain something yourself. And usually what happens is you'll have one person who is super stoked about building the thing and so they build it, they maintain it and then they leave and now who's going to do it? So you hire someone else and they don't know anything about that and so they end up building and maintaining their own thing and the cycle continues on. The biggest argument I hear about buying is, well, I don't want a vendor lock in. It's a very weak argument because that means that you're not considering all of the other parameters around what it means to buy versus build. So I like buying because I'm rich. I like buying.

In there. I'm just kidding.

I like buying usually just because it's. That is an amount of work that I don't know, I don't have to put onto my team. So if I can have like, for example, my team is smart enough that they could, they could probably build a subset of something like a Datadog, we could probably do it. But how long would that take me with my current team, it would probably take me 6 to 12 months to build a small version of Datadog. And so do I want them focusing on that project or are there other priorities that I have that their time and energy is better spent? So then let me bite the bullet if you will buy the product so then I can get that. The problem I'm looking to solve with Datadog is that observability, monitoring, some security, synthetic data, those kinds of things. I can get all of that by just buying the product rather than having to try to build something that is not going to be as good even the first year around. So it's, that's why when we're looking at doing a tech radar, it's, you're also looking at the skill set of your team. The, the time that they have your priorities as a business. And again I keep saying priorities because it's one of the biggest constraints you have is if you have a deadline of three months, I am not going to tell you, please build me Data Dog. Like come on, you know what I mean? Like it's easier for us to just buy the tool, take a little bit of time to integrate it, leverage it. We'll have dedicated account managers that can help us with stuff and use that instead. It may be a more expensive cost, upfront, monetary cost, but I'm going to save costs on time, energy, priority, business objectives, deadlines, features out to all those things I will gain because my team is no longer spending time trying to build and maintain something awesome, awesome stuff.

Time is flying by. I think we only have around 15 minutes left. There's a bunch of questions in here, but I'm going to try to combine the questions the folks are asking with some questions I want to know. So the first topic I think we need to touch on is AI machine learning. I'm going to smash this with one of the attendees questions on how do you decide on choosing the right tool, especially when a vendor is new. So a lot of these new vendors use AI machine learning. What are your thoughts on AI machine learning and if you do believe in it, when a vendor is talking about it, how do you know it's actually legit? I guess. Sneha, let's start with you. Since Carlos ended on the last one.

I think Carlos is definitely a better person since he is working for a company that is AI machine learning based stealth startup.

It is AI based.

Yeah. I have experience actually working with the vendor with model based testing. So the way I would approach this is, you know, just again just making it more. It's always like, you know, you need to understand that quality is a journey, not a destination. And there's going to be improvements, there's going to be new requirements that come. Our requirements are going to be evolving. So what I believe in is if you need to buy, buy.

A tool. What how I approach this is buy it. If you have, if you need to fix a problem or find a solution right now, use it. But then you could always transition into building your own thing later if that's not working out for you or you know, if you need to extend or something happens. Like you can always transition into something else. You're not really locked in to a vendor if you build your framework correctly.

Okay, great. Great advice Carlos. Specifically to machine learning. Any thoughts on it? Now that you're in the like the nollie part of it, working for a company that actually is AI machine learning.

Yeah. So I mean, I'll be biased and say, yes, use it, do it, because it's very powerful and very helpful.

Now, when you're evaluating these products, I mean, I'm working for a company where we are going to be making a product specifically for software testing and so using AI to do that. Now, if some people are going grown because there's another one of these, what I would say is all of these companies are going to have a trial for you. You can try it out and if you have no automation, you're probably going to spend a lot more time kind of evaluating these solutions and really leveraging those trials. If you already have a lot of automation in place, then you're probably going to have a smaller team of people who care about. Let's evaluate this new tool and see if it's something that would help us out either way, take advantage of the trial and see what things they would do. Visualize what it would look like if they were actually doing things. If you purchase that tool, how would it affect your team? How would it affect efficiency, thoroughness, productivity, how would it affect those things? And do you see an overall gain or do you see it as this is another tool I have to maintain. So I don't want to say any company names because I just don't want to. But when we say like AI driven automation or AI driven test testing, that doesn't mean that everything about that system or that product is AI based. And that's something that people need to understand is there may be a small part of it that is AI based. For example, it could be where it looks at the only AI part of their product is looking at test results and looking for anomalies. Right. That that could be the AI part. And because there's a little bit of AI, they stamped AI on their domain and they now say we're AI driven, but in reality it's an outsourced company doing most of the automation. And now you have a model looking at test results. Another, another common thing I see is where it's AI driven, but it requires the tester to do almost everything. To start out, you have to install everything, you have to walk through the app yourself, you have to label all your images as it takes screenshots, and that eventually turns into the data set that feeds into the model. So it still requires that someone maintains the application, maintains the product, so then the AI does the work after that. And so it's, it's, that's why I say use the trial. Because even though it says AI driven, each one of these products is going to have a different flavor of what they mean by that. And so it's good to ask those questions. What part of it is AI? What part of it is not? How much work do I have to do as the tester or do I have to have someone on this logged in all the time? Is it going to run? How many, how many things are run on its own? How do I integrate it into my CI if I have that? If a pipeline fails, what is the process for triaging that? So all of that has to come into consideration just like any other tech radar. Because just because it says AI doesn't mean the whole thing is AI driven.

Absolutely. And this is such a deep topic. You probably need to follow Carlos to really, to really do this justice. I just want to move on, go into a lightning round. Now we have about 10 minutes left. I want to get to all these questions. This font is really small. I can't see the names that. Well, one is, I think Tamika wants to know how much multi language support is important when picking up a browser automation tool. Like I said, really quick snea important, not important. What are your thoughts?

It depends because you know language, I would say, you know, some people think that they need to stick to the programming language that the.

Application uses, but I don't really think so because I think you can use whatever language you want to write. The framework or automation doesn't have to be the same as the product because in a company you might have so many different applications, you're not going to have a framework for each different application based on the language it's written in. I think it just depends on what the QA team wants to use and what, what is what works best depending on the application and the test.

Awesome. Awesome.

And I say for that it, who's going to be maintaining the thing? Who's writing it, who's going to care about it? They should be the ones to pick the language.

I'm going to go in the rant. Yes.

In my philosophy and we did this at Adobe, I had the developers do it. So the UI automation was done entirely by developers. None of the testers were doing the UI automation unless they really had to. And so if a team was really into JavaScript because most of the front end teams were JavaScript or TypeScript based, then yeah, it was usually a JavaScript or TypeScript based thing. And so for the most part, Cypress was pretty popular, but there were limitations Cypress had. And so for those they used Selenium. And I don't know where things are now with Playwright, so I don't know if they've switched or not. But we wanted the team because ultimately I wanted the team to care about their own quality, the team to care about their own tests. And so if I said, well, I want you to care about it, but I get to pick it, that defeats the purpose of that philosophy. And so that's why I would let them choose. But again, still having standards and all that stuff. Make sure that you can't just pick any tool you want. It has to meet these things within that tech radar.

Awesome, awesome. Next one's from Manish wants to know. Hey Kalos, I've used Pynum. It's a great framework. What do you think about the future of Python use in automation testing?

Python is. Python is a language. It's a beast.

Do it AI machine learning.

Python, it's the language of automation. So it's a very powerful language to know.

Sneha, do you like Python? Do you care?

I like Python. I work a lot with Jupyter notebooks and Pandas and all that for data reporting and stuff. So I love Python.

Great. Another person has a good question here about code coverage. How important is it to have code coverage in automation testing?

Is it a KPI that could be abused? Do you care Snea as a director, do you have KPIs that say thou shall have blah blah blah, code coverage and automation, how does that work?

I mean it's good to have code coverage in automation, but at the end of the day, what do you want to know?

Do you actually want coverage for the entire code or do you actually want tests version for the main set of things? Is it necessary to really test the entire code base? That that's where I would, I would be like, I don't think it's necessary to cover the entire code base. There may be some. I think we need to do selective testing, risk based testing and analysis and then take it from there.

Carlos, I like code coverage as a tool to have conversations. It's a tool to say, let's look at our stuff. Do I want 100? Hell no. Because that means that you're probably focusing on the wrong thing. So it's good to be able to say like, I want to know what these things do. Let me write some tests against that. But I'm not going to say, well, this is a bad team because they only 75% code coverage. That doesn't mean anything to me. So if it's used as a tool for collaboration, conversation, be able to have things about risk, then it's a great tool for that. But it's. I use it for that, not as a vanity metric.

Beautiful. Getting some hearts and love there. Andy wants to know, do you think there is value in doing manual testing in DevOps model where efficiencies was believed to be driven by test automation?

I get this question all the time about manual testing. I don't know how you could do any type of no manual testing, but.

I don't know how you could do automation without exploration and experimenting.

Yes, right, right.

So I'll just say that as my lightning round, you kind of have to explore, you're gonna have to experiment. You can't do automation without it.

Absolutely. So, Annie, I think we all agree it is needed regardless of what people are pushing or saying.

Yeah, I don't. I don't think we can do 100 automation like we have. We have voiceover testing, content testing, a lot of different types of testing. So we need manual testing.

Exploration, experimentation on the left side of things. On the right side of things. Chaos engineering is the definition of exploration. That right there you create experiments. I wonder what this will do if it goes down like that. Right there is exploration. You might use tools and automation to help you with those experiments, but it's an exploratory exercise that you're doing. So shift left and shift right both do it.

Amen. Good, good answer there, both of you. So next question is, I don't know if either of you have experience with this or, you know, what tool do you suggest for contract testing?

Oh, I would just use PAX because it's the one that we've been using and it's been great so far, but I don't know if there's another one that's better.

Okay, someone wants to know why is it not suffering if there's not sufficient. Why is there not sufficient budget for testing compared to development? So you build directors, you have to fight for testing budget, or is it development, just testing part of the development pool of finances?

Yeah. I think it depends on the company's culture and the attitude they have towards testing. They need to understand that quality is everyone's responsibility. So we kind of need to bring the budget in. I think it's usually like an engineering budget and then we try to add testing resources also within that budget. So yes, we have to fight for it, but it's about how you convince them and how you make them feel like how quality is everyone's responsibility and we need to make money to do these things and make these things happen.

Perfect. Klaus agrees.

Sweet. Raul wants to know using what factors can you decide the feasibility of automation testing? Obviously everything can't be automated. So how do you even know or judge like something is feasible or makes sense to even automate, I guess is.

The question, how complex would it be for you to automate it? So we talk about like testability and something in our space. How about automatability? How easy is it for you to automate that thing? And again, going back to Sneha's points of what skill set do you have? There's the whole buy versus build again. Are we going to buy something to help us with automation? Are we going to build the automation ourselves?

So yeah, I mean you kind of have to evaluate those things because, and prioritize them. So then you know if something is going to be easy to automate or not. So there, there are going to be some tests that are super easy to automate. They're going to be some. Where you go, well, I probably could, but I don't know exactly how yet or this might be difficult. My environment is ready for that. Our configurations are hard to deal with. I probably wouldn't spend my time automating that. I'd probably spend my time fixing the environments or fixing the configuration first before thinking how hard will be to automate this.

Yeah, I can give a small example over here. So we do a lot of audio testing, video testing as well. So the way we do it right now is like, you know, we, we do it in a very elementary way. We try to like look at the name of the audio file and then the size of the file and then try to do a check. And you know, that's, that's better than nothing. But there are tools out there for video testing specifically. So do we want to buy those tools or do we want to just be okay with the level of testing that we have right now, you know, because it's complicated for us to build something to be able to test it the way the tool tests the videos. So, you know, that's where you think about the cost and the priority of that testing and things like that.

That's a great example. That could be a session on its own. I'm sure people will love it. Call us. Did you just, did you want to.

I was just, I was just going to say that's what I see the responsibility of. If you're like a VP of Quality or a QA architect or a Director of Quality or whatever. I see that as kind of your responsibility. As new things start coming to the market, you're evaluating them and saying, is it worth putting someone on this to dive deeper or do a spike on it? Like, I see that as one of the responsibilities of leadership is to evaluate those things that come out. And sometimes a tester, you know, will will see something on Twitter or whatnot and go, oh, this new thing came out and they might check it themselves and then tell their manager, but they may not have the time to do a deep dive. And so hopefully the leadership is aligned with their team to say, let's focus on this new tool. Let's focus this new strategy and spend some time to evaluate that.

Thank you Kalas's Snare for your Automation Awesomeness for links of everything of value we covered in this episode, head on over to testguild.com a414. And while you're there, make sure to click on the Try for Free Today link under the Exclusive Sponsor section to learn all about Sauce Lab's awesome products and services. So that's it for this episode of the Test Guild News Show. I'm Joe. My mission is to help you succeed with creating, end and full stack automation awesomeness. As always, test everything and keep the good. Cheers. Hey, thanks again for listening. If you're not already part of our awesome community of 27,000 of the smartest testers, DevOps and automation professionals in the world, we'd love to have you join the fam@testguild.com and if you're in the DevOps automation software testing space or you're a test tool provider and want to offer real world value that can improve improve the skills or solve a problem for the Guild community, I'd love to hear from you. Head ON over to testguild.info and let's make it happen.