Future of QA Chat with Industry Experts with Joe Colantonio

Foreign.

Get ready to discover the most actionable end to end automation advice for some of the smartest testers on the planet. Hey, I'm Joe Calantonio, host of the Test Guild Automation Podcast and my goal is to help you succeed with creating automation awesomeness.

Hey, it's Joe and welcome to another episode of the Test Guild Automation Podcast. So last week I was at STA West, I just came back, I got to meet a lot of awesome people face to face. First time, actually meeting a lot of them face to face, which was awesome. And also meeting a lot of the awesome sponsors I've worked with at Test Guild. So thank you everyone for meeting me. I had a great time. And speaking of, last week I had the honor of actually hosting a roundtable at StyWest for karate labs. And this technology agnostic session covered everything. How to make automation more reliable, test culture, API testing, troubleshooting failures, and much more. And it was well attended and we had really great audience participation, as you'll tell by the audio. Hopefully you'll be able to hear the questions because the questions from the audience were not on mic, but people kept raising their hand. So we just tried to make it as informal as possible and we got a really great turnout. So in this episode I'm going to share with you what we talked about for the future of QA with testing experts Sneha, Prasat, Eric and Anthony. You don't want to miss this episode. Really great stuff.

Listen up.

The Test Guild Automation Podcast is sponsored by the fantastic folks at Source Labs, the cloud based test platform that helps ensure you can develop with confidence at every step from code to deployment for Every framework, browser, OS, mobile device and API. Get a free trial. Visit test guild.comSourceLabs and click on the exclusive sponsor section to try it for free today. Check it out.

Once again. Thanks everyone for joining us for the future of automation testing. Really excited about this event. I first learned about karate on my podcast. I don't know many of you know. I have a podcast, used to be called Test Talks, but it's now the Test Guild Automation Podcast. And I get to interview a bunch of awesome experts and one of them is Peter Thomas, who is the creator of Karate. And so that was in 2017 and then I saw him like work on this over and over again and was really excited to see that they recently got actually invited into the Life Combinator, which is a really cool startup kind of place that a lot of companies come out of Silicon Valley. So really excited to see that they've grown with over 400,000 monthly downloads. And this is not going to be a Karate DSL session, but they want us to make this as informal as possible. So it's just going to be more about like I'm going to ask a question and if you have any follow ups, let me know. I want to make it as interactive as possible. So hopefully that sounds good for you all. All right, so let's get into it. We have our awesome experts. We have Anthony, we have Prasant, Sneha and Eric. So I thought start off just a quick easy question, maybe go around and introduce yourself and start with dude, sure.

Yeah, Anthony. So I'm here in an individual capacity.

I'll give you a sales pitch.

I'm really here because I'm a passionate user of Karate ds. So Peter and Kapil have invited me to be here. I've hounded them over the years with questions around Karate. So I think just by virtue of doing that, here's the payoff. But I've been in some way, shape or form involved with quality assurance for a little over 20 years now.

Only recently as an automation engineer. And again I have Karate DSL to thank for that. So you know, I'll keep.

Do you have your own microphone now?

Well, that's okay.

Hey guys. My name is Prashant Patil. I work in Dell Technologies and from Last year almost 13 years architecting solutions for automation teams and DevOps team now and we started at least in my organization, started using Karate around three and a half years back and it has been a tool of choice for all the API testing, functional testing, performance testing and UI automation. So that's pretty much about me.

Hello everyone, I'm Sneha Vishwalingam. I'm director of core engineering at everfi. I started my career with test automation and then moved on to software operations. I've worked in different industries like computer networking, insurance, medical device and right now I'm in the E learning platform sector. And I am not a Karate Labs user. So I'm probably the only one over here who's not used Karate Labs because this is going to be a more agnostic kind of a talk. So I'm happy to be here.

Awesome.

Hi, I'm Eric Driggs. I'm a principal software engineer at Hulu now Disney streaming and I've been using Karate for about six years to solve backend payment, API verification, making sure that we ship software without breaking prod as much. At least don't break prod twice.

Absolutely. So a great point. They all work for great companies, but they're here as individuals, so anything they say don't hold them against that company is just a legal disclaimer there.

So very cool.

So this is called the future of testing. I know how many you saw a recent survey that says 90% of companies actually doing automation, which I was kind of surprised by. How many of you all do an automation is that one? It's about 90%. So another thing that came out of it was that not only is the 90%, but a large majority of them actually struggle with automation, which is kind of hard to get over. So I just want to ask the experts really quickly, like how can you do better with automation? So maybe some tips you can take away that maybe made that test more stable. So let's start at the far end this way and work our way back down to Eric.

Yeah, I think the most underestimated part of all testing is naming. Even just using BDD for your naming. For example, given a user with a free trial subscription, when they try to sign up within 12 months, then they should be re eligible. It's an example of a good name. And if we name our tests right, then we know when it fails, how to fix it. If you just have test API call, I don't know how to maintain that. And so the most difficult part is maintaining and the easiest way to maintain code is give it good names.

So when you think of atomic testing, a lot of times people will name something like patient registration and It'll be like 50 steps and when one fails, they don't know how it feels. So do you make your tests also more atomic?

Yeah, having smaller tests is definitely better. Don't try to do everything in one test. And if you have a clear definition of the operation of their test, if you have a when and a then that are short and to the point, then it's going to be really hard to have those test everything in one test because you can't name it that and not be a liar.

I would say having a strategy for your automation is probably very, very important because most of the time people just think they have to automate everything and end up just automating everything without having a plan on what exactly is needed to be automated. Because not every everything needs to be automated.

Great point.

Right?

She stole my point.

Well, I'm going to probably on similar lines. I think mindset is one thing, right. I think.

Exactly what she brought, like automate everything. But I think we have a lot of product knowledge testers, right, who has been Doing a lot of manual testing. And I think they have great knowledge. I think we need to empower them to see how they can use automation, leverage automation, basically.

Right?

Automation just putting all sequences in one script and that's it. Right. That's the one thing. And the second thing is that worked for us is definitely keeping the test self contained. That means that test is independent of another test. So that if there is an issue, we exactly know where to look for at least.

Right.

It's not like this depend on that one. And then there's a lot of chain of test like that.

So great point. How do you get that way? A lot of times people forget about running tests and parallelization. So they run a test runs fine and they run it, they spin it up and multiple containers and has dependencies all over the place. How do you handle that?

So that's the reason I said that. I think from data creation, from test data creation, everything needs to be self contained within that test. And I think that's one of the reasons also we chose karate because it has the ability to do a lot of things like common tasks can be done ahead of before every scenario. So that or before every test so we can use it effectively.

So I think.

Maybe combining some of the elements of what all of you said. I remember when my career was getting started, everything was manual. And there's obvious drawbacks to that. It's flaky, you could fat finger something, overlook something using your own eyeballs, blah blah, blah. It's time consuming. But I don't think we ever at a time we're trying to hire a QA engineer and thinking I need somebody with a strong development background. That's not what it was all about. It was somebody who could speak both the business and the system side languages, understand what the customer value is behind what it is that we're releasing to production and ask the right questions, especially early on. And again, I'm going back like 20 years here. It was tough to get a seat at the table as a QA engineer early on in a project cause you just weren't taken seriously. QA time was a crumple zone essentially before you get to production. And if the software developers needed more time to get something released, so be it. We gave four weeks to the QA guy, but he probably can get away with three. So it's more about pushing back, understanding that customer value and zooming in on that and not so much worrying about automating everything. I would say what the tools that are available today to an Automation engineer, what they allow you to do is sort of forget about that development aspect of your job and focus on what you do best, which is ensure there's quality in the products that you're releasing to production.

Go ahead.

Do you see a trend where a lot of test automation is outsourced overseas and that there really isn't a lot of domain expertise?

Yeah, absolutely. That just goes hand in hand with it. It's sort of thrown over the fence and assumed that they'll know exactly what they're looking at here, they'll find all the problems and if they don't, well then it's a blame game. Oh, that guy who I never met before overseas screwed up and that's just not a good way to go about releasing software.

Go ahead, follow to that.

When you're throwing it over the wall.

To an outside company and outside country as well, what type of oversight are you having internally to make sure that those tests are not only correct but also maintainable?

Yeah, I mean I'd say in practice oftentimes it's terrible. Even though you talk to some vice president in charge of the whole operation, they tell you it's running perfectly. But no, I mean it's absolutely a great point. Time zone differences, right. If somebody has a question and it's, it's a showstopper, I can't move forward without the answer to this. And the guide to answer it is a 12 hour different time zone. It's, it's a challenge. So I think ways that I, I've gone about tackling that is just asking people to every so often work an odd shift just to make sure that you're available and it's not necessarily going to be face to face with that time zone difference, but just getting on a teams call or slack or what have you and just being available at least for, you know, an hour and a half, two hours a day, overlap every now and then to make sure that things are everyone's up to speed, everyone knows what the requirements are, what they're working towards. That's usually, you know, I think when you run into issues it usually dials back to everyone knew exactly what the requirements were, but then you ask them in isolation and they give you a completely different picture of what they're painting. You have another question? Keep it going.

Sure, sure.

And I've been in this situation so I understand what I'm talking about. You know, at what point in time is the cost to oversee the outsourcing costing more than actually onboarding and having local brown Talent.

Yeah, I mean, I don't think.

If.

Anyone wants.

I can add to a point where I think one of the things that can help is actually having time dedicated to code reviews for someone maybe within the company who can take a look at it before really merging it in. But you're still taking that person's time away from their development by overseeing what somebody else is doing. Yes, but that's the nature of software. Right? Like we need someone to do code reviews and make sure that there was no oversight which could have happened even from someone within the company, not just the contractors.

I'd argue that the only model I've seen work well is manage software quality engineers, preferably full time or contract. To hire even contractors is difficult because most companies will have a policy where contractors are only there for a couple months. And so you onboard somebody for four months and then you're just throwing that money away. So it's best.

There's a 5 or 10x difference between.

The best and you just got enough.

So why not get your money?

You mentioned testing strategies earlier and so could you just. Do you have a couple examples of like high level strategies of what seems to be most effective or most cost effective? Instead of automating everything like in big government, I feel like it's encouraged. Like 10,000 tests are better than 100.

So risk, how does anyone handle that? Because that's a common theme. They have thousands of tests. Oh, we're fine, you know, 10,000 tests, but they're not adding any value.

So you can go ahead. All right. I think APMs are really undervalued as a tool for it. So let's say you've got datadog in your relics and APM and it's looking at production traffic, you're going to find that probably half of your endpoints aren't even being hit. And of that 80% of your traffic is going to 2 to 5 endpoints. And so you can use the combination of look at your non product, look at your product traffic over a period of about a week and you can use that in conjunction with your developers. So you need to have a test plan that's based upon the use cases that the developers know because they're the domain experts as well as you can just use the actual traffic and use that to prioritize because you have such a finite amount of time, you only want to have the tests that are going to add actual value.

Yeah, that and also I think a lot of code coverage tools that help you see if you have enough tests that's covering all of the code instead of just going on adding tests for code that's already been covered and also risk based test plans and assessment to make sure that you're covering all the things that actually needed to be tested instead of the low traffic ones.

I'll just add one point to all this.

Right.

I think a lot of time we miss communication and I think when you get tasks or a team of people to get tasked to automate something, generally it happens that they go in silos and they automate everything or whatever needs to be automated. Right. I think the strategy needs to be where we have a test design session. Just treat the test as a product development. I think if you follow all the principles of those, I think you will have a successful testing code and automation project. Everything.

Yeah, I think it's really easy to have. I think too many tests is often much worse than too little. And I think that gets back to Joe's point earlier of not throwing too much in the same test. If you're asserting every service then your test is wrong.

And also I would think it would kill the team's confidence in your test. If you have a lot of tests that are adding value the feeling and stop ignoring so less is better.

It's cry wolf really quick.

Oh, and I just wanted to add that, you know, testing 10,000 tests and getting those 10,000 results means that someone has to look at those results which cost a lot.

Yep.

Probably just bog you down.

It's really fun to do though, right?

Okay. To Eric's point earlier, software development works a lot better when the test automation is being done in conjunction with the development teams. When you've got an embedded engineer. One of the common problems I faced throughout my career is the problem that the automated test results are seen as unreliable and are not paid attention to consequently by the development team, what can be done to one, keep the test results reliable and two, to actually have developers eyes on those tests and have failing tests be taken seriously and result in code changes to fix them to.

Keep these tests providing value moving forward. I personally think. I don't know if you found this as well. They're trying to say run less tests. I think you should run less tests as well. I think there's a gap here between.

Run all your tests all the time.

In every build something fails, does it have anything to do with the check in? I don't know. So people don't pay attention to it. But if you had a way to map it that when code A is checked in that you need to run test Z and B. Then if those fail, I think you have more confidence in it. So I think it may be just a symptom of running too many tests without knowing what they're doing. I could be wrong.

I agree with you, Joe. Targeted testing, making sure that, you know you're not running the entire test before every single good definitely helps. We've tried that and that's, that's working well. And also we need to do classification, right. So you cannot just keep running 10,000 test cases per week. Right. So you have to have a smoke suite. You have the golden sanity, you have renation suite that will run every night, but then partial testing will run every single PR margin.

Right.

So if you don't do classification, then it's wasted of effort.

I would say only one thing is that let's make sure that is part of your CI CD pipeline. Because when the, when it fails, everybody gets attention.

To get to the heart of. Go ahead.

Sorry, I was gonna add, I think your first comment nailed a lot of this. When you start talking about clean and test case management, just actually conversation about this. Test case management isn't part of everything at the beginning. It's hard to be successful later. You actually have to be thinking about how you're, how you're naming your tests, how you're managing them, how you're proofing them. All these things you're talking about early on in the process before it becomes a problem. If it becomes a problem very manually.

Yeah.

And the problem that I think, John, I'm pretty. I know John, we work together. And so one of the problems that he works on solving is the reproducibility problem. So his solution for a lot of this is mocking. And that's not always something you have control over because sometimes the real complexity you have with services is even. And microservices are worse on this is that they're so tightly coupled. You've got dozens of services that all depend strongly on each other and you can't always mock them out. And a change in one can ripple across it. So it's actually the maintenance problem is the primary problem that you need to solve in automation. How do you solve the problem of maintaining the tests as they break and as the APIs change?

Well, I could do a lot better on that. That's one of the challenges we have, especially for some of our larger projects. It's difficult to keep them on green. And so what we've been doing is we'll do scheduled code reviews on it, we'll have bug bash times to go through and identify the problem tests and work on that. And so polling helps a little bit for eventual consistency, but you need to spend time of identifying the failing test. There's also not a lot of good tools out there for doing analysis on long term failed tests. We don't have tools standalone products to help that.

I think that's a gap. Tools like ReportPortal IO is great because it helps you triage. So when you come in you have a thousand tests but 700 all are due to one failure. Helps you kind of.

You're talking about portfolio. Yeah, but Report portal doesn't know what a branch is.

Well, there's other solutions out there too, but yeah, that's a good point.

Yeah.

If you don't know what an. Org repo in a branch is, then I can't use it because I have to. Anytime I have to. Anytime I have to click a button. It's useless to me for automation.

So you brought up Gun.

I do love the question. I'm curious what some of the issues you ran into with mocking or why?

Well, it's just a challenge because a lot of times you're dealing with stateful behavior in a database. And so like I can't spin up a database every time. Like it's hard. Like I've got to spin up a postgres database in order to run the service. It can be difficult. It's the stateful behavior of the service under test. And so because you're solving separate problems, one is there's build acceptance tests if I'm testing the service under test and then there's integration API tests which is can I ship this to prod without breaking something and build exceptions, tasks or mocks aren't always going to give you that confidence. I can have confidence that my service didn't change, but I don't know if the service I depend on is still working. And so API tests are going to give you the highest degree of confidence, but they're also the slowest and the most flaky next to UI testing.

Has anyone ever used something like test container that helps you spin that out for like different services to go with.

That particular you can, but you still need to know the full like database on it. And so there's a lot of overhead for doing that.

Great, great. So you did mention microservices. I know that's obviously a growing trend. I think it also may help you to minimize on a test or flaky test. Usually you have to do a ui. So anyone have experience with going from UI talking about API testing, I think you're looking at me pretty good stuff.

Well, I've been able to avoid UI testing for the vast majority of my career career aside from manually looking at a page occasionally. So I. I don't think I have too many horror stories in terms of UI testing other than just I've heard it's awful.

Yeah, I don't think it's a horror story, but I think it's just the approach.

Right?

I think so. I can say about one of the team that I work with is they have around thousand test cases total. Out of them, 600 probably are on API test and rest of them are UI test and the UI test. So as she was mentioning that we have the golden sanity, we have smoke. So we have those which should always pass, right? Those are there. Those are our golden test cases. So apart from those, when we run the regression and we do find there's a flakiness, it's not always about test itself. It's just about the network communication. There are a lot of other brittle factors that goes into UI and that's the reason we solely, not solely, but we mostly depend on API. If API testing is great, like all of them are passing, we have better confidence in UI because the amount of changes UI it depends on obviously how many changes UI has been doing. If UI has been not making so many changes, it's more of just a backend changes then you know you can. It's every situation based. But if the UI has a lot of changes, then yes, you'll have to probably regress those more and figure out how to solve that. Because one of the things that we started doing is.

Standardizing a lot of those UI tests so that we can take a page object model type of situation where for per page we have these element locators because those are the most of them, they fail quickly. Right? And we started trans standardization where every page has separate like a PO file, pitch object file with all the elements. So it's easy to make changes when there's a change.

But don't you think that if only Test Automation or QA team is approaching that it's not good enough, right? Because UI development have to follow their best practices. So if there is a page element or XPath or any object naming convention change or anything like that, if QA or Test Automation framework doesn't let like previously told or have a notification, then obviously your 60 test will fail and then Q will be will be the playing one. Right? But it has to be hand in hand that when UI developer developing their web application they have the best practice that if something change in this particular Sprint or polar, these are the list of things will change until I get notification early on. Then we work together and make it happen and test a less flick here.

Exactly. And that's the reason we have now added all the automation testers as part of the pull request when the UI developer is making changes, even though they may not understand everything, but at least they have an idea is element something is changing. At least they have an idea so that helps in understanding. Okay, hey, because ideally, yeah, developer should be telling, okay, I'm changing this element. Okay, communicate. Right? But okay, not every developer is a great developer. So it depends on that, right? I think we have Mrs. And that's how we found out find out if there are any misses.

One thing to add to that is one thing that we started doing. We started adding data attributes to the code so that we have this agreement with the developers that no matter what changes you want to do, don't change the data attributes because our tests are based on that. So that way there's a lot of, you know, we've kind of stopped going back and making changes to the test because of these small changes that we have that is not possible from development side. For example, if you are a growing application, right, you have 10 pages right now, in quarter or two it will be 25 pages. It's not always possible to keep your data attributed, then it's a growing pain. So you have to be flexible as well as follow the best practice. I agree. Because there are situations where we have to go back and kind of change how this component is working. Those are times where we have to kind of communicate, hey, these tests are going to fail because of these things. So that's when we have to communicate. But for the most part, at least when you're in maintenance mode, that works.

When it comes to UI testing.

It'S been my experience that UI testing is the best time to mock the backend. And I've seen it becomes. I've seen full suites that run in seconds because you're mocking it. And so is that your experience? Because I don't do a lot of AI testing, I do almost entirely API testing, but I've seen UI testing done really effectively with mocks. And so are there circumstances where you think it doesn't make sense to mock the backend for UI testing?

A lot of time it doesn't make sense to mock the data when you are solely thinking about UI perspective how our organization work is API is totally API testing. So end to end all API, how it's connecting with each and every page or interaction. So we are dealing with API differently and then ui. So in UI we see the behavior based use case and then if we need to create the mock the data, for example, I have a list of devices. So then we have to mock the data.

Right.

Or some specific attribute change while we are running the test cases. For example, I change the profile in that.

So why wouldn't it work? I'm still a little confused. I'm not a.

No, no, you don't need to mock up when you are using only the UI sort of test cases, but when it's hybrid like you have to get the data from the database or mock services.

So it's more important to like the.

Atomic test type of thing than mocking. I think with ui, because a lot of times I think I forget we.

Mentioned it, but a lot of test cases get rid of a lot of.

Steps because when you run them manually.

It makes sense because oh, I'm already here on this page, I've already done.

All this work because get there, I.

Might as well just check this one here. So there's a tendency to write test cases like that and then when they.

Get, you know, passed on for automation.

Those don't necessarily make good automation candidates.

If you have like lightweight tests that are atomic, we're only checking like very.

Specific things and you have them named correctly.

To your point, then you don't have.

As many issues with UI testing because I think Greg wants to jump in really quick. Sorry. So from a mocking standpoint, it helps you verify the UI works, but typically UI test is failing from the ui. It's not the part of having the data, it's something that underwire underneath and you need to actually explore. So if you mock everything out, you're actually pushing issues into production that you would never caught if you was mocking up the whole. So it really just depends where you want to mock that.

Yes, because my belief, and I would love for someone to correct me on my belief, is that if you have a core set of API tests that are testing your happy path, then you can mock nearly 100% of your UI.

You can definitely mock most of the UI, but I think one of the things you'll see is it could be an actual issue between like React and Redux.

Got it.

And stuff like that. And you can mock that out. But then it could also be that the API served the data really quickly, but then The UI actually had, you know, like react as a rendering issue and you're surfacing that a little bit from that standpoint.

Okay.

Couldn't you also simulate those things?

You could, yeah.

There's so many layers. When a UI test fails, there's so many layers underneath, it could cause that.

UI test to fail.

It's not maybe the data itself. It could be some other technique.

Yeah.

And so that's why they'll just typically retry on it. I get it. I fall. Thank you.

Sorry. So, comments you made earlier, what I'm hearing sounds a lot like the QA team is a reactionary. The developers made this change, messed up an element id, messed up an element attribute, and so the QA team has to fix it. Why is there not more effort being placed on working with developers to say, here's how the tests run. Can you please do X, Y and Z? You add the following data attributes yourself. Can you make an effort to read the tests and understand how they work? Can you take accountability when tests break?

Why are we not doing that more instead of more of this reactionary something QAT needs to fix it.

Right. So what I was saying was that was more of a past. The way we're doing is that whenever there are major feature changes in ui, right? So obviously developer is working in the branch, right? Where he's making code changes. So we are making sure that he has an ability to run the targeted UI automation test on his feature branch, because we can deploy that easily on some environment and we can run the UI automation against that and at least find out if there are. Because here's the thing, right? We can completely depend on developer to make sure that he's communicating exactly what changes he's making. And there are times where it doesn't happen, right? And that's where if a developer gets at least an.

Ability to execute the automation test, right, he has a complete control over that. Okay, hey, the test is not running. Let's work together and find out. Why is it not working? Because I made changes here. Let's figure it out. So those has been the changes, the learnings that we had where. And that's the reason I said right for the pull request whenever there is a ui big changes. I think the QA was very closely with the developer and we test the feature branch, we make sure that the changes that he's made are already taken care. We may not run the complete regression. Okay, that may not happen, but at least the targeted test, we will be doing that. One thing I wanted to just Add I forgot to add about the best practices.

Right.

One of the things that we started doing is that we also applied the CI for test automation code itself. So that means that when the test automation is basically versioned, test automation code repository itself is versioned itself is tested with the golden test cases making sure that the test that modified at least it did not break the golden test cases. Because that's one of the in a stability that we want to make and that's how we could make it.

The fact that most of the time microservices are translated into providers inviters mistake because we end up with.

But the biggest challenge that I talking about it is changing mindset instead of instead translating Titan to.

Shift that down and shift platforms try to build upon different small tests into it and separate that in a way that works and that's a big challenge. But the question I have is a little bit on when you are changing into automation and defining strategy and doing all of that, what are the measurements of success that you miss? Look at. So what Sometimes it's the return of investment because I say to my company, I'm going to invest in automation. Yes. How successful? How do you get measured?

It's a challenge because you know, if I hire a security guard and nobody ever breaks a window, there's the people are going to be inclined to say ah, we don't need this guy. Right.

If you're doing your job as a QA engineer, a lot of times you're looking well, we don't really need this guy.

Production is perfect.

But one thing that's helped me is having an answer when something does go wrong, like what the cost of that was. And I will admit I still, even though I ask questions, I still really have no idea. I could tell you how many users were impacted. But one thing that opened my eyes, I was having drinks one day after work with a friend on the business side of the house.

House.

He's like, yeah man, remember that incident last month? He's like, we lost like a small little customer. That guy was friends with like a big whale. And now they're ticked off at us too. And it was, it was. How do you put monetary value on that? But I think it's having answers, at least the start to an answer to those sorts of questions as ammunition to prove your worth, so to speak. You know, that's how you demonstrate the value. So I think, hey, if we're going to automate things like you don't want to speed to production is great, but really.

Take a have A conversation with your business partners and see what people are saying customers are saying about your product. If they think it's really good, and they used to think it was okay, then that's an indication that a lot of things are going right and probably automation is playing into that. And if you're experiencing a spike in production incidents, then whoopee doo. If you automated everything like you're providing more garbage to your customers than you were before, you should rethink what you're doing. And I mean, I don't have any great examples off the top of my head. Every company's different, right? Everyone's got a different experience with that. But I think it is staying connected to your business partners and seeing what your actual customers are saying and using that as a gauge as to whether or not you on the development side of the house are doing a good.

Job delivering quality products.

Snail, you want to say something?

Yes. So for me, like for example, I work in an elearning company and what matters to me in my product is trying to make sure that the product is learnable. Right. And so my priorities are different from maybe some other product that someone else is trying to test. So I think it's more approaching from what is it that matters to, to you? Like for me, we have a lot of interactive elements. We have media, we have audio, we have video, we have assessments and things like that that we need to test. And another thing that's very important to us as a product, as a QA team to test is how accessible the course is because we want every student to be able to use it and localization testing and multi browser testing. So it really depends. The strategy would depend on what is it that matters to you. And we started with literally no automation. It was all manual testing. So for me, my first phase was to just get some functional testing in, you know, and then the next stage was adding accessibility automation to it and just going stage by stage and then slowly defining goals and adding on to it and building. So that is my, you know, strategy that I would base it on. I think ROI is important if you, if you want to know immediate result, right. You have to figure out what your team needs and define the metrics, right. Like automation P value how many feature bug you find by your automation.

Right.

Otherwise there is no way, you know, that your automation is doing good or not. Right. So that could be one of the.

Metrics that, yeah, you can like write a dashboard with Jira tickets and measure P zeros and that's totally fine and great. What I think most developer teams will care most about is confidence that they can ship code without breaking things. And that confidence, it's hard to know about the bugs you haven't encountered yet. So it's really. I think that some bugs are going to slip through. That's going to happen. Not all test plans are going to be perfect, but the confidence that you want to be able to sell people on is that we're not going to make the same mistake twice. And if you don't have a process in place where when you have an escape and you find the root cause, you need to make sure that you're going to have a really high confidence that functionality is going to be there. And I would dare you to try to like, try refactoring a monolith like try refactoring a monolith payment system without API testing. Because I've been a part of that and I can tell you I can. I can't count the number of bugs that have been caught because of API testing. I literally can't count it. There's so many that have happened at every stage of development that you get with API testing and there have been so many more false positives in that process. But that's the price of confidence and.

The way.

Every organization has moved to Agile where we have sprints and so a lot of time we don't even open the bugs because we are working with the developer to write there to say hey, you know what, this is failing, let's fix it quickly. Right? So it sometimes is not even about that. The other piece that one started helping slowly is. And it's still difficult to do. It takes time is performance testing. I think the continuous performance testing is something we need to adopt so that we don't have to do like thousand million users from day one.

Right.

At least something so that. Because that's where the application break most of the time. There are two things. One of the functional which you figure out where the workflow is correct and not. But the other thing is how does it perform in production? Hopefully we don't find it in the production. We find it before.

Before that. Do you use performance testing as promotional gating criteria? So when you're writing your Gatling scenario or something, you were saying I'm expecting that the 50% and 95% is below this.

So we would like to.

It's not automated getting factor. Let's put that way. Yeah, it's a manual getting factor.

So because I've used like Gatling tests before, like when we're switching data centers and we're able to use that to like pinpoint specific API calls that were around tripping. For example, when we switch did the data center switch? But it's really hard to sort of like, you know, fail performance. It's hard to know like how slow is too slow.

Correct. And that's where you know, if you start performance testing from the beginning, at least we have some benchmark. Okay, hey, we were taking 10 seconds. Now we are actually at seven seconds. Much better. And the seven second became, you know, again, 11, 12 second. Something is going wrong. Yeah, let's figure it out. If we cannot figure out, we cannot like we still want to release. It's okay maybe to release because. But create the bug. Make sure there is an action item attached to it.

One of the things I want to build is I want to learn the Datadog API and be able to query services on the 95th percentile and stuff and send out automated alerts that this service has gotten slower. Fix it. That's what I think would be the cool proactive thing. That's a no code solution that you could actually solve on performance. Because I think in my experience performance testing is really easy to do for benchmarking, but it's really hard to know when you should fail it for somebody.

Using for quants test.

Curious. So we are using caching.

Yeah.

Because it's easy to, you know, because use functional test as a performance test. I don't have to write a separate test. It's just calling a functional test whatever workflow I want.

That's a trend I've been seeing more of leveraging functional test for performance and security. Does anyone else doing performance testing as security?

Yeah, and that's, that's one of the challenges we've had is like when I compare the APM data to the karate data, there's not, you know, it's not, we're not yet at the point of a big value add. And so I think there's opportunities to explore to add more value there.

Where, where are you guys doing performance testing? Do you have a standalone environment?

We have monitors on our agents for the web service. And so when you're using the API monitor, it's already giving you the statistics on your request response time for aggregated over a period of time because you're getting it from Datadog or whatever APM you're using. And so that's, you know, if you're using it for benchmark testing, it makes sense. But if you let like, let's say you have something that's performance sensitive, then you can really say if I'm over 100 milliseconds then this thing should fail. And that makes sense. But most things aren't in that category.

The back corner. Yeah. So I just had a question, right. So been around for a bit and Selenium was going to solve our problems with UI testing and then Cypress is going to solve our problems.

Is that much easier? Right?

Because now I feel like it's a little cyclical in the sense that we're kind of going backwards. It's not backwards like we're moving forward, getting it released faster. So I'm curious if anybody has actually made an active effort to start when those UI tests fail, to really start looking at the value of that test and say do we really need this? And is it covered with API? And start lowering our UI test coverage. And maybe just to your. The react example was perfect, right? Like does it look. Can I click. But do I really have to go through this ridiculous scenario through the ui? Is it acceptable both to go through the just to do everything API to do being sick Sanity. Right. They may make effort from last week I saw someone using GUI automation to hit the swagger page to call you.

Try it now.

So I would just add one point to that. Right. I think categorizing the UI test freely makes exactly what you're saying. Right? We have to continuously analyze and make sure that. Does it really make sense to run this every time or this should be part of big regression. Should this be part of small regression? I think those and I think one thing that karate helps is we can add as many tags we want so that we can make a part of that group. So that easy to say, okay, hey, this time I want to run all small regression or mini regression.

Yes.

I should probably take care of myself.

Actually one of these, these UI automation vendors. But my question is more in line of. Prior to that I was a business architect and this conversation in this room and saying, ah, let's skip the whole UI business. You know, it's only the interview. Let's get the APIs in there. Everything's going to work out great because all the building checks out and from a from an end acceptance perspective, the number of times we've been in conversations where you know, around the table. Yeah, it works here. The only problem is it doesn't ask the test. You know, the application ends up not working and it comes a conversation. Yeah, it's also the low performance thing with the thing over in the back. Super unproductive so to me it's surprising to hear this disqualification of UI testing because that's, you know, that's where all things meet up. That's where it's supposed to work. I get that it's difficult to do to point of it being brittle is valid, at least to get the state is difficult.

But to me there's no other way.

How do you want to address the end user experience.

At the same time.

Discontinuing whole UI testing competition?

Purify that.

What do you think? I don't think I'm discontinuing. What I'm saying is that I think a lot of the confidence can come from API testing if there are not that many UI changes. When there are massive UI changes.

Yes.

You will have to depend on UI testing. Right. It's just that how many times that whole page is going to change? Very few times. Right. You're going to change probably one component. Right. Now nowadays a lot of applications are single page applications, so only one component is changing. So it's easier to target basically only those UI tests which you know that you have made changes. I think that making sure that we do targeted test makes a life easier. If you start doing everything every time, it's going to be painful.

Absolutely.

So what percentage like in your experience? And I'll come out and say I'm a backend engineer, so my back end is API.

You know, I've got a golden hammer, so that's fine. So how would you divide it? You know, because in my experience there's a lot of unit tests, there's a lot of API tests. What percentage of tests do you think need to be UI tests?

Tests?

Well, I think that the conversation is.

Really, I mean the point made that. Right. It's so brittle, it breaks all the time.

Yeah. So.

And that's a fair point. We can't get past that.

Yeah.

It turns directly.

Let's focus that in fairness you can make the same argument on API tests or brittle too. So like there's just, it's just a continuum on it. But like what's your experience when you're working with companies? What percentage of tests do you think they should be doing at the UI level? What percent at the API level?

Just ballpark. Yeah. Some of the clients that we're working.

With did, excluding towards 60, 40 on.

The UI side of the.

Because they are, you know, all the.

Changes that they do underneath from a.

Business processing perspective, which is at the end of the day, what they want to validate, they can safeguard that using the gui.

And once they've got it going, it's.

Fast track to it.

It's not eliminating API testing.

Yeah. It's kind of switching it around, saying, I'm not doing changes on the Google side, so let's just stick to API. It's kind of the other way around. But even the question is the balance between risk and value.

Right.

What is my internal tool? Why do I have UI testing? Obviously.

If there's low risk, if there.

Is a big UI break.

Right. So I determine that's our main usage, all of our customers are using it. Obviously that's. I just think it's just been that whole thing.

Yeah.

Because we have these UI tests or these tools, let's just automate everything.

Right. Let's just do all the user scenarios over there.

There's so many times you have to go through 20 steps to get to a page. You're like, I'm already here, I'm just going to check all that stuff anyway. Which has nothing to do with the. That's a whole different strategy. That's a whole different problem.

Right.

But I think it really, I think the UI testing really depends on your application. So like, like say hi, we, of course it's going to be very ui, it's going to be heavy ui. Right. So it really depends on that standpoint. Right. So everybody's scenario is going to be a little different. And so when you're talking about virtual test, like there's a symptom there and some of it could be you're running too many tests and you're just tearing out the whole infrastructure. So maybe you need to get the comm scheme involved to try to figure what's going on there, add more pods.

And stuff like that.

So there's a lot of chime in here. So I've done both UI tests and API tests. It is completely accepted to run API tests in isolation from the ui. But how many times do we run UI tests in isolation of the backend? And I think the issue here is that the user interface is its own product and deserves its own isolated testing. And people are generally not making the effort to test the UI in isolation from the backend because it's really, really hard to do that. But if we do that properly, if we test the UI in isolation from the backend, we can test each piece of the UI on its own and know that each piece of the UI works. Then these massive end to end tests where we're covering all aspects of the ui plus all aspects of the API are Largely irrelevant. We can get rid of a lot of them and just have some general testing where, yes, the application loads. The application is talking to the UI or to the API correctly and we're getting the data back that we need. And we don't need to test the whole UI with the whole backend. That becomes trivial because we know that the API works on its own. We know that the UI works on its own.

The UI tests are still necessary, though, because the UI is an integral part of the application, arguably the most important part of the application, because it is what the customer sees.

And it deserves its own testing. Does it use visual validation to help with that? It sounds like a lot of these tests could be. I know, Greg, you know, did you do video on any kind of visual validation? Does that help a lot of these issues?

Definitely helps. We use some tool. Can I mention we use other tools for visual testing, and especially with courses, it helps a lot because, you know, we want to take a snapshot of it to see if anything changed. So we've definitely seen a lot of advantages of using visual testing. It's definitely an integral part of our testing clinical.

You can run a lot of API tests and if you're worried about the visual aspect of it, you can run.

You could just. Yeah.

Sorry.

I think one of the aspects that.

We are missing and probably we're treating automation like a silver tool.

Right?

And sometimes it's not. We need also to take a step.

Back and look at the strategy, right. And go to the testing principles, like having component testing, integration testing, and then apply the strategy of what we want to do. Right. Probably we are developing a component which is business and API. Right.

Well, I need to test the API.

And in my integration, I test a small suite of progression with the ui.

We probably should start wrapping up. I just wanted to mention I watched one of your prior shows on your channel and Montebeck was one of those solutions to testing UI without the B. It's been a lifesaver. Really awesome. Good to know. Cool. All right, so speaking of tools, I guess one last thing. This is the future of testing. Anyone have any new cool tools and techniques you recommend? Everyone maybe check out when they leave here. Let's.

All right, I'll pitch myself. I'm working on an endpoint coverage for Karate using the logs and swagger on it. So I want to do more with Swagger on it. So Swagger, gen and Karate is one of the things I'll be working with Karate labs to make happen. I don't know when, but soon.

New tools. So I use test modeler for model based testing. I would say it's definitely changed the way we test things with the courses. I would say check it out.

I would just say just check it out. Karate Labs first, right? If you haven't, if you haven't at all.

If you have.

I think one of the things that I'm looking into it more is distributed performance testing because that's something is important also.

So.

Can I give like a non answer to that?

Obviously I'm a big fan of Karate Labs, but I would say more important than what tools you select, make sure that you know as a quality engineer, you're really focused on customer value, technical risk, test complexity and knowing who your customer is. And you know, we got into a discussion earlier, what percentage would you think UI testing versus there's no answer to that. Right. It depends on like some companies it might be none at all and others it's if you have a spelling error on your homepage, you're going to lose millions of dollars because who's going to give you money? You can't even spell your own type right. So yes, I think it's whatever tool you want to use. There's plenty out there that are really, really good at what they do. But make sure that you understand what you need before you start shopping around.

For tools, so to speak.

Great advice. We'll leave up on that. I want to thank Karate for sponsoring us. If you want to learn more about karate, we have a booth at booth 14. I'm actually gonna have a champagne party from 5:30 to 6:30. All invited.

Thank you, Sneha Prasant, Eric Anthony and the audience at StarWest for awesome audience participation. All your great questions, the links of everything of value we covered in this episode, head on over to test guild.com a4 19 and while you're there, make sure to click on the try for free today link under the exclusive sponsors section to learn all about Sauce Labs awesome products and services. And if the show has helped you in any way, why not rate it and review it in itunes? Reviews really help in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Automation podcast. I'm Joe and my mission is to help you succeed with creating end to end full stack automation awesomeness. As always, test everything and keep the good. Cheers. Hey, thanks again for listening. If you're not already part of our awesome community of 27,000 of the smartest testers, DevOps and automation professionals in the world. We'd love to have you join the fam@testguild.com and if you're in the DevOps automation software testing space or you're a test tool provider and want to offer real world value that can improve the skills or solve a problem for the Guild community, I'd love to hear from you. Head ON over to testguild.info and let's make it happen.

Sam.