DevOps Software Testing Pitfalls and How to Avoid Them with Jay Aigner

Get ready to discover some of the most actionable DevOps techniques and tooling, including performance and reliability for some of the world's smartest engineers. Hey, I'm Joe Colantonio, host of the DevOps Toolchain podcast and my goal is to help you create DevOps toolchain awesomeness. Hey. Today we'll be talking with Jay all about software testing pitfalls in DevOps that I know a lot of people probably encounter and how to avoid them. If you don't know, Jay has an extensive experience founding and running a Quality Assurance Agency. JDAQA.com over the past few years he's worked with multiple different companies, different verticals. So he knows his stuff. He knows how people are doing things. He has over 60 employees across 30 hubs and he has worked in multiple industries, verticals and roles, some including QA Engineering, Product Management, and Senior Executive of Tech in the fintech space. So as you can tell, he's a really well educated guy that knows his place around the software development lifecycle. You don't want to miss this episode. Check it out. Hey, if your app is slow, it could be worse than an error. It could be frustrating. And one thing I've Learned over my 25 years in the industry is that frustrated users don't last long. But since slow performance isn't sudden, it's hard for standard error monitoring tools to catch. That's why I think you should check out bugsnag, an all in one observability solution that has a way to automatically watch for these issues. Real user monitoring. It checks and reports real user performance data in real time so you can quickly identify lags. Plus you can get the context of where the lags are and how to fix them. Don't rely on frustrated user feedback. Find out for yourself. Go to bugsnag.com and and try it for free. No credit card required. Check it out, let me know what you think.

Hey Jay, welcome to the Guild.

Joe, so nice to be on buddy. How are you?

Great. Great to have you. I've been following you for a while on LinkedIn. We've been talking back and forth a little bit I guess. Before we get into it, Jay, I'm just curious to know, why start a quality assurance agency?

Why? Not.

A good question. I did QA for a long time and saw the need. Outsourced QA can be a really good thing for some companies and did a lot of freelancing and it just kind of came about that I had too much work and not enough J to go around and Started hiring people that I used to work with, hired my old boss, hired a bunch of people, kind of just grew it from there. So yeah, I think it's just filling a need that was there.

Nice. So when you work with different clients, different organizations, when they think of software testing, do they ever associate it with DevOps or are they usually two different things?

It's a great question. Depending on who you're talking to in that org, I would say most of the time they see it as separate. Being in the space, I know how close knit those two things are, especially when it gets to automation. So typically they think of them separately. Unless we're just talking about automation then. You know, I always tell people the gold at the end of the rainbow is having all your test automation looped into your DevOps pipeline. So everybody can kind of just hit a button and enjoy automation.

Love it. And I know you do a lot, not only functional automation, but you also do like security testing and performance testing. I would think those will lend themselves more to like the DevOps pipeline. Do you work with companies to integrate them into their pipelines? Maybe just like a security scan or a quick performance test? Like how does that work?

So typically, you know, we have a really strong partner for security testing that's kind of fallen out of our umbrella of things that we actually offer on a day to day basis. But for the performance testing stuff, yes, it can be looped into your cycle. The only problem with, with most performance and regular automated testing is it can get expensive quick if you're doing it in the cloud. So depending on the size of your platform, depending on your budget, depending on these different things, yes, you can loop in all sorts of performance tests to your deployments. It just depends on how important that is versus, you know, budget. Right. And especially typically on the initial build. Right. Like you really want to make sure that this thing you've built is ready for a hundred thousand users to log on. If you've proven that it's not as pertinent every single build to make sure that that can happen. Unless you've done obviously some infrastructure backend changes that could, you know, mess up that functionality.

Nice. So when you go into organization, who do you normally deal with? Like, what are the titles nowadays? It's been a while. So is there still QA people or is they part of a developer team or. Once again, I guess it depends.

Yes, there's still a QA team, you know, under a log somewhere. You can pick it up next to the worms and the bugs. You can find Some qa. Now there's, there's still QA teams. Nothing has changed as far as development team makeup. From our perspective, we typically will work with a cto, a newer CTO that's coming in and you know, has to answer a lot of questions about things that may not, may not have an answer for. Why are the bills taking so long to get out? Why are so many bugs getting production? Why is, you know, QA always the bottleneck? All these different questions. Chief information officers a lot these days, more than I think we'd seen previously. Vp, Director of engineering, kind of engineering leadership level. And then if we're doing more of a staff augmentation situation, we'll work with QA managers or directors of QA who kind of manage the resources that we bring in. So we can either provide kind of a full stack where you don't need to manage our resources, we come in and solve the problem for you, or we can provide you some really good engineers that can work with your team that are already managed by your team.

Nice. So what are some common pitfalls then you usually have to deal with when you walk into an organization, especially when trying to incorporate testing into your build and your CICD pipelines.

It's the most non technical one. Typically it's communication. Anybody who's been in an org of any size probably knows that communication can break down anywhere. But it's, it feels, maybe unbiased, but it feels worse when it's in qa because you're kind of the last line of defense against everything. So if there's poor communication between you and the developers, or you and the product team, or you and the stakeholders, or you and the customers, it will come out and it will, you know, probably cause a pretty big issue or even communication amongst the QA teams themselves, right? The members of the QA team having the wrong people in the wrong seats is something that a lot of see a lot of places that have somebody on staff and they just, they need to free up the rest of the QA team. They need to implement automation. They go, well, you know, Bob's worked here for 10 years, maybe he can handle it. And nothing against Bob, Bob's usually a great guy and he's very smart. But like it's not his expertise to pick the right tool, implement it, you know, implement it in a way that's componentized so that there's individual chunks of it along the way that you can independently execute. Because a lot of times people try to implement automation in DevOps or in QA that's kind of an all or nothing. And if you run out of budget, or you run out of time, or you run out of patience, you end up with nothing. Right. So we try to kind of promote this very modular way to build tests, chain them together to get to your end to end testing stuff done that you need to get done. But you can, at any point, you can stop and run those tests and get the value out of having those tests instead of kind of this big overarching thing. So having the wrong people in the wrong seats that don't really understand how automation should fit into a software development life cycle. And then just expectations, I think, you know, expectations of what can be delivered by when. I think some of the biggest pitfall we see is development team, product team, executive team telling QA how long something will take to test. You will almost assuredly never have a good outcome when that's the case, because, you know, QA people know how long something takes to test. And we're all guilty of giving a little bit of buffer to make sure we have some time to do stuff. But it's usually pretty accurate. More accurate than coming from somebody who has little hands on experience with what needs to be done. So communication people in the right seats and then setting the right expectations for timelines and delivery.

So I would think it's difficult stepping into different cultures and trying to promote that as a company. How do you get buy in then from maybe Bob who's disgruntled now? Like, who the heck's this guy telling me how to do this? Is that something you have to deal with or you just go in, here's what you do, see you later type deal.

That's a great question. Yeah, it's the agency conundrum. I think it's how do you have your own culture as an organization, which we like to have, but also how do you integrate smoothly with somebody else's? We certainly have those situations where Bob's unhappy. I feel bad we're picking on Bob, but.

You know, Bob's not happy and we have to be very diplomatic. But it goes back to the communication and personality kind of piece to it like we come in with. And we've gotten better over the years. There's certainly been times where we come in and we hit brick walls because we didn't set the expectations up front that whoever's hiring us and whatever leadership is writing the checks has to support this, you know, this effort together. Because if they don't, then why are we there? So there's Been times for sure. But we try to be diplomatic. We don't come in to take anybody's jobs. We don't come in to tell people they're wrong. We're just there to kind of lend our expertise and how to set up automation or how to manually test your platform in a way that your customers will come back again.

Love it. So, you know, I don't know why I'm fascinated by having like a QA agency. It's so hard to get buy in as a QA engineer and test engineer because you're always seen as a cost center when you work for them. So who contacts you and say, hey, these QA people are a cost center. Let's get an agency in here to do it. When I think that'd be a cost center. Well, like what prompts them? Does something usually happen poorly? Like something gets released and like, oh my gosh, we need to get quality in line here. How does that usually work?

I think it's multiple bad releases and a feeling from leadership especially that something has to change a lot of maybe bandwidth. We don't always come in and replace a team. We come in and augment teams all the time as well. So it's like we just landed a client that has a full QA team. They have automation, they have manual, but they needed help with tool selection and they needed help with implementation and they need help with maintenance and all these different things that they may not have experience in. So a lot of the time it's, you know, it's explaining to them how to better do the things that they're doing. And not in a confrontational way. But when people come to us, it's typically newer CTOs or you know, that have come into an org and like are starting to kind of get their hands around what's going on there. Or it's high growth kind of SaaS, companies that just got their Series A, Series B that like need people now that are really good and they don't feel like going through hiring. And also it's more cost effective. There's two reasons why it's more cost effective. One is if you're having your project managers and your developers and all these other people test the stuff for you and you don't have a good QA team in place, then you're spending 20 to 30% of their time, which is typically expensive to tests. Right. So why would you, I mean, if you look at the numbers, like it doesn't make sense to have a higher paid person doing something that you're going to pay somebody less to do and then we're not W2 resource and we're scalable. So, like, you're not bringing on 10 QA people because you have a big release coming up and you have to pay benefits and all these you bring us on. We can scale up when you have a big release, we can take some automation done, get all the stuff we need, and we kind of scale back down and kind of go into a maintenance mode where we just kind of help support whatever, you know, the next leg of development is going to be. So it's definitely a. We don't go for the bottom of the barrel. We're not a cost savings kind of led organization because of our expertise that we have. But it's certainly, you know, a secondary benefit that you can save a good amount of money if you don't want to spend the time finding people who know what browsers to test and what devices to use and how to build automation and do all these different things that a lot of times ctos don't really care about. They just know there's a problem and they don't want to build a QA team to fix it. So they come to us and we can help them out.

Nice. Makes sense. So I'm always curious, people sometimes hate the term best practices, but if you're dealing with all these different companies, are there any best practices or are there standard life cycles that everyone follows or do you have to have a group that does different processes for each organization? Does that make sense? Is there actual best practices? If people follow A, B, C and D, they be okay? Or is it just like, oh, this company needs A, this other company, we need to do it this way. How different are gig to gig or company to company?

It's a great question too. I think. I would say best theories, right, Than best practices or best, you know, it's more abstract sometimes than just like, you know, abcd. It's more like what are all the pieces doing together? How are they interacting? How are the requirements being handed off? But there is some, you know, best practices to it, right? I mean, you need, you need information to be able to test. You need to make sure you're testing in whatever cadence matches development. And you need to be able to support deployment, you know, pre and post deployment and feature testing. Right? So it's like kind of just there are standard pieces of every process. Everybody's, I'll say everybody's writing requirements, but not everybody's writing requirements. A lot of people write requirements, but Sometimes, you know, you need to lean on the different areas to say, look, your guys are struggling to test stuff because you're not handing them off the information they need to do it. The requirements are really thin or non existent or they're changing all the time or you know, there are a lot of different reasons why things could happen. But to your point, there are some kind of standard things like I said, like requirements, delivery cadence that you can kind of wrap into any process and it kind of. But yes, to your point, everything is different from the start which kind of makes it hard to productize it service, you know, from a business perspective. But you know, we've done well kind of supporting full stack leadership strategy processes, people and then kind of letting it rip.

Nice. So I know a lot of internal testers or people that have to do testing do a poor, a poor job of kind of highlighting what value they add. And I assume as a quality assurance agency that's something you have to execute on. So even if someone's not an agency or a contractor, is it things they could do to highlight what they've been working on to the higher ups like the ctos to show them, hey, we are adding value. We are doing this because I assume you have to do that constantly.

Yeah, I mean I think everything needs to have a paper trail. I think for us the biggest analog to this was providing daily updates from every person on the team regardless of what they did that day, big or small. And I think if you're doing that consistently, you know, it's hard for somebody to point a finger and go these guys aren't doing enough. If every single day all you're doing is doing what you're being asked to do, you execute and deliver and then you say here's, here's what it was that I did today. Now it's not, I mean if you, if you have management or engineering leadership over top of you, they should be looking at those updates to go. That's either the right stuff or the wrong stuff to be working on. That's not always your job as a QA person. You're doing kind of what's been handed to you. So I think over communicating and leaving a paper trail is probably the easiest way.

Nice. So I think another pitfall and nothing something else a lot of people struggle with. And I think this is like you said, you mentioned this a service you have is tool evaluation. So once again you work with different companies, different verticals, different everything. Is there a go to tool set or set of tools you use or is it once again depend on the organization, what language they're using? How do you evaluate tools, I guess is the first question.

So it has morphed over the years. I think at any given time we'll probably have a pretty standard suggestion list. We are partners with most tools out there. And as I say that, I realized last week that there was probably a hundred new tools in the past few months that I'm not partners with. So like, maybe I can't say that anymore, but some of the major tools companies out there, early on in the company I just reached out and you know, a lot before kind of they solidified their partnership program and we've embedded ourselves as pretty good partners a lot of these companies. So we, if we have a good relationship with, you know, a few of the main tools companies, we know what they're good at, what they're not good at. We can pretty quickly see, you know, what a company could need. And if we have a really good relationship, that typically means we can make sure there's really good support on both sides. Which is I think probably one of the under appreciated things when it comes to picking a tool for automation. Because you will have questions, you will have things that don't work, things will break and you will get stuck. So you need to be able to talk to whoever it is, whatever company you're working with, you need somebody to be able to reach out and say, hey, what do we do here? And sometimes we play that role, that liaison role for our clients where they're just like, guys like, this isn't working, what do we do? So yeah, it starts with kind of just a basic understanding of the platform, their tech stack, their processes and then marrying them up with, you know, one of our tools. And if it's not one of our partners, we would, it doesn't matter. We don't get kickbacks from our partners. We just partner with them because we work with them for so long. So it's not a partner of ours that ends up being a better tool. All for it. Like let's figure it out and go for it.

So I assume a lot of these tools of vendors, do you ever get into the like, oh, we only use open source tools, therefore no. Or do you, they just listen, okay. Or is it, do you usually work random prizes and they just go with a vendor based solution anyway?

You know, again, I think it's based on the time, right? Like 10 years ago, something different, five years ago something different. Now it's something different. If somebody comes to us and Says like, hey, can you write us a bunch of tests in selenium? I would go, why? Right? Like, why would we do that today? You know, I'm going to be honest with people. You could probably burn a ton of hours and you could make a nice, you know, contract with somebody if you just went and just wrote a bunch of selenium test cases. But why would you do that with the plethora of tools available that handle all of that spin up and tear down and all this stuff? Like, why would you do that? So it's really ease of use. It's what saves our, you know, what's most cost effective for our clients, what's easiest for us to use, a lot of different factors that come into it, but it's really a constant revolving door of what's the. Not just, you know, not every day, but it's. We try to keep up with what's the best stuff like, like virtualized browsers, for example, you know, five years ago that was not really a thing where people used it very much. And it was a lot of, you know, Ghost Inspector had the record and playback kind of model, which they still do. And I love Ghost Inspector and I know Justin, you know, over there very well. But you know, some of the more advanced features of something like Reflect and I know, you know, Todd and those guys over there.

Yep, great guys.

The virtualized browser is like to me, the wave of the future. Like you can't. Why would you go back in time to use something else? If you can use this tool that lets you instrument and come up with any browser configuration you want. Oh, by the way, you connect to real life device farms while you're doing that and test on real devices. So I would say it's a constant evolution. There's no consistent stack and whoever kind of comes up with the next thing that's going to help us or help our clients, you know, we're going to check it out.

Nice. And just people know Reflect was purchased by SmartBear, who's our sponsor and Jay, I don't think knew that. So good NJ for mentioning Reflect. Awesome.

I did not know that. That's fantastic.

Cool. So when you get to then you have the right tool in place, do you have to do a proof of concept? Is that something you provide as well? Maybe, you know, while this tool will work, but you don't really know until it's in their, their workflow. Is that something that you normally have to do as well?

Typically we'll do a POC as part of the Sow some slice of functionality. It's typically, you know, something easy, but something demonstrable so we can show off, like, yes, you can log in with it, yes, you can change password. And then maybe something that's a little less canned where it's like, you know, interacting with some part of their system to do something specific to their platform. People like to see that to kind of go, okay, this could, this can fit all our edge cases. It can, you know, do the things we need it to do. So, yeah, we typically will do a, a pretty light POC just because we're confident in the tool and we just want the client to be confident, too.

Nice. How do you know what to test? Is that up onto them to tell you what to test, or do you tell them, hey, we only focus on risk or this test is too, too long, it's going to be too slow, it doesn't make sense to automate. Is there a lot of back and forth that way as well?

Should be. If we're doing our job, there should be. You know, I would say it's rare that we just go in and automate things that they tell us to automate. Part of our process, typically with automation, is basically an assessment, right? So it's like, we'll give you a QA roadmap and we'll say, like, here's where you are today, here's where you want to get to. Here's the steps in between based off of the demos and stuff we've seen from you guys, here's how we should go about it. Now, as we're doing that, if we get into some functionality that's like overcomplicated to try to emulate via automation, then we'll say, look, guys, we just want you to know we would recommend you guys do this manually just because to spin this up and to get the test data where it needs to be and get the environment and the state it needs to be in and all these different things to make this one specific test work. With automation, you know, it may be much, much less of a headache to do it manually so that, you know, part of the initial process is just doing that now in our staff augmentation positions, where we just have people in there that are working with other people's teams, then, you know, they give the direction and our guys kind of go in and just do what they're asked to do. But if we're in that strategic role, we're certainly assessing which things should and shouldn't be tested.

Nice. So this might be A weird question, Jay. You have a quality assurance consultancy and when I talk to testers they're like, oh, our jobs are being replaced by developers, no one cares about quality anymore, and things like that. Clearly if you have a consultancy, that's not true. So are you seeing a demand for it? Is this a hard sell? I thought I saw somewhere where you're doing pretty well, you have a good ARR type of revenue coming in. So how does that work?

I need a better analogy than this, but I always tell people QA is like a funeral home. You're going to visit me eventually.

Right? Like it's, you have to have it. There's no world where it doesn't exist. Like I don't care how great AI is, I don't care how great your developers are. When you get to a certain size and a certain amount of deployments and you know, you need a dedicated person or people or team or in house or out, it doesn't matter what it is, you're going to need it. So even when venture funds, quote unquote, dried up over the last couple years and there's all the panic that set in with the economy and stuff, I felt like we're so far out that stuff's out in the fringes and we have so much green pasture in the middle, there's so many mid market companies, there's so many startups going into Series A, Series B, it hasn't really concerned me. So yeah, I could see people and I think as a current and former QA guy, I think QA people are a little paranoid by design and you know, that's why they're in QA and that's what makes them really good QA people, if they didn't care and they were very like relaxed and they thought everything was fine all the time, then there'd probably be a lot of bugs in production under their watch. But I would say for the most part, man, it's not going anywhere. The demand does not decrease, it's just going up the more software that's made. So I don't see much of a decrease in demand.

How about with the rise of AI, generative AI and GitHub, Copilot, all these things, do you see that reducing it or once again people probably be creating more code that then needs more testing. Where do you see that heading?

I mean, you know, I will stand on this hill until I get struck by AI lightning, but I don't, I'm not afraid of AI when it comes to testing. I'm not I don't think it's going to take anybody's jobs if there becomes a point where like it can actually understand context better, which I think is like one of the key things that QA people understand really well, maybe more so than anybody else in a company, is contextually understanding features of a platform and interacting with developers and product people and all these other like just the context you have to understand to make a software product and then the context of having customers and how they use the platform. Until there's some magical AI that has locked in context about what was made this week versus what was made last week and what changes all this different stuff. Not really worried about it. I think there'll be some cool tools. I've seen some cool demos of some really cool stuff with generative AI when it comes to testing. But I think it's just like every other advance in technology. There'll be people who use the new tools and there'll be people who lose jobs because of the new tools. But like, you know, if people didn't learn how to compute and learn make use computers when computers came out, they were kind of left, you know, on the wayside. So I kind of feel the same way about AI.

Love it. So speaking of AI and the future of testing and DevOps, I assume working being a founder of a company, you need to be a few steps ahead to see where the market is going, where it's heading. How do you stay up to speed on where like future trends, Are there any future trends you see that companies are going to be facing coming up that you're getting on top of anything like that?

I have heard enough murmuring about kind of cloud back to on prem stuff that it has piqued my interest a little bit because, and I mentioned the cost earlier of running just automated testing in the cloud can be expensive. So running your entire business can be expensive, you know, just exponentially expensive running in the cloud. So potentially a world where we're doing more on prem, you know, not physically on premise because we're fully remote, but just the living on servers that are owned by a company instead of the cloud may be something that's kind of coming up, but more, you know, I don't see any massive changes right now. I think the big AI gold rush has hit. You know, I think there'll be a bunch of ones that stand out over the next year or two as the models progress and stuff. But nothing has jumped up in my way that really makes me think there's massive change coming in with Software development.

All right, so you need to recommend tools. How do you know what's smoke and what's not? Because you must be getting up all the time. Hey, Jay, you need to check this solution out when you go to your next client. This is going to solve everything. How do you know what's the real deal?

I mean, we'll try anything, man, internally. You know, let's. I'm a tools nerd, man. I love tools. I love them so much. It's. And especially in the. And I know you love tools because I've seen some of your episodes where you get to talk about these things. Yeah, we try anything. I mean, we've got our stable of things that work and if it ain't broke, don't fix it. But if there's something that's promising and that could, you know, solve some problems that we're having, and we do try it and it works, then, you know, we'll give it a shot. But yeah, I mean, there's something new every day that somebody wants you to give a spin on. So we don't close anything out. But we're not. We're also not. We have so many clients on different tools already that we're not in a hurry to, like, swap anything else into our portfolio.

Great. Okay, Jay, before we go, is there one piece of actual advice you can give to someone to help them with their DevOps testing efforts? And what's the best way to find contact you or learn more about JDAQA.com or JDAQA.

I would say understand the full scope of the project before you start. A lot of those projects seem to get bogged down because of some piece in the middle that you didn't anticipate, you didn't think about. And don't chase the shiny tool or process. Like, there's a lot of things that have worked for a very long time. Stick to, you know, kind of the basics and, you know, develop. Put a nice DevOps pipeline in place with whatever your stack is and just try to understand the scope of what you're doing so you can piecemeal it out, you can have milestones to hit along the way, and you can deliver a kind of a full end to end roadmap instead of so, you know, just a broken piece of half of it. And if you want to reach out to me, LinkedIn, Jagner A I G N E R. And if you want to check us out, jdaqa. Com.

Remember, latency is the silent killer of your app. Don't rely on frustrated user feedback. You can know exactly what's happening and how to fix it with Bugsnag from SmartBear. See it for yourself. Go to Bugsnag.com and try for free. No credit card required. Check it out, let me know what you think.

And for links of everything of value we covered in this DevOps toolchain show, head on over to test guild.com p1 so that's it for this episode of the DevOps Toolchain Show. I'm Joe. My mission is to help you succeed with creating end to end full stack DevOps toolchain awesomeness. As always, test everything and keep the good. Cheers. Hey, thanks again for listening. If you're not already part of our awesome community of 27,000 of the smartest testers, DevOps and automation professionals in the world, we'd love to have you join. Join the fam@testguild.com and if you're in the DevOps automation software testing space or you're a test tool provider and want to offer real world value that can improve the skills or solve a problem for the Guild community, I'd love to hear from you. Head ON over to testguild.info and let's make it happen.