Managing Test Environments like a Navy Seal with Josh Thurman

Welcome to the Test Guild Automation Podcast where we all get together to learn more about automation and software testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Automation Podcast. Today we'll be talking with Josh Thurman all about managing lightweight ephemeral test environments, which are really a must have in today's modern software development. If you don't know, Josh is the co founder and open source lead and head of development relations at Ufizi. Josh has really honed his expertise leading high performance teams as a Navy SEAL before taking those lessons and applying them to software automation and testing. Ufizi is an open source company that provides on demand environments as a service to improve organizational test efficiency, which is even more, more important now than ever. It's a highly module, covers a broad spectrum of test environment use cases that we're going to cover in this episode and was designed to easily integrate with your CI CD workflows. So you don't want to miss this episode, check it out.

The Test Guild Automation Podcast is sponsored by the fantastic folks at Source Labs, the cloud based test platform that helps ensure you can develop with confidence at every step from code to deployment. For Every framework, browser, OS, mobile device and API. Get a free trial. Visit testguild.comSourceLabs and click on the exclusive sponsor section to try it for free today. Check it out.

Hey Josh, welcome to the Guild.

Hey Joe, thanks for having me.

Awesome to have you. So I'll be honest, I don't think I've ever spoken to a Navy SEAL before. So I have to, I can't just leave it out hanging out there. So. So what lessons did you learn as a Navy seal? How do you apply that to your day to day job now?

Yeah, so I was seal for about 12 years and I was a SEAL office and the main job as a SEAL officer is to be a leader. And leadership is a really important skill in every organization. And you can apply those principles and lessons learned just about anywhere. Right? And so the culture of the SEAL team is fantastic. It's about constant improvement. And so a good leader knows what to do. Right? And the way they know what to do is they're asking hard questions about the why and asking questions like hey, how can we do this better? And you know, if you really want to have a high performing organization, someone's got to be filling that role of always asking sort of the higher level questions of how are we taking our organization from point A to point B, improving our processes, improving how well, we perform it, whatever mission it is. Right. And you can apply that to any business company, team, whatever it might be. And so, you know, I'm fortunate getting to have a second career. So I got out as I was a SEAL troop commander. I did 12 years, 11 deployments, had an awesome time and made a lot of incredible friendships, a lot of great experiences, love serving our country and was able to. I always wanted to be in tech, but at first I wanted to do this SEAL thing. So I was able to make the transition and just jumped right into sort of DevOps and software automation. And so eventually took a meandering path, as you typically do in the startup world, to get to where we are today. And it's just a matter of helping teams improve. And it's like, hey, the area that we saw in the industry that needed a lot of improvement was basically everything right? At your laptop and left to production, right? So any team that gets started, you start coding on your laptop and then you have a sort of production environment initially. And then you need to add all kinds of sophisticated testing topology to advance your organization. And so the area we chose to attack, that we saw sort of a gap in the market was this sort of middle area of, okay, how do I get my buggy code, which starts out and then refine it through a seamless process to get it out to my users? How do we make that fast? How do we make that efficient? And the static persistent environment problem is one that is pretty pervasive across the software producing industry. And so on demand environments is sort of our first segue into this space.

Nice. So when you made that pivot, obviously you looked at a bunch of different places and you said you identified this as being a pain point. So why is this such a pain point then? Of all the areas you could have gone in then in tech or to start a company around, why, why this?

Yeah, So I think one, it was just one of the least addressed by the industry, right. So if you look at sort of the four hurdles I have to cross to, you know, producing software, first I have to write the code. I've got to integrate that code with my existing code base. I need to do some type of acceptance testing on it and then I got to ship it to my customers, right? So, you know, writing code tools, there's tons of them out there, right? CI, cd, testing frameworks, there's a lot of those out there to get the integration done. And then deployments, a lot of deployment tools. It's like, hey, what about that middle area? Of this process of iteration and making that faster. Frankly, we didn't see a lot of market solutions out there and what we did see is people building their own on demand environment solution. If you're going to go into a startup when you see the market creating their own solutions, that's a really good indicator that there's a need there. And so that's where we jumped into that space because we saw the need.

Absolutely. I've seen this myself. I've been in the industry for about 25 years. So it's always been a problem with environments, especially in testing. So for folks that may not be familiar with the terminology before we dive into it is what it's ephemeral environments.

Yeah, so ephemeral environments are, they're so. They're stood up, they have a purpose driven life cycle is what I call it. So they're instantiated in response to a testing need and then once that testing need has been met, then they go away. And so environment is the high level construct. Right. Within an environment you have all sorts of smaller things which it makes them hard to put together. Right. You've got your storage, your networking, your DNS record, your TLS certificate. All these elements comprise an environment. And so the higher level construct, the thing you need is a tester, is the environment. But it's comprised of a lot of smaller pieces. We're trying to elevate the level of abstraction. Right. It's the environment that I need. I really just want, if I'm a tester or on the development team, I want that endpoint where I can reach my application and do what I need to do against it and everything below that. I hope I can, it'll be automated for me or hopefully someone on the team is setting that up for me.

Absolutely. So, you know, when I was working in a large environment, a large enterprise company, a lot of types of developers would merge their code and then that's when they would find bugs. So do these type of environments allow you to stand up? And the reason why is because we had integrations with all these crazy things that it was really hard to set up because it was, you needed to know how to set up different applications and all that. So I would assume with an ephemeral environment you could create a template once and then just anyone could spin it up and have the exact environment rather than wait for 10 different team members to do that a little bit. Am I understanding that correctly?

Yeah, that's exactly right. And so we're heavily focused in the pre merge space. Right. So when you can shift QA to the left, your life just got 10 times easier. Because once you merge a bug, then you have a handful of decisions of how you're going to deal with that and none of them are really good or friendly, I suppose. So if you can, I call it parallelizing your feature development. So every developer on your team can effectively have a environment for each branch that feature can be iterated on and without any impact to what every other developer my team is doing. Right. So they have their own isolated environment for their specific feature and they work on it, they put it up, someone can come and test and review it, they can iterate on it, and there's nothing in between them, the tester and the developer, there's no space between them. Right. And they can work on that. And then when it's done, at least now, when you're merging, I know I've got a clean feature, it doesn't mean I'm not going to have any problems later. But I've eliminated a lot of the big ones by testing earlier.

Nice. And on your website you had something called the continuous previews process. Is that what this is?

Yeah. So you know the ability to consistently review code. Right. And for everyone on your team. Right. All the stakeholders. So you know, at the end of the day it's a team sport. Right. So you have developers who are doing their piece, but the more you can integrate your testers and frankly even higher up in the organization, getting stakeholders to buy off earlier in the process, you know, a whole nother set of problems. Is, is the feature that is being implemented like, as intended or is it even what the folks who designed it? Like, is it what they had in mind and maybe they even want to change their mind once they see it. Right. So you can ship a feature and you're like, well, gosh, I wish I could have seen this two weeks ago because I would have given you some feedback on it. Right. And like that's one of the higher level constructs is faster feedback in your organization and continuous previews enable you to do that. Right. There's no point in time when I can't spin up an environment and provide the current status of a feature.

Absolutely. I love, once again thinking of, as a Navy seal, I guess you can't succeed in a mission by yourself. You need your team with you. And so it sounds like this helps with collaboration, your whole team getting in on it with the continuous preview process.

Yeah, for sure. I mean, what I love about on a technical level, we're providing on demand environments, Right. But there's this sort of cascading level of benefits that come down from that. Like some of the higher order benefits are improved teamwork and collaboration. And so when you have an organization, you want it to be highly performant, then you want people to have buy in, right. You want them to give them ownership. And so at the end of the day they want to feel like they're part of something bigger than themselves. And that's just a human desire. Right. And so when you have these on demand environments, you have developers who can truly own a feature, right. And they work on it, they test it and they can test in their own environment and then bring QA in and work with them to make sure it gets pushed all the way through and say, hey, that was the feature I wrote and I worked with these other teammates to get it onto production. Right. And when you can, when your organization can operate like that, you know, everyone's happier. Right. And so that's, that's of course important. You don't want. And it's, you know, in a remote world, it's just harder and harder I think to, there's a lot of benefits to being remote, but one of the negatives is I don't get the sense of, it can feel like, hey, I'm just doing my piece and like I really don't know even know how it fits in the big picture. I pick up my ticket and I, I do this and then someone else takes it. And so with these on demand environments, you can really facilitate better teamwork as well.

Nice. So another thing I found that a lot of people struggle with though with these type of environments is cost. Like they keep them up and running and they find hard to manage. You did mention automation. So is that part of your process where the developer, it automatically will kick off an environment based on code, check ins, locally run it and then if it passes, it'll push it off to, and we merge it up for them.

Yeah, for sure. So you know, we're focused on providing sort of baseline tools to make, allow teams to have a sophisticated testing topology. Right. So everyone has unique challenges, but then there's also this common set of challenges. And like having on demand environments is sort of one of the key. It's like, hey, I need unit tests and integration tests and end to end and then I'm going to have like my front end automation tests and frameworks. Like the core foundation of that is going to be my environments. Right. And so having on demand environments of course is really important to that and so your specific question was about how long do they run? And so you gave me a perfect segue into like one of the big industry problems is shared environments. Okay. But persistence, right. So most teams have environments, maybe a QA and staging. They run 24, 7, like 365. Right. It's not that easy to spin them up and down. And you can do the quick math. If you're working five days a week, eight hours a day, that's only 20% of the time that the environment is actually running. So it's getting used fractionally in these on demand environments. There's really two key use cases. And I've got a customer in mind. I think they're going to come on the webinar later and they have a pretty sophisticated setup. So they will tag features and do basically like a risk analysis on it and say, hey, is this a potential breaking change or is this like something that's not as significant? Right. And so they'll label it. Hey, this is going to be dev test and automated testing. So the developer will test it. Right. Usually typically locally. And then when they open a pull request, that's going to spin up their on demand environment. And I'm talking about full stack in terms of all the services. Right. So we call it production like or production light because it's a lightweight environment. And so they can use, you know, we call it interface to database testing framework. Right. And so you can run your automated tests. And you know, I'm thinking of one customer of mine, they have like 900 tests that they run. But what they'll do on this is they'll just use the most important ones. So they'll run about, I think it's about 60 or 70 tests that they've identified as like, hey, these are the biggest errors we want to catch. They run for 15 minutes and then they shut the environment down. Right. And so that accounts for 75% of all their PRs. Get this. You know, I spin up an environment, I run my automated tests. That whole process, you know, between spinning it up and everything, we're talking like 20 minutes or so. And now that's ready to merge. And you know, they've de risked the merge by doing the automated testing. And then they reserve about 25% of the higher risk features where there's going to be manual testing on those. Right. It's like, hey, these are higher risk. We're going to take a good look at these. They are going to run their automated test as well, but we're going to Have a tester come in and do what they do best. Right. And so that's a really sophisticated setup. And it's, what we try to help organizations do is identify what are high risk, what are low risk merges and you know, the environments. Again, they might run for 15 to 30 minutes or if a tester's going to run it, it might be a handful of hours. And you know, there's a couple ways you can just close the PR and spin it up the next day if you want. Or we have, you know, a timeout. Basically it's like, hey, after so much period of time, we're going to go ahead and spin this thing down and you can, you know, spin it back up whenever you're ready to test again. So that's how we can combat against like they really just don't run persistently. And it's, that's a way to help people on the cost as well. Right.

I love the T idea, especially risk based. Once again, I've worked with companies that have thousands and thousands of tests, automated tests. Not everyone necessarily applies to a code check in. So if you do the risk based, it would only run the ones that really apply to that change. And so you have less time that the environment's up and running, which will save you time rather than just running all your tests just to run them.

Right. Typically there's some sort of trade off of like time and money. Right. And so you know what you're trading for speed, you may say, hey, well this is actually going to cost me more. But the reality is when you take away the persistent, or you reduced your reliance on persistent environments which are running all the time, the costs that may be equivalent, in some cases it's actually less than, because again, I'm running at just a fraction of the time that I was running my static environments.

Absolutely. So, you know, I was just thinking this. I don't know if this is a feature or not. Maybe it was just my company. The environment was so complicated that, you know, everyone knew their little piece of how to make the environment, but no one knew how to put it all together. And so if one person said, okay, I'm going to automate this, I'm going to make an environment for this template. They may not know all the pieces. Is there a way, is there a sniffer you can sniff exists in an environment? So when you create one of these environments, you know what to put in and what to mock what might be missing. Does that make sense?

Well, so we use basically a templated framework where all your, all your environments are being defined by infrastructure or configuration as code. And so we use Docker Compose and we chose it because one, it's something that many or most developers know or can learn pretty quickly without having to understand infrastructure and so on our platform, and even if you're running the open source, it relies on kubernetes, that's the core infrastructure. But to manage your environments, you don't need to know anything about Kubernetes. If you know Compose, you can manage your environments. And so you're effectively just following a template over and over again and swapping out which branches are getting tested against which sort of dependencies are maybe persistent. I shouldn't say persistent. The services that are maybe off the shelf, right. Maybe you have like Postgres and Redis or other things that are effectively. You're not working on those specifically, but they are part of your application ecosystem.

So in the beginning of the interview you mentioned something about a lot of people don't necessarily. They create their own environments, their own on demand environments. They'll do it all locally. The same thing with, when you run something on multiple mobile devices, sometimes for some reason people think they're going to have their own in house lab where they have all the devices that they need to take care of that is not part of what they're actually developing. So I guess would it maybe some things that are holding people maybe back from adopting on demand environments.

Yeah. So, you know, I think it's again, this goes back to sort of like leadership and culture and like the organization. Like, you know, for every, like one, you know, for every team that we help adopt on demand environments, there's probably five who really want to. And it's like, you know, it just seems to be like delayed and delayed and delayed. And I think there's going to be an initial organizational friction. It's. It's new, right? And so like, you know, experiencing all sorts of things. Like a lot of teams are under a lot of pressure, you know, hey, new feature, new feature, new feature. And you've got to have a role on the team who can hopefully look up and say, hey, we need to change our process here, right? We need a fundamental change to our process that will impact everything going forward. And it can be really difficult to get out from under the like onslaught of tickets, right? And just the pressure to produce this new thing. And so you need someone to recognize that and then also someone who's going to carry it through. Like frankly, there's, you know, it can get political within teams, right? It's like, you know, hey, is this new thing, what's wrong with the way we're doing it? And someone's gotta be like, hey, this is better. We're gonna try it out. And we think of it and others think of it as like, hey, we're investing in our team and our core process. And there's gonna, you know, a couple of weeks are gonna be a little bit painful. Cause not everything's gonna work just out of the box, you know, but in those week or two that we adopt these new changes and everyone gets used to the new process, the benefits are forever forward. Right. As long as my organization exists. Like, even if, you know, frankly, we're seeing people improve their efficiencies. Like the number of PRs that get merged in a period of time goes up 30 to 40% when they're doing on demand environments. And even if it were just 5 or 10%, that's forever. Right? Like the next 10 years I've improved my efficiency that much. And so I try to teach them, hey, take the time. It's like in the Navy there's always a saying of, hey, we never have time to do things right, but we always have time to do them over. You know, and it's so true, like in every organization I've been a part of. And so just taking the time to be like, hey, we're going to invest in process and convincing the leadership, hey, this is long term benefits that we're going to go for here. I might be delayed on a few features for a few weeks, but we're going to come out of this and we'll be able to move faster.

Absolutely. I guess as a Navy still used to going into a hostile environment, so I feel, sometimes testers feel like they're trying to make change and they're like, oh my gosh, how do I convince these nuts to make the change and why they're important.

Yeah, for sure. That's. That requires a skill in itself of just, hey, hey, folks, hey, take a look at this. Someone go set it up. Be like, hey, take a look at this. What do you guys think? Get buy in from the team and then be like, hey, we're going to try it out. You know, we don't have to do all the shift at once. But you know, it's most teams who are trying, you know, our service, they're up and running in a day now. It's not perfect. They're going to do things like, well, I'm going to seed my database That's a, you know, that's important. You know, I need to get the timing right, like, hey, when am I going to delete these environments? What are my triggers going to be? You know, and then it comes into, okay, like my ticketing system. What is the process now? You know, as I'm going from in development to where does code review fit in? Where's QA now? QA used to be post merge. Well now it's pre merge. But you know, if you can fight through those battles, you'll come out ahead. And so it's fun to see teams who adopt and are pretty happy with what they've done.

Nice. So I have had a few companies on the show that have a similar solution. So I'm just curious about Ufizi. Besides having an awesome name Italian, what's different than other solutions that may be out there?

I think primarily is all our core technology is open source and if there's another solution out there for on demand environments that open source, I don't know about it. I'm sure someone will hopefully maybe message me after and be like, oh, what about this? But you know, I think the biggest thing is we were like, hey, please stop building this on your own. Like, you just don't need to let other people innovate for you. Let the open source community innovate for you and be able to pick something off the shelf. Because previously teams have, a lot of teams have built their own solution. Right. And so the reality is you're taking a number of engineers offline from maybe the core business initiatives to do this sort of supporting function and they've got to build the solution, then they've got to maintain it. And I haven't talked to anyone who's really happy with their solution. It always is lacking and needs more. And so to keep making it better requires again, more time and effort as well. So frankly there's good closed source solutions out there and then we have an open source solution and you know, it goes, it's like there was a point in time where teams would build like their own ticketing system, right? And then it was like, okay, well Jira, now GitLab, GitHub issues like I'm sure there's some others. There's no way you can build a ticking system as good as what they're doing, right? You just can't do it. So, and we hope this takes off in the same way of like, hey, other people are doing this. Let's, you know, we don't need to reinvent the wheel Absolutely.

So how hard is it to get started with the open source? Sometimes I see companies that have an open source solution and to get started with the open source is like ridiculous to even set up. They make it so complicated that you have to go with their paid solution. So how hard is it to get started?

Yeah, we try to be friendly so and make it. It's not like, I mean of course there's going to be some friction because you've got to maintain it. But you know, so we publish a helm chart with our open source and of course we run through that and on our YouTube channel, me and one of our senior engineers, Adam, do a full like installation with a video. All the commands are there. So if you are familiar with kubernetes and understand helm, you know, you can get installed pretty quickly and then it's going to be a little more work to you know, get your compose set up and then integrate with your cicd. But it's, you know, I don't think it's particularly burdensome to do. I mean so, you know, on the platform there's no core technology really that's not on the platform. We're really helping solve organizational challenges as well as just hosting it for you as well. So you don't have to deal with managing the underlying infrastructure. So the platform has role based access control, SSO, password protected URLs, it's got integrated logging and then it's just generally a management. It's really helpful for like large teams. You can have any number of projects and within each project of course you can have, you know, as many, I would think of it as every feature, every pull request. You might have environments running and you can see all those there.

What I like about it so far just for this conversation is you mentioned testing a lot of. So it sounds like a solution that helps testers. A lot of times people don't necessarily focus on testers, it helps everyone on the team. But it sounds like testing seems like a core competency of this as well.

Yeah, for sure. Particularly on the manual testing side, like you know when you're at a startup you like, you wear like every hat. Right. So particularly early on I was doing a lot of the testing myself and the context switching of once you get in your environment, like there might be four or five features that are in there that need to be tested and you're trying to test this one thing but you just found another bug related to something else. And so it's like, well, am I going to, I needed to ticket it I'm going to have to do a little video and I needed to really document this well so it can be reproduced. And there's a lot of mental overhead that I just spent on this one thing. I didn't even get to the feature I was trying to get to or you play a game, like I'll go back and I'll remember that. And like, that's also exhausting. So I think it really helps to test your out by being like, hey, this is the feature and this is the environment and I shouldn't have to worry about a bunch of other changes that are made because it's isolated. Right. And so you mentioned also it helps the leadership too a lot to just be able to isolate problems and know where they came from. So you can be like, hey, cool, this came out of this developer was working on this feature branch. Just knowing that is half the battle. So in the seals, the targeting cycle is like find, fix, finish, exploit and analyze. And so in software, the find and the fix. Right. Find the bug is half the problem, then fixing the bug is the other half. And so this really helps a lot with the find and fix.

No, I love that once again, A lot of times we'd had seven sprint teams, they check in code, something was broken, but we couldn't tell what team's code broke the build. So it was really hard to roll back. It sounds like this, if you're doing this before you do that, Ultimate Merge would probably find a lot of those type of issues. For sure.

Yeah, for sure. It's a lot easier to point to the problem and not do the whole like root cause, root trace analysis.

Yes. Awesome. Okay, Josh, before we go, is there one piece of actionable advice you can give to someone to help them with their automation testing environment efforts? And what's the best way to find or contact you or learn more about Ufizi?

Yeah, for sure. So eufizi.com if you want to get started with the platform, we're up on GitHub UFizi Cloud EUFizi app. You can go install that with the helm chart and get up and rolling that way. And then so one thing I wanted to mention is so we integrate with any CI CD system or that's the design. Right. And so it's highly modular. Most people use it with a PR trigger, but you can use these as release environments, demo environments, other ways as well. But our GitHub action is really well defined. It's on the marketplace, you can grab if you're on GitHub Actions, you can grab that, add your compose and be up and running like really quickly. And so, you know, we didn't really want to be in the business. Well, I should back up. If you come to our platform and bring it compose, we do have a build service. So if that's where you are and what you want to do, like of course we're happy to facilitate that way. What most people frankly want is they already have CI CD and they don't want to rewrite. And a lot of them, they've put a lot of time and effort into it and so and of course they're running their tests in there, right? So we integrate with CICD, you know, grab our GitHub action. Just add it as a step in your pipeline already, right? You're already doing your build and pushing to your registries. And then one more step is basically our CLI gets wrapped in the runner. It'll run a single command. That single command spins up the environment based on Docker Composer. You get your dynamic URL, your TLS certificate that gets posted as a comment in the pull request. You can test again. If it gets updated, the comment will get updated and say, hey, there's been a new push here. And then once you close out the feature or you merge or close the pull request, we'll delete the environment and take all that down. So come to ufizi.com, hit up our GitHub and then also, or you could also just join it straight from the GitHub Actions. You can just set that up and you can actually create an account that way as well.

Thanks again for your automation awesomeness. The links to everything of value we covered in this episode. Head on over to testguild.com a412 and while you're there, make sure to click on the Try for free today link under the exclusive sponsor section to learn all about Sauce Lab's awesome products and services. And if the show has helped you in any way, why not rate it and review it in itunes? Reviews really help in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe. My mission is to help you succeed with creating end to end full Stack Automation awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Automation Podcast. Head on over to test guild.com for full show notes, amazing blog articles and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.