Automating your Pipeline Using Jenkins and Go with Greg Paskal

Welcome to the Test Guild Automation Podcast where we all get together to learn more about automation and software testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Automation Podcast. Today we'll be talking with Greg Pascal all about automating your pipelines using tools like Jenkins and Go Greg. If you don't know, he's been on the Show I think 10 times at least, is a natural innovator, pioneering new approaches across quality assurance and test automation. He enjoys mentoring others to excel as a craftsperson of manual and automated test. He's the author of Test Automation in the Real World. If you haven't picked up that book, you definitely want to check it out. I have a link to it in the show notes. While he's written for countless other publications and spoken at multiple different conferences like the Automation Guild, Star East, StarWest, he is also the creator of Mets, which is the Minimal Essential testing strategy. And Greg's approach actually has been recognized in Taught by ASTQB as an effective manual testing strategy. Really cool technique. Once again, you definitely should check out. I think I have an episode on that. I'll have it in the show Notes there. Also. So really excited to have Greg back on the show to catch up on all the things he's been working on. You don't want to miss this episode. Check it out.

The Test Guild Automation Podcast is sponsored by the fantastic folks at Sauce Labs. Their cloud based test platform helps ensure your favorite mobile apps and websites work flawlessly on every browser, operating system and device. Get a free trial, visit testguild.com sourcelabs and click on the exclusive sponsor section to try it for free for 14 days. Check it out.

Hey Greg, welcome back to the Guild.

Hey Joe, it's so good to be back with you and the rest of the listeners. I love coming and visiting your show and getting to share some new stuff today.

All right, Greg, before we get into it, is anything I missed in your bio that you want the Guild to know more about?

Joe, you covered it pretty well. I feel like there's always something new I'm learning or teaching on or writing on. I've got a couple new articles in the can I'm working on right now that are probably going to come out in the next few months.

Very cool. So any. Can you give us a little teaser maybe on what they're going to be around? High Level?

Yeah, I've got one that it's going to be called Sailboats and A Journey Into Testing and it's about the idea that as we take a journey into testing things, we can approach it from the sailboat perspective of kind of having our eye on the destination and cruising across the top of the water. We can approach it from the asphalt of a submarine where we. We go down into the. The depths of the application and really study and understand and verify and validate as we're down there. So it's pretty exciting. And I've got a second one I'm working on. It's called the Testing Grand Illusion. And the premise of this article is going to be around the idea of that from another untrained observer. Testing that reduces risk. And testing that's just kind of maybe fun and is not reducing a lot of risk can look exactly the same from a distance and why that matters and that we come, you know, consider those things as we're bringing on test engineers and as we're overseeing them and as we are test engineering ourself.

Absolutely. And I love the concept of a journey into testing. I believe both you and I have been on pretty much a similar path. We both started with vendor tools, get involved with open source. We had a kind of not relearn, but kind of, you know, get acclimated into a new type of testing approach. And then I think we both got sucked into, not sucked into DevOps and those type of activities, which I think you mentioned to me recently. You started doing things with Jenkins with automating your pipeline and go. And I think this is a trend that people are going to see more of in 2021 and beyond. I think people need to get their heads around. So at a high level, could you just talk a little bit about, like what you meant about automating pipelines at your current position? Maybe.

Yeah, well. And maybe I need to redefine that or re. Explain that.

Yeah, we.

What we wanted to start to consider doing is when our product went into the build process, that we could go ahead and kick off our existing automation and get that into the hands of the test engineer as soon as possible. Now, we had learned a really valuable lesson in the previous year trying to, let's just call it automated monitoring. Joe, you and I know we have at times been sent off on what I call an automation fire drill. It usually starts like this. Something bad happens in your company and the next conversation is walking over to the automation team and saying, we need to automate this so it never happens again. Right. Been there and done that. I'm sure most of the listeners have had that too. So that provided Some great foundational training that led kind of to this next generation of what we're doing. I say that because if you think about automated monitoring during especially in production, now we're getting into an area that a lot of us have been told don't do. Don't put test automation in production. I'm of the same, at least background. I've heard that many times we needed to experiment with that. And what we found out is when the mechanics of a process like that is so complex, it sounds great on a whiteboard and it sounds great during that fire drill. But when you start to get to the reality of it, at some point the machine gets so complex that it doesn't become reliable. And I'm not other than maybe a fire alarm going off often, it didn't bring a lot of wind. And we actually back back that process out and we found there was some much better ways to do it. Joe I was able to reach out to a lot of my professional contacts and I found out that folks were using other tools to solve those things in a much more efficient way. So all I have to say is when we started to approach kind of this phase two of how do we get automated results into the hands of test engineers early, after new builds? What we didn't want to do is repeat that lesson. So we said let's don't put our automated testing in the critical path of a build going out. And I think that's a good takeaway. I want the listeners to hear. But instead let's as soon as possible at least kick off the test so that there's the data from our automated results is available for analysis. Because that's really where the great win is from. My perspective in automation isn't that we can reach our destination, but what did we learn along that journey of the test that we executed and what did we provide to the manual test engineer that gave them new insights that they can identify? Hey, we regressed in this area or certainly the test passed, but it took 30 seconds longer. That's telling us we might have a soft fail that's in the works. So I hope that kind of sets the mood of where we're headed.

Yeah. So is this activity separate from your developers, like when your developers checking code? Were your automated tests getting run and because they took so long to run that they were slowing down the code being released to production?

No, that's a great point. And I can see a lot of folks probably have encountered that. I've had conversations with devs around that trying to Solve that problem. We actually use go to solve some of those pipeline build stuff. We've got a great DevOps team that's doing an exceptional job in that. I'll tell you how Jenkins came into this picture because it is too a build tool. As, as most of the listeners know, we initially had to solve another problem we were encountering or another challenge in our automated testing. And that is we wanted to find a way to get test certain tests. We, we have different types of automation in house and we needed a certain type of that automation to run early in the morning. So when our engineers came in, they had some insights into the health of their application first thing in the morning. And as can imagine, we needed a way to execute this test on some sort of timed or you know, scheduled approach. And so for quite a few, I'd say a couple years at least, we were leveraging Windows Task Scheduler to do that in our VM environments. And talk about an awkward tool to work with. That is not something you want to play around in all day. It can get the job done, but it can get very complex. So we had been using a tool like that to kick off what we call our automated sanity check, or maybe some folks would call that a smoke check in the morning. And I can go into that to another day. But what we found was when we were working with Windows Task Scheduler, this was a very painful process and we needed to come up with a better process. So one of our exceptional partners we have here at Ramsey is we work with our IT team gentleman that has been a great help to me. Another Greg, by the way, made the suggestion that we consider Jenkins. It had a UI to it, it's a web based tool, runs in a Java framework and it has all kinds of scheduling capabilities outside of just doing build process. So much like a lot of the work we've done in automation, we've had to find other tools that solved other kinds of problems that we were able to leverage a particular aspect of that tool, which was its scheduling capabilities to begin to execute these morning sanity tests. So that's how Jenkins came into the picture for us.

All right, so maybe I'm way off base here. You remember alm, right?

Oh, of course, yes.

Right. So you know, used to be able to put in your automated. This is why I liked about it, you be able to integrate your automated test there. Someone could log into a dashboard, they can click on a machine, point to a computer and run it on that computer and see the results. Almost sounds like that's what you're doing with Jenkins. Am I wrong?

That's not far from it, Joe. And I think it's come from my background with, with tools like QTP and BPT and ALM was understanding some of those capabilities, finding ways to solve those things. So you're right in that when we did it through alm, we kicked the test off through that tool. In the case of Jenkins, that's actually very possible as well. For those folks looking for an alternative to using something like Quality center or alm, it provides a means of. It's not its intended use, but you can perform that. But one of the things that Jenkins does really well is scheduling things that need to run at certain times. And so we leveraged that capability and started to execute these tests on a regular scheduled basis so that again, a test engineer would come in at 7am and there was our initial run of results for that day. You know, Joe, I'm a believer that you run your automation multiple times in a day, whether there's a new build or not. It those results, when you look at them over time in reporting tools. As we've spoken on our previous podcast about our work with Elastic gives you insights to the health of the application.

Absolutely. So what we used to do is I worked for a healthcare company and at the end of development, we had to go through like a verification process for the fda. So it's really regulated. So what we did until we got there, rather than waiting last minute, is that we had Jenkins and so the developer would check in code the unit tests and everything would run automatically. We'd run some smoke tests, but then we'd kick off a job in Jenkins that goes to a different environment that runs against that new build that was created by the developers that were run on automated tests overnight. So when we came in, we saw the results and people can then start triaging issues to see if it had to do anything with the new check in the developers used or checked in. So is that the same type of process we're talking about here? Like, is there a disconnect between your developers and your testers or is this specifically just for testers?

Yeah, you're right. No, the results at the moment are just for our test engineers. We have test engineers dedicated to each of our products and so they can get it. Now we do have many devs and project people who are interested in those results and get them as well, but the initial target audience is is our test engineer as part of our team. So the way you explained what you guys were doing is is pretty much the same way we're doing that. Although we're using Go for our primary build process, we're not using Jenkins as a company. So we actually. We use Jenkins just for the execution of those automated scripts on that environment that actually Jenkins is running on. That's our. That's kind of our 1.0 of this. But we realize there's even capabilities of expanding that out.

All right. So very. Not very familiar with Go. I always thought it was a programming language. So, like, how does that play into it then? Using as glue or something, or some sort of interface between Jenkins and your test?

That's a great question. So there's something called the Go pipeline process, and this is definitely going to add on my area of expertise, but we use Go to do our build process. It's a competitor, I believe, to Jenkins in the way it's sold. And doing automated build processes, the way we couple those two things is within the Go pipeline process, there is a configuration file, a YAML file that we store a URL in, and that URL just so happens if you go to, it will execute our automated script that is being overseen by that Jenkins system. And so it's a very easy way for something external to the Jenkins box in the automation execution environment to go ahead and kick a test off. So that's kind of our current approach right now as we're evolving it. At some point, we may even implement an API or other things to do it. But this turned out to be so simple and it was built right into Jenkins, that we can do that and pass parameters in that run the test differently. It's actually been exceptional and extremely reliable. I think I've told you, we're running about 65,000 automated tests a week now, and almost a half a million automated audits on top of that. And the process is just going exceptional.

All right. So once again, I just want to bring it back to. I understand. And see if it's the same. So we used to use something called Apache Maven, because we use Java to do this type of build. So it managed the project build, the reporting, the documentation, all that information. You put it in a POM file, and Jenkins would read that POM file through a command line, and it knew what to run, what test to run, and were reporting to create and all that type of information that sounds like Go, not the language, but Go Build, is that what you called it? Is taking the place of what we used called Apache Maven?

Yeah, I think so. It sounds like that.

Nice. So now you Said, you're running all these tests. What's the process then when people come in to view the results? Because usually the triaging piece usually takes the most amount of time. And also, I know you were playing around with elasticsearch a few years ago, creating data lakes and all that. Does this come into play at all with this type of reporting that's created after all these tests are run in the morning when you come in?

It does, Joe. It does very much. And I think again, for those listeners that are in this space of being leading an automation effort, you want to take the time to build great building blocks of your automated strategy. Right. So we have two reporting methodologies in place. Part of this just started because we initially noticed that without reporting, if you don't have anything that's outputted at the end of a test run, you're putting all of that responsibility in the hands of the test engineer to likely interpret results at the command line. And, and you know, when you're first starting out, that might be okay, but you want to move past that at some point. It's not okay that you're consuming all that time for the test engineer to have to read through that. At least it's not for me because it's a. It's a very easy lift to get some reporting. So our first reporting efforts were all done through email. We were leveraging some of RSpec's capabilities. We still do that. We've customized that now so it's much more intuitive. But we leverage RSpec, since we're a Ruby shop, and its test framework to generate an HTML report that we initially just attached to an email at the end of the run. We have a tool that we call Auto Tools. It's just kind of its nickname right now. It's a suite of entry points to execute our automated test and to feed into our framework. And it's actually how Auto Tools got initially put into place was to be a wrapper around this automated test execution to send that email out at the end of it. Now it's become much more powerful the way that we use it, but at the end of the day, it still handles some of the basic things that just need to get done. So, yeah, nice.

So, Craig, just to refresh my memory, what are your automated tests written in? Is it Ruby and does Ruby work well with Go and Jenkins?

Yes, we are writing in Ruby, Joe. That's our framework and our automation is written in Ruby as well. What's nice about Jenkins is Jenkins can touch something at the OS very Easily you can do things even outside of that. But because we're running in a VM environment for our execution of our automated tests, they're all Ruby based. We actually installed Jenkins on the very machine that executes these scheduled tests. Granted, we have a number of other VMs that our manual engineers use all day. I should, I should say our test engineers. I know that term manual engineer can be confusing. We look at those as test engineers. We empower with automated tests. So the test engineers have multiple VMs they can log into and execute their automation. It feeds all into our reporting through our data lake and through a cabana tools with Elastic. But again, going back to our recommendation from it for the. Our kind of our 1.0, which is, I'd say we're in. Maybe we're transitioning into our 2.0 version of this, that we actually have Jenkins running on the very machines that are going to execute the test as well, so they can touch something at the local file system, including kicking off the right Ruby commands or the RSPEC commands in our case, to go and begin the whole automated process of a specific suite of tests for a product.

So, Greg, so it sounds like then Jenkins has a node that reports back up to the main server, I guess. And so when you're saying you have a local on a VM, you have a Jenkins node on these VMs that are then communicating their information to the main Jenkins server?

I guess, yeah, actually we actually execute since since Jenkins is running in a Java environment, I believe this is probably happening mostly, mostly virtually within that VM itself. But there is a node within Jenkins where the automated test executes within and that feeds just wonderfully into our reporting framework, which again is built in Elastic using our data lake.

So Don, why are you doing this again? Is this enabled to run tests in parallel? Like why? Or is that like, is it kind of running in a Selenium grid? So you're able to run your tests and say exactly, run against these particular VMs, spread them out this particular way rather than use a grid. Or what is the benefit of using just the node rather than a node in a grid? I don't know if that makes any sense. Are you almost creating your own Selenium grid I guess then? But just using Jenkins with the nodes on the different VMs, we don't leverage.

Selenium grid at all. No, that's actually a different model than what we use. The reason we have a number of VMs is remember our target audience is our automated test team and we have A pretty large team of 30 engineers now. And so they're executing their automated test all day long. So they have a number of VMs they use to go ahead and log into and manually execute or kick off the automated test they specifically need. When it comes to Jenkins, these are scheduled jobs. That's how it initially started. Scheduled jobs that run in the morning. And now here's really what's at the heart of our conversation. Now we have the GO pipeline. Say, hey, I'm building this new product, Product A over here. And since I'm building this now, at the end of this build process, go ahead and kick off that automated test. The test product A Jenkins can now consume, request and go ahead and get that kicked off, just like one of our test engineers can. So it, it allows us, as soon as the build is done, to go ahead and execute a suite of automated tests. And granted, our suites can finish up in anywhere from five to 20 minutes. We don't have suites that run for hours and hours, so we don't have that particular problem. So that's how we're using Jenkins, is to be a listener to go ahead and execute things. When our GO pipeline says, I finished this build, go ahead and run the automated test now.

So how do you specify what test to run? Once again, I'm going back to my experience. We used BDD and we had a tag called I don't know, smoke. And so we'd have a build job and the Jenkins job would run that build and say just run anything that has the tag smoke and anew to run it at a certain time or after a certain action. Is that something similar?

Yeah, great question. It's something we've been working on for about the last two or three months in our next evolution, if you will, of the way we are organizing our test suite. So initially for the first few years we would call a specific test suite out and we break suites down again using the METS methodology. That's our manual test plan strategy in house. We would say, okay, we want to run the account test for product A. And so those would be in a suite designated account for that product. Those products all live in their own repo. It's all GitHub based repos we use. Again, we actually leverage GitHub a little bit like Quality center as well. As far as being a repository to store things in, it does give us some versioning capabilities, but we use it primarily for distribution and repo purposes. Anyway, going back to your initial question, one of the things that we've began to implement is we now use tagging within our RSPEC usage of our automated test. And so if we have, let's say within that account suite we have a specific group of tests that are related to maybe deleting an account or maybe changing some information for an account user, we can actually provide the specific tagging within that RSpec test to run that. And so what we're now passing in through, through our Jenkins call is a tag that says go run Product A and run all the account tests. And it's been worth the effort to go ahead and take. We have thousands and thousands of custom tests here to go ahead and align them to this approach where we're tagging them well according to our test plan strategy.

Nice. So another issue I ran into back in the day, once again when I was working for a company when I wasn't solo, is that, and I guess everyone runs into this is getting test environments to be clean and, you know, not flaky. So what we did also at Jenkins is whenever we kick off our test run, we create a new instance of the environment clean and it would run the test and when the test was done it would shut down that VM and so it'd be a clean start and end. And we know the environment was pristine because we're using a template of a known good copy that would always be instantiated when the test ran and then tear down when the test ended. Are you doing anything like that? Because you did mention VMs a few times.

Yeah, I am. I love that you bring this up. It's like you read in my mail, Joe. Yeah, we put a high emphasis on consistency of our execution environments. We even put a high consistency in the, in our, our automation development teams environments that they, they have a certain process we put into place so we are in sync. That's by design. We don't have kind of a willy nilly you decide you're going to go use this IDE and this. We have a certain set of IDEs, a certain set of gems, we use Ruby versions. We do that. It gives us a very consistent approach at our testing. When it comes to the VMs that we have, we don't spin up a new one every time we have. They're all exactly the same. They've all actually been cloned off of the master VM we built initially. And so it must have been about 6 months ago I had this thought as we were installing new software on a new automation engineer we hired. I thought I built a script that basically ensured things were Installed well and proper like that gave me this idea, why don't we do something like this with our VMs? So every single morning we run basically a health check on all of our VMs and I get that reported to me and that data goes into the data lake and it tells me things like how many much drive space is still available in that virtual environment, what version of Ruby are we installed, what's the GEM set up? Anything looking unexpected, we'll go ahead and highlight. So in a way we're automating the automation. We've talked about that in the past and we use our automated skills to ensure even the health of our environments there. Now a lot of folks might be saying, well don't you want to have a lot of environments? Yes, there is a case for that. But we found that by having some control of the environments, we reduce certain risk of, you know, knowingly that we're going into it running under these conditions. So that's how we approach it and we run them every day and I look at them every morning and it's helped keep those environments very healthy and in good working order and ensure that the data that our test engineer is getting is accurate to something that they've seen before. Joe, you and I have talked about the personality of an application. This helps them determine, this is the same application I've seen before. Or hey, this looks different than what it normally is. That's telling me because we put so much priority and consistency, something's not right. Our application, you know, it's got a headache this morning, something's not right. I need to dig in deeper and do some analysis.

Very cool. So are you doing the same thing with the test data that the test actually using for data? Because once again we used a database that was fresh from each instance and so if any change we made because we tore it down when we started back up, all those changes weren't baked into the database. So we kind of removed some flakiness caused there any tips around because it sounds like you have a very large number of testers there. Any type of test data management type solutions you have in this particular approach.

You'Re using, we haven't really had too much issues in that. It just may be the nature the products we're testing. We do use a lot of data. Our sanity tests are all data driven and account for a large number of tests that we run every day. That data still to this day managed primarily in spreadsheets and put into a CSV format because it's easily approachable. To the test engineer. All of them have worked into it in Excel or spreadsheets before. And when they have updates to it, it's very easy for them to go ahead and work in a spreadsheet, add a new URL and a few things they want to validate there and just have our team go ahead and update data for them. So we haven't really deviated from that because to be honest, the ROI would be so small that this approach is scaling well for us at the moment, but it's likely somewhere down the road we will, we will need to enhance that process, but why spend that money and added time at this point? We don't have to. One of the other types of tests we have is a truly data driven test. So I love to do data driven testing. These are things I learned back working in QTP and some of its capabilities. We've ported that concept into our automation usage, leveraging the Selenium API and our Ruby framework. And we use that for things like calculators and things. Since I work for, you know, I work for Dave Ramsey and we're helping folks with, in the area of finances and getting out of debt and working through investments and things like that, we have a lot of calculators. So those things we test through true data driven tests where we're taking input data and validating expected results at the end. And so that data is pretty stable and doesn't need to be modified too often.

Perfect. So Greg, you know, you do have a big team, like I mentioned. How do you get everyone on board? Any best practices you have, have that to get this all implemented and everyone on the same page.

Wow, that's a great question. So I'll tell you, it is, it can be hard sometimes, right? You've got like when we started to implement the morning scheduled automation runs, we decided for the team we would schedule one automated run for them each morning. But what can happen is if you, let's just call it over, automate the process again, automating the automation. If you overdo that, it's really important. Important to never remove the eyes of a tester or the brains or the, you know, the intellect of the test engineer. From why you brought automation to the game plan to begin with, to the test strategy, it's just, it's part of your strategy. So if you overdo that and you expect that, you know your automation itself is going to flag when something pass or fail, I do believe you're probably setting yourself up for a bigger problem. Because automation, as much as I Love to do it is still pretty dumb for the most part. It's good at, you know, looking around at some pretty basic things and going from point A to point B. But man, when you compare that to the intellect of a person and a test engineer that really cares about the work they're doing, there's no comparison. I don't care how much AI you throw at it, a person's intellect and their ability to reason through results is still going to win. And so I truly believe that at my core. So we have to continue to go back to the team to talk about those things. If we produce results and we put them in your inbox, you still have to look at them and you have to make a decision to do that. That. So we talk a lot about the strategy of testing, that automation is just a tool in the hands of the test engineer. It doesn't solve all the testing parts itself. It just gives you insight to say things. Look, as I expected or I didn't expect and now I need to dig in further or I can have confidence, things are in good shape and we can move forward with, you know, the new feature I'm testing or whatever. It's interesting. It gets to the philosophy of what we do and it's something I believe pretty firm in and I think it's helped, helped us to keep a good balance. So our testers aren't worried about losing a job because we never pose automation as something that's taking away their work. It's a tool that comes alongside to enhance and help them.

Great advice. Okay, Greg, before we go, is there one piece of actual advice you can give to someone to help them with the automation testing pipeline efforts? And what's the best way to find or contact you?

Hey Joe, that's a great question. The easiest way to contact me is on LinkedIn and I'm very easy to find on the web at Greg Pascal. That's G R E G P A S K A L. You could find find also my materials on realworldtestautomation.com you'll find my book there and about 35 articles now I've written. As far as actual advice, I would say when it comes to the automation engineer, look at the work that you do as a partnership with your test engineer. Even try to separate that idea of manual versus automated test engineers. At the end of the day, a good automation engineer really needs to be a great test engineer because you need to understand the craft of testing and you need to understand when an automation fire drill comes up and someone says this awful thing happened to automate this, you can go through the reasoning of a test engineer and say, you know, while we could do that, at the end of the day, we might miss one of the highest risk I see here. Or this is a better task to be done visually by a test engineer and we can build automation to come alongside and help that test engineer. But to solely automate it, it sometimes actually blinds you from seeing it again. So try to get into that place. A lot of test engineers aspire to go into automation and that's, that's a worthy place to move. But never look at this as you know you now need to. You now get to move into the place with playing with Lego blocks all day or the fun Work Automation's hard. The easiest thing we do in automated testing is writing automated tests. There is a lot of hard work it takes to do that consistently and reliably and give you exceptional results. Back to the manual test engineer or to the test engineer themselves. That's my advice. Let's go around.

Joe, thanks again for your automation. On the links to everything of value we covered in this episode, head on over to testgill.com a333 and while you're there, make sure to click on the Try it for free Today link under the exclusive Sponsor section to learn all about Sauce Lab's awesome products and services. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack automation on awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Automation Podcast. Head on over to testguild.com for full show notes, amazing blog articles and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.