Automation Testing Infrastructure with Adam Goucher

Five, four, three, two, one.

Drop that.

Welcome to the Test Guild Automation Podcast. We're all getting together to learn more about software testing and automation with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Automation Podcast. Today we'll be talking with Adam Goucher all about automation and the importance of infrastructure. Adam is the CTO of the Mobile experience company in element 34. He was an early innovator in contributing to many open source testing projects and one time I believe he was actually the lead maintainer of Selenium ide. He was the creator and maintainer of PY Santer, a Python based automation framework. A creator maintainer for Sonter php, a PHP automation framework. Adam also has ran a number of books like beautiful testing for O'Reilly Media and has spoken at a bunch of conferences like ordev and many Selenium conferences. So listen up and find out why Adam really doesn't care about your automated tests. Check it out.

The Test Guild Automation Podcast is sponsored by the fantastic folks at Sauce Labs. Their cloud based test platform helps ensure your favorite mobile apps and websites work flawlessly on every browser, operating system and device. Get a free trial. Visit testguild.com forward/source labs and click on the exclusive sponsor section to try it for free for 14 days. Check it out.

Hey Adam, welcome to the guild.

Hey, thanks for having me. I think I'm going to be stealing that intro for my conference submissions from now on.

Awesome. Cool. Good, good. I thought I messed it up because you've done so much. I tried to condense it. So. Awesome. So Adam, like I said, you've done a lot. Is anything I missed that you wanted folks to know about?

I think that kind of does cover the breadth of the last 20 years. Really? And yes, I did maintain selenium IDE. I brought it back from the dead originally and I jokingly but kind of not say that. I've been apologizing for that ever since.

So that's funny because I think they revived it again. Yet again, a newer version. Have you seen that version? Any thoughts around it?

Yeah, I've played with it. It looks pretty cool. It does overcome a lot of the shortcomings that the original IDE had.

My concern as was with the original ide, is that for full bore automation.

I feel that too many people are going to use that instead of dropping into the code level to do stuff. But I think that's just an education thing that is ongoing in the automation community.

Absolutely. A lot of people blame the tool, but it's really the technique or people abusing a tool, I guess is what happens.

Yeah, we provide too many. There's almost too many tools and too many ways to use the tools. And unfortunately, there's more ways to fail with the tools that we provide than there is to succeed.

Absolutely. And talking about failure, I think you have a famous saying, it's like almost your motto that you don't care about automated tests, you care a lot about where and how you run them. You've been saying this for a while. I've been binge watching some of your videos and I think I saw some in 2011, 2012. You're talking about infrastructure and it still seems to be a neglected area of automation. People still are talking about flaky automation. When you talk about infrastructure at a high level, what do you mean by that? By that saying?

By that, I mean it's all of the things around executing your scripts, not including your actual scripts. So where your automation environment resides, how it's constructed, how it resets, how you know your Selenium server, where is it running, or how are the, how accessible are the ports, how economic is your environment? Is it just running 247 when only you works during business hours? All those things I think are the next sort of wave of disruption coming to the automation space. Cloud providers like Sauce Labs and such hid that problem from a lot of companies because they just manage it, they move their grid. But a lot of companies still are running internally their own Selenium grids or other technology stacks. And it is my belief that has not been disproven, but neither is it necessarily been scientifically proven that a lot of organizations, they just spin up their own servers. The manager has a credit card that they don't have to ask questions about, so open up their own AWS account with their own. They just spin up the EC2 instance and run it. And they don't know anything about security, about permissions and whatnot. So my, my fundamental belief is that somebody's Selenium infrastructure has been used to compromise the corporate network. If you, if you figure that target, the big credit card breach that target had was through their air conditioning units that were Internet connected, then I guarantee you that somebody's Selenium infrastructure or their testing infrastructure has been used to compromise their networks. And just from a. We didn't know that we could hide things from the Internet. You know, by default, they start up and they're publicly available. We didn't lock down a port or we didn't Patch it or whatever there is. So that's why I say I don't actually care about your scripts anymore. A lot of the patterns that our people are still employing are the ones that we developed 10, 12 years ago. So we've largely figured out what it takes to solve the flaky script problem and how to run our tests to be more efficient and how to select tests for partial execution. We know how to do these things. But what we don't know yet, or what we really haven't spread throughout the testing community is that you need to actually care about your infrastructure as well.

Why do you think it's still such an issue? Like I said, you've been speaking about almost like nine years now. Is it because the skill set or is it always like another team that all they'll handle the infrastructure and people just don't even think about it when they're creating the automated test?

I think it's more of the latter.

Partially. I think it's silos and skill sets. So a lot of people that are doing automation are the. They're not ops people or they've grown up in the cloud and they've not had to do the ops things. Like I'm 42, I think, and so I've been doing this, managing my own infrastructure and computers in the cloud since I was like 15. Like my first automation job was in 98 and we were doing full automation with racks of 1U machines that we would have to reset by floppy disk. So I've always had to be concerned about the infrastructure that things run on. But now.

A lot of that is abstracted away from people. So I was hoping that the sort of the DevOps stuff would leak into the testing space, but it hasn't quite yet. There are some projects that provide sort of selenium in a box type services put inside containers. But even that requires more knowledge than a lot of testing teams possess. So then you have to tap into the ops team and they're busy running production and test is a cost center, so they don't necessarily get the attention that they should require if they knew what the risk profile was.

So I believe you have like five pillars that maybe will help people guide them with the infrastructure. One's operational excellence, one's security, one's reliability, one's performance efficiency and one's cost optimization. Are these the pillars that people follow? They have it better with infrastructure and if so, maybe we can go through each one of these, what they mean.

Yeah, so those pillars are actually blatantly lifted from Amazon's well architected framework and there's an entire team that maintains those. And what's really cool about the well architected framework is that even though it is written by Amazon and all of their examples use Amazon products, it really is cloud neutral. If you just swap out product offering A with product offering B, it's a cloud neutral framework to base your idea, your thoughts around on how you're gonna build these things. They've actually even published sort of a white paper to prove this where I don't remember, the company was able to slash their Azure bill by like 35% by just implementing the policies and the practices in the paper. And they realized, oh, we are running these giant servers 247 when we only need them between 1 and 5 on Friday afternoons or something. And so those are, it's a useful way of thinking about your infrastructure because it is again it is five silos that when in a cloud context are really important. So obviously the security one is a big one. That's sort of the way, that's the hook that you get people to be interested and concerned about their infrastructure. But the cost optimization is also, I think, an underlooked one. And that's the sort of thing where you can scale your environments up during working hours, scale them down after, over the weekends. Are you using right sized machines? Are you using the appropriate technologies? Is there new technologies? Are you constantly evaluating them? So there's the five pillars in the well architected framework and like I said, it gets updated about quarterly. It's a free download on Amazon's website. I don't work for Amazon, I just think it's a really great framework.

The company I used to work for, we used a solution called Skytap and once we started using that, management then was able to say, wait a minute, our tests are taking way too long. So it helped almost drive better practices automation because like you said, the cost was so much for keeping this environment up all the time. So is that one of the benefits you see is if you have a well solid infrastructure plan, that it helps create more reliable automation? Because then you could see the, you know, get feedback from the test running and then to make it better based on cost, try to figure out why your tests are taking so long or may not be as reliable as they should be or things like that?

Well, the advantage of caring about the cost of your automation is that you can then answer the question of how much is our test costing us? That used to be the hardest question to be able to quantify. Oh well, you've got 10,000 scripts. Okay, that's fantastic. What's the cost benefit analysis of that? We have to constantly be maintaining these things. We have to investigate failures, we have to keep constructing new ones. Then there's just the execution costs, there's the person cost of having 10,000 scripts and then there's the execution cost of like running them. And then. But what's the, you know, what's the benefit of that? We don't know that. And we're still missing, you know, part of the equation of what is the actual underlying infrastructure costing. So that should be a very simple number to manage by going into whatever cloud provider billing console and like, here's my automation, you know, bucket and it cost me $76.53. Just like if you used a cloud provider or like a, you know, browsers in the cloud provider, you would know by looking at the invoice how much it cost you. So there's, it helps you have better conversations. It's still going to be really challenging to argue what the benefit of automation is. You know, well, did we catch bugs? I don't know, did it, you know, like you start to get very philosophical really quick, like why do you do the automation? Is it to ensure that the same things worked? Is it that you want to make sure that developers have trained their testing practices to make sure your tests pass? Right. Like there's, it can get really interesting. But the big thing, like I say, is you can answer how much the infrastructure costs.

So what's your go to answer when someone asks you that? What's the point of automation?

Well, to me my go to answer is that it makes sure that it has a baseline level of quality for a human to attack. It is my we don't know. And this comes out of my growing up as a tester through the testing ranks as compared to a developer and ops person. That a human is really what.

Finds the problems in the application. But what the automation does is tell you that these are the parts that you don't need to worry about because they're behaving. But if you run an automation load and it comes back with the errors over here now this is where I should probably spend some time looking at what changed, what interesting thing here has happened so that I can spend a day looking at new account profile management settings. Oh, did that change recently? Because I'm talking with the team all the time. Yes, it did. What was the things that happened? Where are the ripples?

To me that's where automation really what it's for it's not going to tell you that it's perfect and good to go. It's here's this baseline that we have established and we are maintaining that baseline. And so now use your human judgment whether or not you want to do some manual testing or do we want to bump it up to the next environment which could actually in fact be production.

So that applies to functional automation. But I believe also what infrastructure, I believe you said in one year, I don't remember which session, but that you should perform operations as code. So is that something you could automate? Is that like tools you would use like Puppet Chef or Ansible or something to make your infrastructure more repeatable?

And one of the quotes that get thrown back at me periodically is that you know, if you are logging into whatever cloud providers console to make infrastructure changes, you're doing something wrong, you're incurring technical debt. So everything should be, you know, every piece of your infrastructure, you know, certainly beyond just the test infrastructure should be specified in code of some sort that could be the native code platform offerings like an Amazon it's cloud formation or something that's cloud neutral like Terraform, which is the one that I'm currently ramping up on because I, you know, I really enjoy vendor lock in. I think that if you're not locking yourself into a cloud provider, you're doing something wrong. But what Terraform provides is that cross platform capabilities. So especially in like a webdriver context, I could have my Windows and my or like my Firefox and Chrome and whatnot on Linux and Windows running inside Amazon and I can have my Mac Safari on Mac running in something like Mac Stadium or what have you. If I'm just using CloudFormation, I can't configure that third party using it. Whereas Terraform, because it's just a wrapper around APIs, I can do the same set of infrastructure setup scripts, control all of my providers. All that stuff is just, it's just a dsl, a text based DSL checked into whatever repo system you're using. You can write tests, you can write code sniffers and formatters and treat it just like any other production level development project.

One of the things I heard a lot about, and I've been hearing about security a lot but for some reason 2019, a lot of talking on my podcast about security, security, testing and also so I think it's going to be really bigger in 2020 and you mentioned security as well as one way to get people to think more about their Infrastructure. What are your thoughts about security in 2020 and how does it relate to infrastructure that people need to be aware of?

So I think it's still especially, and this is kind of sneak peek stuff for selenium grid 4.

I think the Selenium community specifically is going to have a big wake up call around security and how they may be thinking that they're operating securely, but they aren't necessarily. In the alpha that just got pushed for Selenium 4 last week, we can now communicate with the server over HTTPs again. As a cto, I have to go through vendor vetting processes and they really don't like anything that's not HTTPs. And the Selenium grid since its inception has only talked HTTP. So I talked about this in London with Simon and then he like created a ticket and assigned it to me. So apparently, apparently the, the squeaky wheel has to provide its own grease. So that's available in the most recent release, the most recent alpha of the Selenium grid. So in security for the, you know, everybody's gonna have to look at their infrastructure and start securing those connections now that they can also around node registration for the grid, there's a couple nasty little attack vectors that are theoretically possible that will be closing up in time for selenium grid 4 as well. So in the automation space and infrastructure, the capabilities to secure things are being provided now finally for people. So that's another reason for people to look at their infrastructure. It's not just always been running and it just works well. Now you have reason to revisit it.

Earlier you mentioned not rogue teams, but people that have credit cards and they can open up AWS instances and you don't necessarily may know about it in the company, does that come back to traceability, having a plan in place to be able to monitor your actual infrastructure, what the heck people are doing?

Yeah, and that's hard in that case because you're opening up your own AWS account. So you can't really trace that because it's unknown. I did a bunch of time at HP and they called those all like black ops projects and HP at the time, this is sort of during the carly years where you know, it was this monster company, it had no idea what account it had where and it had no idea what services it was running where. And it basically there was an email that got sent out to everybody that said you have six months to let us know what all these skunk works projects are. Otherwise if we hear about one, you're fired. So because it was just so out of control because they couldn't secure their their corporate systems because it's like, well, we just signed up for this Salesforce account with a default password and it didn't have our corporate password rules. So now somebody could compromise there and all a skilled bad guy needs is a tiny little thread to pull on and they own you.

Yeah.

It's not as if it's necessarily rogue teams. A lot of times when I've seen that happening, it's because they're stymied by corporate policies that it's just like we need to have this type of browser on this type of machine and corporate provisioning doesn't have it for them or it'll be a six month wait for it where they can just get it inside cloud provider X for $20 a month and the manager can expense it. Here we go. So it's not as if rogue teams kind of implies malicious intent a lot of times it's just people trying to get their job done in what they perceive as the best way possible for the company, but it actually works against company best interests.

So one of the last points about infrastructure is when something fails you need to be able to self diagnose it really quickly and have it self heal. Is this something you've seen any advancements in recently like with these AI machine learning so called type of solutions that have been coming out recently?

On the infrastructure side of things, this is the whole notion of chaos engineering, which I think originated out of Netflix. There's a couple companies that bill themselves as just chaos engineering companies. I think Gremlin is one. And so.

If you introduce chaos agents into your infrastructure, I think that might be a little overkill for just your test infrastructure. But if your corporate infrastructure is doing it, then absolutely there's no reason not to be running it against your test infrastructure. And I believe again in a Selenium context the new grid will make that level of. I don't want to say the word observability, but I'm stumbling on a word for a word so I'm going to use that. How do we observe the failures in our systems?

But a lot of that becomes also what cloud provider you are using because again we're not in the actual script levels that are failing. It's a database connection got dropped or somebody tried to roll out another instance and you've hit account limits or something which are all built in ability to monitor using their monitoring tools. So again the testers, if they're monitoring their, if they are responsible for their Infrastructure need to really leverage what their providers are providing them and take off the testing hat for a little bit and put on the ops hat for a little while to figure out how do we do this thing more effectively.

Cool. So I want to switch gears really quick. Adam, I never understood you have like two or three jobs and I never followed up on you're the CTO of the mobile experience company in element 34. Do you also do consulting or and you contribute to open source projects still? What's going on there?

Yeah, I do way too much. My dad was a workaholic and when he retired they replaced him by four people. And so I, I come about that naturally. So my day job is CTO of a marketing tech company.

Outside of Toronto. I also run a small consulting company called Element 34 where less so in the last little bit because being a CTO is a full time job. I have helped people be effective creating Selenium and Webdriver scripts as well as looking at their entire delivery pipeline. Because as trendy as continuous delivery is, we've been in the automation space have been doing continuous delivery for well over a decade because we had to in order to reset environments and get code to run things against. Then when I'm not doing those things, I am part of the Selenium community. I have had the commit bit for a number of years.

And I have been working on sort of creating or filling the security holes that I have been complaining about for a number of years. In time for selenium grid 4, which should be out in June, I think we're targeting. So there's that.

I, you know, and then when I have free time I referee roller derby. So yeah, there's, there's many different things that I do all at once. But you know, the first three are, you know, take up, you know, 30 hours of a 24 hour day.

Cool. So Adam, besides infrastructure, is there anything else you're really passionate about in the automation space that you think has been neglected?

So the after, you know, I feel like the next soapbox that I'm going to be getting onto after the infrastructure stuff becomes more mainstream is really documentation. The new Selenium site looks fantastic and the docs have been rewritten again. They're fantastic, they look great and have been updated. But the one thing that sort of drives me a little bonkers online and this is where I should learn to be offline sometimes is sort of my get off the lawn factor. Because a lot of, I think I mentioned this before, a lot of the patterns that we use to write scripts. We figured out how to do these things 10 years ago, but people are still complaining about flaky scripts. People are still not running their scripts through a proxy in order, a scriptable proxy in order to black hole the Facebook pixel, which will speed up your tests. There's lots of things like this that people are just rediscovering and it feels like a reinvention of a wheel. And I used to get really upset by this. It's like, come on folks, do your history, do your research. We know how to write all these things. There are examples, they're 10 years old, but they still apply. And I realized that that wasn't necessarily fair because what we, as sort of the leaders in the community, if I can call myself that, have done a really bad job is communicating these solutions to people. So the sort of the big challenge after the infrastructure I think is how do we better communicate solutions to people, especially as they're starting their automation journey so that we can just have people be more effective more quickly. That's fantastic grammar.

Than they necessarily are. We don't need to have them struggle with duplicating locators in the scripts because it's like, well, there is this general hand wavy concept called page objects where you can encapsulate these things like, oh my God, this is brilliant. Why didn't anybody tell me this? It's like, well, sorry, like.

It'S out there, but not well enough apparently. So I don't know. I have no idea how to solve that problem. And so that's I think going to be a very sticky problem to solve because it's a people problem infrastructure we could solve with code. So it's a very simple problem to solve. As soon as you start getting into people problems, well that gets a whole lot more challenging too to address. So that's that I want that I'm starting to put on more people's radar. But I really have no answer on how to do this because stack overflow, you know, searchability is a challenge. There's a whole bunch of community involvement challenges, not to mention just the level of toxicity that can exist on that site. And we are flooded with search search results right now. Again, one, when I started there wasn't Google was barely even a thought. So when somebody posted an article on something, it was easy to find because there wasn't 1,000 variants of that same article. So what was the canonical answer? I have no idea how to solve this thing, but there's still a lot more work to go on the infrastructure side, so I'm not even trying to solve that one yet.

Okay Adam, before we go, is there one piece of actionable advice you can give someone to help them with their automation infrastructure testing efforts? And what's the best way to find or contact you?

So my piece of advice is care about your infrastructure, your audit. You know, doing automation is more than just writing scripts and executing them. It's everything about it, including where it runs. So go to a whiteboard, draw out all the little boxes of this is the machine that our scripts execute on. This is where the scripts execute against. This is the machine that controls that execution. These are the people that have the keys to these various machines. How are they configured? Once you have that map, that illustration drawn out, you'll be amazed how complicated your actual infrastructure is. And that should be enough to scare you into looking at how to better manage it. And then the best way to contact me is either on Twitter, my handle is just Adam Goucher, or you can email me at adamlment34ca thank you Adam.

For your automation awesomeness. For links to everything of value we covered in this episode, head on over to testguild.com aelicautomation288 and while you're there, make sure to click on the Try it for Free Today link under the Exclusive Sponsor section to learn all about Sauce Labs awesome products and services. And if you haven't already, make sure to head on over to guild conferences.com and register for Automation Guild, the largest online conference in the world, dedicated 100% to helping you succeed with creating End to end Automation awesomeness. Our goal is that you'll learn a tip, tool, technique or best practice that's going to help you or your team accelerate your automation efforts in the new year. You don't want to miss out. Head on over to guild conferences.com and register today. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe and my mission is to help you succeed with creating end to end full stack Automation awesomeness. As always, test everything and keep the good Cheers.

Thanks for listening to the Test Guild Automation Podcast. Head on over to test guild.com for full show notes, amazing blog articles and online testing conferences. Make sure to subscribe to join the Guild and continue your testing journey. This has been a Joe Calantonio production.