Secure Automation Testing at Scale Leveraging SBOX with Michael Palotas and Lee Walsh

Get ready to discover the most actionable end to end automation advice for some of the smartest testers on the planet. Hey, I'm Joe Calantonio, host of the Test Guild Automation podcast and my goal is to help you succeed with creating automation awesomeness.

Hey, it's Joe and welcome to another episode of the Test Guild Automation Podcast. And today we'll be talking with Michael and Lee all about secure automation testing at scale leveraging something called sbox. If you do anything with enterprise testing, you definitely want to listen to this episode. If you don't know, Michael is currently the head of product at element 34, which is a market leader for enterprise testing grid infrastructure solutions that run inside the corporate firewall, which I know is a big, big deal for enterprise companies. He was the head of test engineering at ebay where he was instrumental in designing, developing and open sourcing of the selenium grid. So he really has a deep understanding of how this all works. He's also worked, I think on SolAndroid and iOS driver. Michael also worked as a software engineer at intel and Nortel, so he really knows his stuff when it comes to enterprise application testing. Also joining him we have Lee Walsh. Lee is currently Director of customer success at element 34, so he speaks to a lot of people. So really excited to get his insight on this. Lee and his team primarily focuses on ensuring that element34 customers are getting the most out of Sbox, which is element34's flagship product, which we're going to learn all about and why you need to know about it, especially if you're working at the Enterprise. And prior to element 34, Lee was the team lead at BrowserStack, so he has some understanding in this space as well. So he has a lot of knowledge, a lot of great information around automation testing, how to scale at the enterprise. You don't miss this episode. Check it out.

This episode of the Test Guild Automation Podcast is sponsored by the Test Guild. Test Guild offers amazing partnership plans that cater to your brand awareness, lead generation and thought leadership goals. Get your products and services in front of your ideal target audience. Our satisfied clients rave about the results they've seen from partnering with us, from boosted event attendance to impressive roi. Visit our website and let's talk about how Test Guild is to take your brand to the next level. Head over to testguild.info and let's talk.

Hey guys, welcome to the Guild.

Hi, thanks for having us back.

Hey Jeff, good to have you. Michael. Hey Lee.

Yeah, thanks for having me.

Absolutely. So Lee this is your first time on the show, but Michael, I know we talked, I think in 2020. So welcome back. I'm just curious to know for people that may have missed that episode, which was, I can't believe it's almost three years ago at a high level, what is element 34 or what is Sbox? Maybe we can just ease into it that way and give people a little flavor what to be in store for.

Yeah, sbox is basically behind the firewall or on prem test automation infrastructure solution that runs completely secure inside your firewall. No data going out, no external access required from the outside. So yeah, that's really in the nutshell what it is.

Awesome. And so once again I want to remind listeners that we spoke to earlier and you also. So you created one of the first implementations of Selenium Grid. So I'm curious to know like how that influenced the solutions that you're working on now.

Yeah, so that's correct and not correct at the same time. So maybe just let me give you a little bit of background on that. So back in the days I used to run quality engineering at eBay International and due to the size and the speed that we were operating at, automation became one of the key success factors for delivering top quality software to our customers. So mind you, this was in the early 2000s and we were looking around for a tool to use and we decided to use Selenium. Now we probably started automation just like everybody else by adding a test and adding another test and then running them sequentially one after another. And what was interesting, we quite quickly realized that this would not scale and then started to look for a way to run tests in parallel rather than sequentially and also across different browsers. Problem was there wasn't really anything out there in the market, also not in the Selenium ecosystem, that would satisfy what we needed. So we just decided to basically go off and create our own solution. Now we were also in touch with the people from the Selenium project, Simon Stewart and the likes, and once we had our internal solution developed, we were discussing, hey, you know, would it make sense to maybe make this part of the Selenium project? And that's how Selenium Grid was born. Now, the reason why I said it's correct and incorrect, so I personally did not write the code for Selenium Grid. That was mainly Francois Renault, who is our VP of engineering today, as well as Christian Rosenbolt and Kevin Maynard. So personally I was more responsible for managing the open sourcing part, getting the product, getting the Selenium grid into the Selenium project and actually open sourcing.

Mike, when did you first work on Selenium Grid then? It's been quite a while then what, 10, 12 years at least.

Yes. Yeah, that was around, I think we actually introduced it at the very first Selenium conference in San Francisco. I believe that was 2011 or 2012. Yeah, so it's been a while and of course a lot has happened in the Selenium space and also with Selenium Grid. But yeah, it's been around for a while and we're happy that we made our contribution there to change the world.

Awesome. So Lee, I want to get you in there really quick. Just a random thought, you're from a infrastructure company, another company that did kind of like what you all are doing now at element 34. What got you interested in the space or you know, what are your thoughts on element 34 and why it might be different from what you've maybe worked on in the past?

Yeah, so previously I would have worked with a traditional SaaS solution in this industry. So you're testing infrastructure or grid, you get access to it, you run tests either manually or in an automated fashion. But where that infrastructure sits is a key component. So that would sit in the vendors public cloud. Over time I saw a use case of some potential customers not being able to use this because they wanted to use real data or they needed to test their early stage environment. So that got me thinking. I did a little bit of searching, came across element 34 and then was obviously put in touch with Michael and Francois. And then yeah, straight away I understood the key benefits of the product and the offering that they have and it was something that I wanted to be part of as well.

Nice. So you speak to a lot of customers. How many people are still using the in house Selenium grid? From your experience, is it a lot of people or.

Yeah, you'll find that there is people that still use the in house solution as well. So they'll have a combination. Now whether that's something like sbox or they built out their own, they'll have their own at a certain point for certain use cases, but they also may make use of the build versus buy conversation. They may be sitting in the buy part as well. So it's a mixed bag but you'll see people making use of their own grid and making use of a vendor, whether that be public cloud offering or something like sbox.

And Michael, what do you think some of the benefits are then from like an in House grid versus an enterprise solution, which I think that's what kind of separates you. I've speaking to a lot of people over the years and it seems like your sweet spot really is the enterprise. And it's something that is critical that a lot of people, I guess, maybe not aware of.

Yeah. Maybe one way to look at it is why do people create their own Selenium grid and house in the first place? Right. Maybe if we start from that perspective and, and what we typically find is because it's actually super easy to get started with. Right. You basically you download the software, you spin it up and there you go, you can, you know, you've got something to show, you've got something to run tests against. So that's great. It becomes a bit more problematic when we look at things like maintenance. Right. So that's a whole different story, actually. So there's a lot happening all the time in the browser ecosystem in the Selenium space as well. New releases very frequently. Right. So it's becoming very cumbersome and very time consuming to actually keep a Selenium grid once you have it up to date. Right. And effectively that's what you want. You want to make sure you're running your tests, of course, also against older browser versions, but very or most importantly, you want to test it against the new stuff. All right. So that's where people usually start. And now where sbox comes in or how we solve some of those issues is basically, we are the only enterprise solution that was built from the ground up to be inside the customer's firewall. Typically, if a company decides to build their own grid, they are probably looking at having it inside their own. Own firewall. Right. And so they may just go off and build it on their own, but of course they end up with all those maintenance issues. And so that's exactly what we solve because we basically give you all the bells and whistles that and the convenience of the SaaS products, because those products are great. Right. But with the difference that it's running completely inside your own firewall, so there's no data going to the outside. You don't need any external access from outside to get back in. So that's the key differentiator.

Nice. So what are some key benefits then that people at the enterprise may need this solution for? We already have something working. I mean, yeah, sure, we have to maintain it, but are there other pieces that they may not think of? Like any sort of compliance issues or things like that would make them say, okay, we need this other solution to help us.

I can probably address that. So there are some advantages, some are more obvious than others, but when we speak with potential customers, we normally mention five key considerations that are maybe more. But those key considerations relate to our offering. But the conversation would be quite similar when we're looking at building versus buying a solution. And if buying a solution, where should that be hosted? Michael's touched on it already, but the considerations, the first two would be the security and compliance piece. If testing within our own corporate network, you're staying secure because you don't need to open that network like you would to a traditional SaaS solution. Remember, if everything involved in testing at the moment is hosted internally, why would our testing infrastructure be any different? And you may have the answer to that question, but a question that needs to be answered nonetheless. If we want to test as early and often as possible as well, we need to know what's involved in opening up that environment. So some will create tunnels and so on, but something that we need to be aware of given where S Box is installed. This is not a concern compared to others. The second piece that I mentioned or touched on there was the compliance piece. And this is pretty straightforward. If no data is leaving my network, then there's little risk of us breaching any data privacy concerns or regulations. Sorry. Also, if we're using real data, we may have agreements with some of our customers that that real data can't be used outside of our network. They're kind of the two main ones that we see come across. Performance is another key consideration. So where is the infrastructure hosted will impact performance? So logically, if I'm testing within my own network versus reaching out to a solution that may be hosted in a different country or a different continent, we're going to see different results. Whether what we're not too sure what that will be, but less latency normally equals improved performance and a reduction in tests potentially failing due to timeouts or other scenarios caused by performance issues. And then the last two that sometimes are overlooked is scalability and cost efficiency. So scalability from an infrastructure standpoint, not from a test script standpoint. I'll answer it from an infrastructure standpoint is what's involved for me to run my tests at scale. How many tests can I run in parallel or concurrently? What kind of queuing system do we get access to? This is all important for me as I grow on my automation journey, but potentially for my organization as more people make use of the solution I've chosen. Okay. And then the cost efficiency part is pretty Straightforward. The cost of us scaling or continuing on this journey. There's other hidden costs outside of infrastructure, but we won't talk on them today.

No, it's a good point. A lot of times people say, oh, I have an open source solution and so it costs me nothing, when in fact it does cost them very much. So I think cost efficiency, they may be like, what? But can you explain a little bit more what that means? Because if someone says I'm using open source, so it doesn't cost anything, what is the cost efficiency? Is it like, easy to maintain as well? And if your tests run faster and quicker, then you get quicker results and less time, which is going to save money.

Yeah. So cost efficiency is if we go from 10 tests to a thousand tests, let's take that example. How much is my infrastructure going to cost me to host that solution internally? Also, do I need a bigger team to maintain this? So the maintenance and management costs and the work that's required to troubleshoot any issues that we see in our own grid, these are all part of that cost that you may not initially consider as you grow or as you start off. But further down the road it starts to become more and more of a pain or a headache and something that starts to go out of control pretty quickly.

Absolutely. I used to work for an enterprise company doing radiology equipment and it was really hard to even get people to, like you said, open up the tunnels and the ports for an outside solution and compliance issues with the fda. So we didn't have an in house solution. But this sound like a perfect thing for that particular environment. So I guess, you know, in that case we had like thousands of tests. What would we have needed to do then to get our test to work with the sbox solution?

Yeah. So from a test script perspective, there's very little changes that need to be made. So if we take someone that has Selenium test scripts already built out, written in the language they've chosen or the framework they're making use of, Selenium is Selenium. Whether I run it locally through a traditional SAS solution or on something like sbox, the important piece is the driver initialization. So where are we pointing our test scripts to? So once I change that from, let's say a local Chrome driver to the S box hub address, I can now start to execute my tests. Other things that you might have in play already is the likes of test altering. So you might be using something to build your test scripts. In the end, tools like that normally create or leave you with A Selenium test script. So within the ui, you can then go and point it to a hub address as well. So very little change required to move across or make use of something like S Box. And I think that's why this market is so competitive as well. How easy it is for me to go from one solution to another means that everyone in this industry needs to make sure they're on top of their game. Really?

Absolutely. So I alluded to some benefits and some issues running in enterprise. One of them also was a security. Security risk. They say, oh, you can't do that for security reasons. I know security is a huge requirement for a lot of enterprises. So, Michael, you have any insights on being on prem how it helps with some of these security concerns you probably hear all the time from enterprises you work with.

Yeah. So just with SBOTs, by being inside the customer's network already, we have naturally much deeper integration points with the rest of the customer's development infrastructure, be it the ability to hook into the customer's IDP or identity provider, like something like Active Directory or OpenID, or else much more complex integrations like Kerberos, NTLM, which may not be possible if you're using a SaaS solution that sits outside of your network. The other kind of talk tracker point to this as well is that SaaS solutions to get into your network from the outside, you potentially need to open your firewall, what whitelist IPs that they provide to you to let them back into your infrastructure or create these tunnels that we spoke on with fbox. You don't have any of that headache given where we're located or we're installed. So we have no idea how many tests you're running or anything like that. It's completely airtight.

Absolutely. Michael, any insights around security from your point of view?

Yeah, I mean, I usually see this security and compliance kind of go hand in hand in enterprises. The compliance part is more about what we touched on, the data. You know, depending on what kind of data you're using, you may or may not be allowed to send that elsewhere. And then of course, the security part is what Lee just mentioned. In order to use solutions that are sitting outside of your firewall, you have to drill a hole into your firewall, you have to let them back in. Right. And that's typically done through these tunneling mechanisms or, you know, essentially that's a vpn. Right. And we all know once you're on a vpn, you can get to anywhere else you want as well. Right. So there's Certain risks involved when you do that. And that's of course if you're maybe a small startup, maybe not so much concern. But if you're an enterprise, if you're a government organization, those things are absolutely key that they are taken care of.

So just random thought, does this only work with Selenium or can you also scale like, like Playwright Tests or any other software? Is this a strictly Selenium based grid solution?

No, it's not actually. I think last time when we spoke it was, and back then it was, the product was also called Selenium Box. And a lot has happened over the last three years from a product perspective, also from a company perspective as well. But so I think probably the biggest pieces that we added in terms of product were the ability on one hand to run Appium as well, you know, for the whole mobile side. But then also we added Playwright about a year ago, I believe, because we saw there's traction in the market for Playwright, some customers were started to ask about it. So we listened to our customers and we implemented that. So we do actually have quite a few customers that are using both Playwright and Selenium and it seems to work quite well. It's funny because, you know, when you look at some of these tool comparisons, it oftentimes sounds like it's an either or kind of a decision. And what we're seeing, it's more of an end kind of thing. You know, maybe for some teams Selenium is better, for other teams Playwright may be the better solution. So what we're trying to do is we're trying to provide one place where all the tests can be run.

Interesting. So if does someone, if they have a mixed test suite of Selenium and Playwright, do you notice how do they run the same on Sbox or do they need to do anything different to get them to work? Or is it just consistently you just point the driver to your environment, you're off and running?

Yeah. So it's a pretty seamless process. For all of the integrations it is where the initialization is happening. So again, if I have test scripts built out already, or if I was to look at a test script right now, large parts of it would not need to change. So what we're actually trying to do, the initialization piece is key. So for Appium as well, we would just point it to the hub address and start. Obviously we'd specify an app that we want to test as well. But the script itself would run wherever it needs to run, really just we're pointing it to sbox in this case.

Nice. So a lot of these other solutions as well, you're able to run on different devices because it is in the cloud, it's a SaaS solution. So if you have it local, then I guess is this solution not focused so much on the devices as well, or how does that work? Does that make any sense?

Yeah. So around the maintenance piece, I suppose Michael kind of touched on this earlier, but one of the benefits of our product is, yes, it is installed within your own corporate network, but we do have the benefits that you would see with a traditional SaaS solution from a maintenance standpoint. So once Sbox is installed, you automate the maintenance part, which will then give you access to the latest and greatest or what your customer base is using to interact with your website or native application. So the first installation is done very quickly and from that point onwards you can run tests in parallel and you can automate new browsers, mobile devices, and so on.

Awesome. So I know the problem, I know a lot of people have is, yeah, I could scale my tests here, but my tests are a mess. They're not going to be able to scale. So do you have any advice on how to help people get to the point where they're like, they can really benefit for something like S Box?

Yeah. So the scaling part is tough. 1. I touched on scalability earlier from an infrastructure standpoint, but as your test landscape starts to evolve, you'll move from running tests sequentially and want to scale and run more tests simultaneously, concurrently, in parallel. Parallelizing tests can be hard. What typically happens is that we take the tests that we have built and we have built them not designed to run in parallel, and we start to fire them all off. The best case in that scenario is that they all start to fail. The worst case is when some of them start to fail, some of them start to pass, we start to see flaky results and we need to start troubleshooting more. So focusing initially when we're building out our tests to make them more atomic is critical because that ensures that we're ready to then run them in parallel or we're preparing for that scale further down the road. So I think it's something that you need to be aware of at the start, which is making sure they're more atomic.

And this is actually something that, you know, even back in the ebay days, we got that wrong. Right. We didn't think about it when we started with automation. We just wrote tests one after another, not really thinking. What happens when you say, I Have a thousand tests now and here we go. Let's just run them all at the same time. Right. So, and I think a lot goes with which data are you using for testing? Right. Obviously, if you're sharing data between tests, that's catastrophic. Typically then your tests will have those flaky behavior. Sometimes it may work, sometimes it doesn't. Right. And that's absolutely not what we want.

Well, that's a great piece of advice. I always recommend people when they're starting, they run right away in CICD and start running in parallel just to find these issues. For sure. Absolutely. So, you know, a lot has changed. Like I said, it's been three years. I know you mentioned some new features since last time. We've been on the ability now to run Playwright, which it seems like that's something I've been seeing a trend in as well as a lot of people are it's time to use Playwright as well. There are a bunch of other things going on. Michael, is there anything other, anything else that's built into S box that maybe wasn't there three years ago that we haven't covered?

That's a good question. I mean, our engineering team worked very hard over the last three years. Right. And typically it's two parts. One is to keep up with what's happening in the browser and the ecosystem space. But then of course the other one is also to add more features. So we've done a lot of work in, for example, adding OpenID Connect as an identity provider system, just like an example, by adding Kerberos NTLM to be able to mimic the user who you're running as and tap into your enterprise identity system, things like that. So this kind of also goes along with what Lee mentioned. Right. Because we are sitting inside your network, we have a lot deeper touch points than what you can have when you're coming from the outside. And so that allows us to integrate much deeper and tighter with your. With the rest of your enterprise development and test infrastructure.

Do you all work with like SaaS based applications? If someone had to like scale a bunch of tests against Oracle or SAP, did you care what the application to test is?

No, not at all. Right. So that's completely up to the customer to decide what they want to test and where that application sits. So of course, you know, maybe one thing to clarify is that when we say on prem or behind a firewall, it can absolutely be in your cloud as well. Most of our customers are running S box in their cloud. So in aws, Azure, Google Cloud or such Kind of infrastructure. Right. So we don't really care where the rest of your development pipeline is. But that said, typically the customers that come to us, they want to have everything in one place, which is behind their firewall.

I think to that there's certain different scenarios that make the considerations that we spoke on earlier more relevant than potentially testing something that's already out in production, available, used as a monitoring tool or something like that. So some of those key considerations become a lot more relevant depending on what you're testing as well.

So I'll be honest, one trend I've been seeing if I go to any testing company, not your site, it slaps on AI machine learning. Even though they may have been around for 12 years, all of a sudden I see AI machine learning popping up. So any thoughts on machine learning or AI, especially how it applies to maybe a grid type of infrastructure?

Yeah, I mean we're watching the space as well and yeah, I think in general it feels oftentimes very confusing about all the solutions that are out there. Right. I think it starts not even with AI and machine learning, but even just test automation. You know, it can be a bit of a loaded and bloated word that's used to describe a lot of things and it's sometimes hard to understand. So what is it exactly? What I'm getting from this solution or that solution and of course AI is one of those shiny new things. Everybody's adding it into their pitch to, you know, make it, the product look attractive. I think what we're seeing with AI is that in fact our customers are using or are starting to adopt AI and machine learning. And this is something that we didn't really think of, to be quite honest. Right. So what we're hearing from those customers that are using AI in their products is that in most cases they have to test with real or at least realistic data in order to train their models, in order to make sure that their algorithms work. And so they're saying, you know, we have, we have no other choice than actually using real, real data to do that. And with that, actually being behind a firewall becomes a must have. Right. So that's something that's quite interesting that we're seeing which drives even further adoption for what we're providing.

Absolutely. Alec, once again, I worked at, for a medical company and we use patient data. You couldn't, you had to normalize it. So this seems like it's a great point. If they were creating an in house model, they definitely would want an in house grid solution to run Run against to test. So great point. So what's on the then based on that and other things we talked about, what is the future of element 34s box? Anything on the roadmap you can reveal or tell us about?

Yeah, I mean there's lots, lots of things on the roadmap. I think our product vision is to ensure that security, scalability and performance that we're always best in class. Right. So that's whatever we do, it's always going to centered around and you know, we're the experts in the space and we understand what problems our customers face and we listen to our customers. So we're going to implement what our customers need. Obviously, you know, technology keeps evolving. Playwright one good example three years ago, wasn't there. Today it's a major player. Right. So we monitor that space. We obviously make sure that as things change, we bring it into the product, we add support for that and but first and foremost, you know, as any I think software product company should do, you should listen to your customers. What do they need? That's the most important thing. And then from that we put together a roadmap. Lee is in very close collaboration with all of our customers. So am I. So we're constantly listening to them, seeing what, what is it that they need and that really also helps drive the roadmap. So all in all, we're super excited for the next phase of the company. We see amazing potential in what's happening in the market and you know, what we can bring to that to solve the issues. Yeah. So we're looking into a bright future.

Nice. So I know I keep mentioning it but like I really think the enterprise is kind of overlooked or a lot of people just pick up an open source thing. So I think if anyone has to do anything or cares about security, risk compliance, scalability, performance, cost efficiency, all the things we talked about, I think it really much applies to enterprise. They definitely should be checking out sbox. So I guess before we go then, what's the best way to find and learn more about s box or element 34? If someone listened to this and said, oh my gosh, I'm an enterprise, I need to try this.

Well, I think the easiest way is go to our website, it's called element34.com and there you can request a demo, you can request to speak to somebody and we're super happy to show you a personalized demo and see if it fits your needs.

Awesome. And before we go, Michael and Lee, I like to ask this one question. That is what is one piece of actionable advice you can give to someone to help them with their automation testing grid efforts. I also created a book on it for folks of the best answers. So give me your best answer to that. Let's start with Lee and we'll end with Michael.

It's a tough one, but to keep things relevant to what we've just spoke on, something like flaky tests that we've touched on a little bit. There's a whole host of different reasons why you might see flaky tests. Try understand what that is. Is it down to performance? Is it down to the data that you're using? And if it is, then potentially it is something like S box that you need to look at. So yeah, review how your tests are or your flaky tests and understand why they're failing and come to a solution then at that point.

Yeah, I like to give one piece of advice to anybody in the software space, not just around automation. But you know, I always say don't just jump on the first bandwagon when you read about a new tool that came, you know, that somebody found or that you read about. I always advise watch the space, see how particular tools evolve because you know, we've all been there. When you read about a cool new tool, you bring it into your company, you start really relying on it and then at some point you find out a it's a actually just one person looking after it and the person moves on and the project dies and you have a problem. Right? So that's one thing that I think can prevent a lot of headache and a lot of rework if you choose your tools wisely and don't just jump on the next best thing that you may have read or heard about.

Thanks again for your automation awesomeness. The links to everything of value we covered in this episode, head on over to testguild.com a461 and if the show has helped you in any way, why not rate it and review it in itunes? Reviews really help in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack automation awesomeness. As always, test everything and keep the good. Cheers. Hey, thanks again for listening. If you're not already part of our awesome community of 27,000 of the smartest testers, DevOps and automation professionals in the world, we'd love to have you join the fam@testguild.com and if you're in the DevOps automation software testing space, or you're a test tool provider and want to offer real world value that can improve the skills or solve a problem for the Guild community, I'd love to hear from you. Head on over to test guild.info and let's make it happen.