Enterprise Selenium Grid with Michael Palotas

Welcome to the Test Guild Automation Podcast where we all get together to learn more about automation and software testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Automation Podcast. Today we'll be talking with Michael all up Enterprise Selenium Grids. We're also joined by Mark Slater. Michael currently works as the CEO of Element 34 Solutions, the market leader for enterprise Selenium grid infrastructure solutions that run inside the corporate firewall. Before element 34, he was the managing director at Grid Fusion, an engineering consulting firm for software organizations that specialize in Lean and agile software development. Also what I think this is really cool. Hopefully we'll dive into this a little bit. He has been or he was the head of test engineering at ebay where he was instrumental in the design, development and open sourcing of Selenium Grid, Solidroid and iOS driver. Michael also worked as a software engineer at intel and Nortel, so he really knows of stuff. Really excited to have them on the show. If you do anything with Selenium and Selenium Grid, you don't want to miss it. Listen up.

The Test Guild Automation Podcast is sponsored by the fantastic folks at Sauce Labs. Their cloud based test platform helps ensure your favorite mobile apps and websites work flawlessly on every browser, operating system and device. Get a free trial. Visit testguild.com sourcelabs and click on the exclusive sponsor section to try it for free for 14 days. Check it out.

Hey Michael, welcome to the Guild.

Hey Joe, how are you? Glad to be here.

Great. Awesome to have you. Is there anything I missed in your bio that you want the Guild to know more about?

No, no, I think that's detailed enough, so that's good.

Very cool. So like I mentioned, I'm really curious to know. I guess I should do better research. You actually developed Selenium Grid? The original implementation, yes.

So not just myself. So I was working at ebay at the time and we were tinkering around with Selenium and this need that we need to run a lot of tests in a very short time frame. And so we said, okay, then let's just build it ourselves. And so a few folks on my team went off and built it or you know, built something that was the initial version of Selenium Grid and we used it in inside the company. And then we started talking to some folks on the Selenium project, like Simon Stewart. And then we said, okay, why don't we open source it and make it available for everybody? And then of course you Know, from that point, a lot of other folks jumped in and added very useful features to it. But yeah, so I think it's fair to say that we were heavily involved back then in the creation and open sourcing of Selenium Grid.

Awesome. So what was the genesis of you developing Selenium Grid then? What was about the software development lifecycle at ebay that you thought, well, we need this solution and here's the solution. And then after that say, wow, this solution is not only going to work for us, but it's going to work for other engineers.

Yeah. So, I mean, at ebay we had a requirement to scale our automated tests, Right. We had a lot of tests. We were looking after a lot of different countries, different platforms back then in my team. So we had this need that we couldn't just run our tests sequentially anymore. I think if we had done that, it would have taken seven days or something like that to run everything sequentially. So we needed a solution that allows us to parallelize our tests and to basically bring it down to a matter of minutes, ideally to run the entire suite. So that was the key driver for us to go and develop a solution for it, right? Yeah. And then so we said, all right, now it works for us. And we were in contact with the folks in the Selenium project. So then we just started talking to them and they said, you know, this would be nice if you guys could open source it. And so we went through a little bit of a process and, you know, inside the company to figure out how we can open source things, you know, without giving away any ebay specifics, of course. But then, yeah, in the end we managed to open source it, make it available to everybody. And, you know, and I think this is one of those nice showcases where you can see that when somebody open sources something, it's maybe in a bit of a rare state, and then other people come in at more stuff, and then it becomes what it is today.

So, Michael, I guess before we go any further, I assume people listening to this are pretty hip to testing and tools, but for folks that don't know, let's set the stage a little. What is a Selenium grid?

So a Selenium Grid is effectively a piece of infrastructure that allows you to run tests across different browsers, across different operating systems at scale. So it provides a central entry point for all your tests that you have in the Selenium space. Of course. Right. It provides you a central entry point for your test execution to run that across different browsers and if you want, also mobile via Appium. So I think essentially also what Selenium Grid allows you to do as an organization is to shift left or to help shift left by speeding up the feedback cycles, by shortening those feedback cycles in the organization. Because typically as test automation matures, there's more and more tests. And if you don't have a way to parallelize those tests, those test runs will take longer and longer and that's not something you want. So that's where Selenium comes in and allows you to run a lot of tests in a short time.

Nice. So if someone's thinking, well, this sounds great, but why would I need Selenium Grid? It sounds like one main reason is to be able to run your test in parallel. Are there other main reasons you usually give folks to why they should be using a Selenium Grid?

I think one other main factor is the cross browser aspect. Usually when you look into our dev organizations, a lot of testing is happening on Chrome only. And depending on who's using your application that you're developing, typically, especially for publicly facing websites, you can't really control what kind of browsers people are using. Right. So you have to make sure that all the different browsers are covered in your testing. And that's also something where Selenium Grid helps. So you can have any browser, any browser version if you want, any kind of operating system, browser and version combination at scale in a Selenium Grid.

So, you know, I've used Selenium Grid in house for certain companies and I found sometimes the maintenance of the grid is really difficult. Are there any known, not issues, but known difficulties you've seen companies or organizations have when they try to create their own Selenium Grid?

Yes. So I mean, you know, the thing is using or getting started with Selenium Grid is typically fairly simple, right. You download the jar, you spin it up, you attach a couple of nodes to it so you can show results very, very early on it, you know, maybe it takes you half a day to get something up and running. What becomes very labor intense and with that, of course also expensive is the whole maintenance of the grid as it grows. Right. Having a couple of browsers supported, that's easy. But once you have a Selenium Grid that serves an entire enterprise, for example, where you potentially run hundreds or thousands of tests in parallel, you need all those machines, you need all that infrastructure to support that. And then I think it can become very time consuming and very expensive with that to maintain that. Sometimes it's an entire team that does that. Sometimes it's also just a one man show or a one woman show who does that in a company. So such a central piece of the CI pipeline like Selenium Grid is sometimes completely dependent on one person in the company, whether that person is there or whether that person is on vacation. And when the person is on vacation, then this piece of infrastructure doesn't get maintained. So I think typically there's the cost aspect. Of course, when you talk about, about building and maintaining your own grid. I think there's also scalability issues in the open source Selenium Grid version today. So you can only scale to a certain point. That's at least what we hear from customers out there. Right. And then it's also just, you know, from a business continuity perspective, if you want to make sure that this piece of your CI pipeline is up to date or up to speed and up and running all the time, then you know, having that relying on one person can prove a bit difficult.

I absolutely agree and that's why I think a lot of folks go to cloud based solutions like Source Labs. But you know, I spoke to folks earlier and you said, hey Joe, there's another option. Not only is there your own grid and cloud based, but there's another kind and I believe you came up with Selenium Box. So could you just tell us what Selenium Box is? What makes it different between say a cloud vendor option or a in house type of solution?

Yeah, I think in general when we look at the Selenium Grid space, there's two main paths you can choose. One is you have it running inside your firewall, inside your network and the other one is the outside and with that the SaaS option. So I think for any enterprise building a Selenium grid on your own, I would count that as not really an option for them, you know, because they typically get to a point where they say, okay, this is too much. So we need something vendor managed with proper SLAs around it and things like that. As you said, there's numerous cloud or SaaS solutions out there which are by the way, fantastic products. Right. So they are feature rich products, they have all the bells and whistles that an open source homegrown Selenium grid doesn't have. And then as you said, so then there's this other path that you can also take. So if you want to stay inside your corporate firewall for, and there's various reasons why you would want to do that, but if you decide that you want to stay inside your corporate firewall. Then there is for example, Selenium Box. That's the product that we develop and which is in the end you can think of it, we bring that experience that customers have with the SaaS providers, we bring that inside, inside the corporate firewall. And with that, of course there's a lot of aspects specifically around security. Right? Because if you go out to a SaaS solution, you're sending out all your data, your test data to the SaaS solutions, they can potentially see that data. And for some organizations that may be okay, and for others it isn't. And for those where it is not okay to send things outside their corporate firewall, that's exactly the space where we come in, where we bring that rich feature set inside your firewall without having to do any of the maintenance that we mentioned earlier in the homegrown and self maintained Selenium grid space.

Now I used to work for a very large, large company healthcare division and it was very difficult to get any type of cloud based solution that we could use. And even if we did, it would take forever to find out who we would even go to to get permission to open up a security port or something that the utility might need to use. So a lot of times what they end up doing is they take the SaaS solution and they do an on prem implementation of it. So is Selenium Box just an on prem solution of a cloud provider? Is it something else as well? Does that make sense?

Yeah. So I mean, I think from a feature perspective you can look at it. So you know, all the features that you are used to from the SaaS solutions, like you know, live view video, getting all the logs and things like that. So of course, you know, that's also something that we provide. So all the features that you see in SaaS Solutions, we take that and we bring that in house. But Selenium Box itself is not a SaaS product. It's just, just we basically we provide all the bells and whistles that people expect with the added security that everything is running inside your firewall. And not only security, but with, you know, with running inside your firewall you also have aspects like performance. For example, you don't have to cross the big pond or go from the east coast to the west coast and around in for every Selenium command, if it runs inside your network, of course the latency times are very short and with that you will see a massive increase in performance.

I would think that'd be a key feature because that's one of the main complaints or concerns folks have when they run in these cloud based solutions, it does tend to be slower. So it sounds like that's one of the main, another main benefit is if you have an in house, it runs quicker as well and more secure.

Absolutely. And of course that will depend a little bit on where you are and your servers are and where that SaaS provider is. Of course if you're next door then probably the latency is not that big of a deal. But typically also let's not forget in an enterprise environment you have to, in order to get outside of your network to, to a SaaS provider and back in, you have to navigate several proxies, several firewalls and things like that, which typically slow things down. So that's very common that we get the feedback from our customers that especially the ones who were with SaaS Solutions before where they say wow, this is by a magnitude faster than before.

Very cool. So are there any other features that maybe differentiate you from any other type of solution out there there in this space?

Well, I think just because we are running inside the customer network, we have a lot tighter integration points into the existing enterprise infrastructure, be it certain authentication mechanisms that we can support by being inside. So I think there's a lot of enterprise level features that just come because you are inside the network and that cannot be exposed to the outside. So as a SaaS provider you are not in a position that you can offer those services because you don't have access to them. Right. Whereas when we're inside the network we can actually provide a lot of those additional features that I just mentioned.

So another popular option is Selenium, which I believe uses like Docker or Kubernetes in the background and puts up things, it just makes it quicker and faster to tear up and tear down environments for your tests. That's supposed to help maintenance. So what's the difference between say Selenium Box versus Xillenium?

So I mean one of the key differences is that Selenium is an open source project and we are a commercial product. Right. I think that's one of the key differences. There are no SLAs around open source projects. Right. If you have a problem, you may or you may not get a response. So pretty much all our customers are large enterprises that need SLAs around. If something goes wrong, they need a one hour response time for us to get back. And then I think also we with Selenium Box, we or when we look at a project like Xenium so that supports Chrome And Firefox. Right. So everything that you can spin up in a container, whereas we go way beyond that. We can do Internet Explorer, we can do Edge, we can do Safari, we can do mobile emulators for iOS and Android, and we can also do real devices. Right. So it is really the entire spectrum of what you may need as an enterprise, rather than just the Firefox and Chrome containers.

So I know I asked you about Selenium, but you know, I just realized that it actually has been discontinued. So I guess I was just using Selenium as an example. But this, whatever you said about Selenium would apply to any other solution now or after that is open source. When you talk about what I think a lot of people don't appreciate is when you work for an enterprise, security and being able to get tech support is very important because those SLAs is usually mission critical for certain functionality for sure.

And you know, I think also, and not just about Selenium, but there's various other open source Selenium grid solutions out there. I think one thing with just like with any open source tool, when you want to bring that into the enterprise, I find it always makes sense to look at how active that project is. Right? You know, when has the last commit been made? If it's five years ago, then there's a pretty good chance that, well, either the product is, or the project is really mature or it's dead. And also I think what's super important to look at is how many people are behind it. And I think specifically in the Selenium grid space there's certain really cool solutions out there, but it's maintained by one person. So if that person moves on and you've brought that into your enterprise tool chain, you will have a problem. And that's what we're seeing. That's why companies are looking towards commercial solutions like, like the SaaS providers, if that's okay with them from a security perspective or of course, and also at solutions like what we provide.

So once again, not a small point, people should dwell on this a little bit. I've been blogging for 10 years and I just looked at my broken links and a bunch of them are two projects that are no longer supported on GitHub because like you said, it had one or two maintainers. And that's a great point that whenever you invest in any type of technology, especially open source, you want to make sure how active the community is for sure. That's a great, great tip. So I guess the next question folks would have then is say for some reason someone does invest in Selenium Box, is anything they need to do in order to make their tests work with Selenium Box and then if they don't, they no longer use Selenium Box. Do they have to make any changes to the code to work normally again against a normal Selenium grid?

No, there is no changes required at all to the tests. So. Because just like the SaaS solutions out there, they support the W3C standard. So do we. Right. So the Webdri protocol is W3C and we fully are compliant to that. So if you are running your tests on your homegrown Grid, on the SaaS solution or on us, there's absolutely no changes required. That said, of course the change that you have to make is where you send it to. Do you send it to us? Do you send it to one of the SaaS solutions? So that's a one liner in a configuration file that you need to change. But from a Selenium command and from a Selenium test perspective, there's zero changes that are required. And that's the beauty of open source technology. Right. Because I think for a customer there is no vendor lock in. Right. If they can choose the solution that works best for them. And it's not that because you bought Selenium Box, you're going to be locked in for the next 10 years to use it. If there's something better out there, you can go there. Right. Which is also one of the key drivers for us to provide a fantastic product. Right. And for everybody else as well. Right. Because it is easy to switch.

So that's awesome. You don't need to change your test to run against Selenium Box. I guess another question would be, since you've been dealing with Selenium Grid since the beginning, are there any best practices folks still need to think about though, as they're creating automation tests that allow them to run maybe better or more optimally in a Selenium Grid type of environment.

Yeah. I can particularly think of one thing, and this has actually nothing to do with Selenium Grid. But what we're seeing quite frequently is that companies who start out running tests sequentially, they come to that point when they say, all right, we need something that is able to parallelize our tests to run them faster. And then they plug in a Selenium grid and then everything explodes. So their tests explode, nothing works. They get flaky. Sometimes they pass, sometimes they fail, you know, and then oftentimes, unfortunately, the blame is on the tool itself. So either, you know, on Sauce Labs or BrowserStack or US or on Selenium as a whole. But deep down what typically happens is that tests are not designed with, with parallelizing them in mind. Right? So tests are designed to be run sequentially. And when you take 1,000 tests that were run sequentially before and then you say okay, now run them in parallel and they are not independent from each other. So let's say they share the users, they share the test data that they are using. If you have a situation like that, then of course you will get a horrible outcome. So one advice that I can give, and this is also just from working on the side where we created a lot of tests, is design your tests when with parallelization in mind from day one. Which means ideally you are able to have test data that is atomic to each test. Each test should have their own data. And of course if that's production data or production like data, that's fine. It's just that every test should have their own set of data rather than sharing the five golden user accounts across a thousand tests that you're going to run in parallel. I think from my perspective that's one of the key success factors. To run run tests at scale and to get those short feedback cycles.

Great advice. How does Kapir maybe price wise? I know we're not selling this. I'm not getting you to say how much something costs, but is it comparable to a cloud based solution or any other maybe price benefits of going with a in house solution?

Kyle, that's a very good question. So our solution is as Michael said, an on premise solution. So it's designed for use across the whole enterprise. So if you're looking for maybe one or two users as an introduction to commercial Selenium grid, then we would be outside of your price point. Until up to a certain level we'll be equivalent to other commercial SaaS spenders. But then the beauty of our solution is we allow companies to scale to an infinite amount and when you can scale to that amount using our product with an infinite license, then we become become very, very commercially attractive relative to other solutions. So really there's an initial price point where we get into the game and then the more you scale, the more the commercials work in our favor.

So I guess the question should have been build versus buy. And it sounds like you're saying if you're a small team and you're using a Selenium grid, I know a lot of small teams are happy with it. That's Fine. As I said, I work for enterprise and I worked across multiple verticals against multiple teams doing multiple things. Solution like this might be the perfect kind of option for them. So definitely they should check it out. I guess one other thing I want to cover before I ask my last question is updates. I guess there's pros and cons to having things on prem and such, but one thing SaaS products are good at is if there's an update, you know, they handle it for you usually. But when it's on prem, you know, you have to coordinate with a bunch of different teams and then make a plan when you're going to do the update. So how hard is it to update a solution that is in house in general, like yours?

Thank you for that question because that is actually also one of our key strengths because the update is exactly as transparent to the end user as it is on any SaaS solution. So what we do is when there's, let's say, a new browser that releases, we of course pick that up, we test everything, we make sure that everything works just like how we expect it to work. And then we make that binary or that image, we make that available in a private repository for the customer and the customer just pulls it automatically from there. So the customer has actually no interaction whatsoever with the fact that there is a new browser that released or that anything changed in the Selenium ecosystem. So from that experience perspective, it's exactly the same AS using a SaaS solution. And that's also what we want. I mean, we want to bring that SaaS experience inside the file.

Okay, Michael, before we go, is there one piece of actionable advice you can give to someone to help them with their automation Selenium grid testing efforts and what's the best way to find contact you or learn more about Selenium Box.

So you can find us on either seleniumbox.com or element34.com right? So if you have any questions, you can find us there. There's also a lot of resources out there that you can go through if you want to learn more. I think piece of advice that we give to any kind of customer that we start interacting with is really, you know, think about do you want to build or do you want to buy? And if you want to buy, do you want to go outside of your firewall with all the implications that come with that, or you do want to remain inside your firewall that will guide you in terms of which solution is the best one for you. Then from there, of course I think it's important to understand none of the Selenium Grid providers make it easier to write your tests. You still have to write your tests actually in a much more robust way than if you just run what I mentioned before. If you run everything sequentially, right? So the Selenium grid gives you that opportunity to run a lot of tests in a short timeframe, but writing the test is still on you, right? And that I always say test automation is software development, right? You develop software. In the end you should apply all the techniques, all the patterns that you would apply in a regular software application development process. You know, you should never say, oh, this is just a piece of test code. And that's why we can get sloppy. So I think when we look at how to write good tests, that's typically what I take as the baseline.

Thank you Michael for your automation awesomeness. For links to everything of value we covered in this episode, head on over to testguild.com a312. And while you're there, make sure to click on the Try for Free Today link under the Exclusive Sponsor section to learn all about Source Lab's awesome products and services. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe and my mission to help you succeed with creating end to end full Stack Automation awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Automation Podcast. Head on over to testguild.com for full show notes, amazing blog articles, and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.