254: Making Load Testing with Real Browsers a Reality with Tim Koopmans

Hit it.

Welcome to the Test Talks podcast, the place to go to geek out on software testing. And now your host whose mission is to help you succeed with test automation, Joe Colantonio.

Hey, it's Joe. And welcome to another episode of Test Talks. Today we'll be test talking with Tim Koopmans, founder of Flood IO, all about his session at this year's Perf Guild, making load testing with real browsers a reality. Because I've been so busy planning and putting on the online conference Secure Guild because I haven't had time to interview someone new recently, for today's episode, I took part of Tim's session that he gave at PERF Guild 2019 and some of the answers he gave for the live Q and A during the conference. And it created almost a mock interview for the full awesomeness of Tim's session and to see things in action. If you missed the Perf Guild and you want to actually see Tim and all the other speakers. Awesome sessions. You can get all the videos automatically when you register for Perf Guild. Now, even though the event is over, you still can register and get instant access to all the recordings of the event. To do that, just head on over to perfguild.com.

Test talks is sponsored by the fantastic folks at Source Labs, the cloud based automated testing platform that eliminates the need to maintain your own Selenium grid and test infrastructure. Try it for free today. Visit testalks.com and click on the sign up now link under the Homepage Sponsor section.

Hey Tim, welcome to Test Talks.

Hey Jay, thanks for inviting me.

Right, Tim, right off the bat, could you just first tell us a little bit more about yourself and also give us a little teaser about what is Flood Element.

G'. Day. My name is Tim Koopmans and I'm.

Really happy to be talking to you today about making load testing with browsers a reality. I'm presenting this to you in the context of my current job, which is Director of Engineering here at Tricentis Flood. And for the past couple of years we've been using browsers as an alternative to more traditional load testing tools at the protocol level like JMETER and Gatling. And one of the tools that we've been doing that with is called Flood Element. It's a browser based load testing tool, it's open source and it's built on top of Puppeteer. And you can find out more information about it at Element Flood IO.

So Tim, in your session you did give your definition of what is performance testing. A lot of people sometimes think of performance testing differently. So what is your definition of load testing?

To me, load testing really is the art of simulating demand on a system and then measuring it for performance. But performance isn't the only dimension that we care about. And I like to use the mnemonic pairs to remember the other dimensions, and that is performance elasticity, availability, reliability and scalability. You see, load testing really is just a means to an end, and that end really is to observe these different dimensions under load. And it's really what you do about those dimensions that makes all the difference to your load test efforts. So in order to do something about them, you have to be able to simulate load in the first place. And this is where I see many people, both new and old to performance testing spend a lot of time, and by that I mean disproportionate amounts of time actually spent, you know, building and maintaining load test scripts, typically at the protocol level, in order to simulate that demand. And this is all at the cost of execution and analysis, which is the pointy end of load testing and where we should be spending the majority of our time. But unfortunately, we kind of get sucked into this enormous time sync of load test creation. And that's why I really want to talk to you today about the reality of using load tests with browsers.

So, Tim, can you just tell us a little bit more? Like, what is browser based performance testing? What does that even mean?

So to be clear, when I'm talking about browsers, I'm talking about the thing that you're looking at right now, which is probably Chrome, could be Internet Explorer for example, but not only are there different browsers, but there's also different devices and platforms that you can use it from. So maybe it's Windows, maybe it's Linux, maybe it's your desktop PC, maybe it's a laptop or a tablet or even your mobile phone. You see, the browser really is tightly.

Coupled with the modern web and it.

Feels like every application these days is going to be browser based.

So Tim, in your session you also have a reference to the Matrix about Spoonboy. What does Spoonboy have anything to do with browser based performance testing?

Because I love that quote in the Matrix where Spoonboy says, do not try and bend the spoon. That's impossible. Instead, only try to realize the truth. And then Neo asks, what truth? And spoon boy says, there is no spoon. You see, in the Matrix, the Matrix is the perceived reality by most of the humans in the simulation. But Neo, he knows that it's a simulation and he can literally send unexpected data into the matrix and bend the spoon. And as load testers, we sometimes behave exactly like Neo. We bypass a simulation at the browser.

Level and we do whatever we want.

To do at the protocol level. And this is quite powerful because we can fool other players in the system, like the system under test, into thinking that they're actually serving real users and real browsers, for example. But what is the cost of actually bending that reality? And should we perhaps conform to the simulation at the browser level instead?

So to me, I actually started my career as a performance tester and I actually found creating the scripts were a lot easier than analyzing the results. So why did you feel like you should be investing more at the browser level rather than the protocol level?

The biggest reason for Flood to sort of go after browsers these past couple of years is because most sites these days are becoming increasingly complex when it comes to the protocol level underneath. It's not uncommon, for example, to see over 300 requests on the first visit to Amazon.com for example. And not only are there tons of requests back and forth between the browser and server, but the browsers themselves are doing a lot more work for us these days than they did 20 years ago. So everything from parsing and executing JavaScript frameworks through to processing client side business logic, all the way to caching responses and even handling security, this all comes into the domain of the modern browser. And as anyone that's ever tested something beyond the primitive pet store example can tell you, that load testing a modern single page web application at the protocol level can be an absolute nightmare to build and maintain.

So Tim, how does this work? Is it either or do you see people using both browser based approach and protocol approach together in one scenario?

Yeah, I think it's really good as a hybrid approach. I mean, I mentioned some of the style of testing, which is obviously resource consumption. It takes a lot more resources to spin up concurrent load with a browser level approach over a traditional protocol level approach, for example. So we have to be considerate of the environment and be wary of punching more holes in the ozone layer. So I think it's always good to consider both approaches. A common approach that I see with customers is that they will often use protocol level stuff for targets that are particularly easier or I guess, how should I say, more well defined. A good example might be, say a RESTful HTTP based JSON endpoint for example, might be testing like a mobile site and it has pretty well articulated requirements or specs around how to use the API. Say they might use a protocol level approach for that and then offset some of browsers as well to see what it looks like from more of a user perspective as opposed to just underneath the hood at the protocol level. I really think using both approaches is fine. We also have some customers that just only test at the browser level because some of the implied complexity that I hinted at, particularly with single page applications or I guess more complicated business or user flows throughout the application, it just becomes a whole lot easier to sort of build and maintain the scripts at the browser level. So, you know, ticketing, gaming, any sort of transactional kind of types of clients, I guess in the browser, you probably guess even video streaming is pretty common as well. So it's really just.

Picking the tools wisely and making sure that you're aware of all the different options that are available to you in the open source space.

How does Flood Element actually capture timings as you're running the script?

Yeah, the concept of timing in Element at the moment is what we would call step boundaries or simply wall clock time. We have timers that happen around the individual step boundaries, which you would have seen in the script that I was building on on the screen when you're running it from the command line. The command line does give you information in terms of. It's more to sort of help you with debugging. So it's going to give you incremental sort of wall clock time or elapsed time for each of the steps. You can also, obviously, I'm pretty sure from top of my head, you can interact with that via the API as well and dump that out if you wanted to. We have had customers that have asked for, I guess like the sort of traditional reporter formats that you get with different testing tools. So being able to basically run Element as is and spit out information into some type of format that you can then consume in your tool of choice. And it can also of course be used with Flood, which relies on the same wall clock timing methods that were built into Element itself. We also have plans to add some other timing metrics because we are interacting with the browser, we have full control of Chrome. We're really interested in say like Lighthouse metrics for example, but we can kind of intercept and do whatever we like thanks to Puppeteer at the network level. So we are sort of interested in trying perhaps some other useful timing things, typically like what you would do when you're sort of debugging front end performance issues. So you're looking at say.

Interactions with JavaScript or.

What am I saying? XML, HTTP requests, that sort of thing. So being able to sort of see summary of those broken down at the step level or at the page level, let's say, would be also really useful, we think, for customers.

So what's happening under the covers? Is Selenium used to do the driving of the browser? How does that work?

Yeah, actually Element. So for a while Flood was using Selenium and we still actually support Selenium WebDriver, the Java version. I guess the problem that we found with that approach, and I'll just quickly talk about that, is that that was even more resource hungry than say the Puppeteer approach, which is what elements based on. The main reason is because we have to run a separate JVM for the standalone server for it to coordinate. That eats into the CPU and memory footprint of the load generator itself. So kind of indicative benchmarks, the similar benchmarks that I showed during the presentation. We're getting around like 5 to 10 browsers per call with a selenium approach versus like up to about 40 within reason, depending on the script with an Element approach. So the answer is like we used to support. Well, we still do support Selenium because there's many customers that just prefer to write Selenium, but with Element we kind of went in a different direction because we were really excited about like Puppeteer kind of answered a lot of requirements for us when it first came out. So when it came out, we saw this really nice way to interact with programmatically with Chrome without a lot of the sort of bloat that I was talking about before. That's when we started to spike the idea. Well, wouldn't it be really cool if we could sort of borrow the DSL from say, Selenium js, which we really like and we think a lot of customers were already familiar with. Wouldn't it be kind of cool if you could bolt that together with, with Puppeteer and add the necessary sort of instrumentation and sort of controls that you'd need to be able to run it in a distributed environment. And that's pretty much how Element was born. So yeah, it's really sitting on top of Puppeteer at its core. It doesn't really have anything to do with Selenium, aside from borrowing some of the syntax in the dsl.

So, Tim, is this an open source solution? Is this a paid solution? How does that work?

Yeah, it's fairly new. Element itself is definitely open source. You can find the links, it's on our GitHub repository. You can fork it and do whatever you like. Like I said, it's based on Puppeteer, it will remain open source. I think the things that we might modify would be how we actually executed on Flood. Obviously Flood is a paid or subscription based platform, but I think having that capability, like a lot of customers have really liked being able to build low test scripts locally on just their own laptop. It's kind of like a novel concept to just be able to build and test as you go. And definitely customers have been asking for more ways to sort of run Element locally. I don't think we'd go as far as trying to implement a, you know, like a completely standalone distributed solution for Element itself, but we would welcome any, you know, any pull requests in that space as well if people have some good ideas there. But I certainly don't want to try and dictate the way that you would use it locally. I mean, I think customers and users are really creative when it comes to using tools these days.

Does this work like with package solutions like SAP? And if so, do you have like a quick get started guide to actually how to do that?

Actually yes. Like if you look on. When we were sort of releasing Element for the first time, we thought it would be good to have a number of examples for the common sort of things that people are trying to test. And to that end it was more sort of packaged app based. So we did. This is how you would test YouTube. This is how you would test. What was it? WooCommerce, Magento, WordPress for example, just to give people some starter scripts. Then in the course of doing that, we actually released what we call the load testing playground, which is just once again another open source repository on our GitHub and that has some example Element scripts in there. I think the intent for our load testing playground is to provide useful examples that we find for the main tools that we support. It includes examples for Element for SAP, Fiori for example. Salesforce is definitely on the to do list. But likewise we also want to have examples for Gatling and JMeter as well because we love those tools as well.

So Tim, not in a way that one tool is better than another, but how would this Flood Element Compare to say LoadRunner's true, true client protocol or even. I think SmartBear came out with a new awesome tool called Load Ninja.

Yeah, I mean I try and keep an eye on the competition, you know, building a product, but I don't try and spend too much time there. But I am aware of the competitors. I would say things like TruClient has been around for a long time and certainly I feel like that's the granddaddy of the browser approach. There was another competitor that sort of came and went, which was Browser Mob. I don't know if you remember those.

Yes.

Yeah, I think it was Patrick Lightbody that. That was kind of like some of my early inspiration, I guess. But I think what's common to it all is really just acknowledging that browser level is a viable approach in some situations. And more often than not it's to sort of reduce the complexity around building and maintaining scripts. I think all of those solutions have that in common. I'm not sure what was the. The other one, the load ninja one. I have a feeling that was still kind of Selenium based.

Brand new.

Yeah, I think, I think, you know anything. My caveat for the caution or the lesson learned, I think for anything Selenium based is it was kind of never really designed to be like a load testing tool first and foremost. I mean, it had some fairly opinionated decisions in there about, you know, why would a tester need to care about the protocol level. Like it was strictly like browser only. And that can get in the way sometimes when you want to change it without changing the core code. Obviously things like True Client are going to be a closed source approach.

As far as I know, unless they've changed recently.

I think what I'm trying to look for in a load testing platform first and foremost is open source and being able to extend it, being able to get my hands on it easily without too many restrictions around how you use it. I think that's a bit of a differentiator for us. We want to still take the open source route there, I think just trying to keep it modern.

Our choice to use types is in line with Puppeteer, as you could probably see in the demo. It provides really nice helpers for building scripts. You can take advantage of IDEs like Visual Studio code or just decent text editors for that. For that matter, extending it means that you can use any number of node JS libraries, for example, like we're using the fake library to generate random data that's immensely useful. We have like a test data element there as well, which we can suck in. Things like CSVs. I just feel like open source is always the easiest path forward and gives you more options, more flexibility basically. So that's why I like to think tools like Element are a bit of. A bit different to the other sort of competitors.

So, Tim, I know the tool is fairly new, but you did mention sometimes when you get the actual product in the hands of your Users, they sometimes do things that you didn't expect. Have you seen any of those type of issues and if so, what are they? What other scenarios might they be using this tool for?

I think the use case has been, I mean there's been some attempts to do some.

Not malicious but you know, to fake some traffic to things like Google and stuff like that because we are using a real browser and I think some.

More dubious customers figured out that they could use this as a bit of a platform to vote brigade and thumbs up things. So we had to sort of, we do monitor that, so we had to sort of stamp down on that a little bit. Yeah, users always surprise me in terms of what they do. I think the biggest surprise though was just how quickly users were willing to ramp this up. I honestly thought it would take a bit of time before we started seeing large load tests, but we had paid users going straight for Hero like running 40, 50,000 browsers at once and you can imagine that's around a thousand load generators. So I think, I mean it's good for us, it's good for business of course. But yeah, that was maybe the.

Exciting thing about it because we quickly had to make sure that we could scale to that as a frequent thing, not as a rare thing, which was, I'd have to say more rare to want to launch a thousand load generators at the protocol level. I think just usage and just keeping on top of the more malicious intent is probably the two things that are sort surprised me the most about Element.

So Tim, another question just came to my mind is does Element only support Chrome?

Yeah, at the moment it only supports Chrome because it is built on Puppeteer. We obviously interested in other browsers and I can tell a little war story there because we are owned by Tricentis as a company and we also have an integration with Tosca which is Tricentis test automation platform. A valuable lesson that I learned there when we first rolled out our engine in support of Tosca for example, was that many enterprise customers that were also interested in.

A browser level approach don't have the flexibility or can't even use Chrome. It's just simply not in their SOI or their application doesn't even support it. And quite often the the requirement was Internet Explorer for example. So at the moment with the Tosca side we use a completely different engine. It's a source, it's a completely different automation engine and that's talking about lower levels of concurrency, but it's to satisfy a different customer requirement. I of course would love to just be able to support all the browsers. But it really comes down to what's available.

In the market in terms of tooling, what's kind of like a viable approach. So we always have our eye on these things, but unfortunately for things like Internet Explorer, you kind of have your hands tied there in what you can do in more of an open source route.

Okay, Tim, before we go, are there any parting words of wisdom you'd like to leave us?

No, I just. I think I really want to encourage more people into the load testing space. I think it's flood is really about removing barriers to creating and executing tests. In terms of roadmap, we have a really strong focus on analytics this year, so you can expect to see us expand in that space because we think we've got the execution stuff down pat now. We're certainly providing lots of options in the creation side of things, and so you can see a concerted effort from us in revamping how we do analytics as well. That's coming from the flood space and that's all I have. Thanks very much for inviting me.

Thank you Tim, for your performance testing awesomeness. Links to everything and value we covered in this episode. Head on over to testtalks.com254 and while you're there, make sure to click on that. Sign up for a free trial link under the exclusive sponsor section to learn all about Sauce Labs awesome products and services. So that's it for this episode of Test Talks. I'm Joe. My mission is to help you succeed creating automation awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Talks podcast. Head on over to www.testtalks.com for full show notes, amazing blog articles and other automation awesomeness.

Sam.