227: Fuel Your Performance Testing Using Loadster with Andy Hawkes

Hit it.

Welcome to the Test Talks podcast, the place to go to geek out on software testing. And now your host whose mission is to help you succeed with test automation, Joe Colantonio.

Hey, it's Joe. And welcome to another episode of Test Talks. In this episode, you're going to discover how to fuel up your performance testing efforts. And as we test talk with Andy Hawks. Andy is the creator of a new performance test tool called Loadster. So listen up and discover how Loadster can help you lower the barrier of entry for newbie performance test engineers and help find performance risk in your application before your customers do. I actually met Andy at a conference, had nothing to do with testing whatsoever in Las Vegas and during lunch happened to talk to him at random and he said he created a load testing performance tool. I was like, oh my gosh, what are the odds? Funny, because I was in Vegas, I'm like, I have to get you on the show to talk all about it. So that's what we did. Hope you enjoy it. Let me know what you think. Check it out. And actually, before we get into it, I also want to mention that I will be doing a Perf Guild in 2019, dedicated 100% to performance testing. I did do one in 2017. I didn't do one this year, but I will do one next year. It's probably going to be in April, so. So to find out more about that, check out perfguild.com.

Test talks is sponsored by the fantastic folks at Sauce Labs, the cloud based automated testing platform that eliminates the need to maintain your own selenium grid and test infrastructure. Try it for free today. Visit testtalks.com and click on the sign up now link under the homepage sponsor section.

Hey Andy. Welcome to Test Talks.

Hey Joe. Good to be here.

Awesome. It's great to have you on the show today. Today I'd like to talk about all your experience performance testing, but more specifically about your tool Loadster, which I think you created. So I want to get a little background behind that. But before we get into it, could you just tell us a little bit more about yourself?

Yeah, sure. So I've been in software industry for about 18 years now. Most of the time I've been doing software development work, but there was also a few years in there where my title was principal test engineer, focusing especially on load testing and performance testing at a pretty big company. And then since then, it's kind of been a recurring theme for a lot of my employers and clients over the years. There's often things that need to be load tested or performance issues. It seems like pretty much every app has them and so it's definitely been something that came up a lot and that's what led to me creating Loadster a few years back. It was a side project for a while and since then has become my main thing.

Well, that's awesome. I'm just curious to know why do you think it became your main thing? Is there more of a demand, you think for performance testing now or is it just that folks that are abandoning those older time tested tools we used to use back in the day and looking for like more modern type testing, performance testing tools?

Yeah, I think it's probably more the second thing. I don't know how the overall demand has, has grown or shrunk. There's still plenty of demand obviously and very much still a need for it. But I think yeah, there were a lot of like the older tools. Some of the problems with those at the time were number one cost. So only the big enterprises could really afford some of those tools. And what ended up happening was there was like this ivory tower situation, at least at the place where I was, and I was actually like one of the guys sitting in the ivory tower and acting as sort of a gatekeeper to all these projects. You know, it was kind of like the worst excesses of like the waterfall model. So, so like, you know, these teams would have like a tight deadline and they'd be trying to get their, their, their release out and then one of the last gatekeepers was us. And it was like the performance team and everybody kind of dreaded going through that because like they usually weren't prepared and it was usually late in the development cycle. So if there were issues, it was like there was always this big showdown it seemed like. And so that kind of gave me the sense that, you know, something in the situation is kind of broken. But it was a number of years after that that I actually got going seriously on Loadster. I had moved on to smaller companies, smaller medium sized teams and several times we had tried to like get budget for some of these tools and it was just really hard at some of those companies to do that. Also the open source tools were not up to the level that they're at now, so it was just really difficult at that time. So yeah, that's kind of a long answer, but I think it's more the second. I think there's been a change in the industry where some of these big expensive tools that cost as much as a house are kind of like Giving way to some of the more affordable tools as well as open source.

Awesome. And so Andy, a lot of times folks focus just on the cost and how do you decide on what performance tool to use and why, if there's a lot of options for open source and there's other vendor tools. So where do you place Loadster? Like when would someone choose a tool like Loadster over say, a pure open source type solution?

Yeah, yeah. So JMeter is a great tool and some of the other ones, Gatling's really good. There's a lot of good, very capable open source tools. What Loadster tries to do is make make you the most productive performance tester you can be. So if you're willing to trade a little bit of money for potentially a lot of time savings, then it makes sense to go with Loadster. And that's the feedback we get from a lot of customers as well.

Awesome. So Andy, usually when someone develops their own tool, they're usually scratching their own itch. So one of the itches sounded like it was cost value, but also some features. So you mentioned your customers mentioned you have some features that help them do performance testing better and quicker. So what are some of those features?

Part of it is a really easy to come up to speed interface. It's a graphical interface that you can drop down to JavaScript code when you need to, particularly if you need to do any validation or capturing of a response. You can put in a JavaScript function. But it's entirely possible and easy in fact to create a script without really having to know or having to do any coding. So it's kind of a good fit for a lot of people who maybe they're just getting into automation testing or they don't come from primarily a development background to create scripts and get going, get up to speed.

So Andy, do you know if there are any common misunderstandings you always find people have about performance testing you think is a blocker for most folks?

Yeah, I would say one of the challenges a lot of people have that that often just results in them not even getting started on it is feeling like you have to performance test all the things all the time and able to have in order to have like a valuable outcome. I think what it's really about is reducing the risk of a production failure. And even if you can run a day's worth of tests and reduce the risk by 50%, that's a pretty valuable business outcome in a lot of cases. And in fact it's probably impossible to reduce the risk 100% of a production failure, we can get pretty close, but not 100%. So I think one of the big impediments to people, especially as they're just starting out, or maybe they're evaluating whether they should do any load and performance testing is just that.

I definitely agree. I've been noticing some people using automated tools to help them with their exploratory testing. And I had thought maybe this is something performance testing could help as well. Like really early in the lifecycle, someone's trying to just make sure the code is optimal or the architecture they think they're going to go with is working. Is that someplace you think performance testing could help?

Yeah, absolutely. I think it makes a lot of sense to get started on that earlier in the development cycle whenever possible. I mean, you don't have to shift left and load test all the things all the time. But it's definitely exploratory. Testing can, can pay off in a big way, save you a lot of time down the road.

I noticed another thing a lot of people get hung up with is load testing or stress testing. They don't know what it is. So what's your definition of what is load testing?

So my definition of load testing is anytime you test a system to see how it behaves under load, and typically with web apps and websites, that means with lots of concurrent users or lots of concurrent requests. Load testing is kind of a general term, I would say, like stress testing is a more specific term where you, you want to see how the system behaves when it's put under stress. Whether, you know, you get data corruption or whether it crashes or, you know, if it bogs down or it crashes, does it come back up when the stress goes back down?

That's what I would classify as stress testing.

Do you have any recommendations on how to create a realistic load test? I know back in the day I was the only guy doing performance testing and sometimes I get a God complex, like how shall it handle 10,000 users with no think time? Ha ha. And then cause all crazy chaos in the office because people think that their application isn't working. But it just was totally unrealistic what I was doing. Any thoughts around how to make more realistic scenarios?

Yeah. So the test is only as valuable as the assumptions that went into it. So if you start off with incorrect assumptions about user behavior or incorrect assumptions about the amount of load that you're going to get in production, then you're not going to get a very valuable test. So I think one of the challenges is getting a good answer from the business side of things, on how many concurrent users you need to test or how many transactions you need to test, what's the worst case scenario you need to test for? And then working backwards.

So you mentioned a lot of times requirements are hard to get. Is there anything built into your tool that you thought, hey, I'm going to add this to help people, guide them along. Is there some sort of life cycle you recommend when using a tool like Loadster?

Not really.

Our tool treats everything in terms of concurrent users. And what a concurrent user is is a simulated single, basically a human user going through a transactional flow on your site or API. Basically, like what you need to do on the requirements side is figure out, let's say you have a requirement that you have to, you know, process 10,000 shipments in an hour or something like that. Basically you figure out how long each shipment takes as far as a user flow and then divide that by an hour to get the number of concurrent users that you'll need to peek at.

Awesome. And I think we talked about this. We actually met at a conference not related to testing, which was kind of wild. And I asked you if the tool had any sort of monitoring software built into it. Does it? And if it doesn't, like, what sort of metrics do you like to go by when you're running a performance test? And do you have tools that integrate with Loadster to get that for you?

So at this time, Loadster doesn't have any server side monitoring. It's all from, from the client's perspective. So the main things that we collect are response times, obviously N10 response times for every request and you can see a breakdown of the response times and also throughput metrics. So there's bytes per second, bits per second, upload and download, there's total number of transactions throughout the test, there's transactions per second. So those are kind of the main stats. We're looking at doing some partnerships and integrations probably in the next few months here with some of the APM providers.

Awesome. And a lot of times also that I see people struggle with is not emulating a network or where the users are actually coming from. So with the tool like Loadster, I believe it's in the cloud. So are people able to know how response times might be for. From China compared to, I don't know, London or something like that?

Yeah, definitely. So Loadster Cloud, which is our on demand cloud testing side of things, has 15 different cloud regions. They're basically all the public AWS regions. And you can distribute the load between those regions however you want. So it definitely makes sense. Like if you have like a global user base, it would make a lot of sense to choose several regions and then compare the response times from those regions. Basically because the Internet pipes are so huge, you don't really get bottlenecks that way, but you'll definitely see a difference in terms of latency. Then another thing you can do is you can simulate a slower connection. So in your Loadster scenarios you can say, well, okay, I want this group of virtual users to simulate like a DSL line or like a cable connection or something like that, so you can see the effects of slower networks.

Cool. So also, what kind of reporting is put together from the tool? Any special things in your reports that you think makes it easier for folks to understand what actually happened during the test?

Yeah, so the reporting, the reports are generated for you typically within seconds of a test wrapping up, there's a number of different charts and tables. Some of the charts are like average response time by URL or by page if it's a web app, which that's probably one of the most valuable ones because you can see as you add more load, you can see if there's a slowdown. There's also all the throughput charts that I mentioned earlier. There's bits per second, bytes per second, upload download, there's the total amount of data transfer during the test. There's total amount of transactions, which includes hits, pages, errors, if there were any HTTP errors or validation errors or something like that, total number of virtual user iterations, which is valuable especially if your requirements are in the form of like, you know, 10,000 transactions per minute or per hour or whatever. And then there's also some stats on Loadster's engines to make sure that you didn't have an overloaded engine or anything that could have possibly affected the validity of the test results.

Andy, how about protocols? I know a lot of times folks aren't only concerned about the back end, but they want to know what, what it's like, especially with JavaScript, how it's rendering on the front end, and also APIs. So is someone able to create some sort of workflow where they have all these different types of users and protocols running at the same time?

So it's all HTTP. You can definitely do like lots of different user behaviors at the same time, but as long as it's HTTP, we don't do any like TCP or any low level stuff like that or any of the streaming video protocols.

Also I know a big Trend Here is DevOps and shifting left testing earlier in the life cycle. So have you seen any companies or folks actually try to plug performance testing into their like two to three week sprints? And is that a practice you would recommend or what would you suggest for folks that want to get performance testing earlier in the development life cycle?

Actually, I'm not really seeing too much of people who performance test as part of every sprint as like a manual task, but I am seeing a lot of interest, interest in wiring it into like a CD pipeline to where every build gets a set of performance tests run against it and you can potentially even fail the build if there's a regression or something. I'm seeing a lot of that and I think that fits nicely into like the agile methodologies.

Awesome. So does Loadster actually have a Jenkins plugin or can you call it from the command line or something like that?

There's not a plugin, but there's a simple hook where you can call it with curl or something. Basically you just make an HTTPOST to a trigger URL and then that'll kick off a test and then we've got a couple of customers doing kind of like a private beta with a lot of guidance from myself on setting up some of the things to fail to build if there's actually a regression.

Cool. I think also you just came up with Loadster 4. What's included in Loadster 4 and do you have like a roadmap on where. I know you probably can't commit to certain things, but things you'd like to see in the product in the next year?

Yeah. So Loadster 4 came out earlier this year. Basically it's a big architectural change. With Loadster 3 we can only go up to maybe 50,000 or so concurrent virtual users. And with Loadster 4, because it's been re architected, more of the processing and metrics gathering has been moved out to the individual instances, the different engines, we can probably get up to a million or more. With Loadster 4 there's also been like some UI changes and just streamlining a few things, but mostly it's been around that. As far as what's going on now, we're actually in the middle of rolling out Loadster for the web. That's entirely in the web browser. There's been more and more demand for that. Originally it totally made sense to have a desktop app, a native desktop app, because a decent percentage of our customers a couple years ago had in house test environments behind the firewall and they wanted that sort of thing. We're seeing less and less of that. There's really obviously, as everybody knows, been a huge shift to the cloud and so that makes less sense. And some people are in an environment where they don't even get to install desktop apps. Some of that, some of the larger corporate type jobs. And so having it on the web just makes a lot of sense. So that's kind of the release that's in the works right now. Definitely in the next few months. I would also like to get more integrations and more monitoring as far as like integrations with APM providers.

Now Andy, you did mention there's a free option. What are the plans? Is the free forever or is it capped at a certain amount of users? Just talk a little bit about like how people could get their hands on this and actually start using it.

Yeah, so the free plan is up to 50 concurrent users. You can run unlimited tests if it's self hosted. So if the traffic is originating from your local loadster, workbench or a self hosted engine and you can run up to 10 cloud tests a month as well on the free plan.

All right, Andy, I know besides the free model, I know this is not a promotion for you tool. This, you know I approached you for the interview, so this is not an endorsed interview. So just curious to know though, for folks that are interested in learning more, you did mention something about fuel. Are there any other paid options? What do they get you if someone wanted to really go all out with this particular tool?

Yeah. So Lodestar, there's actually two different pricing models that make sense for different people. There's the monthly plans and annual plans where there's unlimited testing. So you and your team can run an unlimited number of tests up to a certain number of virtual users as set by the plan.

Those are recurring plans. And then for a lot of people where load testing is more of like an occasional activity, we also have a fuel model which is basically like you buy credits and then the fuel represents virtual user hours and Those start at 99 bucks. So you can get a bundle of fuel and run, run a bunch of tests for a pretty, pretty small investment. That is if the free plan doesn't meet your needs. Also for listeners of your show, I'd like to offer 500 units of fuel for free. So if you try out Loadster and you'd like to get some free fuel, you can just email, actually you can email me directly andyoadsterperformance.com and I'll hook you up with 500 units of fuel that's 500 virtual user hours. And by the way, the fuel is down to the second. So like if, if you run a three minute test, that's only a 20th of a unit of fuel per virtual user.

Wow, that's awesome. Thank you so much. And I think that's a great offer. I love the name Fuel and load up on Fuel. That's a great, great branding, I think.

Yeah, I hope so. I mean, I'm not really like a branding expert or anything, but like, you know, Loadster kind of sounds like roadster. So we taken like a kind of a fast cars thing. So fuel sort of fit in there as well.

Very cool. Okay, Andy, before we go, is there one piece of actionable advice you can give someone to help improve their performance testing efforts and let us know the best way to find or contact you or learn more about Lodestar.

So I would say the one piece of advice would be it doesn't have to be perfect to be valuable. If you can reduce the risk of a production failure by 50%, totally go for it. Especially if it doesn't take very much investment. It's definitely one of those things where there's like diminishing returns, the more thorough you want to get. So you can get a decent amount of confidence for a pretty small investment of time and money. You can get more confidence for a little more investment and so on. It's definitely worth doing. Even if you can only increase your level of confidence in the app by 50%, 75%, that's still a big business value in most cases. So I guess, yeah, in a nutshell, my advice would be just do it. And yeah, you're welcome to reach out to me. I'm on Twitter. I don't really tweet a lot, but I'm Az Hawkes, a Z H a W K E S on Twitter. Or you can email me@andyosterperformance.com thank you.

Andy for your performance testing awesomeness. For links to everything of value we covered in this episode, head on over to testtalks.com.

And while you're there, make sure to click on that Sign up for a free trial link under the exclusive sponsor section to learn all about Sauce Labs awesome products and services. And if this episode has helped you in any way, why not share with a friend or coworker? Or even better rate it within itunes. The more people that rate and comment within itunes, it can help with the rankings, get more people involved and helping create automation awesomeness. So that's it for this episode of Test Talks. I'm Joe and my mission is to help you succeed with creating automation awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Talks podcast. Head on over to www.testtalks.com for full show notes, amazing blog articles and other automation awesomeness.