Performance Strategies for the People with Corey Bailey

Five, four, three, two, one.

Drop that.

Welcome to the Test Guild Performance Podcast where we get together to learn more about performance testing and site reliability with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Performance Podcast. Today we'll be talking with Corey Bailey, all about a host of performance testing and performance strategy topics. Corey is a principal consultant with more than 15 years of experience in IT, focusing on performance testing, tuning, automation and team building. He specializes in a bunch of things like Performance Testing, LoadRunner, JMeter, Gatling, UFT Performance Center, New relic, and a whole bunch more. We have a lot to talk about. You don't want to miss this episode. Listen up.

Let's face it, performance testing is tough. Traditional load testing tools create scripts that are bogged down in data. It's hard to read and require a lot of work, even for simple playback. Load Ninja cuts out the dirty work by using AI to inspect and debug your code almost instantly. No coding necessary. No changing of hands or relays between teams either. Give it a shot. It's free and easy to try. Head on over to testguild.com smartbear and click on the link to learn more.

Hey, Corey. Welcome to the Guild.

Hey, thanks for having me on.

It's great to have you on the show. Before we get into it, is there anything I missed in your bio that you want to add? Tell us a little bit more about yourself?

No, I've been doing performance testing for a long time. Kind of meandered into SRE on occasion, but I really, really enjoy the performance engineering aspect of things because you really get to know how the application works.

So there's a full name of this podcast, the Performance and SRE Podcast. So just curious to know, do you see SRE as a growing trend that more performance engineers are getting into?

Definitely. I think performance engineers have a lot of knowledge about how the systems work and kind of an operations mindset that maybe devs didn't have. And so I think as dev and ops have started to combine, people have started using performance engineers as, all right, you can. You already have the skills necessary to evaluate and monitor different applications. And so I think there's a general trend to kind of merge those professions.

I think a lot of things that people struggle with nowadays, especially as people shift left a bunch of their activities, is coming up with a strategy. And I think you have a unique experience around coming up with a performance strategy. So I thought maybe we'd dive into a few areas of that. The first one is you send me an outline ahead of time of an automation strategy or things to talk about. And I found the first point really interesting on training transparency and visualization of production performance above all. And back in the day that's something I think I neglected as a performance engineer, but it seems like a growing trend. So could you just talk a little bit about what do you mean by transparency and visualization of production performance?

Back in the day we would do performance testing and there would be. That was prior to you going to production and then once it was in prod, generally it was a monitoring group would sort of acknowledge, take a look at, make sure that things aren't down. Those are generally pings, right? Is the app up? Over time the different tools or diagnostic tools have gotten better so that we can run that all the time and we can also see the intrinsic many layers to a particular application. It's more useful to have like if you were not going to have a performance engineering group, it's more useful to be able to see what the actual performance of your application is. So real user metrics is more important than testing because you have to know if your app is up or down. There are things in production that are just different than how you thought they might occur in a lower environment.

So Corey, another thing that's changed since I started is that a lot of the logic now resides within a JavaScript front end on a client machine. So to get a real user metric, it's kind of difficult. Do you use a real browser in production or what do you mean by how do you get it like a real user front end metric?

Well, so there's a couple of different solutions. So the two that are probably most used are either a synthetic monitor which tracks what that. Well, that's not a real user metric, but there's JavaScript libraries that you basically attach to the front end. So most of the big RPM frameworks, they have something that you snippet or something that's embedded within the application. Kinesis has JavaScript libraries where basically you can track whatever is going on on the browser side, log that information on the client and then send it to the third party. Either that's something like New Relic or Dynatrace, or if you're using something like Kinesis, you can send those metrics to yourself and aggregate them elsewhere. So again, you're tracking what real users are doing. There's a host of UX information and just how users are actually using your system as opposed to here's how we think they're going to Use it.

So I guess a lot of things also is when you're in production, you're doing real user metrics. Is where do you focus in on. Is it just certain areas or do you try to focus in on actual workflows or user journeys?

The way you start off with is essentially at first clicks and then you move into those user flows and then what is my how long does it take to do a checkout as opposed to how long does it take for a page to load? That's a lot more valuable because maybe your end user, you know, you don't want. People don't want large lines if they're using an application in a store. So if you're able to track that, you can show immediate value to improving performance of the application. So it creates a virtuous cycle.

So as you're monitoring real user metrics, are there any APIs you can tap into to get any sort of statistics or data?

Most of the APM products or logging. So if your front end is using APIs in the back end, you can both monitor those performance metrics from the APM or combine it with the user metrics. So you can, you can create that user journey of the browser. Took X amount of milliseconds, the API took as many milliseconds.

Combine those into your workflow, which is here is the end to end flow and where are your bottlenecks?

I guess as you're monitoring in production as well is a lot of people forget this step, but it seems like the obvious what are they measuring? They can measure so many things. So any tips for creating like sle's or slos.

So the sla's and slos, sometimes they're for third parties, sometimes they're internal. To say, hey, here's if we were running a test, this is where we would say that the test failed. And so because we're in production, this is kind of okay. Production is failing at this point, so we need to act on that. It's really important to at least have a threshold where like, hey, this is the point of view where we think it's unacceptable and we're treating it like a production defect. So the developers would come on board and either do an immediate fix or depending on the severity, put it in the backlog and work on it soon.

Now, Kurt, I spoke to a few people and this may seem wild, it sounds wild to me, but I haven't done real performance in a while. And that is they monitor production and they somehow are able to create what real users are doing and then take that information and then generate workflows that they then use for testing their application. Do you do anything like that? Is that something that's realistic?

It depends on the kind of app. So there are different frameworks or basically different streams where you can stream essentially the real what the users are doing and send it to a secondary system, sort of like a blue green. And so if you're able to set up your application in such a way and it typically requires cloud native or it's much easier, that's ideal because you can make a change in that, in that secondary environment, see how functional and performance is impacted and then if it's good, switch it over. You can also, you know, there are ways to query the various web server logs and essentially replay them. So there's a lot of data harvesting available out there.

Another thing I've seen is this is not just performance, it's like security testing as well. And other activities become a lot of times a bottleneck for teams in that you have like a team that's like we own cio, we own performance or we own security. How do you make it so you can make teams be able to interact with performance testing and security testing, things like that. Unlike a self service type or model.

Those teams performance especially you have to create frameworks or different repos where other teams can use those repos and kind of bootstrap and be able to create their own performance testing or security tests if it's just centralized. Unless it's very hard to not be the organizational bottleneck at that point, it's much safer to deploy very often in a continuous delivery model than it is to have this long delay in testing. So it's really important that the different team or the central organization allows the outside teams to self service. One of the ways we do that has been creating different frameworks for different sort of app types. So we might have developers who are focused on java or in APIs versus another team might be using Node and do browser development. So for those groups we kind of set up different applications and different repos and different styles of testing where they can use their existing knowledge as much as possible. But we've kind of put the training wheels on as far as the performance testing goes.

I like that approach, but I'm thinking for a large organization, how do you help with training? Like I think you list out like seven or eight different options. So do you actually have like eight different frameworks for performance testing based on say I like Java, so use Grinder or I like Python, so use Locus IO.

It kind of depends. So this model sort of requires more of a consultative engagement model where like, hey, this team is for open source. I'd probably go to JMETER right away. But as you learn what different teams are using, you can slowly add different kinds of frameworks and that's typically been the case. So if you're in a Windows setting, you might want to focus more on loadrunner or neosys or something like that. But generally speaking, you have to grow that kind of those different frameworks over time. There's no single way to do that.

So I was speaking to James Pulley a few weeks ago and he says Open source is great, but he brought up a point that a lot of these just focus in on creating stress or load. They don't necessarily have monitoring built into it and analysis pieces built into it. So if you work for a large enterprise, do you still think there's a place for commercial tools or do you think most companies now focus in on open source?

Oh, definitely.

I think the constraint with closed source systems tends to be it creates an organizational bottleneck. It's not that the tool is, I mean, I love loadrunner. If there was a Java version of loadrunner, I think it would have gone, it would be more successful than it is now. So essentially something like loadrunner or neosys, they're fantastic tools. Organizationally, it can become a bottleneck because everyone has to get a license. Here's what the right amount of users are for a particular external, external team. It can definitely be overcome and there's a lot of, you know, it's really nice to have the monitoring and the testing combined, but you know, especially in a continuous delivery model, it can be a little bit of overhead. That being said, like most of the open source stuff that we've done is just you draw in the monitoring some other way. So it's just not embedded as one. So it definitely takes a lot of. It takes some work to do that. And so one can make an argument that you're paying as much in resources doing the work as buying the off the shelf. But it allows the teams to self serve. I think in a lot easier way.

Maybe I'm being a snob. It's such a specialized skill. How is it when you make it more democratic and everyone can have a go at it? Is it effective?

Yes and no. It depends. I should say so. Giving a kid a Gatling gun is not always the best strategy. Again, it kind of requires an assessment of who's going to be running These tests, how much do they know their app? Are they just, they just want someone else to do the work, then it might not be right for them to do that work. On the flip side, a lot of developers are going to know their app better than you are. And again, if you have those APM tools, the transparency, the visualization of, of how that app is working, with some consultation on what a test should look like and what is good and what is bad. I think it's pretty easy for the teams to pick up.

So you do have a few things listed under browser that I haven't seen before. So I'm just curious to get what you mean. One is Puppeteer using Crane IO for scaling. How is that used in the teams you've worked with?

So Puppeteer is a Chrome framework for test automation. It's used for a variety of uses. One of them is Testing, but it uses a headless browser. Crane IO is just a tool I was playing with that uses Lambda so it can scale up as many browsers as you want. Lighthouse is also another framework that is done by Google. So in your dev tools within Chrome you can run an audit. So it'll audit your performance and number of other things. So you can use both Puppeteer and Lighthouse to automate sort of an audit of how performant your app is. The great thing about Lighthouse is it gives you essentially a grade. So there's pros and cons to Google kind of coming up with that grade. But I think everyone wants to be high on a Google search list and Google is now using these kind of metrics to rank you in your searches. So I think using that as a model is fantastic. Plus you can add this stuff to a pipeline, run three or four tests and you know on the front end how performant it is.

So you do have Appium listed, which I wasn't expected to see in a performance thing of application under test needs. So do you actually use Appium for performance testing?

Performance testing mobile apps can be tricky for a number of reasons. Some of them are offline, meaning that you couldn't open them with a browser. Even if it was, it wouldn't work. And to do that sort of scripting requires network, proxy and then kind of understanding if those, when those offline apps are. It's very resource intensive from a, from a, from a person perspective. So what we do with Appium is really kind of record those network or record those. The UI performance, not that, not the server performance. So this is more like the browser performance we were talking about before. So it's is the front end responsive and then have a separate test for the back end. So if the mobile app is connecting to APIs you would have separate API performance tests. But this is again to make sure that the developers haven't done something which is slowing down the device. You can also use the developer tools as well. So if you're using Android anyway, so you can connect, you can get the various the front end metrics.

Most of the things we talked about seems to be more front end performance related. How about backend performance? When you do a performance testing do you focus in on any performance backend type of activities and what tools do you use?

Quite honestly, most of the work I've done has been backend. It's APIs, stuff like that. So typically I've used LoadRunner, JMeter, Gatling or K6 if it's an HTTP application. Any one of those is fine.

Especially with AWS and Azure there are custom frameworks you need to test. You can't just do or it's more difficult to just pick up a performance testing tool and do that. But Java is very portable and it's pretty easy to import into JMETER and create a custom test to test sqs or I want to test Kinesis streams or something like that for instance. The tricky part about that is those APIs tend to be very granular and so again you need to go back to the what does that end to end workflow look like to make sure that you understand how often that API is going to be called and then understand how the front end is it calling 20 APIs trying to do it concurrently, which it can't. So you really have to know the application end to end to know how to test that API to begin with.

So Corey, I'm just seeing like a lot of parallels in performance testing that I saw with automation testing. It just seems like performance testing is kind of lagging behind. And one of them was what I loved about tools like loadrunner or QTP and ALM was they were all integrated. So I ran a test, I have edited everything in one place. So it seems like all these open source solutions there's a need to kind of get all this data and analyze it. Any solutions for people for like a centralized kind of performance testing data hub for analysis for performance engineers.

If you have lots of teams and they're all running their own tests, one of the things that is missing that you got from loadrunner was here's performance center, right? So here's what everyone's doing but most of these tools you can use something like if you're Grafana or influxdb which are semi open source databases, which basically you can stream your metrics there. So it helps visualize real time what's going on. If you have something like datadog or Splunk or elasticsearch you can do the same thing. So what it allows you to do is stream metrics from performance tests from all over. Essentially have to tag the performance tests when you set up those various jobs. But it allows you to see a centralized view of what performance testing is going on and where and how. Like we can we tag based on here's the app, here's the performance test type, who's running the job, what was the change? So basically it still usually requires some ask because you know someone might just be playing around with performance test, but it allows you to see from a high level what everyone's doing and how often people are actually doing performance tests.

Awesome. So another big no no when I started this is 20 years ago was don't mess with production. If you do something production, make sure it's non destructive, but you actually have listed out experiments in production. Can you talk a little bit about some experiments folks can do in production that gives them performance data? That's really helpful.

Well, so what we talked about at the beginning was.

How do we make sure that the build is performed in production before a lot of people see it? One of the ways to do that is that blue green environment where you make a canary build, meaning that some of the traffic starts going to that new architecture, the new code, the new infrastructure, the new whatever, it starts hitting that particular application. And so you can evaluate it live, it's real people and then create some sort of threshold where you decide this is acceptable or not acceptable and either turn it back off or flip everyone over. Another way to do it like some of the. It may be more difficult for legacy apps, but if you can set up certain kinds of users like either a subset like a particular Office or particular, like 10% of the users, they get the new code. So again this allows you to find out if there's any functional defects. If you see anything performance wise, you know that all right, this code is bad, you need to revert it. And then probably really importantly is, you know, it's better to deploy code more often rather than sort of of wait to have these monster releases. Because if you're having, if you're doing a release once a quarter, you know there's going to be a ton of defects because it's just a lot of code has built up and you haven't tested it where you don't know the impact in production until it goes there. So it's better to do small changes quickly than to wait to do it once a quarter, once a year.

Oh, amen to that, that's for sure.

Another thing you have listed here that it's a term I heard recently. Well, for the first time, maybe the beginning of last year was chaos engineering. Is this something that you see as a growing practice and how is that helping you with your performance?

Well, it's strange. So like this was something we did constantly 15, 20 years ago because people did have these. There's this long lag time in order to get essentially new machines, whereas now you can procure them in a virtual way very quickly. So you did. You had these really long times to run performance tests and part of that was always all right, what happens if one of the servers fail? And so we did this all the time before. I think as we got into a more continuous performance testing, it kind of went away a little bit and then has kind of come back with the as SREs have taken over because as we've started delivering code more often, we don't have that amount of time. So I think it behooves organizations to set aside time to to do that, which is what happens if particular node goes down. If one of your dependencies no longer responds, you need actual traffic to do that. So performance engineers have both experience generating that traffic and also reviewing how it affects because it's really interesting to see what the app does. You have a model in your head. Well, if you this particular API goes down or this database goes down, I know the app will do this, but in real life that may not happen. And when it happens in production now you know what to do.

Okay, Corey, before we go, is there one piece of actionable advice you could give to someone to help them with their performance testing efforts and what's the best way to find or contact you?

Again, I think the most important thing is to know how the app is actually used. Estimating what someone is going to do or thinking how it's going to work is not as useful as actually having. So it's super important that you have metrics on your app to know how it's actually being used. You can't model your tests, you can't model production unless you know that performance has really evolved over time. As long as I think it's super important. But just as organizations come less centralized, you have to give the tools to the people. If you want to get in touch with me, I'm on LinkedIn and Twitter with the handle Citizen Crane. I don't post super often but like to be in touch with people.

Thank you Corey for your performance testing awesomeness. For links to everything of value we covered in this episode, head on over to testguild.com P24. And while you're there, make sure to click on the Try Today link under the Exclusive Sponsor section to learn all about SmartBear's awesome software load Ninja. So that's it for this episode of the Test Guild Performance and Site Reliability Podcast. I'm Joe and my mission is to help you succeed with creating end to end full stack automation awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Performance Podcast. Head on over to testguild.com for full show notes, amazing blog articles and online testing conferences. Make sure to subscribe to join the Guild and continue your testing journey. This has been a Joe Calantonio production.