Simplified Performance Engineering with Stephen Townshend

Five, four, three, two, one.

Drop that.

Welcome to the Test Guild Performance Podcast, where we get together to learn more about performance testing and site reliability with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Performance Podcast. Today we'll be talking with Steven Townsend about simplifying performance and engineering. He's done a lot of different sessions around performance, so he has a lot to share. He's a performance specialist based in New Zealand and has a passion for getting the basics right, which he explores with his YouTube series Performance Time. So you don't want to miss this episode. Check it out.

Let's face it, performance testing is tough. Traditional load testing tools create scripts that are bogged down in data. It's hard to read and require a lot of work, even. Even for simple playback. Load Ninja cuts out the dirty work by using AI to inspect and debug your code almost instantly. No coding necessary. No changing of hands or relays between teams either. Give it a shot. It's free and easy to try. Head on over to testguild.com smartbear and click on the link to learn more.

Hey, Steven. Welcome to the Guild.

Hi, Joe. It's good to be here.

Awesome. Great to have you. As I mentioned in the intro, you have a lot of experience talking about performance testing, but can you tell us a little bit more about yourself that I may have missed in your intro?

Yeah. So, yeah, I've been specializing in performance testing engineering for the last 11 years now. It's the only thing I've ever done in my career, which is a blessing and a curse. And most of the time I work for consultancies, so. So I got to work with a lot of different customers and technologies and business domains. And in the last year and a half, I wanted to put my roots down, work for a place where I could build something lasting, build in some improvement. And so I'll be working for IAG Insurance, and it's going really well.

You said this is the only thing you've done. Your career. What got you into performance testing?

Well, see, I did computer science at university and I wanted to be a developer. When I got to my third year, I remember sitting in front of a computer lab with a bunch of people touching screens and trying to solve problems. And I just thought, I can't do this the rest of my life. So I actually auditioned for drama school to become a professional actor. And so I went away for three years and did that. And when I finished that, which was an incredible experience, I wanted to get back in it, but no one would hire me. But there was this one crazy Belgian stainscapers, he's another performance expert and he. I interviewed with him and he. I applied for this role of a junior performance engineer and he liked me and he hired me and it was just pure dumb luck, I guess. And I've thoroughly enjoyed it as a career so far.

Very cool. And one of the things that comes up over and over again when, when you're speaking is that you really want to focus on the basics, get the basics right. Why is that like one of the main points you hit on all the time? Do you find that people make performance testing over complicated?

Yes, there's a couple, yeah. I've seen quite a lot of situations where there's performance testing being done, but it's not necessarily providing a lot of value or is understandable by non technical people. And to me that's just a huge waste opportunity. But the other thing is that right now we develop software faster than ever. And I just think there's a real push to automate and to be fast and to try all these fancy new things. But there's no point in doing that until we get those basics right. Someone once said to me, there's no point in automating bad performance testing. And I think that's very true. And so that's why I think right now more than ever, getting the basics right is very important.

Awesome. So what do you think are some basics that folks need to have before they really stop putting. I'm going to put a load of 1,000 people on my system with all these different transactions. What do you think they need to know before they actually start doing something like that?

I think it starts with thinking about performance, not as a testing discipline, but as a risk mitigation exercise. So it's just about understanding you're about to implement some software, there's some business need. And so where is the risk? Which technical components in the system does the risk reside in? What would be the impact if there were performance issues? And then just thinking holistically around what you can do to prevent performance issues from happening or to mitigate that risk. I think that's the first step. It's having that connection back to some real business need, something that actually matters in the real world.

So how does a team know that? Do they speak to the product owner or is it understood like here's our main feature of our application. If that's down, then obviously that's a risk area.

The process that I picked up from so just a quick shout out. There's a guy called Neil Davies and another one called Richard Leake. I don't know if you've heard of either of them. They're incredible performance specialists based in New Zealand. And so I learned a lot of the way that I think around performance from them and the process that I the. The basic approach that I picked up from them is to do a risk assessment around performance at the start. And that involves looking at any implementation of any software, thinking about what are all the different components, what technologies do they use, Are they things like, are they new systems, are they bespoke, are we, are we buying them off the shelf and configuring them? How do they integrate together? What are the protocols, how difficult will they be to test doing a workload model, how much, what volume of load do we expect on each of these different systems? And what would be the impact if each one of them went down or was slow and using that kind of information to prioritize what needs to be, what risk needs to be mitigated first. So that's kind of the approach that I generally take for a traditional engagement. And it might be a little different if you were working in a complete agile way, but you would try and take that risk assessment approach to the user story level, that micro level.

So once folks know the risk areas that they want to performance test, I think you have a different approach once again than what people usually jump to. They usually jump to like a protocol level testing tool. But I believe in one of your recent presentations or an article that you're about to publish, you speak about UI automation to help you with performance testing. And to some people that sounds kind of controversial. So can you just tell me a little bit more about what you mean by UI automation? How does that fit into performance testing?

Yeah, I listened to your previous podcast with Mark and Scott and I think Mark said be careful. And I agree. I think the thing about UI automation is that there is a particular kind of system and situation where it thrives completely. So there's a particular document creation system that I need to perform test. The network traffic is absolutely diabolical. There is hundreds of kilobytes of data full of hundreds of dynamic IDs. All of them have to be correlated and played back in the following requests. Within the hundreds of kilobytes of payload, there are base 64 encoded components. Within those base 64 components are URL encoded components and there are dynamic IDs within all of those, which you have to unpack, extract and Then pack them up again later. Now it's theoretically possible to performance test that at the protocol level, but the maintenance cost required to keep that going is just astronomical. And so the simple solution that I've picked up is I'm using in my case, JMeter and the Selenium We WebDriver Sampler. So I'm spinning up 30 real Chrome browsers in headless mode and interacting with them at the UI level. And those browsers handle all of that dynamic traffic in the background. For me now it only works in that situation because the load that I need to apply is quite small. I'm creating 600 documents an hour, so 30 browsers is fine. If you, so if you, if you need to generate very large volumes of load, then UI automation is probably not the most cost effective way to do that. You, you can, and I, some of my colleagues are able to spin up many hundreds of browsers through that infrastructure they've built. And if you go to the cloud, of course it's always possible. But yeah, I think, I think it works. It shines completely when the load you need to apply is not that high. The protocol traffic that you need to simulate is very complex and the UI of that particular web or mobile app is not very complicated.

So Stephen, I guess what other people may trip up on is making UI automation reliable has been an issue. So do you have any tips on how do you make it more reliable when you do it at load like that?

Yeah, so that is a tricky thing. I don't have a single answer to it. I've just had to understand the application really well and know when particular issues are going to occur and make sure that the timeout period for waiting for elements is not too high. That's the best that I've been able to do in saying that the success rate for any transactions are running through. It's still in the high 90s, so it's not such a big deal. But I guess it's a depends on the application and the context.

This is actually the first time I heard about the WebDriver Sampler plugin. Is that something you use often with JMeter?

It is and it is a little outdated in terms of some of the functionality that Selenium can provide now, but it does the trick and it's available online just from the JMeter plugins website. So you need the WebDriver Sampler plugin and you also need to choose a driver for the browser that you want to simulate and obviously the browser itself. I've chosen Chrome because I'VE found that is the most reliable and easy and there's a lot of support for that. You can also use Firefox and other browsers if you wanted to.

Very cool. So also talking about the basics, JMeter has been around for a while and a lot of people find it confusing. So do you have any tips on how to make it, how to focus on just the essentials of JMeter? Maybe so people won't get sidetracked with other things that may throw them off.

I think the key is to not think, is to first of all you have to understand the basics of performance test scripting. You know what I mean? And so I think it was helpful for me that I had used two or three other tools first before I went onto JMeter. So if you're going directly into JMeter, that's fine, but you need to understand the basic structure of what a Lotus script looks like, understand the principles like parameterization and correlation and putting some validation in to make sure your tests are doing what they need to do. Naming conventions, all of that basic stuff around what a good load test looks like. And then be thinking about how can I use JMETER to achieve that as opposed to how can I use JMeter? Because you can do anything you want with JMeter, it's completely open ended. Maybe that's why it's confusing. It's because there's too much choice, too many possibilities.

Very cool. So besides using UI automation tools, performance testing, another kind of controversial topic is mocked environments. So any thoughts on whether someone should use mocked environments when they're doing performance testing or what are your thoughts around that?

It almost becomes a religious debate in some organizations.

As of maybe two years ago, I was completely of the opinion that you needed to have a fully integrated end to end environment.

I still believe that that is the way to get the best realism. Of course it is. But as time has gone on, the solutions that we performance test have become more and more complex.

And we deliver software faster and faster.

It gets to the point where you need to test the system, which depends on 39 other systems, and if any one of those systems goes down, you can't run a performance test that's just not going to fly. So if you're in that situation where you need to run performance tests every day or multiple times a day continually, then absolutely I think mocked environments are the way to go.

I think the caveat is that there is a subset of performance issues that occur relating to when a system calls another system and that second system is slow. So the Worst performance issue that I ever saw was an old. It used to be a mainframe system and it got migrated to a mainframe emulator and every call that system made to any external system, including the database, went through. We didn't know this at the time, a single threaded external integration point. And so we opened this system up to a cloud based repository of data. And some of these initially, some of these searches that it was doing were taking 25 seconds. And when you initiated a search in the system, it was doing eight of them in a row sometimes, which meant that the entire system would lock up for multiple minutes at a time. And every call it was trying to make, the database was also failing because it was waiting in a queue and the system would crash. You could reproduce it with just one user or two users. I think to avoid that kind of issue, there's a negative test you have to run where you force your mocks to be either very slow or to force a timeout and to see what happens under load. So that would be an important caveat.

Awesome. So what are some other benefits you think? More control over your environment, I guess would be another one.

Yeah, more control.

It's cheaper, potentially a lot cheaper to have your environments going.

Maybe more consistency in the results that you're getting. Especially if you've got downstream systems which. There are systems I test which behave differently at lunchtime than the end of the day, maybe because of the extra network traffic from people browsing Amazon or whatever.

So a popular term as you know, is a Shift left. Everyone's talking about Shift Left but performance testing. More and more people are talking about the Shift Right, not just monitoring, but there's other things you could do with Shift Right. Any thoughts on. In your career or your situations, have you done anything in the production environment more than what you may have done earlier in your career?

So I mean full transparency. I have not been able to do as much as I'd like to in the Shift right space. The one thing which has really helped me is the move in our organization towards cloud because in the cloud world I seem to find it easier to get access to production data read only access, obviously, and only performance related data. We do a lot of work at the moment in Microsoft Azure. Having access to log analytics and application insights in production has completely revolutionized the way that the involvement and the level of value I can add for performance testing, even just workload modeling, I can just today, right now I can go to production, find out how many insurance quotes are being called per hour over the last 12 months, break it down by per hour. What kind of products are they? Multi risk or single risk quotes. That level of workload detail I can immediately get access to and I can also, after a new release, they maybe release a new product or service. I can immediately tell them what the impact is on the load, how it's performing and to feed that back and say, we ran some performance tests, these are the results we saw and it's actually happening the way we expected in production, or it's not. And then we start asking questions. That level of monitoring and I guess there's a big move right now towards APM and having that access as a performance engineer and I actually think that all developers and all testers should have access to monitor production. Of course, I mean, it makes sense. If you really want to go DevOps, then you have to get away from this idea that there's a special operations team and they are the only people that have access. That kills DevOps completely and it also kills any kind of accountability for the performance of whatever you're implementing or building. So from my perspective, sort of APM and monitoring and production is the only shift. Right. Activity that I've actively been involved in at this point in time. What I would like to move towards is canarying, if possible of course, that, you know, releasing to a subset of users. That depends on the technology and what your system is. There's. I don't know what your thoughts are on Synthetics. I'm not sold. So essentially, from what I understand, you're essentially running very simple tests in production against endpoints, constantly in production. And if there's an issue, you get eluted. I don't know.

Why are you not sold? Because I'm older. So back in the day, in order to do this, I used a product by was Mercury. Back then it was called Business Process something. And what you would do is you run like a winrunner script or a QTP script. You load it up on a desktop computer, you would ship it out to a remote agent and you'd have them put them on the network and connect to the Internet and just run the test on a schedule. And then if you got like three thresholds in a row that caused a performance issue, you get alerted and it seems to be fairly, fairly slick. So that's what I think when I think Synthetics. Is that the same?

Yeah, that does sound pretty good. I think what I'm not sold on is the interpretation of the data. So if you were. Yeah, obviously if at any point's down or it's completely not responding. Absolutely do something about it. But if it's a little bit slower than it was before, but you're only capturing one sample every five minutes or I don't know, however often you set these things to. It's just being aware of.

How much confidence you can have given the sample size and how often you're polling and knowing how to interpret that data. Because I think it could very easily get to the point where people where you have some data in front of you and you think something's really, really bad is happening or there's a change, but maybe there isn't. So I don't know, I'm just yet to see it in action. But after what you said, maybe I'm sold. The other. Sorry. There was one more in the shift for ice space that I think is awesome would be rapid rollback. Of course that's application and technology dependent. But why not? If you can go live and then there's a problem and roll back in a minute, brilliant. Why not?

I completely agree and I've heard about this for a while, the canarying and the rapid rollback. I haven't actually met anyone that's worked at a company that's implemented this though I don't know why. Is this fairly new? Is this hard to do? Are people you think kind of. Have you seen companies actually doing the canaring and the rapid rollback in your experience?

No, I'll be honest, no, I've not seen that happening. I've heard about companies doing it in New Zealand as well, but it's always companies who develop their own software from the ground up. So they got complete control over the technology and everything. Whereas in an insurance company you tend to be buying off the shelf products very complex monoliths in many cases and it's just not necessarily possible.

What I love what you said earlier was everyone should be able to look at production. I think that's actually great feedback because that's how you learn. You delivered software, you did the best you could and went into production real world in the wild and like oh boy. That didn't go so well. And why did that not go well? Maybe when I develop the next feature I'll keep that in mind. Is that supposed be to something you would one way that you help collaboration with your delivery teams and your dev teams. Yeah.

I don't know whether you ever heard Andy Grabner talk about the ufo.

I have not heard him talk about that, no.

So this is going to the extreme. Basically they had a UFO above each of the teams of developers at Dynatrace, whenever they checked in code and it would get picked up if it failed any, including performance.

Testing, then basically a bunch of red lights would appear above that team.

Everyone would know which team was the one that released the code, which broke production or whatever it was. So I thought that was quite interesting. But yeah.

Having that shared, I mean, I don't know. I haven't done this yet, but absolutely. My plan is in the next year or two is to be having production metrics. And I guess my job as a performance engineer is to be making that data presenting in a way which is meaningful and which developers, testers and bas can all understand. And we can all talk about it together. Because it's one thing to say, hey, we've implemented Dynatrace or New Relic or whatever it is, but then you need to have skills and using that tool before you can understand the data. So it's, you have to have. It's a really important part of our job, I think, is analyzing data, understanding the impact to the business or to the, you know, to the business, and then being able to communicate that in a way which everyone can understand. It's.

It's a difficult thing to do.

Yeah, it's the ability to. You have to almost tell a story with the numbers rather than just throw up these charts.

Yes, exactly. It's all about telling a story. So, okay, let's say you do a random testing for a particular application and there's no performance issues and maybe it's for a big project. I think there's absolutely no point in building a 60 page document full of tables and charts and all this pretty colors. Who cares? No one cares. The story is that there are no defects or there is no degradation since the previous build. That's all that you need to say. With just enough evidence to back that up, why do anything else? And if there are issues, just report those issues.

What do we understand about the issue so far? Where's the investigation got to? What do we know about it? Nothing else matters. Who cares about the 279 transactions, which are fine? It's this 202 or 3, whichever problem that we should focus on reporting on.

Absolutely. I love giving executive summaries and just attaching all the other stuff. And then you always get those nerds that actually want to dive in. They can, but just keep it high level for everyone else. For sure.

I've been dabbling with the idea of a visual executive summary. I literally just have a simple diagram. Sometimes it's a Simple solution diagram. And I just color code the components which are bad or good with maybe a few bubbles above it saying this transaction slow or this has got no capacity, you need to scale it up or whatever, that kind of thing. It's gone down pretty well so far.

Very cool. You have another concept, do more simple stuff. Sure, do more simple stuff. What do you mean by do more simple stuff?

In the past we tried to the way we used to do performance testing, a lot of the time we tried to imagine performance risk is like rain falling from the sky and we would try and catch all the risk. So just before it hit the ground or before we went to production through these big complex end to end performance tests, that was the way that we used to operate or some people did in the past. Now there are so many more opportunities to catch risk at different levels. We can be involved in architecture to help make sure we design software which performs well in the first place. We can work with developers and collaborate to do things such as early component testing, single user tracing, building in performance, acceptance criteria to use a story so that we're thinking about performance from day one. And we can do a lot more in production, like we said, rapid rollback, canarying apm. And so because we are able to mitigate risk at so many different layers now.

The need to do big, complex end to end performance testing is slowly diminishing. And I don't think it will ever completely go away necessarily. But that massive, complex end to end performance test suite is something we need to move away from. It's too slow, it doesn't keep up with modern rapid delivery, it's too expensive and it just doesn't add enough value anymore. So I'm saying by do more simple stuff, I'm saying if you can do some early component testing, maybe get your developers to do some tracing using an APM tool or a network capture just to see what is the impact of whatever feature they developed. Have some more monitoring in production. You can reduce the scope of that end to end performance testing to something small and manageable and pragmatic. That's really what I'm saying there.

Okay, Stephen, before we go, is there one piece of actionable advice you could give to someone to help them with their performance testing or reliability testing efforts and let us know the best way to find or contact you.

I think it would be if you're a very beginner. Just pick up a tool and learn it and have someone to guide you to make sure that you're building performance tests in a way which will serve.

You in the future.

But when you do get to the point that you have mastered a tool, then start thinking about performance as a risk. Think about how where are performance issues going to occur and try and predict that in advance and focus your efforts in those areas. Then think a bit broader. Broader around maybe there are things that we can do which don't involve any testing at all that we can do to very simply eliminate performance risk or massively reduce it. Very simple example, there was a system that I did performance testing of once and the users had a choice. They could run a report on their client or the they could run a report on the server. And we identified some bottlenecks very early on and we just trained the users to generate the reports on the client. No performance testing required. That particular performance concern was eliminated immediately.

In terms of how to contact me, probably the easier I don't really do social media apart from LinkedIn, so LinkedIn Stephen Townsend I believe and it's Stephen with a ph and Townsend with with an sh Taunshined almost.

Thank you Steven for your performance testing awesomeness. For links to everything of value we covered in this episode, head on over to testguild.com pplikeperformance12 and while you're there, make sure to click on the Try it Today link under the Exclusive Sponsor section to learn about SmartBear's awesome performance testing solution. Load Ninja and if you haven't realized, I moved the weekly podcast from Monday to Tuesday, so Test Guild Performance Podcast is going to be released every Tuesday going forward rather than Monday. So that's it for this episode of the Test Guild Performance and Site Reliability Podcast. I'm Joe and my mission is to help you succeed with creating end to end full stack automation awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Performance Podcast. Head on over to test guild.com for full show notes, amazing blog articles and online testing conferences. Make sure to subscribe to join the Guild and continue your testing journey. This has been a Joe Calantonio production.