Signal and System Testing with Paul McLean

Five, four, three, two, one.

Drop that.

Welcome to the Test Guild Performance Podcast where we get together to learn more about performance testing and site reliability with your host, Joe Calantonio.

Here it is. Here it is. Here it is.

Hey, it's Joe. And welcome to another episode of the Test Guild Performance Podcast. Today we'll be talking with Paul McLean about performance testing around Daedalus testing. Email into performance testing, OIDC and Paul's unique approach to performance testing, which he likes to call signal and system performance testing and much, much more. Paul is an IT Performance Consultant at RPM Solutions and Test Data Services with over 30 years of performance testing experience, helping folks with their load stress, system performance and scalability efforts. He really knows this stuff. We're going to cover a lot of things that I even haven't heard about before. So you want to check it out. Listen up. Let's face it, performance testing is tough. Traditional load testing tools create scripts that are bogged down in data that's hard to read and require a lot of work, even for simple playback. Load Ninja cuts out the dirty work by using AI to inspect and debug your code almost instantly. No coding necessary. No changing of hands or relays between teams either. Give it a shot. It's free and easy to try. Head on over to testguild.com smartbear and click on the link to learn more.

Hey Paul, welcome to the show.

Thanks, Joe. It's so great to be here.

Awesome, awesome. Great to have you. Before we get into Paul, could you just tell us a bit more about yourself?

I'd love to. So I actually started my performance testing work 35 years ago. That sounds like a long time, right? I was in Sweden in 1985. 86. As part of my degree, I had to do real life engineering style testing. And I worked on an HVDC transmission test. So it was actually for Itaipu, which is a high hydroelectricity plant, supplies half the power to Sao Paulo in Brazil. And it was all about making sure that there was resilience in that power transmission. Because if the power transmission went down, there were big problems. And have you watched the Chernobyl series recently on tv?

You know, it's so funny. We just ordered it from Netflix. It's coming on its way, so we haven't watched it yet.

It is so good. Well, the control room in that series looks exactly like the control room was testing in. So there were a whole lot of tests we had to run Monday to Friday, but on the weekends they let me come in and run ad hoc Tests myself. Right. And it was so fun because it was just like that room. So when you watch it, you can think about that 1985, 86 time frame. In fact, they evacuated us from there when Chernobyl happened because we were near it. They thought it was a nuclear plant next to us. So how's that? So it's a place near and dear to me. And that's a great example of what not to do in a load test. Because they were actually doing load testing and they were trying a scenario and they stuffed up production, to put it mildly. So that's super interesting. So my background is signals and systems style testing and with reliability as its focus. And when I run a load test, I know the traditional thing is to ramp up and hold and you basically work out what your model is. But where I can and where a customer allows me, I play with some signals and systems. I'll have a bunch of sine waves, I add them together and then you have all sorts of ups and downs and you get a pattern. It's like a modulation sort of frequency. When you run a test and you do that and you look at your infrastructure, you see the same pattern and where the same patterns represented, you know, everything's awesome. But where you see distortions in that pattern, whether it's in iOS or utilization on a particular node or something like that, you know you're heading toward issues. It's amazing. So that's, that's a bit of background on myself. I've been doing load and performance testing in it since 1999. I started with X Runner, then moved to Loadrunner. And I use Loadrunner still every day actually.

It's actually the first time I ever heard signal and system performance testing. Could we go into a little more detail what that means?

Well, so my degree was before. Well, I got my degree in 1986. 86, 87. I can't remember which year I graduated. So the degree I got was in signals and systems and computer science. So it's a double degree. And I did testing of electronic circuits. I remember doing some tests to work out how a system would respond to a step change. When you want to figure out if something is going to work in electronics, you often give it a burst of energy. You put a step of energy into it and then you see how it responds. So it was all about response time. If you're testing an amplifier, for instance, for music, then it needs to be able to track the little signal that's going into it to make the signal bigger. And that is Actually hard to do, especially to do it in near real time. And there's an awful lot of electronics involved, so I was interested in that space. It's an engineering basis. So instead of worrying about the outcome too much, you worry about the risks and you worry about the process and the fidelity of what you're doing and what the system is doing.

So how do you apply that to, say, a web application? Do you just hit it with a bunch of transactions and then look at the patterns and look for anomalies?

Well, if you look at a typical web app, it's got visitors, it has a population of visitors visiting it. And let's imagine that they were visiting at 100,000 visits per hour. So a decent rate, you don't have that 100,000 spread out evenly, and they're not all doing the same thing. So the key thing that I build into any load test is the appropriate amount of variability on that load. So with loadrunner, there's lots of things you can do with loadrunner that are awesome that lets you build that variability. And you don't have to do a simple script that just does everything exactly the default way. That's in the training. You can, you can use Rendezvous to. So when I use a rendezvous, for instance, what a rendezvous is it? It's a place where you synchronize a whole lot of running scripts, even different ones for different business processes at a particular point, and then you release them. So my normal process for doing that is I have a rendezvous or many rendezvous through a script if necessary, and then after the rendezvous fires, I have a random think time. And if you had a random think time of 10 seconds after rendezvous, then all of the scripts that get to that point will then distribute that next step over that 10 seconds, which is very, very nice for deadlock detection and all sorts of things.

I've often done my load tests not in a development sense, but in a production reliability sense. Someone will have an issue with deadlocks that are occurring and causing havoc, and they can't reproduce them in a test environment. And I might get asked to come in and within a few days I can usually reproduce them. And I'll use my understanding of.

Variation over time to be able to do that just using some simple tools. But say Loadrunner, it's amazing what you can do. So your original question was, how do I apply it? So if you have a certain amount of activity over an hour.

Then what you'll find is that the busiest 10 seconds in that hour is about 2.1 times the average for that hour. And if you have a system that you're testing and you don't make sure that you've got that in there, you're missing some of the spikes and the peaks that can actually cause issues. So I'm one for doing a bit of statistical analysis to find out how long we should run our load test for or our endurance test for. Because if you run a particular test for one hour, you might have a 1 in 20 chance of seeing a catastrophic failure. If you ran the test for over a whole long weekend of 60 or 70 hours, you might see that catastrophic failure three times. Now if someone wants to run a full on test and put all the effort into scripting and you only run the test for an hour and you miss out on seeing the problem, that's bad. But if you get permission to run it for 70 hours over a long weekend, awesome.

So this is really cool, Paul. I think most people though do like very basic scenarios. They don't look for anomalies, they just try to prove a positive almost. So how do you create what sounds like a complex load profile? Is there an easy way to make these type of complex load profiles?

Well, I find it very easy because I've been doing it for so long. So basically you use statistics. So can you imagine a sine wave? Right. That's the opposite of ramp, up and hold. It's going up and down, up and down. And the up part is actually quite steep. That's very nice when you're running a load test to have that steepness. Because often a system doesn't come unstuck just by static load. It comes unstuck by a dramatic change in load. So the way I think about it is with a bridge, a lot of load testing has the concept of it's like a bridge and you get your traffic going over the bridge and if your bridge can't handle the traffic, it collapses. That's bad. Well, when a locomotive goes over a bridge, in fact, there's some great pictures of locomotive load tests on bridges where they'd actually line up 60 steam locomotives to test the bridge. The engineers have signed off and they've said, it's good, let's do our load test. And they actually, at a reasonable speed, send 60 of these locomotives across. So there's a whole lot of locomotives on the bridge at the same time. I mean, how, how fun is that for a, for a load test in production? Yeah. So.

When you have static load on the bridge, the bridge handles that easily. But when you suddenly have 100 tons appear on the bridge. That's a dynamic situation. From an engineering perspective, that's tough. So what I use is a bit of simple statistics. And I basically set up a load profile so that for every second of my test, I define in a data structure what the likelihood is for load to be applied in that second. So the first second of the test, it might be half a percent the second second and 1%, the third, 1.5%. And I'll use this sine wave function or a bunch of them in a spreadsheet, produce out my Test duration, say 7200 seconds for two hours, and it will have numbers in it, floating point numbers from 0 to 100, 100 means full load. And then in my test scripts, whether it's Load Runner you could use, you can use it with anything you can use with Cron. In fact, in the unique system, before you apply the business process that you want to run, you look up the percentage of workload you need to apply in that second, and maybe the answer is 50%. And then you do a simple random number generator between 0 or between 1 and 100. If it's 50 or less, you say, okay, I'm going to do the load this time. If it's not, you skip doing the load this time. When you have a whole stack of scripts running or load generators running, wherever they are that all do the same thing, you end up with an authentic representation of that curve that you started with in Excel, and it works a treat. I did a post recently on LinkedIn and I actually showed a triple sine wave. And I showed with LoadRunner how amazingly the transactions per second mimicked the load. And what I've done for some customers, I've said, draw with a pen and paper for me what your load profile looks like for this, for this workload. And they'll draw it. And I'll say, I'll just whack that into my tool, into my load testing with Load Runner that I've already set up, and they will then see that exact load profile done. But it's not just that. You can say, give me the load profile for this thing for sign up and for this thing checking out and this thing signing up to a policy or whatever it is, and then you can just add them all together. So instead of having some complicated alignment of scripts and scenarios in a load test controller, you just have this bit of statistics doing the magic for you. It's fantastic.

No, it's very unique. It sounds very, very cool. I love it. I'm Getting excited. That's what I love about this podcast. I'm learning about things I haven't known about and I've been doing testing for 20 years myself. So it looks like based on a lot of your videos and your LinkedIn, you're doing a lot of things with REST API calls. So you talk a little bit about that approach. Is that like a data store that you have your information, you just make calls from your load runner script to get that information on the fly?

Yeah, I do. So goes probably back to whenever you want to run a load test, you need lots of data. So even with modern testing, cicd, if you load testing, you need stacks of data to do it. So I've had lots of situations where I need the data. So I've created, over time, many records, identities, populations of data, and I thought, well, I'll actually make these available. That's why I made test data services. So I've got two and a half million identities, they're Australian ones, but I'm building 20 million US ones because I know a lot of people would rather use US ones than Aussie ones. But this data persists in a DynamoDB database on AWS. And you have a simple REST API call to say, give me a random identity, please. And it will give you back some JSON data, up to 56 value pairs, which will have their name and have a bunch of suggested passwords you can use, have an email address. It'll have a whole lot of other stuff in there. That simple REST API call gives you access into random information with a physical address as well. And that's actually quite useful. But where it becomes more useful is that you can, you can do some data tagging with another API that says, okay, I've just got this random person, his name's Peter Smith, and I've signed him up to my test website and the test website's given me a user ID of X3552. Now, how in the heck am I going to remember that? You've got to save it. That's a real test. I don't like websites that do that, by the way, when they give you a user ID rather than letting you specify one. But that still happens. With a simple call, you tag that, record that identity in the population, a simple REST call with that special login, and then you can say.

Give me one of the, the user IDs that I have created recently and it will randomly give you one. So when you want to log in and do a transaction, one simple REST API call And you get that weird ID and you get the grid that matches it. So you can then find out all of the other information and you can log in or let's say the signup process involved an email. The email goes out. So if you use the email addresses in that data set, that population data set, a simple API call lets you register to receive emails from a from domain and then you do your registration to say, okay, this particular email I'm registering from example.com so any emails from example.com to this person I want to catch, hit your submit button to sign up. And then you wait 10 seconds and then you say, okay, was there an email? And it'll say, yes, there was. And it arrived 4.3 seconds after you registered, which is really nice because then you know how long that asynchronous back end step actually took to go from the system you're testing into AWS ses, which is where it got processed. And it will give you the payload and it will even give you the, the unencoded mime, if they've got that. So that you can then grab the link that's embedded in that, click on the link to authorize.

That endpoint to acknowledge that that's a real email address. So I've used that to sign up tens of thousands of accounts as a pre step for a test in an evening, and it works a treat. I've actually validated that I can do over a million 1.3 million signups per hour.

With a real system, with Amazon Cognito, in fact. Amazon Cognito is amazing.

Nice. So is this what you mean by dataless testing? A lot of times I'd have to set up a table, put in all the data ahead of time, set it to random. Is this what you mean? It's dynamic. Do you have code that does this automatically?

That's right. So dataless testing for me is all about population oriented testing. So when you have an app, if its end users are a subset of the population, then instead of trying to manufacture that yourself and keep track of all that data, what you ought to do is point your testing scripts and scenarios at that fake population of data.

That's what that's about. Then in your testing, if you have in your scripts what I would call a Daedalus test, then you've got no references to absolute test data in any of your scripts. It all fetches everything just in time, when it needs to, with references. So the way I've set it up for myself, and anyone can read the white paper and then they can reproduce what I'm doing themselves if they like. That's why I put a white paper out. I'm not trying to hide the idea, I'm just saying if you don't want to do that, you can use mine. It's all ready to go and you can actually access it for free at low volumes. So you can, with the idea of subscriptions, there's a subscription on the web page you can just use. You can create a subscription, that subscription to key. I then use that key in a lot of the calls to keep track of things like that data tagging I told you about. So you can create as many of these sub subscriptions as you like. So for every release in a CI CD situation, you could.

Randomly grab a whole stack of random people, sign them up to your test system, and you get your user IDs or you acknowledge what the user IDs are, store them with tags, grab their emails. You can even do MFA for Google Authenticator or Microsoft Authenticator, store the secrets as you get them. And because they're all linked in with this key, the subscription key, then for the life of that test run or for however long you want, you've got full access to every bit of data that you've used so that you can reuse it. So you can have one part of your test that sets up a thousand people, another part of your test that actually generates load using those thousand people, and actually signs up a few more at the same time, and suddenly you've got a whole lot of tests that can run every time someone commits code to a repository and there's a build that fires and you haven't got any data anywhere. And you can actually run them all concurrently, because with randomness and millions of identities, you have a small chance of an overlap. And even if you did, it wouldn't matter because they're all different subscription keys. And voila, you've got all this data. And.

When it comes to data, it's important you get a lot of variation, because sometimes apps don't do what you expect them to do with that variation.

So, for example, a typical tester when they're doing an E Commerce site might just assume someone's got a credit card, but not everyone has a credit card. And what do you do if they don't have a credit card? Now maybe, maybe there's nothing you can do, but maybe there's an alternative payment, and maybe you should validate that that alternative payment method can be invoked, even if it's a rare one. Because maybe there's some error that happens when you try and invoke that. And maybe that error leaves a connection open and doesn't close something, right? And maybe after that's happened 128 times, you blow some connection pool limit. They're the sorts of things that you find when you do really high quality tests, when you don't stay to the extreme happy path.

What it sounds like for this approach as well is because you're using a REST API, as you mentioned, you tend to use Load Runner, but it could be used for any type of performance test tool. It sounds like.

Yeah, yeah, so that's right. So they're almost all get. So I've ignored all of the proper orthodox approaches to how you're supposed to use gets and puts and posts for rest APIs, and they're almost all GETs. That means that let's say you're doing manual testing in a room and you've got 50 user IDs that you often use for some terrible system that's got data that it's hard to work with. You could actually set up shortcuts on a Windows desktop to catch emails, any of those 50 users, or do MFA tokens for any of those 50 users. Because in a Windows shortcut you can put a REST API that's a get has to be a get there, right? So user 43 needs to log on and you got to log on with that user. You double click on the icon and it'll say what the MFA token is, the current one and the next one, and how long that one will last for. And if you need an MFA and then an email goes out to that person, that test user, instead of having to go to some central repository or have some special email address that everyone shares, you just double click on.

That desktop icon. This is an extreme example of what you just said. It's a manual test and then you can actually see the email that went to that person, or you can see actually as many of them as you want. You can step through them. If five got received, you can actually look at each of them.

So rest APIs that it gets make it very easy to do anything from functional manual testing to automated testing in anything. So I've been having a bit of fun playing with Locus, which is Python based testing, and it allows you to do tens of thousands of requests per second really nicely. It's lovely for load testing of API endpoints. And all these calls work awesome in there, which is nice.

So Paul, I think it also a lot of the Examples you use something called Test Data Services. Could you just tell us a little bit about test data services as well?

So it's TestDataServices.com and it basically is me trying to share out what I've been using myself to make myself efficient. So if I want to do a high quality test that has some variability and has some clever stuff in there, then. And I'm never going to get any work if I have to charge for all of my time each time because it would take a lot of time. So to make it affordable for my customers, I built a whole lot of tools so I'd be very efficient. And I used dataless testing approaches a long time ago. But I thought instead of keeping that to myself, I should really let other people in on what I'm doing and maybe lift everyone's testing a little bit. So I've made a population available and a whole stack of REST API calls available which can do everything from like I said, email, MFA and data tagging to even OpenID Connect. When you sign on with Google or sign in with Facebook that's usually using something called OpenID Connect. That's a way you secure APIs and that's going to be happening on a massive scale soon. It's really hard to incorporate that into a load test because how the heck do you log on with 100,000 different people to your system over the course of a load test run? If you have to use.

Some Google account you have to set up 100,000 Google accounts, which is actually hard, possible but. And they don't like it, but it's hard. So you can use test Data services to.

Be the identity provider and you point your OIDS table endpoints in your app to my ones and instead of sign on with Google you're effectively then signing in with test data services and you can get a random logged in user whenever you like or a specific one whenever you like. So that's probably all I need to say that it basically is a whole lot of APIs that let you do dataless testing so you don't have to worry about test data management and you get the added benefits of simul of being able to incorporate those identities in that population interacting back with you as well. It's not just one way, which is quite cool.

So Paul, another topic I have written down to ask you about is oidc. Can you tell me a little bit more about what OIDC is and how.

It relates to performance testing with OIDC, that is what FAPI or financial APIs and health access APIs are going to be built on. So when we're going forward with open banking and with a lot of healthcare, that if we don't have a means of testing with OpenID Connect, we're not going to be able to test open banking stuff and healthcare stuff. So with open banking, for example, you might want to give permission for someone, your accountant to look at the last two years of history of a particular bank account, but that's it. And OpenID Connect or OpenID Connect actually lets you do that with Scopes. So how the heck do you do a load test where you want to validate load with that sort of functional test case without having a means of, of doing it? So there you go, that's, that's where I think we're going to have some issues in the future if we don't have some ways to test with OpenID Connect.

What are your thoughts on how you would do that though? Would you use the REST API as well? Test it?

Yeah, yeah, that's what I'm building my one for. But I'm hoping that other people will build similar ones so that we can do decent quality healthcare testing. Testing with production data in test environment is to me a terrible thing. But it's what most organizations doing in finance and in healthcare. But what it means is because they've anonymized the data and they say it's safe, they'll run their tests on it, but often it's de anonymisable, like someone can unshuffle that data and figure out who's who. And that's a big problem. So because of that they can't then do integrated testing with other organizations that they have interfaces with. And healthcare is all about making it so that everyone who needs to has access to the same records and can share them. Well, how do you test that if no one in their test environment is letting anyone share any of the data? Because underneath it all this production data there. So.

I want to help move people away from using production data and test environments. And having a population of test data that's available is one of the ways I'm seeking to be able to do that. And I'm just giving an example of how I would, how I would do it, making that available.

Okay, Paul, before we go, is there one piece of actionable advice you can give someone to help them with their performance testing efforts and let's know the best way to find or contact you.

So I would say that the most important thing, actionable advice I would give is to think about risk when you're thinking about a load test, not thinking about a formal SLA that they put in front of you. That's what I would say. I know that doesn't sound very actionable, but thinking. Thinking is an action because when you're thinking about stuff, it leads to things.

And LinkedIn is probably the best way.

Unless it's a sales thing. I'm pretty good at responding and chatting and giving information and sharing stuff to people directly on LinkedIn.

Thank you Paul for your performance testing awesomeness. For links to everything of value we covered in this episode, head on over to testguild.com P21. And while you're there, make sure to click on the Try It Today link under the Exclusive Sponsor section to learn all about SmartBear's innovative performance testing solution. Load Ninja so that's it for this episode of the Test Guild Performance and Site Reliability Podcast. I'm Joe and my mission is to help you succeed with creating end to end full Stack Automation awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Performance Podcast. Head on over to testguild.com for full show notes, amazing blog articles and online testing conferences. Make sure to subscribe to join the Guild and continue your testing journey. This has been a Joe Calantonio production.