Modeling Test Data For Performance Testing with James Walker

Five, four, three, two, one. Drop that.

Welcome to the Test Guild Performance Podcast where we get together to learn more.

About performance testing and site reliability with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Performance Podcast. Today we'll be talking with James Walker, all about borrowing principles from functional testing to help performance testing and also to maximize your testing transactional mix. James is an experienced software developer, engineer in test, and an experienced software inventor. He has helped design and build multiple model based testing and test automation tools and has collaborated closely with a wide range of organizations to solve their QA needs. James was a manager of software engineering at CA Technologies, is now Director of Technologies with Curiosity. So you don't want to miss this episode. Check it out.

Face it, performance testing is tough. Traditional load testing tools create scripts that are bogged down in data that's hard to read and require a lot of work, even for simple playback. Load Ninja cuts out the dirty work by using AI to inspect and debug your code almost instantly. No coding necessary. No changing of hands or relays between teams either. Give it a shot. It's free and easy to try. Head on over to testguild.com smartbear and click on the link to learn more.

Hey James, welcome to the Guild.

Amazing. Thank you so much, Joe for having me. I've been a long time listener, so it's great to finally be getting involved. Very excited.

Awesome. Great to have you. This is a topic also I think most people probably aren't familiar with, so I think it's going to be a great help to everyone. Before we get into James, is there anything I missed on your biography that you'd like to share with us?

When I first entered testing maybe five, 10 years ago, what I found was actually that it was amazing how unstructured everything was in the industry. There wasn't really any good data around. The ways we're testing, the way we're storing our tests are very textual, not really very granular. There's not a lot of structure there.

My background coming into it from very data centric point of view has been looking at the industry. You know, how can we start applying structure, how can we start, you know, creating these kind of high quality data sets to start actually moving us forward inside the testing industry?

Absolutely. And I think you're in a really good space here because one of the biggest challenges I see with any type of testing is data, right? Using the right data, having the correct data, not just doing happy path data. And a lot of Times I think people associate it with just with functional testing. But I think you have a different take on it as well, where you believe it can actually help with performance testing as well. So what can we learn from functional testing to help us with our performance testing with data?

No, I think that's a great question. I think before we get into that, we've probably got to peel back the layers a little bit.

Sure.

I think if you look kind of at any kind of testing.

Really, it's very unstructured.

The way people go about actually creating tests, whatever those tests are, performance, ui, API tests, there's not really so much science behind how we're testing what's being tested and how good the test cases are that we're actually creating. So this is kind of where my career started, I suppose. So this is where we started diving into model based testing, which kind of, coming back to your question, it's a rigorous, structured approach, you know, applying science, creating good data and overlaying that into, you know, testing practices and principles. So for those of you listening who aren't familiar with model based testing, that's really just, it's very simple. It's creating a flowchart basically of how the application should work. And then we're basically applying algorithms, you know, to generate good, high quality test cases for you. Okay. Now the funny thing here is that when most people start creating a test, you know, whether that be performance or UI test, they start creating kind of a mental or physical model of how, you know, the system works or how the operation they're testing works. Okay. And then they basically kind of peel back the layers and they start actually using that model to generate the test. They just don't really know that they're using modeling as part of that process. So really what we're doing here is kind of formalizing that process. And as you said in the beginning, we launched probably one of the first ever tools in the market for model based testing. We transformed many organizations testing practices using this. Now as part of those activities, we started obviously talking to testing teams.

And you know, as part of that, we would also go in and we would start talking to performance testing teams in particular. And whenever we show them, you know, this wonderful world of modeling and all the benefits, it was always kind of the same response. They always look very unimpressed. They didn't really know what it was. They couldn't really comprehend how the, you know, modeling approaches could really help in the performance testing world. And when we started digging deeper into this, we actually realized that most Performance testing as we saw it in these organizations was actually just a product of applying the same request at load. So, you know, we're applying one singular API call or maybe a couple with very small deviations between them, and we're applying them at load x amount of times to see how the system reacts. I would say now we've kind of entered, you know, a new era with software. Right. Almost every company we visit, you know, when we're out on the road and we're talking to them, usually they start off with a little pitch explaining what their company is about, what they do. And the first thing they usually come out with is saying, we're a software company. And that's true across all the industries. We talk to insurance companies, financial companies, aviation companies, they've all transformed themselves and now consider themselves tech companies.

And as part of that, there's massive, massive challenges. Users are expecting more and more functionalities. Companies are having to move much, much faster to deliver these new features. But also there's a massive consequence in demand of that from the user, that they're expecting a really high user experience whenever they use that application. So if companies aren't delivering the user experience the customers expect, the likelihood is they're going to make a decision to change service provider or move elsewhere and moving fast and maintaining quality so, so hard. You know, that's probably one of the hardest pieces of developing good software.

So, you know, I would say by and large, testing was kind of seen predominantly as kind of a compliance function. And now it's really moved into the heart of the software development life cycle. And I would say this is mostly just true of functional testing. So we were in Holland two weeks ago and basically over there the test manager role has almost disappeared now. And basically all of these test managers are now transforming into quality driven scrum masters. Okay. So that's just one example of how, you know, testing is really moving away from this compliance function. And really what they're trying to do there is get testing in every single part or aspect where we're creating, testing and delivering the software. Quality's got to be at the heart of it, right? So performance testing in our kind of experience was kind of the worst offender for this compliance box checking. You know, high volumes are very repetitive requests. You know, just throwing the same request over and over, which is fine when it comes to stability testing. I think, you know, if you're checking the wheels are on and let's monitor the system to see how, you know, it reacts to these very specific instances and tasks and that's absolutely amazing. But the reality is that we should be simulating our user activity. We should be thinking about the functions in there, how the user is using that functionality, and we should be replicating that user activity as closely as possible for specific scenarios to understand how our system is actually going to react.

When it's under that real user load, which is basically what we've been doing in functional testing. We think about the different user scenarios, how users are going to use our system, then we start to automate those scenarios and pick them off and start to apply our test cases in that way. Really what we're thinking about here is how can we apply performance testing, bring it into that realm where we're thinking about how users are using our system, trying to simulate that behavior, and then try and understand is our system reacting in the way a user, what is expected to react when we're getting those real kind of levels of load and activity which are happening?

Yeah, so that's a great observation you made how most people, when they do performance testing, it's more stability testing. And back in the day when I focused more on performance testing, we do a stability test and we do our performance testing and then we put into production, we think, wow, why is the. Exactly. The behavior is totally different than what we've been testing in our test environments all this time. And, and I think it might be because like you said, we use the same type of data, low data, not much variation with it over and over again. And when you go in production, because you have users doing all kinds of different things with different kinds of data, not just happy path, that can cause issues.

So because we're modeling earlier before we even create our test, I assume, does this help us know what kind of transaction mix we should create when we're doing a performance scenario?

So I think if you think about kind of a real world example here, so, you know, a couple of weeks ago it was Black Friday and you know, every user hitting the website was not doing exactly the same thing. You know, they might have been going through, you know, clicking on particular items they may have been going through, posting some returns. You know, there's probably some complex chains of activity there. Right. So, you know, when we think about it like that, we've actually got quite a rich transactional mix there. You know, we've got a mixture of different activities and tasks which users are performing on the page. Right. And I think Black Friday is a very good example because every year, even though a lot of these companies think they're doing fantastic Performance tests. We still see a lot of system crashes right now. The way we're prescribing this here and we're talking about modeling, which is what you just said then, you know, you kind of firstly need to understand the system and how the users are actually using it, right? So you've got kind of two ways you can kind of navigate your way through this. Firstly, you could come in and you could start from the left, you know, before the system has even been developed, before it's been created. We can start creating models of how that system should operate. And at that level, we're kind of thinking more from a requirement level, you know, what is the requirement of this system? And let's draw it out visually. Now, what we've actually seen in the industry is a lot of product owners and a lot of business analysts are kind of doing this activity already. I think in the latest product owner course, they've actually introduced the concept of modeling in there for defining requirements. And they're typically doing this kind of through Visio diagrams and BPMN like tools. So they're defining how the system should work already. Okay, now, once you've got that model, what you can then do is once the system has started to come into fruition, it's gone through development. Literally all you're doing is overlaying on that model the different API calls or the performance actions like JMeter, Taurus or Gatling commands that need to take place for each step. He's literally going into a block and saying, I want to go up and perform this API request, you know, against this endpoint. Okay, Now a route through that model is one scenario, right? So I may have a route which is going through a registration. I may then be performing a login. I may then be going through and ordering some products. I may then go for a checkout process. But that, by and large, is one scenario which would represent one instance within that transactional mix. If you look across the whole model, what you would then do is you could use that model to create a whole world or create the whole transactional mix for you with a whole world of different scenarios in, and you can actually weight different parts of that model to create different transactional mixes. You may say, leading up to Black Friday, most of our users are going to be hitting.

The sales part of our application. We're going to be hitting a lot through our payment processing system. But the actual other parts of the system, like our returns portal.

You know, creating a new account, they may not be hit as hard or have as Much volume going through. So when you think about it from that context, you can start to use a model to create a whole world of different transactional mixes from that same model that has been created once. Okay? Now, that model doesn't actually have to be created manually. It could come all the way from the right hand side. And if you speak to Jonathan Wright about this, he calls this AIOps. Okay? So from his point of view, the way he talks about this is you're basically taking.

You're monitoring what's happening in production and you're learning about the activity that users are performing inside those production environments. And then what you can do is you can take these logs, take this monitoring data, basically use that to drive our product development and our testing practices. Okay? Now, Dynatrace is one company doing this. Another company who we've worked with quite a lot is called Probit. And basically what they have is the capability to monitor production, understand exactly what users are doing in the system, what pages they're hitting, what elements they're interacting with, what inputs they're entering data into, and they can actually use this to create models of this activity. Now, from that extent, what you're doing is you're coming all the way from the right and you're taking the actual user behavior and you're using that to create a model automatically for you.

Now, obviously, there's no magic wand. There's no magic wand behind that. You do get a lot of noise. Users are performing a lot of behavior in there. It can end up being quite a big data set. So what you will find with these tools is they have techniques to go in and cluster the data. You can set particular thresholds. So you may say, I only care about the top 50% of user activity. Everything that falls below that waistline, we're not interested in. And we will not use that to build the model. But what you do get is a fantastic starting point where you can actually go in and you've got a model which is created there, which actually represents the system and represents what you're testing. And what we've actually found is quite a lot of companies are actually doing this themselves. So we were speaking to a financial services company about a month ago, and we were speaking to their performance team, and they'd actually built a custom harness themselves to go in and start mining the logs they had in their production environment. And what they were then doing is they were using this, they were using process mining to go in and automatically map this back to a model which represents the user behavior. That they were performing on that system. And then what they had was that as part of that they were capturing all of the requests that were being sent backwards and forwards and they were basically creating their own model with the user behavior which they were then using as their kind of in their performance testing environment.

So when you say it's modeling because functional testing is usually going against a browser, I think at performance testing you're usually doing more like an API or some sort of headless type of protocol. Does the model actually then generate the performance test for you and then you can just consume it within like JMETER or Gatling or Taurus or something like that?

Yeah, absolutely. So, I mean, from our point of view, what we've seen in the industry is most companies have got some kind of open source initiative now. You know, they're moving towards JMeter, Gatling. Taurus is another great example from the model. You're literally going in and you're overlaying these actions on top. So you could be saying, I want to create a new thread pool, I want to go up and perform this request. If you're testing a REST service, you could be firing at it a JSON request, which is absolutely fine. Then the combination from that of overlaying this on the model is when you go through and you're basically creating different paths through the model, one of those paths represents a test case in the world of performance we're actually generating, we could be generating JMX files, we could be creating YAML files, if you're talking Taurus or you will get out the end is that complete chain that has passed through each of those blocks in the path. If we go back to what we were thinking about earlier, if I have a model where I've modeled out the registration page, I've modeled out the login page.

I've modeled out the order products page, and I've modeled out a checkout page, okay? As I'm passing through each of those blocks, we're going to pick up and we're going to basically start collecting and aggregating all of those actions that we've overlaid. The aggregation of those actions formulates the resulting script. Okay? Now the interesting bit here is obviously we're talking about, you know, performance testing, right? So we're talking about, you know, tools like JMeter, Gatling, etc. Etc. Now if you think about this from other types of testing, you know, if you're thinking about UI testing, you've got tools like Selenium, you're doing mobile testing, you've Got tools like Appium, you know, Selenite, Cypress, etc. Etc. You could be talking about standard API testing. Now, each of these is a different type of model, but when you strip testing back to its rawest form, really it's just all about the data.

If I'm doing UI testing, I am basically plugging my different data into a web front end. If I'm doing API testing, I'm plugging that into some endpoint, I'm sending it to some endpoint. So if I'm testing a REST service, I'm going to send some JSON or I might send some XML to that endpoint. If I'm doing ETL testing, I might be sending it some stored procedure to go up and run that transformation.

Now, when you think of it from that level, we're creating a model of the different data combinations and the different ways and different data or different routes that I can work my way through that activity. There are different types of models, but really it's the same process, just overlaying different actions. With performance testing, you're doing exactly the same. We're just overlaying the requests we want. We're also specifying on here some weighting, which is saying for this particular part of the system, I want to have 1,000 instances or I want to multiply this up 1,000 times. You're not just saying what I want to perform, you're specifying as well the amount of load that I want to apply to that particular part of the system.

Another thing I think people are worried about with modeling is how can they trust the model? How can they trust the data being generated by the model?

Yeah, I think this is a fantastic question. The truth is, when you're modeling, you're only as good as the model.

If I've created a model, I'm only testing what I've actually specified in the model. It's not possible for me to go off and test something that I don't know about. So you're only really testing the knowns, if that makes sense. The real interesting piece here is obviously the known unknowns, or the unknown. Sorry, not the known. The really interesting piece is the unknowns. What don't we know about? Now the good thing with modeling is actually you've got a visual image which is showing everyone exactly what we're testing and how we think the application works. Now, if you think about this from a classical kind of testing point of view, I may have my test cases stored inside some kind of test case management system. And that level it's very hard for me to actually understand what my test suite is actually doing. And you've almost got a sense of technical debt encapsulated inside that test suite because someone will have to physically go through, walk through all of these test steps to actually understand what we're covering with our test suite. With a model, I would say it's actually beneficial because you've got that visual image that anyone can instantly look at and see. This is how we're testing the application. Now, the other part to this is there are ways around this as well where you can actually start to identify where the gaps are. In my models, we're talking to a company, C Lights, just this week, actually, I think it was yesterday. And they've got some pretty cool technology where they can go in and they can start linking between test cases and the actual underlying application. So, you know, for them, it's all about, you know, looking at, you know, what parts of an application is a test case covering. How much coverage are we getting from these test cases? Now, the interesting bit there is they've obviously got mappings that they can look at change requests. You know, a change has come into the application, what test cases are affected. We've also got the inverse in there as well in the, you know, you know where the gaps are in your test cases because you know what's not being covered by the application or what your test cases are not hitting in that application. So when you think about it at that level, you can start looking at the models and you can say, oh, hey, you know, this model should be covering this. But actually we're only covering a very, very small subset or a very small slice through that particular sub component of the application. So there's definitely ways around it. I would say the biggest or the most powerful piece of modeling, though, is just having that visual image. And then other stakeholders in the project can also come in, look at that model and say, oh, we've got a massive gap here, or this isn't actually how the system should work. So it's kind of a communication collaboration piece as well.

Awesome. So I worked for a large enterprise and we had an environment that was made up of multiple tiers, right. Multiple servers. So it sounds like the model then help, you know, that, hey, 90% of your actions just going, maybe this is unrealistic, but it's going just against your web server. It's not touching the seven other servers that make up the environment. Is that something modeling can help? You know, that, you know, we have a deficiency in testing this piece of our application, then.

Yeah, I think so. I mean, if you. I would say modeling by itself cannot do that. You've got to have other tools in the mix to help you. So I mean, if you, if you look at the monitoring tools that are available now, they're actually phenomenal. You've got tools out there that can track requests through several eight layers deep of microservices, all the calls that are connecting between all of them, which is actually very, very complicated. Now, if you think about it at that level, and you can map between those calls back to the model and you've got that traceability there, which I think is the core word here. Then you can go from the model to any of those calls and you can map backwards at any time. And you know, exactly, you know, this test case is hitting, or this model is hitting these parts of the application, these different services. And likewise, we know that for these services, these are the models that are representing it. So you've got that full kind of backwards and forward traceability. But you need to have something in there which is actually logging the data in the first place. And that's really where, you know, monitoring tools kind of come into play. I think they've evolved massively over the past kind of five years. And I think for any kind of microservice development especially, you've got to have these tools in the mix. Otherwise it's such a logistical and architectural nightmare. There's so much going on. It's so hard to keep track of what's happening and where everything's kind of moving towards.

If you start injecting not happy path, how do you know that you're not creating a false situation?

Yeah. So I think, again, a fantastic question. So I mean, for every model there's kind of two bits. And I would say for every test suite, it should be split into two. You know, you've got your positive test cases, which are your happy scenarios. Typically, when we come in and we start to do an audit of an existing test suite and we look at it, what we actually see is that most test cases are actually happy path oriented. It's not until you reach the realms of actual, you know, looking at tests that really good experienced testers have created that you start to realize they're the ones who are creating these negative test cases. Now, in the world of modeling, you've got to think about two things. The best place to start is usually creating a happy path model. And that will literally just be probably a linear model. Okay. Now where negative comes into Play or negative testing comes into play is you've got to start thinking about the bad data that I can inject into a field or the bad interactions I could perform on a web page, for instance. Okay, so if we're thinking about, you know.

So if we're thinking about.

You know, an email address, okay, obviously a happy email address. We're all very familiar with that. It could be my email address, which is James Walker at Curiosity Software. For a negative email, we could have an email with no sign. It may not be, you know, syntax semantically correct. You know, there's a whole world of different negative data that could be injected into that email field. Okay? Now, through the model, those are just different decision points. So you would literally have a block on the model which is saying, you know, email with no sign, email with no.com at the end or no kind of domain name or some kind of incorrect domain name at the end of the email. Okay? Now in the model, the nice thing is once you've got that on the palette and you've dropped it on instantly, me as a modeler, I've got to start thinking about how the system reacts to that bad data that's being injected. So it may be that if we're hitting an endpoint, we may come back with some kind of 400 error code for a bad request. If we're going for a web front end, we may be coming back with some kind of error message that happens. And it's then up to me as the tester, as the modeler to come in and start dropping those assertions on the model and saying, because we've entered this bad data up here, all the way down here, we should be seeing this error message being returned by the service. In the context of happy and negative testing or positive and negative testing, it's really the same concepts. All you're doing is just formally defining it in the model. You're literally just creating these different decision points. You're specifying what those negative scenarios are. Then in the model, you're also dictating afterwards. This is what we see afterwards. That's true of performance testing as well. When we're thinking about moving performance testing into this more functional realm, we should still be applying these same concepts. Just because I'm doing it a load, I should still be applying those assertions. If I'm applying this and scaling this API request up, I should still assert that every time it comes back with this response or it's still rejecting that bad Data with a 400 response code.

Okay, James before we go, is there one piece of actionable advice you can give someone to help them with their performance testing modeling efforts and let us know the best way to find, contact you or learn more about Curiosity Software.

Okay, so I think the one piece of advice I would give anyone is to actually, let's try and move the needle, let's try new things, let's experiment and let's really get in there and let's try and move performance testing forward to where it needs to be. I think if we think about this from kind of a challenge point of view, what is the main challenge to actually getting into modeling? It's actually creating the model itself.

Actually getting into model based testing. Whatever type of testing you're doing historically, they always used to be a very kind of high barrier to entry. You know, someone would have to sit there and physically start creating this model, which if you're dealing with complex systems, can take a very long time. I think we're kind of now in the realm where we've got a whole world of accelerators out there for creating models. So, you know, if you look at what we were talking about earlier, some of the monitoring tools, you can immediately create a model out of that and just do an export and then bring it into a modeling tool and start to apply model based testing. You also don't need to actually be that granular. You know, a lot of people come into this and they start thinking, oh God, I've got to go through and actually map out every part of my system. You may start from a very high level and you may just have a block which is literally registration screen, or you may have another block which is login. I don't actually have to model out every single piece of granularity which sits behind that part of the system. I could just keep it very high level. Later on I may come in and I may say, I want to dig deeper into that block. I'm now going to create a whole model in itself for my registration page. But initially you can start very simple and really kind of, you know, just get the wheels moving, start modeling and start experimenting. So if anyone's interested in learning more about modeling. So our company is Curiosity Software and we have a model based testing tool called Test Modeler which supports all types of testing, UI testing, API testing and performance testing. If you'd like to learn more about the tool, you can visit our website, which is testmodeler IO. Or you can feel free just to drop me an email, which is James walkeruriosity Software.

Thank you James for your performance testing Modeling awesomeness links to everything of value we covered in this episode. Head on over to testguild.com forward/p plike automation1 4 and while you're there, make sure to click on the Try it Today link under the Exclusive Sponsor section to learn all about Smart bears. Awesome Load testing tool Load Ninja so that's it for this episode of the Test Guild Performance Podcast. I'm Joe and my mission is to help you succeed with creating end to end full stack automation awesomeness. As always, test everything and keep the good. Cheers. Thanks for listening to the Test Guild Performance Podcast.

Head on over to testguild.com for full show notes, amazing blog articles, and online testing conferences.

Make sure to subscribe to join the Guild and continue your testing journey. This has been a Joe Calantonio production.