Journey towards continuous performance testing in a complex financial Agile organization

Welcome to the Test Guild Performance and Site Reliability Podcast where we all get together to learn more about performance testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Performance and Site Reliability Podcast. Today we'll be talking with Stephan, a performance consultant, all about his journey towards continuous performance testing in a very complex financial agile organization. I think a lot of people struggle with performance, especially when they're in regulated areas, like I was with healthcare and as Stefan is with finance. So really curious to know how he implemented performance in a way that was awesome for the team and awesome for the company. Definitely don't want to miss this episode. Check it out.

This episode is brought to you by SmartBear. Listen, load testing is tough. Investing in the right tools to automate tests, identify bottlenecks and and resolve issues quickly could save your organization time and money. SmartBear offers a suite of performance tools like Load Ninja, which is a SaaS UI load testing tool, and Loaduipro, an API load testing tool to help teams get full visibility into UI and API performance so you can release and recover faster than ever. Give it a shot. It's free and easy to try. Head on over to smartbear.com solutions performance testing to learn more. Hey Stefan, welcome to the Guild.

Thanks a lot, Joe. Thanks for having me. Shall I introduce myself a bit or.

That was my next question. I had a very small bio, so we get into it. Is there anything else you want the Guild to know more about yourself?

Yeah, just like you said, I'm a performance engineer, performance automator, also a reliability advocate. I work for ing, which is a great working space. I could all advise it and work worked for the last five, six years on performance automation in a quite a difficult environment. So I wanted to tell you something about the principles we use and what challenges we had and how we tackle them.

Awesome. So you mentioned you were a reliability advocate. I know software reliability engineering the past few years has been really, really a hot topic and I almost see performance in SRE merging. Are you seeing that as well?

Yeah, yeah, definitely. Before the performance, the reliability advocates were led by a SRE agent. But now they ask me, I think and reliability and availability are all very closely related. I'm also working on the topic of sli SLO error budgets trying to implement it. But maybe that's for another talk. The margins, it's come closer together. All those environments. Indeed.

Nice. How do you juggle what you're doing during a day? Then if you're doing SRE Type activities and then like more traditional performance activities.

Yeah, well, automation I think is the key there. Right. Also don't try to solve everything yourself. We also have an excellent group of engineers who do a lot of stuff. So yeah, that's I think the key.

Absolutely. I know at performance for some reason it's really hard to find someone with skills that understands performance testing. So how do you get that help? Are you involved in training as well or is it just. Does your team have like a playbook when people join how to implement performance or SRE type activities?

Yeah, it's a bit different. In the past, before we went working In a BIS DevOps model, we worked in a project based organization and then we had a team of performance test engineers. And now because the principle, you build it, you run it, everybody is supposed to do everything themselves running their components and we just provide automation for that. So it's more like a self service but with some help of us that that makes it work.

Interesting. I'm sure a lot of companies have implemented this kind of structure. So are there any downsides? I know before when I worked on a team because I knew the application from beginning to end, it was easy for me to performance test it. But working as a almost like a consultant within a company to all these other organizations, it was hard then to jump in for different projects. Once they something in place. Do you find the same struggles?

I think one of the downsides is of course getting people doing stuff which is not related to their day to day job. So taking responsibility to also help others. ING has the orange code, so it is promoted that you do that. But like what I said, I'm a product owner as well. So I could also say I don't do anything with performance, but I chose to do so also on the effort of reliability. But it's more like in a guild structure, say. So yeah, it has benefits, but it also has some concerns. But I think the benefits is that you really have motivated engineers that also work outside their comfort zone to. To really make a difference. And that's also being awarded.

So I've spoken with other engineers from your company. For some reason it seems like a really good culture there. And that's a lot of times where people struggle. They are in an organization that maybe doesn't have a culture of quality or performance. But for some reason your company has a quality of performance. It seems to me already. So why is that?

I think it's all related to the past. I think innovation. Looking at the ING as a bank, we are very Innovative compared to other banks. Also the bank that I started was actually the Post bank, which is the predecessor of ing. At least it's merged and we had like the first online banking system and those kind of things. And now I think we're really a trendsetter. With mobile, for example, it is in the engineering culture.

Great. So like I said, it's really rare to find a company like that. So if someone is listening in, maybe they're not in that type of culture and organization, where would they start to maybe plant the seeds of performance and let's start from there and then build up on that.

Yeah, I think it's all in the mindset. The thing is, I think in general engineers say they want to do the right thing. Right. They want to run it, but sometimes they lack a bit of help or knowledge. So I think how we started, we started when we migrated to a BIS DevOps architecture. Of course the teams themselves, some teams struggled having the knowledge, especially because we went from a monolith application to microservices that also gave some challenges. But when we started automating it, using it in the pipeline, engineers starting to understand what we were doing. So they started to work with us in it. So we got also a lot of contributions from teams that we even didn't expect. Like the inner source model, for example.

Awesome. So once you have an understanding going from a monolith to microservices, I guess, does that change your CI CD process as well?

Yeah, because we are of course a customer focused bank. Right. Because if you don't have customers, you can have the best of it, but nobody cares. So we really felt that we had to increase our delivery to be able to deliver what the customer needs. Because the world changes, things got faster. So once we started doing Biz DevOps, we really saw that we were more agile delivering the right software to our customers. So I think that was the start. But when you are coming from an approach that you had a couple of major relationships release on your applications per year and now you have applications that do multiple releases per day, the CI CD process is probably the most important. Right. So yeah, you want to integrate that as well. You want to provide the teams the feedback and the confidence that they are able to do that. And one of the most important parts in my mind is the testing aspect. Not only performance, but also of course, functional and security wise. And that automation of it, that was the real fun part. I think that gave the real benefit.

I guess there's a kind of a myth that you can't incorporate performance into your CI CD pipelines because it takes too long to run. So any tips for someone that maybe is starting to try to incorporate performance into their CI CD pipelines, how they can get started?

I think the first thing is more on a management level, right? You have to have the commitment from the management to make a change. And we got that, we got the trigger and then the real challenge is getting the frequency right. You want to have your load testing. Frequencies should support also the frequency of going live. And that was the first couple of months we worked on that. We looked at which frequency worked. The daily frequency was already on the horizon. That made the most sense. But also the timing. We do load testing on an acceptance environment which is near production. Like so it's used, it's hybrid, it has all kind of systems and also all kind of processes. So the first thing we had to do is figure the best timing. And that is of course from a management perspective. Just before everybody is in the office, you want to have the test done. You don't want to block any timings. Yeah, we did it. We did it early in the morning, first by hand in the first couple of months. But then we, we managed to also automate other tools like a bit older tools like Performance center, which are a bit more tricky to automate than I guess the Gatling tools from this world. But we managed to do it. We managed to start test, we managed to parse the results. But I think the one of the big changes that from a manual perspective on load testing towards an automated way is also data sources. The monitoring aspect is way more important from a component than the numbers you get from your load test tool. So you use the load test tool to put load, but it doesn't really give a lot of weight on your analysis anymore, at least for my how we ended up doing it.

That's interesting. So do you actually build in an instrumentation into your code that could somehow give you those numbers? How do you, how do you get around that?

Yeah, we bought, we, we created an application called Auto Analysis which is basically a dashboard and it has several sources like Prometheus, but also the Performance center counters, the Gatling counters and like system metrics and so on. And these data we parse, we parse it on a granularity of 1 minute interval and we have 120 data points and we store them in a database and we do analysis on those. So we're looking not only at high level results, but also at patterns within the data ranges.

How do you benchmark it? So do you have logic built in that when it runs, say today compared to a week ago, that if there is a change in performance, that do people get alerted or is there always a human that looks at it first and then contacts a team or the correct engineer?

Yeah, we started very simple. We started just saying, oh, this test looks nice to us. So for this component, this is the benchmark that we use. But of course that's a bit that works if you have only four load tests, like in the beginning, but we now have 40. So we use machine learning to do that. We're looking at the last 60 days, looking at all the data points, how they relate towards each other, and we use that to have an expected outcome and we compare it with the expected outcome. So the test of today will be compared with the expected outcome of the last 60 days. But you can also manually say, okay, I just want to compare it to a certain day. For example, if you just had a release or you had, in the worst case, maybe you had 60 days of bad testing, then you expected outcome is also not that good and you have a good test and you don't want to compare it to the bad scenarios. So you can override it. But. But for us it made sense to use machine learning because you can't handle it by yourself.

Interesting. I know machine learning has been a buzzword in automation testing, but I think it applies better to performance, especially because it's analyzing a lot of data. So how does that work?

Yeah, I think we do about 1 gigabyte per test or something. So it's a huge amount of data. It's not something you like to do yourself.

Yeah. How reliable has it been then, actually doing that?

Yeah, that's a good question. I think it's all in the mind of the implementer. Like for a performance tester, some things are logical, but once you start building it, you might not think about it. So like a couple of months ago we saw for a component that we a benchmark that I didn't expect that covered a bug that we had some, that we also looked at the test of the same day as a benchmark. And the chance, of course that is the benchmark is not that high. But on that occasion it actually compared the apples with the apples. Right. The day of the test with today. So that was a bit funny. We had to laugh about it. But I think in general it does work once you know what you're doing and you start to learn what you're doing by just doing it. Right. So it's A trial and error thing.

What are some benefits of doing this? A lot of people may be going, well, this is a lot of work to get performance into my CICD pipeline. Why can't I just do like the old days and just wait until everything's developed and run it?

Yeah, it's an excellent question, Joe. I think I used to do this test manual once. It was far smaller. Ahead of the performance test team. We did it with four people. We did four tests, four people. And we spent the whole day doing it and then looking at the end also we went to load Runner analyzer and we cross results and we would spend another day and it was really in the eye of the beholder what you would see because I could look at something different than other people. But once you automate it, you know that all the patterns are covered. We use the rule engine, which is basically an automation of looking at patterns, like peaks, like slopes, like anything you want to look at, look into that's already automated so you don't need to look at graphs anymore. So I think that's a real benefit of doing it. And also of course the time you spend on it is way less for me. It's probably an hour work per day to look at the results and send maybe some, some emails about some things I noticed.

Absolutely. I would think another, some complexity that nowadays is probably involved is infrastructure because back in the day I had everything on metal on a raised floor. I'd have to log in and I'd have everything under my control. Now it's like, I don't know, some, some using aws, some are using third party services. So to do a performance test, how do you handle the infrastructure in a CI CD pipeline?

Yeah, I think what we try to do is give meaningful feedback to the teams without really understanding how they run. For me it's not important if a component runs on our private cloud or on another data center or maybe at Amazon, as long as the team knows where it runs. So they do the translation with the feedback that we give. And I think that's the only way you can do it because things change, you know, you can think, you can, you know everything, but next week it's already somebody migrated to another server, for example.

So also having an SRE type of role now as well. When you're doing performance, do you ever do experiments then to say what happens if this microservice goes down? How does it affect these other components that may have a dependency on it?

Yeah, we did have, we did have it running during our Daily test for a moment. We really like the output but they're looking at it from a different way. It's now more like doing it isolated. Isolated chaos engineering testing. So more specific, focused on a component and an infrastructure instead of just blasting it on the whole environment. And I think that's also because you want to get this feedback directly to the teams and not via us. So we do of course experience a lot of things, but not like a controlled chaos engineering thingy at our test, but we do at ing. We do use it a lot of course.

So you know, with the whole pandemic thing going on, a lot more people using their mobile devices and being a banking app, it's not just a trivial app. Like people are probably doing real transactions now from their phone. That's even more complexity. So how are you handling that in your CICD or in general? Do you test on all the devices or do you have the number the most popular devices you always test on, how does that work?

Yeah, that's more from a functional perspective because we are looking more from the APIs onwards. So the device is not included in this load testing setup. From a functional perspective. I'm not sure exactly how it works. I would guess a bit. I know they made some setup but I'm not sure exactly all the details. So if you want to know more about it, I can ask somebody to do a presentation about that.

I was thinking more of the WI fi conditions, perform it from a UX type of perspective where they're in different locations so their WI fi might not be as strong and things like that where there's a low.

Yeah, we used to limit the bandwidth due to our non functional requirements were also set for a specific bandwidth but we don't do that anymore because we don't control that much. And the setup, at least for mobile, because you have your application and the API doesn't do a lot of data. So I am sure that some customers with the bad WI fi will receive worse response times. But it is not something that our performance approach is covering. It's more covered by like a real time user monitoring and those kind of principles.

Great. So you know, you were talking about how your team, the teams are more and more responsible for a lot of these activities. You're like an outside consultant that actually helps them. But a lot of the heavy load is done by the teams now. So do all the teams have performance as part of the definition of done? Like oh, that needs to be performing and they're thinking about it as they're developing yeah, definitely.

They do have their non functional requirements, of course, which is not only related to security but also to performance. So mobile has a general rule of thumb that the 99 percentile needs to be below 1 second. For example, for ING, it's mobile first. Now the other channels are having less load and mobile is still growing because I'm not sure how that is in America, but when I do banking business I mostly use the app and if I look at it five years ago, I mostly use the browser. So it does shift, but it also means that the peak periods, like for example the old peak periods we have shift. So it's a way more challenging way of working. Absolutely.

How do they manage the backlogs then? Say they deliver a piece of software, it goes on the CICD and then all of a sudden the performance is bad and some automation tests failed. Functional automation. And there's a security alert on, I don't know, on one of the check ins. Do they have to prioritize?

I guess, yeah, yeah, absolutely. The chances are you can't go live. You will probably see this issue before going to production in our setup. But of course there are situations that you miss something in your test setup or things are different in the real world. Right. It could be maybe test data or you miss something in your idea in your use case, for example. So there are situations possible that your safety net. Because the test I'm talking about is it's actually also used as a safety net and it gives feedback to the teams, which is great, but it also should cover some safety for issues that teams on a component level don't see. But we do see it because we see the whole picture. So yeah, I think the chances are that they can't, they won't go live, but if they do, then you have the like depending on the, on the impact you would refer, you would refer, go back to the previous version or you would fix it in production. But then you have a major incident and then everybody is involved. Right. You have the team around you and the management. So it's not something that you can just say, well, I do it when I feel like it. We take it very important. Reliability and performance. That's our license to operate. To be honest. Features are nice, but if it doesn't perform and it's not available as it should be, then your customers are not happy. And that's, that's what we do it for, right?

Yeah, I guess. Also what makes it. It's like you're not testing a simple application, a simple web device it's this is hardcore financial banking type. How do you handle the data? I mean, I can't imagine. I know when I worked in insurance like we had jobs that only ran at night and it was hard to performance test because it was all these kind of older infrastructure we had to still deal with. I don't know if that's still the case with financial systems. I'm sure you have to deal with older kind of processes as well and merge both together to get a. How does that work?

Well, it works two ways. Of course you can't use production data, so you need to depersonalize it and you have. What we use in our test approach is having the accounts. So we know the accounts on acceptance environment and we have the passwords and so on, which are not related to production, if you want to ask. We just randomize the usernames and the passwords and we create them on the acceptance environment and for the data itself. So the transactions, the products, those kind of things are also from production side. But it's like a data refresh and a database depersonalization. So the data should be more or less as it is in production, let's say from a month old. But it's not retraceable to any customer. And that's the most important thing. Else we do have a lot of issues because that's the risk you don't want. So that's always on our mind. But the load test of course also generates data, right? The load test does like transfers and we try to have them of course as production like as possible. But over the days, over the weeks, your environment is of course different. Every test you do impact your data. So I'm not sure what the impact would be, but I guess we had for example in the past an issue that we were stupid enough to transfer $0.01 to the same account, for example. And yeah, working on mainframes and ASP non stops that gave performance issue because you had one account with a million accounts with 2 million transactions, for example. So it did mess up a bit. But we try to keep it as close to production, but we know it does differ a bit. And that's the thing, when you go to production, you always have a slight risk of not getting the test circumstances as right as possible.

How do you try to make it as accurate as possible? I know a lot of companies have monitors in production, they create tests based on that, but it's totally nominalized, like it doesn't have any personal data or anything. But it's legitimate transactions that type of transactions that occurred in a normal day.

Yeah, there are two levels. One is the refresh cycle, which is monthly for most systems and then of course getting the anomized data on it. And the other one is actually in our auto analysis dashboard. But you normally do as a performance tester, you don't really look at production. Right. It's just what you see. But now we also look at the same metrics on component level, on production. So we see a component X get on endpoint y, it gets 1 million requests and it had this kind of characteristics. And we also look at the yesterday. In this case we have a job running, look at all components yesterday and compare it a bit. Of course, the load could be different because we tend to load as peak load and not. We don't deviate a lot about load. And of, of course, production on a Monday it could be X and on a Friday it could be more. So there are some. Your load will differ a bit, but your percentages between the endpoints and your response sizes on the endpoints and those kind of things also tell you if you are doing the right stuff. And we automated that also we looked at actually four different levels. One is do we miss components? Like we run our load test scripts from all the teams, but maybe there's a new component that we miss or something else we can sort that on load. So if we see a new component with high load, we know we have to work. And then if the components matches, then per component we look at the endpoints. Do we miss certain endpoints of those components that we see on production but not on acceptance? It could be like a changing of an address endpoint, for example, that's a low load, low impact transaction. We don't do that in our test, for example. So then we look at do we get all the high load endpoints that are important to us? And then we go one step further, is from those endpoints that we match, what was the load, what was the response and those kind of characteristics. So it gives you a pretty good idea of your test coverage because we calculate a score on it. The score is totally. It doesn't say so much. Only if the value is higher in time you're doing good, if the value is lower, you're doing worse. Let's say we aim for 80% coverage and that is the quality cycle that we drive on. So we see does our test still make sense or do we need to step up and add new stuff, like more mobile, for example. And that is, I think one of the things that are missing. When you do things manually, you don't have all that overview and the detail that you can do when you automate.

Does this help to account for any type of seasonal transactions? I'm just thinking insurance, like RE registrations, people upping their policy every year. It's like there was an uptick in activity.

No it isn't. Our seasonal transactions is more based on the salary schedule of the Netherlands and it's probably an old method which is should look into because we are more international and we we grow over our border borders also on a system kind of way. But the old way was looking at the month that everybody gets their holiday money which is somewhere in May, end of May and around Christmas New Year. That was the busy period on my ing and we didn't have like high load first of January things. But the teams have the most view, the best view on that. We don't know what we're looking at as a performance test. We look at data but that's probably it. Right?

Okay Stefan, before we go, is there one piece of actual advice you can give to someone to help them with their performance testing efforts and what's the best way to find or contact you?

Yeah, the best way is just via email. Stephan.mousetng.com I think you can maybe share it somewhere and yeah for me it's the best way is automate it. Start small but have a plan of what you want to do. And don't be afraid to step on some toes doing it because once you start, not everybody will understand what you're aiming to. But once you on your way there are many more people want to join. I mean it took me ages to get some countries involved but once the first country was there, now you get so many more requests because they see the use. So don't get discouraged when things don't seem too popular because you know why you do it and you will see that it does bring the value in the end. And always be flexible. Also from an organizational standpoint like you build it, you run it principle is for example very important. So you you need to try to incorporate it also in your processes. But don't be afraid to change things.

Thanks again for your performance testing awesomeness. Links to everything of value we covered in this episode. Head on over to testguild.com P85 and while you're there, make sure to click on the try them both today link under the exclusive sponsor section to learn all about SmartBear's two awesome performance test tool solutions, Load Ninja and Load UI Pro. And if the show has helped you in any way, why not rate and review it in itunes? Reviews really do matter in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Performance and Site Reliability Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack performance testing awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Performance and Site Reliability Podcast. Head on over to test guild.com for full show notes, amazing blog, blog articles, and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.