Introduction to Performance Management with Arthur Luimes

Welcome to the Test Guild Performance and Site Reliability Podcast where we all get together to learn more about performance testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Performance and Site Reliability Podcast. Today we'll be talking with Arthur all about introduction to performance management, performance testing in general, and much more. Arthur is a principal performance consultant with more than 30 years of experience in the area of performance and capacity management, infrastructure, organization, security, network management, design, auditing and troubleshooting of large enterprise networks and business applications. Really knows his stuff. I met him on the PAC for NIOTIS call maybe a month or so back. Really, really intelligent guy. Really knows his stuff. You don't want to miss this episode. Check it out. This episode is brought to you by SmartBear. Listen. Load testing is tough. Investing in the right tools to automate tests, identify bottlenecks and resolve issues quickly could save your organization time and money. SmartBear offers a suite of performance tools like Load Ninja, which is a SaaS UI load testing tool, and Loaduipro, an API load testing tool to help teams get full visibility and its UI and API performance so you can release and recover faster than ever. Give it a shot. It's free and easy to try. Head on over to smartbear.com solutions performancetesting to learn more. Hey Arthur. Welcome to the guild.

Hey Joe. Thanks for having me. Wonderful. It's good to be here.

Awesome. Before we get into it, is there anything I missed in your bio that you want the Guild to know more about?

No, I think you've completed it. That's very complete. Thank you very much.

Very cool. So I thought we'd start off at a really high level. You have a really good video on this as well. But, you know, how do you explain to people what is performance or what is performance testing?

Yeah, I think performance generally, well, it depends on who you ask. So if you would ask a normal user, they would say performance. Well, I only bother about performance when it gets really bad. So if the response times are getting very high and I can't move on and I get stuck or I feel the site is unstable, then I start to not trust the site. However, if you ask an IT manager or a traditional performance tester, they will tell you, well, performance, that's the response time of the HTTP calls that we measure in Splunk or in our other dashboarding tool or that we load test. So it really depends on what you ask to which person what performance is. But generally I think performance means that things are Fast or fast enough. And also that things are stable, so predictable, so roughly always the same.

Great. So when you see people taking on performance activities within their sprints, how do you know that everyone's talking about the same thing then? It seems like even just the term performance is kind of not understood between everyone and the team itself.

That's a good question. Actually. I usually ask them, so what do you actually mean about performance? Do you mean response times? And then they'll say, yeah, yeah, yeah. And then I ask, well, can you show me something about what you mean with response time? Are you monitoring it in production? Are you load testing for it? Do you have non functional requirements for it? So then generally the discussion comes up and then it's more clear about what they actually mean with performance.

Absolutely. And a lot of times those non functional requirements, you get something like it should be fast or you get reports that it's not fast enough. So how do you drill down and to make that a little more granial?

Yeah, that's a very good point. So, so usually if you ask the business what is a good performance requirement? They'll tell you it should be blazingly fast, or it should be as fast as Google or faster or it should be zero seconds. Well, in case it doesn't have to be. In most cases it doesn't have to be 0 seconds. In most cases, a good response time, a response time where the user will feel that it's actually a pretty good site, is roughly one second. So if most of your transactions are roughly one second and most of your user actions are roughly one second, you're doing pretty well.

Cool. And another thing that you talked about is performance management. Now I don't know if I never heard this. Is this an official term? What is performance management?

Yeah, well, I'm not sure if it's an official term. I've looked for it, I've looked for books for it, but I'm afraid that there isn't very much available for it. I would actually call, to be more precise, application performance management. And application performance management is everything you do to make sure that your applications that you send to the end users are doing well, are fast enough and are stable. And that's not just load testing. I'm hoping that we can talk about a little bit about that later.

Great. Yeah, so I thought we'd dive into maybe the different areas of performance. When people think of performance, the first thing they think about is load testing. But I think you have like seven areas or eight that you focused in on and one of Them was a performance infrastructure and software architecture. So what kind of activities go into that particular area of performance when someone's trying to plan performance for the teams?

So the architecture really not only determines the performance of your site, but also determines the stability of your site. So if the architecture only holds, say, one web server, then of course, when that web server goes down, then this whole site will be going down. Of course, a lot of architects understand that. So they go for a distributed architecture where there are multiple websites and multiple application servers, multiple databases also, but very rarely this is actually tested. So if one of the servers goes down, what actually happens to the end users and what happens to the end user response times? So that's one of the things. Another thing is that if you think of an application that is actually dependent on many, many components, then each of these components will have a small contribution in the response time, and each of these components will have a small variation on that response time. So you can imagine if you look at the end user, what the end user sees is a response, then that can be greatly varying, much more variation than what you see of the individual components. And then if you go to the IT managers of all of these individual components, they say, the managers of the web servers, they'll tell you, no, my site is doing fine, my web server is doing fine, there's nothing wrong. And they ask the database, people say, no, everything is fine. But I have a small variation, but not a lot. But very rarely you can see the effects of all these, of all these, the different components on the response time of the end user. That is something that is usually not investigated, but the end user is really suffering from that. So I think that's very important part of having a large chain in architecture, to make sure that you do something so that it's not just fast, but it's also stable and predictable.

So it's a great point. There's so many points of failure that performance could occur in, especially when you have an end user reporting something and you don't see it anywhere within, in house. So how do you troubleshoot those type of issues and how do you prepare for those type of issues?

Yeah, it's a very good question. And that's the one that usually gets asked the most when it comes to how do you do proper performance management? I think it's very difficult to troubleshoot an individual user unless you have tools which can, can monitor individual users, which are sometimes expensive and sometimes not. But, but it's usually more difficult to pinpoint an individual user. However, when all of your users are suffering, then it's usually much easier. And then you can do multiple things. Most people just look at logs and web server logs, look at database logs, application server logs, and to try to figure out where in the chain things get slow. However, there are also a chain based application monitoring tools that can help you, that can tell you exactly, okay, this much time is spent on the web server, and this much time is spent on the application server. This much time is spent in the middleware layer. This much time is spent on the database. And that really helps you to zoom in on where the actual pain is, what is really causing the higher response time.

When I did performance testing full time, it was probably 20 years ago, you know, 18 years ago. And what we did for that is we used to ship machines to our clients and then we'd have a little agent that ran and then if there was performance issues like runtime, just as a one user, we'd get pinged and we'd get alerted before there was a big issue. I'm thinking things have gotten more sophisticated than the way I did it then. So how would someone do something like that nowadays?

I think you're from the same generation as me, because I remember doing that. Also I worked for IBM and we had different PCs running on the desks of people in different areas of the world. One in South Africa, one in Israel, one in different parts of Europe. And they would just log on to the application and then report the response times to a central database where you can show graphs on it. Now that kind of functionality, which is called active performance monitoring, is still available. Vendors still sell it, you can still build it yourself. But I think we're moving more and more to a situation where applications are just getting so complex that it's very difficult to figure out what the user is going to do. So that it's also very difficult to script then that active monitoring script. And also we're going to a situation where the individual users can have a lot of fluctuations that your robot will never see because your robot cannot log on with 10 times per second, for instance, like your end users are doing. So those good old times where you had a lot of active robots sitting in different places of the world, I think they're still there, but we need more and more to monitor what the real end user is doing.

Cool. So I think another area where maybe we could build things into it to make it easier maybe to troubleshoot when people have an issue in software development. So how does performance actually fit into software development? Once Again, back in the day, used to be I was part of a separate team. They develop the software, they give it to us and then we test it in a staging environment and then it would bomb in performance because it was never like it was in staging. So how can you build performance into the software development piece?

It's a very good question. And I think you can do all kinds of things with tooling and with automated software checks. You can build your own rules to check for the quality of your software. And Sonar has some of these rules. So that will alert you if you write code that is not performance. However, I think it all starts off with the software developer. I think the software developer these days, they don't have a good picture of what it actually requires to write performance software. And I'm not blaming those developers because I think software development is a very difficult topic. It's getting more and more rich. It's not just about knowing Java. You need to know the full stack. You need to know about JavaScript, you need to know about style sheets, you need to know about how to interact with your databases, middleware, you need to know about cloud. So software development is a very, very difficult topic. Yeah, it's not a big surprise that software developers will make mistakes. And then I think it's good first to have a sort of a sense of sort of automatic automated check to make sure that, that the most obvious mistakes don't go to production. And then of course you can also automate your load tests to be part of the CI CD chain to make sure that software that is much slower or uses much more CPU or much more memory than before than the previous release will not go to production.

Very cool. So is there any way more developers embrace actually building these type of performance things into their software?

Yeah, yeah, it is done. I've seen a number of companies where that is done successfully. However, you need to ask for priority. You need to constantly ask for priority because a lot of teams, most teams are under a lot of pressure, there are a lot of deadlines, there's huge backlog. And so the idea is then to make sure that the product owner in that backlog has enough visibility for performance and also puts performance onto maybe not the highest priority, but at least give it some priority. So what I recommend usually to development teams to DevOps teams to make sure that at least say 10% of your sprint or maybe 20% of your sprint goes into non functional requirements testing and fixing and fixing bugs. So that's not just performance, it's also about security, maintainability logging, monitoring, but just to make sure that the non functional requirements aren't forgotten in the whole urge and need there is for new software these days that needs to go to production really fast.

Well, I don't know how you sell that. I used to work for a large organization, that's the first thing they never would estimate is non functional. They'd add it like extra work. How do you sell that to them?

I think the difficulty is in the word non functional. So once you say the word non functional then they think, okay, that doesn't deliver anything for the client. Right. So it's about talking, it's about talking to the possibility to have a stable performance or to have a stable application. So this is where usually the words I use, it's about making sure that the application has a high uptime and is available when the customer needs it. So when I use these kind of words then of course I'm talking about non functional requirements. That's sort of more easily accepted than using a word which has the word non already in it.

Very cool. So another area that you mentioned with performance is I think it was configuration management. This is another area I think that folks don't necessarily focus in on. Can you talk a little bit about how performance fits into configuration management?

That is a really cool topic John. Very happy that you ask about it because configuration management in the times that before we had cloud was already a hot topic. Configuration management as we know is about to have a database or at least some way of knowing what of your infrastructure is, where, so where are your CIs, where are your configuration items? So where is your database located? Which version does it run? How does it depend on the application, which services are calling, which other services. So that of course is absolutely necessary not only to tune your application to make it faster, but also to solve problems in production. Now these days with the cloud, configuration management is getting hotter and hotter every day. So a lot of customers that I see that have an infrastructure as a service provider and then have another application as a service provider or a platform as a service provider and they don't match, right? The company X and the other one could be from company Y. So how are you going to figure out if your application is slow in this kind of cloud microservice environments? So what is causing what? So it's getting very, very difficult. Especially because microservices by nature they run wherever they want and there is a platform underneath it of course that roughly looks at where there is least CPU usage, where there's least memory usage and then I put a new application on that particular server. However, and to go into the depth to understand why your one particular microservice is now slow, while before it wasn't slow, and if it maybe depends on a hardware upgrade that your infrastructure as a service provider has done for that is that is really difficult. And I'm very excited actually that there are some vendors who really understand this and now make sure that they have cloud monitoring solutions that are not just about following a request through the chain, so from one microservice to the next, but also can look at the depth. So how is this particular microservice on which Java version is it running, on which spring version is it running and which microservices running, and then on which physical machine is it running underneath all of that? So that is really exciting and I think that's the future of performance tuning and performance monitoring to have software solutions that not just you vertically so in the chain, so horizontally so in the chain tell you where the response time is slow, but also tell you vertically. So why this particular microservice is slow because, well, maybe it was just put on a physical machine that was already overloaded or just has very old hardware.

You know, that's a great point. A lot of times people think the cloud is going to save them from everything, but you're saying it almost adds some extra layer of difficulty because you don't necessarily know if it's in the cloud or on another service or another cloud provider.

Exactly. Yeah, they say that troubleshootability. I'm not sure if that's an English word, but.

Oh, I love that word.

The troubleshootability is getting less. And of course cloud means that some of the issues we had before, namely that you didn't have enough servers and that those can be solved. Right. That is a big, big, big advantage of the cloud. However, cloud also introduces, due to all these different virtualization layers, new problems that will not just affect the response time of the end user, but also the thing I just mentioned in the beginning, the predictability of the response time. So we'll probably, with more and more cloud solutions, see more and more variation in the response time and we'll have a very hard time to figure out what is causing that variation in the response time.

So speaking of cloud, I think one lost area that people, well, I don't hear much about, I don't know if it's been renamed as capacity management. I don't know if that's the same as chaos engineering, but back in the Day when I used to work for an insurance company, we always did capacity planning for our performance. So what is capacity management? How has it changed over the years within the performance realm?

Yeah, capacity management is very exciting. It has a lot to do with using the right amount of storage, the right amount of cpu, using the right amount of memory, using the right amount of network bandwidth. And I use the word right amount a couple of times because it means about not using too little, but also not using too much. So and then the question comes, what does that mean, too little or too much? Specifically, when we go to the cloud, before it used to be very clear, right, you had your application consisted of 10 different servers, web server application databases, etc. And then, okay, the server, they cost, I don't know, thousand euros per month. So you can calculate what your infrastructure costs. However, with microservices, this is all much more vague and much more unclear. And it could very well be that you introduce a microservice that just uses a lot of CPU a lot more than is actually needed and that therefore uses a lot more microservice instances than you might need, uses a lot more of the underlying hardware and therefore costs a lot of money, which might be completely unnecessary. So I think that is a challenge with the cloud because it scales so easily. You can just get new instances roughly in a couple of seconds. However, it doesn't mean that developers should now just relax and build, just build inefficient applications. You're looking at the amount of CPU that you have is still important. Looking at the amount of calls that you do to backends is very important. Looking at the amount of memory that you use, what you put in, in garbage collection, for instance, what can be garbage collecting and whatnot is still very important.

Yeah, absolutely. I think what really highlights this is when you say we need to help making this more performant, they're like, oh, we can just run it to infinity because it's in the cloud. But then you say, yeah, but it's going to cost us an extra million dollars doing that. So at least now you have some kind of cost benefit. You can say, let's say if you help performance, we don't have to utilize the service as much and therefore that's why we should invest time in fixing it.

Absolutely, yeah. In the old days, what I used to strive for is to have some sort of a measure of at least how much CPU one request per second would take. We used to call that the CPU per hit per second. And then we say, okay, there is a certain number. If you have more than 3% CPU per hit per second, that means that you're writing a very inefficient application. However, percent cpu, what does that mean in a cloud environment? So there it becomes more difficult to have these kind of measures, but at least you can look at. So how many instances do you actually need to run your load? Right. So then you really talk about, they say you have 100 hits per second. How many instances would you need for that? So that would be instances per 100 hits per second or something like that. That is something that we should strive for. I'm really looking forward to discussing these kind of topics more. Also with performance engineers who do a lot of testing and a lot of monitoring in the cloud, and also some cloud architects to see how we can find a sort of a framework on how to define efficiency CPU and memory efficiency in the cloud environment.

Absolutely. And talking about monitoring, that's another area that you focus in on end to end monitoring. There's so many things that could be monitored. I think someone showed me on a front end they had like multiple APM solutions with the JavaScript and actually they had so much that actually slowed their site down because they were trying to collect too much data. How does someone do end to end monitoring correctly in a performance type of approach?

It's definitely true that you will always have some sort of overhead when you monitor anything. It doesn't matter what you do, even if you just log, okay, this took so much. You write something to a log file that will have some overhead, of course, but the overhead should be low. Right. Usually I consider 2, 3% sort of a maximum for a monitoring solution. I think if you have more than that, then you should really look at. Are you not measuring all kinds of things that you're not interested in anyway? And the question on what to monitor depends really on what you're interested in. Right. What are your non functional requirements? What are they about? Are they about just roughly response time and then what kind of response time? In browsers, for instance, there are multiple metrics which you could call response time. There's the page load time, there's the time to first paint, there's a time to first meaningful paint and there are a huge bunch of other criteria all having to do something with time that can be measured. It's really important to pick only those measures that you're interested in and report only on that because otherwise it will just confuse people and it will confuse operations and it won't help you when you have A real problem on your site.

Great. Now, another thing that people chip up on, I think, is the different types of load testing that are available. Performance testing you mentioned at the beginning of the show. Hopefully we'll talk more about performance testing as being more than just load testing or stress testing. So what are some different types of load testing? You think people should know more about that Maybe they're not focusing in on that would really benefit them.

Yeah, just like performance management, it's not just load testing, but also there's more than one load test as well. For instance, like the load test that is mostly used is something called the peak load test, which makes sense, right? So you ask people in the business or you look at logs from the past and you start to wonder, so what is your, what is the peak load that you can be expecting? So that could be, I don't know, 100 hits per second or something like that. And then you test for that, you write a script and you create those hundred hits per second and then you run it for some time to make sure that you have a stable load and you have enough measurement points so you can say something meaningful about it. But I think peak load tests, even though they're cool, they don't tell you everything. There are other tests which are also very essential. For instance, stability test. So what we usually do, what we usually mean with stability test is that you look at how much request your application can actually handle and then you do a little bit less, say 60% of that, and then you let that run for a couple of hours, say at least eight or maybe over the weekend, so that your application gets hit by maybe millions of requests. And then you look at long term trends, right? You see are the response time slowly increasing, are your logs filling up faster than what you would like? Maybe if you look at your memory, is there anything in the memory which is slowly increasing and doesn't get garbage collected? You could have a memory leak. For instance, you can look at your amount of connections that you open to backend systems. Are they being closed again? So is the number of connections is that stable? Otherwise you could have a connection leak. So there are multiple things you can look at during a duration test. A third type of test I'd like to mention is a breakpoint test that basically you have a linear increase in the load, basically until your application breaks. So that braking that usually happens in certain phases, usually at roughly 60% CPU, you'll see the response times increasing. But if you wait a little bit in the load test, then you don't increase any further than usually the application stabilizes. And then you can continue on, right, Add more and more and more load to it until the application really creates a very high response time to give errors. And then what you can do is can slowly go back, so lower the load a little bit with every time 10% less and less and less until your application recovers again. Meaning that the response times are under what you defined in your, in your SLA or in your non functional requirement. And that point, that load, which is the load that your application can handle even when it's been overloaded. That's what I call the breaking point load or the maximum throughput of your system. I think that's a very important test, this breakpoint test to tell you what the maximal throughput of the system is. And then there are other tests which have to do more with the stability of the whole infrastructure, which are very exciting. There can be called a slow or no backend tests. So a slow backend test is basically you take your application and you. Most applications will have a backend, so say an application server or a database or something and you make that database or the application server that's behind it make it slow. It can be done in multiple ways. We can talk about that later. And you do it in small steps. You make it slower, slower, slower, slower until there are all kinds of errors and then you make it fast again. And that will tell you a number of things. But one of the most important things it will tell you is that when your database is slow or has been unavailable and it comes back online, that your application recovers by itself. So it's basically about the recoverability of your application in the chain. And these kind of tests are getting more and more important, especially when you talk about a microservices landscape where a lot of services are dependent on a lot of other services and this sort of auto recoverability should be built into the microservices satho.

You have a wealth of information and this type of information to me is it's almost like a specialized skill set. But what I see is companies go more and more agile DevOps Site reliability engineering now is the hot topic. Where does performance fit in? Is there still a role for a performance engineer? Or how do you see companies being able to soak in all this information, actually know about it and then know how to apply it without having, you know, a team called like a performance team, like back in the day?

Well, it's not just back in the days companies I have seen, they usually have still have a performance team, a performance competence center you could call them, where people have this kind of expertise and they can train others. They can train developers for instance to run little load tests for themselves, like unit load tests for themselves. And they can teach developers on how not to make some basic mistakes in coding. They can teach operations what it means to have a proper dashboard to make sure that the non functional requirements are being checked in production as well. So I think this kind of performance competence center is needed and they could also educate for instance reliability engineers who are actually part of the team. So I think there is still a lot of work and there will be only more work for load testers and it's just that load testers will be more and more automators and teachers. So I think being somebody who communicates very well not just is able to write a good performance script, but communicating, teaching, advising, having an advisory role, those are kind of skills that are being asked more and more of performance engineers these days.

Okay Arthur, before we go, is there one piece of actual advice you can give to someone to help them with their performance testing efforts and what's the best way to find or contact you?

They can contact me through LinkedIn. My name is Arthur Lames. L U I M E S I'm a Dutch person. I work for a company called altosys Performance and I'm looking very much forward to having all kinds of discussion, really very looking forward to also to train people in the area of performance management because that's really where my passion is. And I think when the first thing, I guess if the question is like the first thing, what should I do in order to ensure a good performance of your application? The funny thing is is that the first thing you should do is to ask yourself what is a good performance requirement? So what is a proper time that is acceptable to end users? If you don't have that, you can try to tune, you can try to monitor, you can try to load test, but it doesn't really bring you anywhere because you don't really have a goal to work towards.

Thank you out there for your performance testing awesomeness. For links to everything of value we covered in this episode, head on over to testguild.com P44. And while you're there, make sure to click on the Try them both Today link under the exclusive Sponsor section to learn all about SmartBear's two awesome performance testing tools, Load Ninja and Load UI Pro. So that's it for this episode of the Test Guild Performance and Set Reliability Podcast. I'm Joe and my mission is to help you succeed with creating end to end full stack performance testing awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Performance and Site Reliability Podcast. Head on over to testguild.com for full show notes, amazing blog articles, and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.