Performance (And Load) Testing In Production with Leandro Melendez

Get ready to discover some of the most actionable DevOps techniques and tooling, including performance and reliability for some of the world's smartest engineers. Hey, I'm Joe Colantonio, host of the DevOps Toolchain podcast, and my goal is to help you create DevOps toolchain awesomeness. Hey, it's Joe and welcome to another episode of the Tuskill DevOps Toolchain podcast. Today we'll be talking with Leandro Melendez all about performance and load testing and in production. This podcast audio was actually taken from an Automation Guild session from earlier this year with Leandro, where he shares why the traditional methods of performance testing may not be suitable anymore. So you're going to learn several recommendations to help embrace these types of tests in your production environment, and Leandro delves into this info as usual with fun analogies and fantastic stories. If you don't know, Leandro is a performance testing advocate with Key6Grafana, helping everyone to ramp up on their performance practices. He's over 20 years of experience working with some of the top companies in the world and he's also a popular author at his blog Senior Performo. He's also an international speaker and he participates in multiple conferences, events and webinars with keynotes, workshops and multiple talks under his belt. And lastly, he is the author of the Hitchhiker's Guide to Load Testing Projects, which is a fun walkthrough that will guide you through all the phases or levels of a standard IT load testing project. You can check that out in the show notes@testguild.com forward/p1.25 if you don't know, since 2017 I've been running an online event called Automation Guild, which this session was taken from, and the first three days are dedicated to front end testing, with the last two dedicated towards more DevOps automation, performance and security topics. So the reason why I bring this up is that I just opened up a call for speaker for the 2024 event taking place in February 5th to the 9th. So if you have a DevOps related topic you'd like to speak about, share your idea@guildspeaker.com hey, if your app is slow, it could be worse than an error. It could be frustrating, and in my experience, frustrated users don't last long. But since slow performance is a sudden, it's hard for standard error monitoring tools to catch. That's why bugsnag, one of the best production visibility solutions in the industry, has a way to automatically watch for these issues. Real user monitoring, IT detects and reports real user performance data in real time so you can quickly identify lags. Plus get the context of where the lags are and and how to fix them. Don't rely on frustrated user feedback. Find out for yourself. Go to Bugsnag.com and click on the free trial button. No credit card required. Support the show and check them out.

Hello amigos. Welcome to this Automation Guild 2023 session where here your amigo Leandro Melendez. We will be having some fun and some lot, well not some lots of knowledge in this presentation that I have prepared for you today. So let's get ready. Let's move aside so that you can see what I have prepared for you today. Welcome as I mentioned, to this Automation Guild from the Guild conferences of the 2023 version. Today I am going to be talking about performance and load testing in production. Something that has been considered like a taboo or something that hidden. So for the ones that do not know me, I am Leandro Melendez, also known as Senor Performo. I am a Deborah, a performance advocate with K6. I have this not so secret identity of the Senor performer. Starting writing blog posts in the Senor Performo blog, but as well I am a podcaster, I am the host of Pervites Espanol and as well I have a YouTube channel, Senor Performo esp for Espanol and Senor Performo Eng for English. And last but not least, I am an author of a book I published not long ago, the Hitchhiking Guide to Load Testing Projects. A fun walkthrough over traditional and load testing projects, not to confuse with performance. On today's session, what are we going to be talking about? I divided today's presentation in three main blocks. First, I'm going to talk about traditional performance testing. You know that one that we do not touch in production, which is a topic for today. And next we're going to get into production. We are going to analyze production and see how this environment is working and to check what differences are there and what happens to it so that we can start testing on it. And last I'm going to give you some tips actually how to start testing on it. Let's get into it. First, I'm going to talk to you about some ancient situations, ancient techniques around the performance, how it was being done in the past.

Once upon a time, as you may be seeing here, performance testing was different. First we had physical servers, those big bulky machines that we had down on the basement. In my days, performance used to be different from how it is today as well. We had the waterfall methodologies. These phases were the software would not be released often only once or twice in a year. And it was a big event. Not much could be changed after you released. So as well, because our server was in a box in the basement down there, we had somewhat limited or fixed resources. Probably you know what I am referring to nowadays in these modern days, we have some differences in terms of our environments as well. Had to do with these monoliths. Again, everything had to do with this single box that we had down in the basement. Inside of it was the whole solution. It was packed up inside of it. It was like a single item that we had to take care altogether. Because of this, the tests could not be done Generally, performance tests could not be done in production in that single box that we had. We had to test in another box, separate pre prod qa staging all those good things. And probably the reason why you are watching this presentation. We had some situations to record outside of to work in production. So production, talking about it, as I mentioned earlier, was large. Was a huge environment, powerful machines, too expensive. Was the most powerful and largest machine probably that you had in them in the company. And once you had this production machine ready and running, it turned out to be somewhat fragile, not so stable.

Was a little bit complicated. You didn't want to touch it. Being this situation that once you released it was fragile, it was, well, something that you should not touch that much. It was tough to release. It was a huge process. You had to deal with lots of loopholes or Gandalf would step in front of you. Though shall not pass telling you every time that you're trying to release into production. There were several problems around it as well. On other times. Performance testing in the past was as you know, load. I know, I know. I have repeated this every time you see me here in the Automation guild and any other places. But generally in the past, performance testing meant load, burn it all, load those ones that we're trying to bring down the system. And because of that, because often we brought down the system doing our performance. So we had to test somewhere else, not in production. That's the main reason why all of this was avoided. Because people was afraid of us testing our performance in production. And somewhat justifiably, as I mentioned, these load tests were focusing on trying to bring down the system and why were they afraid? Right? Again, as I mentioned, performance meant load. We were pushing the limits of the application. We were making so much noise, so much movement in our application. Not just Measuring and testing the performance. Pay attention to this because this is what's going to bring everything that we're going to be talking about. But because of our tests, the system may stop working, be non productive and not productive or non working. Production was bad. How bad? I mean really, really really bad. We couldn't deliver services, products, sales, internal working capacities for our internal processes. Something that we would like to avoid. So another situation is that in the past waterfall and the applications that we had these monolithic thick solutions that had to be opening the desktop or everything was inside of a browser. It was hard to see the performance, to know the performance. Because of that we tended to try to know the performance of our application through load tests through the purpose of automations was to load test the application. But if the only way to know the performance, to see it and experience it, to know it was was through Automations, which meant load test, there was somewhat of a disconnect, right? As well because of that we had to run the tests in advance before the application reached production. Because as I mentioned earlier, it didn't reach production often, had to do it seldom and prepare a lot before we even reached there which a multitude of other problems. Doing our tests in an environment that is not where the system is going to leave eventually, like permanently live Automations were the only way that we had to know what was the performance of our application. It was seriously difficult in the past. So there were several roadblocks that probably you have been hearing. Production was untouchable because of that special snowflake situation we had with production. It was untouchable, you shall not touch it. That was the biggest taboo that we had in the past. Because of that we required frozen code that was not moving, that our automations took a lot to work on to be stable, to be able to load test the environment. Any little change could break them. And it had to be as similar as possible to production. Otherwise we had to do some interpolations, correlations and see, okay, if the CPU is half the size, if I have half the ram, does that add geometrically or exponentially? So we had some several situations in the past with production and even as if we were doing tests in productive environments, which rarely happened, or if we were in the pre product environment or staging or whatever you want to call it. We had to do it in weird times, weird hours, especially because some other people were working on some of those other environments. They were precious, they were expensive, hard to configure as well fragile and shared many of the situations and characteristics that I have been mentioning so many, many more roadblocks that we would hit getting into production. As an example, we had data. What happens if we burn all the data? If we touch real user data or we delete records? There were several problems and not to mention security because often some of the data in production, well, always is real data. Again, we may be looking at things that we should not and because of the processes that we may be triggering in our productive environments, maybe productive processes, we were triggering them for real. What happens if we are shipping something, if we are sending payments, something serious that if we triggered it, what happens? I don't know. In a hospital we have the ambulance problem and we're load testing. Suddenly a bunch of ambulances leave the hospital going to pick up. I mean there were so many problems, so many situations with production in the past. There was much, much more around it. Now that you saw what are the situations of the past? Let's get into production. Let's see how we have like evolved surpassed some of the limitations that I mentioned in terms, in many, many terms and be able to start to think to safely get into production and start working with it first. There are some reasons, right? I have been mentioning this. Why not? Why shouldn't be doing performance test in production first is because again everyone performance testing is load testing is capacity planning, is trying to bring down the system. That's not right. If you know me, you know performance is different from load testing. A big concern was that our load, our performance tests or load tests would bring down production. And that again, as I showed you earlier, is bad, seriously bad. We needed a lot of automations, there was no other way. So if we wanted to know the performance of all the elements, we needed to do lots of automations to figure it out. Also had some drawbacks. Unstable.

Data. That was another the limitations of the automations. If you didn't have enough data, how am I going to populate the data? Am I going to burn it in production? Am I? It's secure, usable as well. The hardware was limited both in two situations production. If we pushed it a little bit too much or a lot much, it may go down. And as well for generating the load that we needed, we needed some hardware. And the situation is that it was already in production. I mean when we hit the productive environment, usually it already was too late. I mean it's already in production. Why do you want to test it? We already know that it survived from the previous phase of performance testing. You did, right? Oh, it was early, but lots of things have changed over the years. The future is not coming. We're already in the future. We already have several situations where our productive environments have evolved. Our systems, our processes, everything around us. To start off, we don't have more thick solutions like these thick clients that everything was inside. Now we have tiers, now we have services, we have microservices, we have components of production in the cloud, in the ever elastic cloud, which is another situation that we shouldn't be too confident about. But nowadays the cloud is somewhat elastic, is kind of difficult to bring it down. Another thing that we have today, we don't have any more the waterfall processes where we will release seldom once, twice a year tops. Three like crazy environments will be three times a year. No, now we have frequent releases into production. Hopefully we are doing them at the very least once a month, optimally twice a week. If you are awesome, you're releasing into production again, keyword into production, we're releasing elsewhere. That kind of doesn't count that much. But releasing into production every two weeks, There are some organizations that do it even more often, more frequently. The other thing, I have some acronyms. Balt, B A L T Big ass load test. This refers to the ones that are trying to bring down the system, those humongous load tests. When you do those huge load tests in your environment and your solution being in production, being in pre product, whatever, you have some metrics, you know, some validations, you know that the capacity of your productive environment is this in the next sprint, when you release a change, those metrics that you got are invalid, things changed and you cannot rely on the metrics that you had before. So these huge tests are kind of pointless. And aside so being pointless, because now we are continuous and releasing often you don't need them that much anymore. Because as I mentioned earlier, in waterfall days, when you released into production, that's it, no touching, no more. Nothing will happen in the next six months, semester, the next year. So that's it. Nowadays we already have an mvp, a productive environment that is growing naturally, continuously agile ways, in agile manners. Now we are appending, it keeps changing. So we already know that with production load the environment survives or behaves in a given way. When we add up, we just need to make sure that change. We don't need those huge bulks anymore. Pay attention to that. Because that's a big obsession that has been affecting a lot. This term of testing in production and performance. I mean the productive environments has changed in all these terms, how we work with it, it has elasticity. As I Mentioned it can grow with the cloud. Some components are not so easy to measure the capacity. If we are just thinking about capacity, we have these continuous environments now. We are interconnected with services and microservices. Our solution may not be in a single. It's not in a single box anymore. A service may be here, a component may be there. The third component may be separated in three different worldwide locations. So it's a huge tanglement of situations that we cannot bring down to a single box anymore. We are distributed. And another key component of production nowadays is that it can or should. You better have some mechanisms to roll back changes of your productive environments. And that's a big thing that we have in modern software methodologies. And all we can roll back our productive environments should be super simple. Performance is as well evolving and changing. Performance is not only load, not these bolt tests anymore. We have ROM real user monitoring. We can know what is the performance that a real user is experiencing. We have synthetics and unit tests which every little component is just triggered here and there and is reporting to us how is it perform? It's performing it's performance. We don't need that much like that automation component. Even as unit tests are automations. We can and should with performance now measure variations in terms of performance. Release to release to release. Not just like big big load test. Trying everything together. Why am I testing everything together if my release of this sprint is this little thing doesn't affect the old hundred things that I have automations to do. So you don't have to do an everything load test and everything performance test all the time. Or you could depending on what is your distribution. And another key change that we have is observability. With observability we can know how is the performance of our processes, of our users, of our code automatically without needing the automations. I mean in these modern techniques for performance we do need automations to be integrated as I mentioned as synthetics unit test. But not just trying to bring down the system. Some processes, some applications, I mean you can tell back here K6 is a very good solution for those synthetics for continuous performance. But again you can even use it in production. Automations in production. How's that? Synthetics. You can leave a synthetic process running here and there every 10 minutes, 1 minute, 30 seconds, whatever period you want. Just don't turbo load test. What do I mean? Well, you can do small loads. I like to call those chihuahua loads. I mean we have those San Bernard load tests that are humongous. No, no, no, no. No Tiny Chihuahua test. These little bodies, cute, little wiggly, mimic a little bit of load testing, but at a single component. Just 2 users, 5 users, 10 iterations. Tiny. Keep it small. You don't need to bring down the system anymore because you can just iterate as well. There are some ways in which your users, real people, can do load testing. Remember when that was like, it's ridiculous. Are you going to have a room full of people clicking at the same time? Nowadays you can. Nowadays you can. Those are your users and you can do it with what we are going to be looking in a moment. No more bolts in production. No more big load, big ass load tests. Well, at times, at times we will have these Black Friday events, big releases, promotions, big sale, new products. There are situations where, yes, we need to push as far as far as possible and probably even in production, try to push it to see if we will survive that humongous event that is way larger than the current users that we have. We'll get into that in a moment. What are the tips and tricks that we need to achieve these production tests that I'm talking about? First, you will need some minimum requirements to prepare your productive environment. Get ready, pack it before you even think of testing in production. Don't go mindlessly testing in production just because your performer told you, because you may get in trouble. Be careful, be very, very careful with your productive environments. So first thing that you need, easy to rollback code changes. That's, that's one of the cornerstones of being able to test in production. If you release something that doesn't work. Oh man. Doesn't work. Rollback, rollback, rollback. Now we need some release techniques, some ways in which your code will arrive into production and you'll be able to test, we'll see in a moment some of those techniques you need. Great. This is a great power. With great power comes great observability. You need. Observability doesn't work that well. The phrase, I know, but you need to be able to observe what is happening not only in production, but on your environments as they move into production little by little. But always, always know what is happening with production. Observability is key. It's a cornerstone here as well. Big, big, big. I mean big time. Okay, clear. We need automations and synthetics, some of these automations to be running in production continuously. Because if in a productive environment our users have issues, bad performance and they leave. We want to know with observability. We don't have an automation triggering here and There our key processes we may never know. And if our users do not call us, Twitter complains and all those things we'll never know. We need those automations to know how production is doing. And from this we need historical metrics to know from the past how things have been to know if there's a change that may concern us. So first observe again as I said is key. We need great observability in our system. How do we achieve this? With an apm, an application performance monitor, manager, general monitoring. Not only with an APM you can implement logs, traces, monitoring, all those good things and have visibility with tools like a little bit like well back there Grafana and as well you should instrument your code so that you get telemetry. That's also a big first step. All your production should be instrumented so that you know what is happening inside of the code. Several keys, components, elements on your code, on your infrastructure, on your solution, everywhere now metrics and automations. This is something that you can or must have in your application. Short small automations that bring you those metrics that you can receive. Again as I said, chihuahua load tests, not loading, not humongous, not that will bring down the system continuously and if you can have them bring them from QA into production. Some automations that you used in QA pre productive to release are useful here. Just don't overload it, don't burn data, don't do those things, prepare for those things as well. Iterate on important ones, keep them running here and there and keep those metrics. Those are key to know over the story of your environment. I mean you don't need like five year old metrics but let's say the last six months the likes last year to know the response time has been suddenly goes Here we have an issue and that's how you know that there's an issue. You can mix it with ROM real user monitoring which will tell you what your real users are experiencing when you have a new version on your system. Something in a conference I remember someone told me oh you know there are these freaks I think in Facebook they said they just released into production without performance and load testing before. Yes, that's possible. You can just release into production and see what happens. But be careful, don't just recklessly release into production this right away one is do releasing to production run your automations and see that there's no deviation. Again you should have metrics from the past and see okay, what's good, good, good. New release keeps getting good, good, good, good Very good. Monitor everything. If your users receive a new version, you will know if something is wrong because you will see the spike from your real users and your automations. Fix it right away. If you notice that there's a deviation, fix it. Snappy. Roll back. Be ready. Almost like a cowboy in a duel. Ready with the gun. I'm going to roll back as soon as I see something. Be ready. Okay, now another key component. Here are the release techniques. How are you going to release this software in a way that will give you the triumph in this duel of samurais with swords? There are several techniques. Av Canary blue green region release gradual growth. Many of these will allow you to have productive versions to your users and test them and see how your users are doing without affecting everyone. You can have two versions of production and see the new one, the old one. You can have two versions in the same production and the new features. Show it to some users and not to others and see what happens. You can release it just on a world location, a tiny country, a tiny city, something that will not bring down the entire system and you can start seeing how unique change did key, key, key also to have good release techniques and you can let as I said, your real users load test. Production already gets load. You already get productive load in production. It's super easy. You should do it. Please try it. And you can also do it as I mentioned with beta releases, subgroups just divide and conquer. Divide your releases and put it on different environments. Bots, big ass load tests. Stop obsessing about them only when needed. Only when there's a crucial Black Friday situation that you're gonna have. Seriously a big event. That's when you should probably. You have created a lot of things that is easy to see. What is the situation? Sometimes you will have to do it in production. Just be careful. You may be able to create a sub productive environment and see what happens with it still being production. But sometimes the event is so big that you may have to do it in production. Good luck and well, some last thoughts. Performance tests are about speed and efficiency. Don't obsess about these big ass load tests. Don't obsess of trying to bring down the system. If your system is performing well in these agile and continuous days, you'll be good. If you have a big event, do a big ass load test. Prepare production to be robust, to be testable, to be.

Repairable, rollbackable easily and that you can fix it quickly if something happens. And last but not least, observe. Always observe and know what happens in production. And with that I want to thank you very much for enduring this presentation to be hanging on with us. I hope I didn't exceed too much of the time and with that I'm giving it back to Joe. Thank you very much everyone and hope you have happy, productive tests. Adios.

Thank you Leandro, as always for your performance testing DevOps awesomeness for links of everything of value we covered in this episode, head on over to testguild.com P125 and while you're there, make sure to click on the SmartBear link to learn all about SmartBear's awesome solutions to give you the visibility you need to deliver great software. That's smartbear.com hey, thanks again for listening. If you're not already part of our awesome community of 27,000 of the smartest testers, DevOps and automation professionals in the world, we'd love to have you join the fam@testguild.com and if you're in the DevOps automation software testing space or you're a test tool provider and want to offer real world value that can improve the skills or solve a problem for the Guild community, I'd love to hear from you. Head ON over to testguild.info and let's make it happen.