Lessons From a Performance Journey with Leandro Melendez

Welcome to the Test Guild Performance and Site Reliability Podcast where we all get together to learn more about performance testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Performance and Site Reliability Podcast. Today we're talking with Leandro Melendez all about his lessons learned from from his performance journey. If you don't know, Leandro has more than 20 years of experience with IT business from developer, DBA, project manager and others. The last 10 years he's been working in the QA field, specifically on performance and load testing using tools like LoadRunner, WindRunner, JMeter, HP Diagnostics, NeoLoad, AppDynamics, Dynatrace and many others. He knows them all. You probably have also seen him from awesome content that Perfbytes puts out around performance. If you don't know about Perfbytes, you definitely want to check them out. I have a link to them in the show Notes from this episode. He also has an internationally popular performance testing blog called Senior Performo where he curates a diverse set of learning material for performance testers. He's also upped his video and audio game, so he's been releasing a lot of different audio and a lot of different videos. As I mentioned, all that will be in the show Notes of this episode. You don't want to miss this. You're in for a special treat to learn all about performance testing and how his journey relates to how you can learn more and get better in the craft of performance engineering. You don't want to miss it. Check it out.

This episode is brought to you by SmartBear. Listen, load testing is tough. Investing in the right tools to automate tests, identify bottlenecks, and resolve issues quickly could save your organization, time and money. SmartBear offers a suite of performance tools like Load Ninja, which is a SAS UI load testing tool, and LoadUI Pro, an API load testing tool to help teams get full visibility into UI and API performance so you can release and recover faster than ever. Give it a shot. It's free and easy to try. Head on over to smartbear.com solutions performancetesting to learn more.

Hey there Andrew. Welcome back to the Guild.

Hey Amigo Joe. It's as usual, a pleasure to be back here and to be able to share with all your awesome community and all the knowledge and the things that I think we both love to distribute all over the world.

Absolutely.

Yeah.

It's an honor. But before we get into it, Leandro, is there anything I missed in your bio that you want the folks to know from the guild more about.

Well yeah, there are a little bit, a few more details that we me and all the Pervites team we have been working on. First of all, you might have seen here and there Senor Performo has a few YouTube channels now where I'm trying to push some little snippets of shows, short ish videos where I try to explain not only in blog form or in podcast form but in video form where people can see my ugly mustache showing up all over the place.

It's iconic. That's an iconic look for you now.

Yeah, I would try to keep it. And that mustache is stamped everywhere. Almost a trademark I hope, right?

Yeah, absolutely.

And yeah, I have been pushing a few series of videos called. Well there's a channel in English, a channel in Spanish Senhor Performo just like that is the Spanish one and Eng at the end, that's the English one where I am putting up a series where I explain all these type of things related to performance testing fun ways with silly images and directly from the couch in your house, yours truly, for everyone who is listening and explaining in this video series called in Spanish Nosotros los Performers or Os the Performers which is more or less in line with Mexican telenovelas. It's a performance novella for us and that's what have been publishing new from what was the bio that you mentioned? And soon to come the Pervites team are going to be pushing some learning material around performance from the performacologist. We're going to be trying to give more scientific, medical ish specifics on performance with it's going to be video content. We hope everyone will like it and we will have a Spanish version as well with yours. Trying to share that rest of knowledge in El Performa. Colojista sounds a little bit more convoluted in Spanish but I think those are the news that are coming up soon. I keep saying soon. Also load testing projects Journey best practices video game walkthrough title and work book coming up soon that I hope to have finished anytime now and start promoting it.

That's awesome. I'm really excited to hear about all this content you're on produce in there and like I said definitely it's a must go to resource if you do anything performance testing is check out anything by perfbytes and what you're doing also@seniorperformo.com as well. So Leandro, I like to know how you got here, how you got to be the senior Performo. So I thought maybe we dive into A few points you gave me earlier on a session we were going to do, but I thought it'd make a better podcast. So I guess let's talk about your first time with performance. Your first lesson maybe learned like how did you get into performance? And maybe it was the first lesson you learned that on your journey to becoming the. How do you say, great in oh, grande, grande senor performo.

Well, it started all with, as you mentioned before, my life as tester. I was a developer project leader and doing all sorts of things in the other side of testing, more on the dev and management side. And when I jumped into performance testing, or jumped was an overstatement, I was thrown into performance testing. The very first thing that you learn in those lessons is 101 in scripting. Those are the first things that almost every performance tester starts doing. You are thrown. Here's a set of steps. Automate them and make them work. And on those days my boot camp was on loadrunner. I was a loadrunner for a while and that's where I started to learn the trade, where things were thrown at me. You got to automate them. Fortunately, given my developer background, I developed several web applications, web services, all those fun things. And I understood more or less how it works behind the scenes to automate a script at a protocol level. But the initial indications, you started to record a script and gather this code that was generated by the application and start changing wherever it needed a correlation, a parameterization, anything that you needed to update on it, you would go line by line on it. And I think that was my first learning because the first I would say even one or two years as a scripter, because I started a performance tester scripting, I was just let's say running whatever I recorded as recorded up till the first point where it broke and then start to see what did. I had to change to keep the flow on the script. And definitely that was not the best practice because it was lots of trial and error and many times my scripts would have lots of things that I didn't know that were needed to be automated or correlated, parameterized and they would broke a few months later or even the day of the execution where even some customers I remember that would say, yeah, it's normal, all the scenarios have like 20% of errors and problems from the scripts because they are per se unstable. And I would buy that. And I think that was the first learning. Some few best practices on how to make sure that your scripts are well done, that they are not missing any points, any Correlations, any parameterization, even client side generated timestamps, JavaScripts, any of those fun things that we have to include in our automations. I learned that you should focus on the differences, things that change from script to script on the same steps for the same business process. So there I developed like a ten step guide for bulletproof scripting and eventually it would become like a badge of honor for me. Like hey, my scenarios don't have 20%, hopefully have only 0% errors related to the scripts. If they start to show errors, most probably it's because of your applications dying. And quickly those learnings were around. First record whatever process twice, first get familiarized with it, go manually through it, click it, do it a couple of times before you even start recording, start the recording, do it twice, changing any parameters that need to change or would change during the process. And from there just compare those two generated scripts, see what changes and what do you need to make sure that is up to date. And with that, most probably you will guarantee that your scripts will be bulletproof. Adding the cherry on top was to make them iterate, make them run like five threads or virtual users on those days of the same process so that they won't step on each other and at last that they can coexist with other scripts that you might have generated. So I think that was my first huge learning, like whoa. How almost everyone else that I know does it is if not the best way, not so efficient. So I came up with those 10 steps and that was a big revelation at the moment.

Nice. So the first hurdle obviously is creating the actual script to emulate a user interacting with the real application. I guess the next journey when someone's doing a performance test is actually creating a realistic scenario. So any lessons learned then once you started creating scenarios for these performance tests utilizing the scripts that you created.

Yeah, I think that was the very next big lesson that I got. As in a beginning when I was thrown into a project, the project would have defined already some goals or specifications for the scenario. And I started to notice that they were to my opinion a bit random. I was like, why should every process be triggered a thousand times in an hour with 100B users per process, like hundred here, thousand here, specific round close numbers. And I started to question that as well as some executions. Like I was like why am I automating this process that happens once a month or is not even done by a real user? I was requested to automate scheduled jobs, cron jobs, triggers, SAP. I can't remember right now how they are named like cron jobs in general that are automatically executed in periods of time. And I was just creating automations that would access the admin console and trigger them. And I'm like, don't they just are already automated or scheduled? And I think that took me to understanding or trying to better design these real life scenarios or as close as possible possible and understand some of the processes that you need to focus on. Even as a customer might tell you hey, this transaction, this business process is super, super important, has to be automated. I started to wonder like hey, why are we automating it if it only happens once a month if. And there was my epiphany moment like hey, load test. Automation is to load test. That's why it's called load runner, neo load. Everything was load something. And I was like, this is not just to receive response times, this is to load the processes. So I started to learn and investigate. I came up to the Pareto principle to understand that you should focus only on the tasks. Very little, few number of tasks that would generate 80% of the load in your production environment and automate only those. And generally it wasn't like those 50 hundreds of automations for a load test that some customers would arrive me and five, ten more scripters and create gazillions of automations for load testing where yeah, this one is only runs once in the scenario and that's it. Why do we even bother? So I started to understand, learn and very much instruct the customers. Like what is important might be super important process for you. It's I don't know, payroll, everyone depends on it, but it happens once a month. You'd better just manually execute it or bring someone that executes it and let us automate the ones that happen zillions of times or several times per minute and include them on the scenarios in a modeling way that truly resembles. And that opened another can of worms probably. I'm jumping into another part of scenario creation where I started to notice this rule where you had to divide the load in between what business processes had to be included in your scenario, together with how many threads or virtual users you had to simulate in total and how to distribute them and also to hit the number of the throughput or the targets for load in your scenario. So all those three elements started to play together. I even generated some not equations. That sounds too fancy. They were just a simple division. Like If I have one user that has to complete the process 10 times in an hour, then it has to wait this time in between Every iteration or vice versa. Three elements that you can work with. And from there I started to model the scenarios to make them the most accurate possible. I think that was the other big learning and understanding. Whenever I get a request for a load test that seems very random or close numbers, it's 1000 iterations and 100 users right away. Like why 100? How many users do you have in the system? And that sends me to another rabbit hole with the difference in concurrent users, active users and registered users in your system. You generally. Yeah, we have 10 billion users in our system. They always accessing it now only 10. Okay, our load test should be with 10 users, not the registered users. I had another example for a gym where not all the people that is registered on the gym goes at the same time. They cannot even go at the same time. And even the active users are the ones that go often to the gym. But the concurrent are the number of people that you have inside of the gym at that given moment. And the gyms cannot even take all your users or the active users. So some cool topic or definitions and ways to explain them that I think was the next set of epiphanies that I got from those exercises of starting to model scenarios.

Nice. So once you run the test, you have the scenario. I always found the most difficult piece my journey with performance testing was analyzing the results. So in your journey, any best practices you found on how to analyze results better in order to give accurate information, not jump to conclusions based on some anomalies or not actually testing the correct thing.

So in result analysis at the beginning, I would say my journey was very rudimentary. Whatever I could gather from as performance metrics from the four horsemen of Pokelypes, I like to call them disk, ram, CPU and network. Those were my main observations and was doing only that whenever allowed. Because again, many customers would have their IT infrastructure team that would be monitoring all that. And it was just like you just deal with your response times and that's all we care. And generally, and even nowadays are some of the few things. The only few things that I can analyze when I don't have more access. But I started to understand some of. First the correlation process where I think the first ones came pretty natural to me. Like when you start to see the CPU burning down and the response time going up. Well, there you have an evident correlation. But I think the next big realization to me and goes a little bit coupled with performance. And I might have said this on the show before and this is a rant that I have Always performance testing is not load testing and vice versa.

I was waiting for this. This is good. Yep. Go, go.

I had to throw it. But that realization came as well with most of the projects asking for load tests when they just needed performance metrics. They had no idea of how their system was doing. And the monitoring and interpretation of metrics was just I just need to know how my system and the only way that all the industry knows how to push them is through a load test. So I started to understand that I think in the side of data analysis and correlations mostly was to see something starts to go haywire on one side, I see impacts elsewhere and start to do connections with the limited graphic resources that I had at the time. Again, on those days I only had whatever loadrunner was displaying on the controller or was outputting on the analysis graphs and that was the first set of tools and processes. But I think the gates of heaven opened when I discovered the apms. Those things just changed completely. My perspective on how was performance testing a tool that could, you could just install the agent on your server or servers and then you would get all the performance metrics without having to do automations or load tests for it. And I think there was when my mindset started to change and I started to understand like, hey, I'm doing performance testing but I'm not using loadrunner, what's wrong here? And the short circuit started to happen in my head because I was trained and indoctrinated that performance testing load runner, that was it. You just slam the system and gather metrics. And together with the scenario modeling that was another problem because you were arriving at customers where they had no idea what was their current performance, what was the utilization, what was happening in the system. And I think for metrics interpretation gathering, the first big part was that the APM would log and keep all of those metrics and relevant information for you. And the next one was the capability to port that information to wherever I wanted to, if I wanted to send it to an Elastic database, to send it to Influx, to even keep it in an SQL or internally. LoadRunner stored all that in a Microsoft Access database. So being able to port that anywhere that I wanted, present it live and put it through, I don't know, Kibana Grafana and all the other Annas and probably Tableau, any visualizing tool that on the early days you were locked up to the controller, whoever had a controller machine was looking at it and later being able to export that show it live and hey, whoever wants this information from the performance Metrics, go nuts with it. You will be helping me. I think that was the next big realization. Nowadays, any customer that I arrive at, my first question, do you have an apm? Do you know your performance metrics? Because that's the departure point. Otherwise it's very difficult to start working on the next steps for performance testing or assurance.

Yeah, absolutely. And I think James Pulley says this all the time, if you don't know you have one user performance, why are you trying to do a stress test or a load test? So I think that's really important. You go in and say, okay, what's the baseline we have here first? Before we get crazy with all the tools and ramping up a bunch of different users for sure. So I guess that goes to project management styles and as you're talking with clients on your journey, maybe when you first started you were told we're going to do 100 users and you just did 100 users and you did it. I sure over, over time you learned how to kind of address the client, say well this isn't the right thing, we need a baseline. So any kind of project management styles you've learned along the way that you think have served you well.

Well that's another one that when I started, as you mentioned, I would just receive an email or a meeting, a phone call from a customer specifying what they needed and we would just jump in to start scripting. Many times without even a formal test plan, any documentation or even test cases. We were just, there's an alm, Go nuts with it, figure it and generate animations for it. And thanks to the background that I had before I jumped in the testing life, I was a project manager and I understood and was trained on the PMI principles, how to segment your projects so I could do a jump. And I think it was HP's project guide. Probably it was even still Mercury where they indicated those steps and best practices for facing your load testing project. I am still tempted on calling it Performance testing Project, but it's just a load testing project. So on those ones I think I became even a bit OCD anal retentive with it. If my project doesn't have a test plan, formal test cases and all the things involved with a good project management practice, I'm not going to do it. And that was the next big learning that I had. Understanding, adapting the steps to the and sadly all that was waterfall times, waterfall process, very phased, you jump from one step to the other, very hard to go back to the previous one. And all those fun Limitations with waterfall. But it made a difference. Most of my projects, I had everything documented. The customer understood everything was presented of what was going to be done, what were we going to be working on or what did we need? Where was it going to hurt? Because often it would hurt a lot. But all those phases also draw me to understand where to optimize processes, time and little things. Like as I mentioned, some customers would throw us at an alm full of functional test cases trying to figure out what to use and up to the customer's likeness for us to automate everything that was under functional. And some of those others were like, hey, we cannot use this. Or the simple realization when I'm recording this, this step loadrunner is not recording anything. What's the deal? Why am I following that step? What does it have to do? And many functional test cases and steps are oriented and testing the front end functions that do not do the round trip to the server. And I started to realize like hey, if we include in our projects functional test cases, they are garbage. They don't most of them. I'm exaggerating 50 steps on the test case and only 4 things recorded in our script. I was like, why are we wasting so much time with this? And no, no, no, no, that was the other project management. Why interview SMEs, ask them for the most direct steps to hit that save button, which is mostly what we want to do in our automated load test scripts and jump over hoops, loops and anything that we don't need. I'll give a quick example. This customer, we had to do payments on a payment system and the system had an account with a given amount of money. The customer was like, yeah, the payments are around $100 per month and you should do that. But the problem is that we ran out of money in that account when we were doing load tests. And we're like, what happens if we pay a cent every time? Database wise, it doesn't make a difference if it's a million dollars or a cent, it's a record. If the name is an A or all the full name, it's still a database space a cell occupied in the database. So what happens if I don't click here? What happens if I skip this tab? I just want to get as straight as possible to the save button. So I think performance load testing project wise, that was the other big learning. Good test cases documented for dummies. That's another one. Your grandma should be able to pick up many customers that we would arrive at required specialized people that Know the I don't know purchase module for SAP perfectly and I'm like dude, I don't even know how to purchase in SAP. I just received the steps. I am seriously good at automating those steps, but I have no idea what am I doing. You should document steps that everyone can follow and not have weird Instructions. Configure the Transfibrillator 3000 for, I don't know, an extraction. I'm just using the weirdest words that I can come up with and that together with the faces. Good load testing oriented test cases and writing them up for any tester to understand because for sure I was going to be gone in a few months because the contract was due and the company probably in the future will have to bring someone else to follow it. I didn't care telling them I don't mind if you don't call me again and you need someone else to do this and even if you call me again, I want to easily remember what was done here so that I can follow the steps again. So get the test cases done for dummies, for your grammar, for a toddler to understand and make sure that you grab them oriented at low testing and at the end face those steps with documents, a test plan, documented and hopefully signed off test cases. That's very powerful. If the persons or people that are telling you the steps know that their name will be on its signature and they can be well responsible for that, they will pay more attention. The phases for the projects you initialize, you discover all what is needed for the load test, you design everything. The test plan, the steps, the scenarios, the test cases, what is going to be in scope, how many times do I have to hit what on each scenario? Then you go to create your scripts and automations, your scenarios, put together everything needed. Next phase, the fourth phase, you execute, grab all the results from those executions and the last step, analyze results, put together reports and present them. Those are the key stages for a load test project.

Perfect. I love the grandma documentation advice for sure.

So I guess to wrap it all up now like myself, like you, you know we started in a waterfall environment and obviously things have changed a lot in 20 years and we're now doing Agile shift, left testing and production. So any lessons learned with implementations in an agile environment now for performance engineering or performance testing.

Yeah, that was I think one of the last biggest ones that epiphanies and change in mindse that I got when I was forced to jump into agile. As I said, the indoctrination at the beginning Was low test, low test. This is performance. Low test, low test. And my logic was like how am I going to be able to put together a low test on a two week sprint when everything is changing, moving and all the pieces come and go? It's impossible. Who in their right mind would propose this agile garbage that everyone's to do? And I must admit I was very much against all agile methodologies until as usual, I was forced to deal with an agile project and I was like oh damn now what should I do? So it brought me lots of transformations in the mindset and helped me to have that epiphany that I told you. Performance Testing and Assurance Performance assuring performance is not done necessarily only through load tests. There are a myriad of other activities that you can and most probably should do before you even think of a load test. And in Agile especially is even more important. Again like you mentioned, James says it you need to know baseline, single user, each process, how, what is the performance of each of them. And that was another one that it jumped at me several times during a project. Why am I requested to automate this process to be load tested when I already see manually with a single user? That sucks big time. The performance is awful. I mean I already know that if I run these 10 times concurrently the system will crash. So some of those mindset changes started to occur and then going to conferences trying to figure out because it was the first big challenge where I was like I have no idea how to proceed here. The load testing mindset was so again indoctrinated into me that when I started to understand that you assure performance by making sure that every little piece performs on itself and then you can put them together and then try to see if they perform still put together and at the very very last step and only when it is strictly needed that you are going to have a blast of users slamming the system. Only then you start to think on a Lotus, but before that you start to do everything that you can. Asides of Lotus, this includes also an epiphany from reading Lisa Crispin's book Agile Testing the Pyramid of Automation. Everyone was focusing on performance and load testing at the front end level. Extremely inefficient, time consuming and especially to maintain on an agile sprint by sprint pace was impossible. So you had to start to see what were other things. And I was like oh I can get performance metrics from unit tests. Oh I can get them from just service testing automations easier calls than maintaining those big ass end to end scripts that were I don't know several thousands of lines of loadrunner code. I remember that maintaining those monsters and especially when the communication in the protocol pattern were very random like matrix code, huge chunks of matrix code coming and going that was awful to maintain like finding the proverbial needle in the haystack.

Right, right, right.

I'm sure that you were even having some of those moments and remember Citrix, Sivil, SapWeb where you would see those big chunks of objects and it was to a regular person garbage. And I was like no, I'm Neo in the matrix. I can see there's a girl in the red dress perfectly. I don't even need a graphic interpreter anymore. So all of those understandings and learning and figuring out that I could get performance metrics from everywhere especially as I mentioned earlier through that big breakthrough that were the APMs and on top of that putting the right coverage in automations in the right places by the right people because not everything has to be done by the tester. Nowadays I do very few automations for load testing. I mean just instructing hey developer, please like flossing your teeth, don't leave your house without brushing your teeth in the morning, don't document or check in code that you have not created a unit test that reports performance to me before you even check in in so that everything is done automatically before any checking happens. And after that as soon as any check in happens a bunch of mini load tests. Mark Tomlinson mocks me a little bit for me a little for calling mini and load test is like a little bit counterproductive each word to each other but that should be like just put five threads on this right after you check in the code and make sure that it can take a little bit of load that there's not too much deviation and deviations. I think that's another one that you need to pay attention to in performance terms nowadays with the Agile methodologies and I think that's a huge other kind of worms that opened up when I started to do agile and I could consume several episodes but I think I will leave it there because there are so many things that I learned trying to tackle that that it's so long.

Yeah, absolutely great advice and that's a good point. About a lot of times people they usually when they're on a system they test as one user or two user to get but sometimes they don't do a third user. So when I used to do a performance I'd start with the 1 user baseline then go to 2, then 3, then 5 then 10 and then ramp up from there because you usually find weird issues around three to five for sure. So you dropped a lot of knowledge here Leandro. I guess. Before we go though, is there one piece of actual advice someone could put into place right away to help them with their performance testing efforts? And what's the best way to find or contact you?

I will go to my regular rant Low Tests all load tests are performance tests but not all performance tests. Tests are load tests and you should do very few load tests whenever you need to assure that your system performs. The only critical cases that I have seen that you truly need to do load tests or big ass executions is when you have systems that hundreds or even thousands of people, many customers will request. Hey, can you load test this environment? I'm very worried that it will do well, but yeah. How many customers do you having it? Oh, 10, 50, something like that. Just do a release practice that you can load test it through your users and see and gather the metrics. I mean there are so many other practices that load testing is only when you are going to have so many users that you cannot test it otherwise. But with automated load tests that's cost efficient, time efficient and all efficient in all the areas after that rant and recommendation or training guru I don't know on my content everyone can find me on LinkedIn, Twitter, Facebook, even Instagram and now YouTube as I mentioned as senorperformo s rperf. That's my hashtag or handle for almost all the networks. And if you want to go and take a look at the Senor Performo channel and channels because the English and if Si habla espanol Bienvenido. You're welcome to if you want to practice as well. Might be useful useful. But all the YouTube stuff, don't forget to give it a thumbs up. Subscribe, share and let your friends know that there's a mustache guy trying to share knowledge over YouTube with videos.

Thank you Andrew for your performance testing awesomeness. Links to update and value we covered in this episode. Head on over to testguild.com P57 and while you're there make sure to click on the Try them both Today link under the Exclusive Sponsor section to learn all about SmartBear's two awesome performance testing solutions, Load UI Ninja and Load UI Pro. And if you like this episode, why not rate us in iTunes ratings really help in the rankings and help spread performance testing to the masses. So that's it for this episode of the Test Guild Performance and Site Reliability Podcast I'm Joe. My mission is to help you succeed with creating end to end full stack performance testing awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Performance and Site Reliability Podcast. Head on over to testguild.com for full shop show notes, amazing blog articles and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.