The Story of Load Testing (A Hitchhiker's Guide) with Leandro Melendez

Welcome to the Test Guild Performance and Site Reliability Podcast where we all get together to learn more about performance testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Performance and Site Reliability Podcast. Today, in for a special treat, we have Leandro Melendez joining us, talking all about the Hitchhiker Guide to Load Testing, which is his new book. We're going to dive into it. You don't want to miss it. Leandro is a Performance Testing Manager providing testing and consulting services to worldwide companies. He has over 20 years of experience in IT and over 10 years in performance practices where he served multiple multiple S and P 500 companies all over the world from the US Mexico, Canada, Brazil, India, you name it, he's done it. He's also the author of the popular performance blog Senior Performo, where he curates a diverse set of learning materials for performance testers and engineers. He is also the producer and host of the Spanish language version of PerfBytes podcast. If you don't listen to that, you definitely need to listen to it. I'll have a link to it in the show Notes after this is done. And he's also known as Senior Performo. If you go to YouTube, you could see all kinds of learning he's been doing there. He does a bunch of different webinars, bunch of different kind of education things that you don't want to miss. Leandro also is an internationally known public speaker. He participates in multiple conferences, events, webinars, keynotes, workshops and multiple talks he has under his belt. And he is now also the proud author of the new book brand new book the Hitchhiker Guide to Load Testing Projects, a fun walkthrough that will guide you through the phases or levels of an IT load testing project. We're going to dive into this step by step, so you don't want to miss this episode. Check it out. This episode is brought to you by SmartBear. Listen. Load testing is tough. Investing in the right tools to automate tests, identify bottlenecks, and resolve issues quickly could save your organization time and money. SmartBear offers a suite of performance tools like Load Ninja, which is a SAS UI load testing tool, and loaduipro, an API load testing tool to help teams get full visibility into UI and API performance so you can release and recover faster than ever. Give it a shot. It's free and easy to try. Head on over to smartbear.com solutions performancetesting to learn more.

Hey Leandro. Welcome to the Guild.

Hello amigo Joe how's it going?

Awesome. Awesome.

So good to be back.

Yeah, it's great to have you back on the show. You seem like you've been really, really busy, so I can't wait to hear about what you've been up to before we dive into it. Leandro, is there anything I missed in your bio that you want the guild to know more about?

Oh, no, you pretty much cover all of it and more and it's quite extensive and I really appreciate that you went through pretty much all of it. That was very, very good cover.

Awesome. Yeah. So as I mentioned, you are an expert in this field. I'm really excited about this new book. I can't wait to get my hand on the physical copy. I know. Pre order as of now is now available on Amazon and we'll have all that information in the show notes. But before we actually dive into the book, why write a book? Leandro, I wrote a book in 201415 and it was a pain. So why take the time and effort? I know you work full time, you're doing all these other projects. Why write a book?

Oh, wow. Yeah. I've heard someone saying, like writing a book is like getting pregnant, going through all the pains that go through it. But in the end you get a lot of joy out of the, let's say, not experience, probably the outcome of the experience. But I have to say the main reason was that the content of the book and the teachings that I tried to do. On one hand when I started thinking of writing it, it all started from some sort of frustration. On one hand, trying to explain the complicated topics and often a little bit too technical elements of this trait of performance testing, which on those years when I started with this idea used to be only load testing or most load testing. And I came up with interesting ways to explain some of those complicated topics. But as you well mentioned, I'm a consultant. Every other month or several times a year, I would be with a new customer trying again and again to explain these things. And these practices took many people, Sea level management, seasoned performers, and even novice or people that were just starting in the trade. And I started to repeat myself a lot. And I started to think what would be an efficient, good and cool way to share this knowledge. Well, there are many ways which I'm trying to probably attack all of them at once, but one of them is a book and I wanted to give it a try, try to explain all of these topics in a way that would be fun, entertaining, and that would wrap you in the process, that would be like an adventure. And that's why I follow it more or less in that template. Mostly I wanted to share knowledge and make it as fun and enjoyable as.

Possible and I think you succeeded. But I love this book because it is tool agnostic. So it is almost like an adventure. You could read it on a couch and get a laugh and you learn like there's no math in it. There's nothing specific about a tool. It is like you said, like a manual, a get starting guide or cheat codes for people to get into performance testing for sure. So I love it. I guess to start, the scope of the book is since performance testing is such a loaded term, what is this book specifically about?

So it's specifically and as the title not so short title implies, it's just about load testing projects. Where your project is we have this application, we need to slam it with some load and see how it does. And I would say another characteristic of what it covers these load testing projects by nature, most of them even in these modern agile days, most of them are not agile evolved. If that's a word they follow or the initial practices or the common practices for this type of projects is in a please don't be scared. I'm going to say it is in a waterfall model where you have phases that you have to go through and complete tasks, get deliverables, get some items like special weapons to kill the boss to move to the next level which are specifically the projects that I'm trying to cover and to explain in this book something that I probably am still right stating are most of the projects that performers that listen this show are still doing load testing projects. They engage into a requirement, they start to go through the phases, do the big time execution, analyze the results and deliver. Those are most of the products that still nowadays even in these agile days. They should be changing I got to say. But it's good that people understand it since many of us are still doing that. Even me, I still right now I'm doing those traditional load testing projects and it's very, very important that everyone understands the steps, the concepts and what involves all this beautiful trade that we're into.

Absolutely now because it's load testing. The only way to do it is as a waterfall because I know you can introduce it early in your software development lifecycle. Doing a shift left with one user, you check in some code, maybe you gather some performance statistics. But in general for a load testing project doesn't it have to be waterfall?

Almost most of the time it has to be waterfall. If you are brought up to a project that is already on one hand probably on production, that was no automation done that it has not been. And again this is why I'm pointing this out. It's sadly most of the projects as a consultant that I walk into that do not have instrumentation, do not have telemetry, do not have that many unit tests that you are just dropped into it. Here it is our project do load testing on it. We need to this helpful question. We need to know how many users can the system sustain or serve without having any degradations in terms of performance. And again that's why more or less from this process waterfall is the arguably I would say the only way to do them. Of course, projects, systems applications that have been generated with the best practices on agile testing probably may not have to go fully into these steps. But as an introduction and I would say as the base for all this work, I think people will get benefits on understanding and following these best practices for these projects. And as well the ones that are fully engaged in agile processes will learn and understand from where most of the practices that we do come from. And if all goes well, part two will be around Agile performance testing. Hint or spoiler alert. Next second part is in the process where we will dive into those differences on how agile continuous DevOps projects should embrace performance testing. But this is mostly for load testing projects where you are set. We need to know how many users can we support, how far can the system go, etc.

Nice, nice. So the book is divided into five main sections and I thought we touch on each one. Just give each one a teaser. You have to buy the book to get the full awesome info here, but I thought we'd give a taste of each chapter so people know why they need to grab this book. So the first one is level one and it's called Discovery. I took tutorials, an introduction and you introduced something called the why, the how, the what, the who, the when, the where. I'm just curious to know out of those was that six of them or some of them? Which one do you think most folks skip over again?

Arguably. But to me and in my experience the most important and that is most overlooked is why. Why are we jumping into this effort, into this project of doing a load test? Why are we trying to do this? What brought this? Probably the question is not why, but. But from the why it comes all the rest of the information that we need to defer because this is the key element that most just oversee. We just to do a load test and start Scripting, slamming the system and give me some results and why. The why is a key component. This defines how are we going to proceed for the rest of the project, most of the actions that we will do. What are we going to do? Why are we going to do it?

How?

Because if the reason for this is a new project that we're going to release, we may need to do some type of load testing. If the reason for this load testing is that we already see issues in the system, we may try to do a set of actions and so on. The variations if we need a spike test, if we need a load test, a stress test, a larger test, an endurance, if we need to push it further and from things as simple as what type of processes are we going to include in our test? Is it a spike test? Well, we will just focus on the processes that will spike and not pay attention to 99.9% of the other processes in the system. If it's an average low day, we may need to segment differently. What are we going to pay attention into? So in this initial findings in this tutorial level for your adventure, we need to clarify the questions around all the project or around all the system, around all the needs, around all the actions. But most importantly is why are we embracing in this adventure and talking about adventures and quests and movie plots, novel plots. That is the main reason. Why does our hero has to be the hero? Why is he embracing this thing? Why is it like, nope, Neo, I'm the chosen one, I won't embrace into this. You need to know he is the chosen one, he's special and he needs to embrace the adventure. So this is the main question. Why are we becoming the heroes of this adventure?

I would say absolutely. And you mentioned like performance test lead must find out the requirements or reasons for the performance load test before they do anything else. Do you find as a consultant, maybe someone thinks, oh, I'm being hired as the expert, so I should know. I'm the one that should know what the requirements should be for this app application. Is that why sometimes people get tripped up?

Oh yeah. There are some sort of expectations on both sides. On one hand, the customer usually will think that you will come as the almighty and all knowledgeable performance engineer that will magically clap your fingers and erase half of the performance errors right away in the universe, should I say. And on the other hand, as a consultant, you come in with lots of knowledge, with lots of experience and understanding. But here's where also on both sides we have issues. The organizations expect us to tell them everything and come knowing everything about and probably even about their system, about their processes, about what they do. And for us, many of us are already, let's say, very involved into this, I would call it vicious cycle in which we are already like, okay, yeah, these processes, automate, automate, automate, execute, execute. Slam the system, get out, deliver the report, we're done. Where we should be doing a little bit more analysis, understanding what is the need. It's like a joke. We have here in Mexico a bad doctor. You will come into the office and the doctor will give you paracetamol or Advil. For every illness that you may have, where that's not the right thing, a good doctor will ask you about all your symptoms, how you feel, where it pains, what is happening, when did it started, all that. And then it will decide he or she, what is the best course of action to take for you. So I would say there are some of those misunderstandings. On one side, the customer should have lots of information or the consultant should be able to ask many of those questions and be very precise about asking where it hurts and how to measure those pains.

Absolutely. You just opened up a memory pocket. My grandma was Italian and she used to swear by witch hazel for everything. She used it for everything. So it's kind of funny, it's kind of like Advil, I guess in Mexico. But anyway, what I thought was interesting that you introduced here also is proof of concept. So how does proof of concept come into play at the very first stage?

So for a load testing project, what we are going to do is generate multiple automations and trigger them and simulate user activity. But in my experience and many places that we have been providing services, not all the solutions, not all the applications will go over standard automation practices. It won't be easy. They might be ciphered, they might be encrypted and even silly. Examples where I had a project where we had to measure more or less the performance of cows, they were being milked and we should measure not only the device, but how much milk would be part of the metrics. And you need to figure out the device had some sort of controller that you could extract that information, but you need to make sure that you can automatically generate that activity. And there are multiple examples where we have had.

Was that a real example with the cow?

Yeah, yeah, yeah, yeah. There was a system where I had to gather the measurements from, you know, these devices where they plug them and they extract the milk. And these devices also have microcontrollers where you can Gather lots of information from them and performance.

So Leandro had straw nipples after this PoC.

Oh, no real interaction with cows happened there, thankfully.

Yeah, that was mostly playing with the device, playing with how it performed and the measurements that it would bring. The people in charge will take the devices, plug them and do what they were supposed to do. And I was safe in an office just looking at the numbers. But yeah, so many of these things. Windows applications that go over the wire in binary or encrypted, ciphered old applications that use components, WCFs, Java remote processes in, in old ways where even if you can capture information that's just gibberish, it's encrypted, you won't be able to mimic or forge the messages, which is a key component. So yes, it is a very important and at times overlooked step. You know, if the application, you know it well, you have played with it or you know what type of application it is, you say yeah sure I can, but if it has also I don't know if it's using Kerberos, if it's using a specific authentication, I don't know some of those issues that we have encountered and that you need to find a workaround, a way to dispatch that project and if you cannot, and if you cannot measure how much milk comes, what's the performance of those cows, you may not be able to do that project. And that's why I brought it up. Some other experiences, video games. If you cannot simulate the action on a video game and silnoff that was my first ever performance break. We had to automate some sort of AI routines that would respond to the video game playing against each other. Can we do that? Is it possible? How can you generate such actions in a performing manner as well? So multiple questions, can we do it? That's one of the initial things that you should be able to tell. Certainly we can. Or let me go and do a POC and figure things out. That's really, really important.

Nice. So we're on level one, we're jumping to level two, which would be chapter two. It's called Design Writing It Quest for a Performance Story. So I thought this was interest, like why I think a lot of people overlook now writing things. And one thing you said is you need to prioritize what's important to get documented. Why is that?

So there are multiple elements in a performance testing or load testing project that I have seen in multiple organizations that just are not covered. Go on an email or are just wide open, like you will just have a test for 10,000 users. And that's it. That's what is said in the description of the project. So a reason to have everything documented. It's very important because on one hand it will give you guides, it will give you specific instructions of what is needed, what has to be done, how much will we do, how much we want to do. And I've seen many organizations where all the performance test plan, all the indications, all the documentation, it's just, hey, just simulate a thousand users and that's it. They won't say anything else. That's your only indication, that's all what is in the guide, in the plan. And being a consultant, more or less like a contractor or someone that needs to have every, everything, all the T's crossed and I started. It's very important. It's important because if I had the experience when some people selling the project did not scope everything very well and the customer was like, oh well you're gonna be doing all that for me. I was like oh, what oh. And in just two weeks, wow. And we're like what, uh, oh. So some of those things where you need to plan, specify and define exactly what you are going to do, how far are you going to push the system?

System.

Some other elements that at this phase you need to describe, well, what actions, what counts as a business process or a business flow where you have multiple business processes, how to do it, how not to do it, that you want fix defects or that you won't be updating the database. Silly things that of times are overlooked. And some customers were like wait, wasn't the performance engineer going to fix that query? What, why? And from silly things like that to huge things where it was overlooked. How much would we take to script? How many scripts were we going to do? And some customers expected a script per available process in the system. And it was a huge system with houses and processes. There's no way, so it's really, really important like a contractor. If you're going to build something, if you are going to put together a building, if you're going to build an airplane, detail by the bolts, what is going to be included in that airplane in your load testing project. So and in the end that's going to be a document, I call it, well, two documents. One the performance test plan and on the other hand the test cases which also have perks. I don't want to spoil the book, but it's very different a test case for performance and load testing than a functional or everything else that you may have around with those Two deliverables. It's recommended for you to get a sign up and to make people accountable for what you're going to do. At least that they read it, that they are aware or at least that they received it and they are okay with whatever you said even if they didn't read it. Same for the test cases that you documented what is rightly what the process should do. So those documentations are incredibly important.

And what I like, it's not just one sided. It's not like you're just trying to cover yourself with that signature. It's also I think you mentioned something like it's almost a map then after that. So once you have agreement, you almost have like a map continues to go forward. Which I think you stress in the book as well.

It's a little bit like when all the sages and the old guys in a tribe used to tell the story. The prophecy, how things were going to happen is more or less that's your guideline of how things are going to happen during your adventure as well. It paints on day one. What am I going to be doing? You can paint it at the detail per time. So yeah, it's really important.

Nice. So you know the requirements, you have your map, you're ready to go. It's time to get busy and actually create the planning phases. Done. It's time to construct. So should testers just jump right in now and let it rip or is there anything else they need to do at level three?

So in level three where you start building up things, as you mentioned many things is just building the scripts and there you go. There are multiple other elements. You have the scenarios you have to implement monitors and you have to go through multiple steps on each one of those tasks. Because as an example with the scripting you don't just click record on your recording tool or your load testing tool and gather all the information. I have also developed and I have been sharing a set of 10 steps best practices of how to generate. I like to call them bulletproof scripts. Often load testing scripts would be kind of fragile. I used to call them diva scripts when for weird reasons some of them would break. You have 100 iterations and on two or three of them they would fail and you would have no idea for that. So I will describe you as well. What are the best practices? Why it is important to know the process to not to jump right away recording to not to try to correlate on the go. On my first years and I say that from experience, I was just record replay where it breaks. Okay, here I have to fix something. Record, replay, play again, fix something, play again until I got to the end. That's a very bad practice. So on this stage you will start to construct and have a bulletproof not only scripts, scenario monitoring and everything in place ready to go.

Nice. Now, when I first started, I actually started off as a performance Engineer. Like in 1999, 2000. You mentioned correlation. I'm just curious to know, is correlation still a thing or do these tools handle for you automatically? Now.

Many tools promote that they have automatic correlations, that you can create rules or that they will recognize what are the most common or that even would have them packed into this and you can tell it is this type of system or is that other system system. And the tool will come up with some predefined correlations. And yes, many tools already aid you, but and this is a personal opinion and preference, you still need to know what you are doing, how a correlation works, how to build those rules just in case the tool is not catching it out of the box. And this is where I am a little bit paranoid and suspicious. I rarely use them. I like to come up with clean correlations to make them the best possible. If the tool has a variation or something that doesn't match up, I know where I know how to fix it and an advantage. Once you identify them, they repeat all over the system. You can re implement them. Create these rules yourself and the tool there will help you a lot. But the process of identifying, scripting the correlation and implementing it, it's I would say priceless and one of the most difficult. And when you master it, most entertaining tactics tasks in automating, performance testing, the Stockholm syndrome that I mention all the time, that's where I would say it is.

Absolutely. And this is covered in the book. So that's why I brought it up. All right, we're going through it really quick now, but it's just a teaser folks. Like I said, you need to get a hold of this book. So just a few flavors of each chapter. I'm going to jump to level four, which you actually put into action. It's the execution piece. I like how you mentioned the importance of a sound check. So I know back in the day a lot of times they'd say, oh, the first transaction always took the longest because. Because the database server was warming up or something. So what is a sound check? Do you have to do anything before you actually jump into executing the full plan that you already have up until this point?

So a little from the previous level, you did some sort of sound checks, you did some shakeout runs, you did some tunings, and you did some low scale and volume executions to make sure that your things are okay before the great day comes comes up. But on the great day, most probably you invited or you asked to be aware or available to the ATS in your organization, the dba, the network administrator, and many other people. Probably even the project lead is around. And you don't want to waste time from them, from the resources, from the people. So it's recommended that before everyone shows up, very much like a band would do, would check the guitars, check the microphones, check all the gear, check all the equipment, and make sure that at the moment that you start the big play, nothing, absolutely nothing will fail. If it does, you will have some time to fix it, tune it, or put it in place. But. And this is another step many teams overlook, they are. It's the day I just saved up everything yesterday and today I'm ready to run. And they click run. And oh, the load generators are down. Oh, why are they done? Oh, please, Mr. CEO, wait for me a little bit while I figure out why that's the load generators rebooted or I don't know, silly errors. That can be a big embarrassment or a big waste.

Absolutely. And so like I said, you jump into a bunch of other things in level four, but then you get to level five. I love this. This. I guess it's an analogy is you call it solving a crime because I always thought creating a script was easy. Actually finding out what to do with all those results was. Was really hard. And it is like almost like a crime scene. My wife makes me watch these creepy crime shows and it's always like, oh, I know who did it. And then after a while you're like, oh, that guy didn't do it. And case goes cold. And it's all these weird branches where you finally actually find who actually does it, where it's someone you didn't even think. And it's just like a performance test. You think, oh, it's the cpu. Oh, wait a minute. No it's not. So like, why, why is it. I guess. Let me stop talking as you. Why is it like, why is this called solving a crime or solving the crime for level five?

So and very in tune with the cop and police TV series and superpers that you mentioned. It is like coming to a crime scene where you picked up multiple elements from when the crime happened. The crime here being the low test execution. Lots of stuff was generated during that execution. And you need to put together and make sense of those of the evidence that you will find the blood that is in the wall, the angle that the bullet was crusted inside of the wall or the body or creepier things. Right. And you need to be able to deduce what is causing and looking for these other correlations that happen in between the metrics that you already gathered and be able to say yeah, General Mustard is guilty and he killed it with a pipe. And I don't remember what other elements could you say include. But you have to put together and give an estimate or a diagnosis of what is the problem. And that is like solving a crime pretty much. You have to get your pipe, get your investigator hat and say elementary. I have the answer.

Nice. So for folks that don't know, there's also a bonus level where you'll have to get the book to find out what the bonus level is about. I'm gonna wrap it up, I guess before we go. You've given a bunch of actionable advice. So is there one piece of actual advice you can give to someone to help them with their load testing efforts? And what's the best way for folks to find or get their hands on this book?

Okay. Oh wow. There are several points and picking the most important one of the book, I can go with one. But I will say a few. Just names them the reason for the low test. Super important. Have documentation and sign off so that everyone's aware and you're protected. The good old cya, good scripting practices and make sure that once you execute you get the best results, the best information and analyze them. One key element for each step of the process. And about the book which for the video version I cannot wait to show it off. Here I have it. And for the ones that are just listening, here's the sound for the pages.

Sounds beautiful, isn't it?

Knowledge. I love the sound of knowledge.

So the Hitchhiking guide to load testing projects is right now on pre order state in Amazon and it will be released in the electronic version pre order electronic version. On September 10th the printout will be available. We couldn't put it in pre order format. We're newbies on this publishing trade, so.

So.

But the paperback format will be available on September 10th as well and to be purchased after that date, physical format on the us, uk, Germany, France, Spain, Italy, Japan, Canada and Australia. All those Amazon sites. You'll be able to order it and receive it for the rest of the countries of the world. The electronic version is available and trying to figure out how to get it on the other places of the world that the paperback is not distributed via material Amazon, but probably you can order it from the nearest as an example, from Mexico, you can order it from Amazon usa. And I'm working out on some other ways to get that done.

So yeah, is there two versions? Is there an English version and a Spanish version?

No, for now, just English. Well, excusing the Chinese people. It's the most widely spoken language in the world, I believe, after Chinese, of course. And that was the initial step. It's in the plans, it's in the works to get a Spanish version. And who knows, I'm thinking if it's worth it, probably other languages, let's see which ones get more demand right away. But right now English.

Nice. And I think it'd be a good audiobook too, because like I said, it's really written at a level where it's not. You're not going to get bogged down.

It's almost a story.

Yeah, exactly. Yeah. So little.

It's in the, let's say, the to do list or the backlog.

Cool. Awesome.

Narrated by yours truly, hopefully.

Thanks again for your Performance Testing Awesomeness. Links to everything of value we covered in this episode. Head on over to test guild.com P74 and while you're there, make sure to click on the Try them both Today link under the Exclusive Sponsor section to learn all about SmartBear's two awesome performance test tool solutions, Load Ninja and Load UI Pro. And if the show has helped you in any way, why not rate and review it in itunes? Reviews really do matter in the rankings of the show, and I read each and every one of them. So that's it for this episode of the Test Guild Performance and Site Reliability Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack performance Testing Awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Performance and Site Reliability Podcast. Head on over to testguild.com for full show notes, amazing blog articles and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.