Integrated Performance Assurance (IPA) with Leandro Melendez

Five, four, three, two, one.

Drop that.

Welcome to the Test Guild Performance Podcast where we get together to learn more about performance testing and site reliability with your host, Joe Colantonio.

Hey, it's Joe and welcome to the first ever episode of the Test Guild Performance Podcast. Hopefully you had a chance to listen to the pre show earlier. That explains why I started the show. So for the first guest, for the first ever Test Skill Performance Podcast, I thought I'd pay tribute to some of the folks that have inspired me over the past, especially in performance testing. So one of the one podcast I listen to all the time dedicated to performance testing is Perf Bytes. So James Pulley, Mark Tomlinson are some of the folks that really have inspired me with performance testing and some of the reasons why I started this podcast. And also Leandro Melendez, who is part of the Perfbytes family. So in this episode we're going to be speaking with Leandro all about his framework, Integrated Performance Assurance, which includes actual suggestions and practices you can implement in your projects. So listen up and discover several performance practices you can integrate into your solution right away to help you with your performance testing efforts. Let me know what you think of this episode and how you want to see future episodes of the direction of this podcast. You can always send me an email at joe@testguild.com let's face it, performance testing is tough. Traditional load testing tools create scripts that are bogged down in data that's hard.

To read and require a lot of.

Work, even for simple playback. Load Ninja cuts out the dirty work by using AI to inspect and debug your code almost instantly. No coding necessary. No changing of hands or relays between teams either.

Give it a shot. It's free and easy to try.

And head on over to testguild.com smartbear and click on the link to learn more.

Hey Leandro, welcome to the Guild.

Hey Joe, thank you for having me. It's such a pleasure.

It's great to have you as one of the first guests ever on the podcast. So really excited to have you. I really am inspired by what you've been doing for Perfbytes and what you've been doing in the community, especially in the Spanish community. I guess you've been really spreading the word beyond English, which I think is really cool. But today I'd like to talk all about your framework you showed at the last Perf Guild on the Integrated Performance Assurance Framework. But before we get into it, Landlord.

People, I don't know, can you tell.

Us a little bit more about yourself?

Sure. Well, first of all, I have been around it for the last 20 years and out of those the last 10 years I have been involved in performance testing, learning a lot, growing a little bit hopefully, and through experience in several projects I started to notice biases, ignorance and certain difficulties and several places where I found people that were not following best practices or knew about performance testing, which as a performance engineer you could figure drove me nuts several times. So I decided to start.

Trying to teach basic topics about performance and if possible a little bit more advanced topics. With an alias in Internet I'm known as well as the Senor Performo. I started with a blog and writing up descriptions on performance testing, best practices, practices guides and whatever I thought could be helpful to not walk into some of those issues or peculiarities that I found on several projects. And after that I was invited by the Perfvites team to contribute in Spanish as you well mentioned, I started to work on the Perf Bites Espanol version of the show where as you mentioned earlier, I am trying to help all the Latin American community and Spanish speaking whoever can pick up Spanish, I'll be happy to give some information.

The Latin American market especially has a lot of needs for performance testing and sadly for many of them the language is a pretty tough barrier. So if I can help them to cross those barriers, that's my mission and that's what I have been doing for the last at least with per fights and Espanol. It's been about a year now that I have been trying to spread the word of perfect.

Awesome. So you did mention you have a lot of years of experience and you still see a need for basic topics. So I'm just curious to know like at a high level, what's the IPA framework? Is that something that helps address maybe some misunderstandings that we have in the performance community?

Yeah, I started to create it departing from the point where I noticed that many companies, not only Latin America but in the USA and Canada where is where I work on. I'm a consultant so I'm at different customer every month or two months and I started to realize that many of them had some of these vices. As I mentioned, everybody's trying to provide performance metrics through low tests and even it's something that also that drives me nuts. Many places I have found that they do not know or.

Are aware of the difference in between a low test and a performance test in general. So the IPA framework that I started to devise helps many companies that do not have any of these best practices, many are trying to gather some metrics through load tests. Again, as I said, the IPA framework helps to gather or have those metrics, if possible, out of the box. And the solution in the case that it's a solution that you are developing and trying to help to not touch, have that need all the time. Many companies, I would say I need the response time for this, let's say a stored procedure, something that a cron job or something that is not triggered by a user or a load test. And they try to do it through an automation for load testing where the IPA framework can help to receive some of those metrics without.

Having to go mindlessly, if I would call it in a way to a low test right away.

Very cool. So that's one of the questions you do address is performance is not load testing. It is a great misunderstanding. What is the difference for someone that doesn't know?

I've come up with a definition for performance testing in general where I usually teach it as three pieces. Performance testing. As the name says, it's testing. And that's another confusion where you have performance optimization, performance planning and several other tasks that are afterthoughts or are around performance testing, which are confused as well. Performance testing consists of three parts. First, you are testing, of course, you are validating the response time of any action or any input that you put on your system. This is.

Recorded in seconds, milliseconds, several times. But this is the first component. You always measure time. How fast is your application? The second component is to check your infrastructure. When you are triggering this application, you want to see how the pieces of your system measure. You gather metrics generally through monitors, and that's generally the second piece. You try to check every piece of your solution. Servers, load balancers, switches, routers, networking databases, anything that could be involved in your solution. You want to see how it does. The third component is a given use.

This is where the load test falls. The given use that you would like to measure in that case is load having several users. But a performance test can come out of just having a single user clicking here and there and you can gather results. That use could be taken as a baseline or single user test.

It is not necessarily being done through automation. You can have.

Real person just clicking at the application and gathering a response time and checking your hardware, how it measures under that single load.

That's why I try to encapsulate everything. Every type of use that you can give to an application can consist of a performance test. A load test is a Specific subfamily of a performance test. Sometimes I say that performance testing would be an equivalent of Microsoft Office. And a load test is Word, just a component of it. Word is heavily used, probably is together with Excel, two of the most heavily used components of Office. But it's not Office.

So I guess one of the big things I think you mentioned is when someone engages you for performance testing, they usually want to load. They assume it's a load testing type of scenario, but they also don't seem to know their infrastructure or what they're even doing. So I think the first step in your IPA framework was know yourself, just talk a little bit about about that before you actually jump into any type of performance testing.

Yeah, so as you will mentioned, when I'm invited to help performance testing an application, the customer most of the time thinks of a load test. They usually refer to this performance something, something they are not even sure at times. It might be even a checkbox in their project plan or something that they are required to do. A first step when you have to do a load test is to know the numbers, know how much you have to simulate, know how much you have to hit a given.

Action or a function or something that the system to be tested can do. And several times it's so frequent that I could count it almost as a rule when I get to a new customer, they do not know this, they do not know how is their system doing. And several times they look for us to try to figure these things out from the low test. But we end up in a situation like the chicken and the egg, which one came first? And generally it's a recommendation by several performance engineers, gurus. And where I am basing this or gathering this recommendation is from Bob Westcott's book, the Ebery performance book I believe is the name. And there he mentions that number one, the step number one in any performance endeavor is to know the system, to know how much is it being used, how many times an action is being triggered, how it is doing under current circumstances. And some of those metrics, again you don't need a load test to know them. There are several other.

Routes that you can take to be able to gather that information rather than a load test. And on top of that load test.

Gives you many times the response time from a client side perspective. So you don't have the best metrics. When you try to gather some of those through just a script in a low test, it's better to know it from the server what is happening and then you can add some of the Latency that you will see many times you will have the processing time and the latency and you will not know the difference. And you are just like tapping and trying to figure out.

Crudely if I can call it in a way. So that's why I am always suggesting you can know these metrics many times and you should have them from scratch without needing a load test to gather them. Generally speaking, this is I would say the wrong approach or not the most efficient one.

So how do you get the metrics then? Is this where you come into. I think you have a shift left approach where if it's in development, you should catch it in development before you even get to the full blown load type stress test. Is that how your framework works, where you shift it left so that you are capturing metrics earlier in the software development lifecycle first?

Yeah, this is a set of steps that I came up with from the need from Agile projects, but I think this is something that should be switched left.

Even in waterfall projects. I mean it's some information that should be available.

As the code is developed and released into the QA environment or even the debugging environment. There are metrics that you should be getting right away without waiting to get to the point where we, the performance engineers are trying to automate it. And many, many times I have being in projects where I get to the phase where supposedly all the previous testing has been passed and has been done and I start to automate and find processes that are considerably slow just manually and I get to a point where hey, how is it possible that this process made it to this point, that it passed unit user acceptance, functional several levels of testing and no one ever raised the question like this is slow and I'm the only person running it. So that's why I started to figure things out and trying to mitigate because again, for most of the industry performance testing is the low test. But to me it's an incremental quality assurance process where the point is to decrease the risk of having something slow or breaking your system. If you start checking that first at a unit testing level and check that the process or your function, your method is not slow, you are passing the first gateway of quality in terms of performance, to call it in a way. But the point here is how can I know that I started to think about it. How can a developer, which is most of the time not so interested in knowing, they just want whatever they are programming to work, sadly they are under pressure and they have other requirements, so how can they have that Quality gate right away, even just checking in the code, it should be possible for them to tell I'm checking in code that performs well, that runs fast. And probably even Ron Mark Tomlinson made a little bit of fun of me calling, calling this way. But I call them mini load tests where if you know how fast is your code executing under a single thread and you can somehow call that code from the developer machine parallel. Let's say I run a 5 to 10 thread test, which counts as a low test, but it's tiny, which sounds like words that you shouldn't put together. But.

You pass the first gateway of quality in terms of performance by assuring that your code did well, you had a mini Lotus, the code executed and you could send it several times. The developers just checking code, they managed to make it work and they more or less say I'm done with this next functionality that I have to program.

The IPA framework that I worked on helps a little bit to at least on the first step for them to be aware of that quality gate or even requirement in the phase gathering requirements for an agile team. When you have that in your to do or to pass check marks, you guarantee that at least that first step is being fulfilled. It's incredible how many times we performance engineers get to points where we wonder how we see that no one else notified this earlier.

Absolutely. I love the concept of these mini load test. Almost mostly when people are developing, you have one or two developers banging on the system, but then you get to like 3 users or 5 users and all of a sudden you see really degradation quickly. So I don't know why more people don't try that approach.

Yeah, if you do that, you avoid several.

Issues that we find that some, well, issues or errors that developers have at times when they are creating code. And several times I've seen the code is not optimized to be parallel executed. It shares variables, it shares there are some bugs that can be even detected at that point that sometimes they make it to production and that's big trouble when that happen.

So your approach is kind of very developer centric almost. And I think one of your phrases was don't automate everything.

Does this tie into that or what.

Did you mean by don't automate everything in your IPA type framework approach?

Well, my approach generally in terms of performance, I like to measure everything but not automate everything.

I follow up religiously the Pareto principle where 80% of the load in your system generally is caused by 20% of the tasks that your system can do. So Focusing on the other tasks that your system can do in trying to load them. It's a mistake when again, many companies confuse load testing automation in the goal that they can achieve or.

Why it came up to be load testing automation. As the name says, it's generally for load testing. If you have a process that happens once a day, once a week, or I would say even once a minute, it's very easy for you to manually generate that load and get away with it, rather than spending several hours of a scripter, a performance engineer, which we are not that cheap to get us automating a process that you can easily invite a user or someone that might not be as expensive as a performance engineer and ask him, hey, could you click this a couple of times while we are running the big load test? If it is a process that you are concerned about, but you should measure everything else and know how it is doing where, even if it's a single click or through a load test. And that's another component of the ipa trying to have everything measured and know how your system is doing, even no matter how small is the amount of clicks, load or interactions that you have with it.

So I guess the sticky point here is you can measure everything, but how do you know what's important? Like what metrics? I think you have a saying metrics should be a fact. You'd be collecting all this information, but how you know which metrics should you should look at that are really telling you true information that you need to know about the performance of your application to test.

So again, I would say the most important ones where you do not want something not performing. The critical ones are the ones that consume that 80% of your load.

And it's usually very easy to identify the ones that you have the most hits, the ones that happen the most often. Those are the metrics that you should put special attention to. There are others where it's good to have and to know how your system is doing, but it's again, not so critical.

I don't like to say this, you could skip some of those, but my personal taste is to have at least a baseline for everything that your application can do through the framework that I proposed, the IPA framework, you can activate and deactivate some of this based on the importance of each one of the actions that can happen in your environment. But again, I would like to have as many as possible. I understand that there are sometimes environments where.

The load is so big that the metric gathering and logging can become a bottleneck itself, but I like to give the example that if you want to push the system that high and you're starting to have the metric system as a bottleneck, you should consider expanding it. And the example that I give to exemplify this, these metrics could be like all the instrumentation that your airplane has, generally altitude, latitude, speed, I don't know, fuel. And there must be thousands of metrics that an airplane gathers. And if you are trying to put more people in your airplane, let's say users in your system, and for the sake of pushing more people, you start to take out these metrics, these instruments in your airplane. Personally, I would not like to be on that airplane. And something similar should be applied to your applications. It's crucial and very important for you to know how your applications are doing even in production, even if you have a very busy production, you should consider expanding it rather than.

Saving resources on something that.

I might be a little bit biased as a performance engineer, but I think it's critical.

So you did mention the IP framework, both the metrics, how you're getting it with code. I don't think we actually talked about it though. So for the get these metrics, usually you start off as like coding something simple as like a stop watch or something approach to a method. Like how do you code or instrument your code to give you these metrics? If someone's just trying to do a homegrown type approach.

So most of the implementations that I have done around this, it's as you say, homegrown. And they start from the application that we are creating. It depends on the code, what version, what they are doing. But as you well said, generally speaking it's just to put a start timer and end timer in most of your methods or functions in the code that you are creating that can be reported somewhere else. Another first or initial step is to try to create method or function or a call that is easy for the developers to implement and to put more or less like brackets, start here, end here and inside of your code to be able to track all the steps that you are doing and that becomes.

A method of a class or something that you call in every piece of your code. And that even on some instances we have placed rules for the checking that if the code doesn't have it is rejected.

At an initial state. The very basic that I would recommend is to have that start timer and end timer and the function that is able to report those metrics elsewhere. Sometimes it has been reported to databases, to plain text files, as logs, or even something more complex to have the splunk forwarder to pick it up and put it in a splunk server or with elk the elasticsearch to be enabled. So that's why this proposition of framework is very mutable and it depends a lot on how your solution is being implemented and what are your needs. But the very basic and minimal it's to have the timers and somewhere to send the logs you can add on top is like the cherry on the cake. You can also add utilization information like what user was running that process, from where was it, what data did it use, what process was it, and some other things that even exceptions that help you to trace errors. And if this functionality makes it to production, you have a mechanism to easily activate some alerts if something happens, if something goes slow or.

If you need to trace anything, you can be one step ahead of a customer call saying I have this problem. You can even say I know it's this, this and this and we're working on it.

Cool. So I could see developers, some developers given pushback this approach. Are there any open source APM type solutions that could pull up these metrics for you?

Yeah, there are several APM tools that can do this. As you mentioned, open source and paid. I generally recommend to have them not only in production environments but but in QA and dev. Many of these solutions give very similar metrics. From what I propose, the basic ones are response times. Again, what this approach has above some of these solutions is the.

Utilization patterns or information that you would like to track. Some implementations, we even added an incremental counter for for you to know what were the steps of the user going through your system and to give you the opportunity to create test cases from real user patterns in your system and to mimic them. So.

I would even recommend to have both an APM and the IPA or any version or similar approach that you can apply.

Alright Leandro, before we go, is there one piece of actual advice you can give someone to help improve their performance testing efforts and let us know the best way to find or contact.

Okay, well I think that word of advice would be keep learning, keep growing. Read as much as you can regarding testing topics or performance testing in general. Again, know your terms. As the book says, let's call a spade a spade. One is performance testing, the other one is load testing and make sure that everybody knows it and as contacting routes. I am on LinkedIn, Facebook and Twitter as SenorPerformo. We have also groups on all of those platforms for perfbytes, Espanol. If you are a Spanish speaker and you want to learn a little bit more about performance, you can always access through any of those. Again, if you have any questions or anything that I could help. We have several channels with Perfights teams and any of the groups we are always answering helping. And if you want to send a question, if it's in Spanish and if you would like to have it in the podcast, it's preguntasperfightsesp perfects.com, i'm sorry. And.

If you want to learn a little bit more, there's my blog SenorPerformo. It's www.srperperf.com.

Senior perf or something like that and I think those are the main channels that you can reach me.

Thank you Leandro for your performance testing awesomeness. The links to everything of value we covered in this episode and head on over to testguild.com p likeperformance01 that's testguild.com p01 and while you're there, make sure to click on that exclusive sponsor section to learn all about SmartBear's awesome performance testing tool load ninja. So that's it for this episode of the Test Guild Performance Podcast. I'm Joe and my mission is to help you succeed with your performance testing efforts. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Performance Podcast. Head on over to testguild.com for full show notes, amazing blog articles, and online testing conferences. Make sure to subscribe to join the Guild and continue your testing journey. This has been a Joe Calantonio production.