The New Performance Game with Cristiano Cunha

Welcome to the Test Guild Performance and Site Reliability Podcast where we all get together to learn more about performance testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Skill, Performance and Site Reliability Podcast. Today we'll be talking with Cristiano all about his recent blog post on the new performance game. If you don't know, Cristiano is a solution architect and test advocate at the X Ray team. He's passionate about testing automation and delivery process improvements and he started his career as a developer and evolved to be a tester and he's now been over 10 years in the testing area. So really interesting background, really curious to know why he got into testing and even more why he's even into performance testing, which is even rarer. So really excited to have him. His performance path has been on both sides of the trenches as you heard and provide him, and it really provides him with a unique view of the process and challenges it puts forward. So really excited to have him on the show. You don't want to miss this episode. Check it out.

This episode is brought to you by SmartBear. Listen. Load testing is tough. Investing in the right tools to automate tests, identify bottlenecks and resolve issues quickly could save your organization, time and money. SmartBear offers a suite of performance tools like Load Ninja, which is a SAS UI load testing tool, and LoadUI Pro, an API load testing tool to help teams get faster full visibility into UI and API performance so you can release and recover faster than ever. Give it a shot. It's free and easy to try. Head on over to smartbear.com solutions forward/performancetesting to learn more.

Hey Cristiano, welcome to the Guild.

Hi. Thank you for having me.

Awesome to have you. So what caught my attention was your recent blog post you wrote about the new game of performance testing. I guess. Before we get into that though, is anything that I missed in your bio that you want the Guild to know more about?

No, I think you cover it really, really good. I just want to add that I have a journey that has led me from development to testing. I even get into infrastructure. So I was working a little bit on pipelines and things like that, focusing always on testing. So that really opened up your eyes and your mentality to the different challenges of the software delivery life cycle.

Awesome. So you start off as a developer. So is. Is that what got you into automation and testing was DevOps and the pipeline automation type stuff?

Yeah, a while ago we started to we were developing mobile applications Far ago and we were working together with Microsoft in delivering some software and they already had heavy testing processes, I don't know, eight or 10 years ago. And that started to, because in my area in Portugal, where I'm from, the companies were not that evolved. So we didn't have at that time testing teams present. And not only in Portugal, but everywhere. We didn't have that function as a separate team or even within the teams. But Microsoft for some reason already has the processes in place, the nightly builds and everything because they are bigger. And that triggers the need when we join Microsoft to have a testing, testing team. And we converted, some of us developers have converted to testing and because we were developers focusing on automation and that kind of put myself or light my bubble, my bulb a little bit and just I never get out of it after it. So I really enjoyed the validation, the process, the understanding of the thing. It's a pretty different job, pretty responsible, different responsibility than the development side.

Awesome. So I guess what's even more odd is you actually go into performance testing, which really is on the pipe, you know, on the ladder of where testers focus their attention. It's usually really like people don't even get there. So why do you think performance testing then is like an important part of DevOps?

I think DevOps is really bringing the attention then that the performance and also security is needs because of the need of bringing these pipelines together. It's not only to test functionality anymore, it's not only to test if you are going to deliver this new feature and if it's working okay. Because now DevOps and observability are bringing the attention of what is happening in production to the teams. Before it was just, I'm going to deliver this feature, new release. Here it goes.

Production.

I'm going to focus on the next thing to deliver. And now with this new responsibility, the team is now looking to what is happening in production. So they are more aware of how the application is behaving in production and they have new concerns. So they want the application to behave well. They want to not be caught by any security loopholes that may be have passed through the testing. So I think all of this together has put this new responsibility closer to the teams and the teams are finding ways to couple with that. And I believe that that is why performance now is appearing. Because like I said in my blog post, when I started performance a while ago, it was really, really about after going to production, let's see what's happening. Let's put together an environment production like so with everything that production has and we were at it for months in a row just to have an environment that was close enough to production so we may have some meaningful conclusions. And then the tests were really heavy, running for nights or days just to extract some conclusions. And the reports were not easy to understand. So even when the test is over, we gather all the reports regenerate. At the time was an Excel file where we try to compile all of the results and reach a conclusion, understand if the performance is there, is not, if it's not why. And this is another set of questions that come about.

Yeah, I mean that actually was my next question is why do you think or how do you think performance testing has changed over the years? So you mentioned it was like higher, like an afterthought almost. So once the system was already built, you do like a stress test. So has the type of performance testing you do changed as well? Is it more of a user type of activity now? Has it react for one user on a client?

Yeah, I believe it has changed throughout the years. And the first experience that I just described is it's still around. So we still need those heavy performance stats in production, but the teams now have the ability and the will to do that sooner in the pipeline. So the shift left approach and bringing the tests more close to the code, code compilation and code development and these tools that we were talking about, this different approach will allow the teams to define more concise performance tests that will not validate everything that the application will do, but they will actually focus in one or two critical or critically identified parameters that they want to validate their application with and they will be able to run those performance tests in the pipeline. So instead of being heavy and taking all of them all night to run, they will need to be shorter, they will need to be faster and they will need to have one thing that the heavy performance stats at the time did not is they need to understand if they have best or not. So they need to have some criteria to understand automatically if the performance is best or not. And having this built in in your pipeline will bring you also an historical view of the performance of your application or your features, depending on what you are focusing on.

So I see a lot of teams struggle actually with this part getting it in the pipeline, making the performance test, as you mentioned, shorter, faster and have a pass fail criteria in historical data. So any tips on how to do that?

Yeah, actually one of the. I give an example on my, on my blog I'm using Gatling and there Are other tools around. You have K6, we have Locust, we have another set of tools that are now available for you to programmatically define some performance tests. And they allow you to define your scenarios, your load and everything. And they allow you to do one thing that in Gatling is those failure criteria that you just talked about in Gatling, they are assertions. So they allow you to define how I will for instance ramp up my users for how long giving in which scenario that I want to ramp up the users. Of course I'm not going to go overboard and have 10 million users in the pipeline running tests. Of course they are more a sample of something that I want to do. And this is it will depend on the team, of course. And once you have that program, you can execute that on the pipeline and the assertion will tell you if the test these are not valid and you have a success or failure. Because their session will allow you to define if you don't want for these requests to have any 500 error for instance, or if you want to reach a certain request time, you can define some thresholds. You can say that the 19 percentile of this, of that cannot be more than this. And every time you execute those tests in the pipeline, it will validate against those failure criteria and will give you a pass or fail. And of course you can go a little bit over that with some tools also within the pipeline, because Jenkins has also those plugins that allow you to see the trending of the test that you are performing. And more than the failure criteria. You can also define a criteria. That is if my tests using the approach of baselines, like we do in visual testing, you can define a baseline saying that for my application checkout, I'm used to have this value on every build that I have so far. And in the next build I want to remain on the 10% more or less of the values of performance that I have. So I don't want to be worse 20% on the next build. That means that something is wrong. I don't know why, but the performance is not the same as I was expecting. So that also is on an alert for the teams. And this approach of having the performance tests closer to the code running in each build will also allow me to reduce the scope to analyze what went wrong. Because once we have all of the feature in production, something is wrong. We have to go to look everywhere to understand where is the culprit. But if you have it closer to the code and in each commit you are able to understand if the Performance is better or worse. And if it's worse, you can say, okay, this is the list of commits that I have. Let's see from this list if we can find the culprit here. So I believe it's an advantage. Of course there are also things that now the team needs to worry about. So it's another type of test, non functional test now that the team is responsible for and needs to maintain also.

All right, so as someone so much experience with automation as well, you know automation can be flaky and so when it runs in the pipeline, people start ignoring it. Do you have the same issue with performance testing? How do you make sure your performance is like not giving false issues and it's really when it fails, the team's like, oh, we definitely have a performance issue because these tests are rock solid. Yeah.

So the basic principle should be to have the first thing is don't execute the performance test on the same machine on the target application. Because otherwise you can have the load of the application impact your tests and vice versa. Of course. The other thing is mock everything that you can so everything that you is not on your scope to do validations, mock it because you don't want to be validating if external API is not performing enough. It's not your focus. You need to focus on what you are delivering. So mock everything that you that you can even data. So focus on exactly on what you want to do on the performance test. And you have a lot of so unit testing already has a lot of mocks available. We have also other things available like wire server, mock server. So a lot of services there that allow you to mock everything that you have in order that you have an isolated environment where not a lot of instability will be brought because of the changes that happen. And don't be over ambitious regarding your tests because if you think you're going to validate everything regarding performance of your application with the test in a pipeline that's not the target and you probably go overboard. You're going to do too complicated tests that will take too much time, too difficult to check. So keep them simple, keep them focused on what you want to validate, keep reviewing them and make sure that you have a shield environment. So mock everything or do a preparation before four of that and use in DevOps as you are you are focusing on automation. So infrastructure as code. If you want to spin up an environment containers use mock everything that you can use to bring the ability to replicate every time your test. Because the worst thing that can happen is if I Run two times a performance test and I have two different results. Something is not good. So we need to, at least for the same version, the tests should be within a percentage, more or less the same. We should have the more or less the same results.

Awesome. So you mentioned earlier that you are using Gatling in the post, which I'll have a link to in this episode, is you are using Gatling, but you mentioned locus and K6. Did you do a proof of concept? Why did you choose Gatling as the tool for you?

Yeah, for this particular case, I use Gatling because of Scala. I just want to try out Scala. Scala. Scala. I don't know how you pronounce it right, but I never tried it out before, so I wanted to try it just to check out the languages. Of course, Gatling has support for other languages, but the native language that it was written on was Scala. So I kind of went for it and it was the first tool that appeared to me when I was searching for tools a while ago. So it's a really old tool, Gatling, at least from my mind. I kind of remember when I was searching, I was. And the name is catchy. So for me, yeah, I was just searching for something new to learn. And that's why I like blogging. Also, it allows me to do my personal investigation. Of course, if I was searching for something for my team, my advice should be use the language that your development team is used to use. Otherwise you'll be alone on your performance test and no one will help you because they are not used to the language that you choose. So get together with, who knows, things that are the development team that I use to develop things and use the same language because if you need a helping hand, they will be there for sure.

So are you saying you're alone because you're using Scala and your team's not?

No, I'm saying I just used Scala.

For the blog purpose. I don't use Scala.

Yeah, in my working environment.

Yeah. Nice. So I'm just curious to know, you mentioned that the traditional performance test hasn't gone away. We still need to do that. So who does that though, typically? So a developer is actually involved in the unit level, I guess, performance testing and then do you have a separate team for the full stress test type of activities?

Yeah, from my experience, yeah, we still need. Because we need guys that are used to define performance tests that I use to live and breathe performance to try to define that. There are guys that are really heavy knowledge regarding technical issues because they can approach the team the development team and talk directly one to one with them, understanding why they are using Kafka for messaging and doing those questions that are really deep technically because they will find bottlenecks in databases, they will find bottlenecks in messaging systems and they will need to understand why the application is using the way that they are using. Because more than just say oh, the performance is worse, we need a performance team that can say that and can lead you to an answer to a solution. So they say they should know the performance is worse because of the bottleneck we find in the database that is used by the service. And it's, I don't know, doing too much requests. They are doing not using indices in the in the database. They should hint or advise on where the solution may be and they should work with the development team in order to enhance the performance and check after the changes were done if the things are enhanced or not. So I believe we will still need those kinds of deep understand professionals about performance. But what we are going to do is now make the team available for them so they will have a notion of performance also within the team that will allow them to work closely with those performance engineers. Talk one to one with them also regarding performance. Because the performance concept will start to build in within the team with those performance tests they will do on the pipeline. And with that they will also will make the build more performant and the test that will be, the performance test that will run at the end will be also will be more valuable and will be more performant, for lack of a better word. Because the build will have better performance overall.

Absolutely. So also in this blog post you mentioned not only DevOps but observability briefly. So I'm just curious, how does performance testing relate to observability?

Well, I believe observability will bring all of the data that you have in production will allow allow you to understand how will everything is behaving in production. And that will have a direct correlation with performance. Because when we talk about those heavy tests of performance tests that we are doing, they are not just looking to the performance of your application, they are looking at the performance of your application. But they are also gathering all of the data of the machines, of the network, of everything. Because if something happen on those loads, heavy load performance tests, they will need to correlate everything to understand. Well, if I'm running these tests and the checkout is not working for some reason, I see now that the memory on this machine, that is a database is really high or the consumption of the disk in here and there, it's really high. And they will allow them to understand what really is causing that performance issue. So observability is more of a window that now is open to production that we didn't have before. And of course observability is another investment because we need to have the metrics available. We need to have the correct metrics available. It is really not just logging things or putting things in everything available. It's really looking at what really matters and what we can extract conclusions about. So understanding how your application is in production and in the future, I believe that we can also predict with observability possible behaviors of our application.

Cool. So you also mentioned as the pillars to make your test more reliable in the pipeline, you mentioned shorter, faster. You also mentioned historic data. So I'm just curious. So observability and historic data. And then later on in your post you mentioned InfluxDB and Grafana. Are those the tools you use to help you get that information?

Yeah, I use InfluxDB and Grafana also to help me because as a developer I want to follow things as they are happening. And sometimes just looking at a log that is just spanning on my window is really hard to read. It's not that intuitive to understand if my performance tests are doing what I expect or not. So pushing all of the data that is passing on the log regarding the performance indicators that are just spinning around into a database like InfluxDB and putting a Grafana over it and defining a dashboard will allow me to check in real time what's happening to my application while my performance tests are executing. So I can see the amount of requests are growing as I was executing, expecting to, if the visual users are or not growing, if the number of errors is or not there. So it's just a matter of imagination. After you have all of the data there, it's just putting the dashboard together that answers your questions. So it's really a cool tool. And getting used to those tools will also help you afterwards with observability and what happened, what's happening in performance. Because from my experience of infrastructure, Grafana is the de facto tool for the dashboards everywhere. So if you are used to do that in your tests in the pipeline and looking at those with that, it is really easy to go and define a dashboard to look at what's happening in production. And sometimes it's really, really helpful because in Grafana you can do things as complicated as defining an SLO and check it, put a dashboard to let you know if you are or not over that objective and give you warnings and launch warnings directly to you or other things. So I believe it's really, really cool. And having all of the data there will allow you to have an overview and go a little bit back on historical data. If you grow window size of the data that you are looking at, be careful with that of course, because if you are saving everything and you go 10 years back to look something, probably something will not go good or you are spending a lot of money in saving those data.

Absolutely. So you mentioned a bunch of different things, different testing, DevOps performance, testing observability. So it goes into like SRE type things almost infrastructure. How much autonomy do you have at your job? Like it seems like an awesome job to me because it seems like you're able to, are you able to focus in on whatever you want within the testing kind of criteria?

I'm able to do different things because in my actual job I'm investigating possible tools to integrate with the tool that I'm working with that is X ray. So with this I'm searching, I'm investigating and I'm looking at tools, where they work, how they can integrate with the tool that I'm working with today. So I'm not focusing in performance, I'm focusing basically in everything in testing, which is really, really cool. So it's my first time in there because my previous job I was focusing on infrastructure. So I learned a lot about pipelines, about messaging, about infrastructure as code, everything we need to know. And that was really strange for me. Before that I was leading an automation team. So I grew an automation team myself. We built a framework, a custom framework for all of the company. We put some best practices. I was an evangelist, I grew team. So it allows me to have all of these different experiences. And today I will say that my go to answer is depends because nothing is simple enough that we can say well it's because of this, so use this, use that. It's. It's always, it depends. We learn a lot of of menus out there. So we have a lot of recipes that we know that has work in this company in the past. But it's hard to apply that directly in other companies. So the important is as you go on your career, learn everything that you can learn with the guys and thank you Joe for having the Test Guild. So it's a really, really cool fellowship of guys and, and they are really, really contributing and talking about things, having experiences within your area. I will say that don't think that a small experience is not teaching you something because I was not really, really happy on the infrastructure area because I needed more of the. I missed the testing area. I was really, really focused on infrastructure. That's why I'm now in this, in this place where I can explore everything regarding testing. Not only the performance, the functional, non functional skills that is, that is needed and understanding also the community. So, so my job I think for now is really, really good because I'm, I'm where I want to be. I'm able to write a blog, I'm. I'm able to be here with you talking in a podcast. So I'm in a phase of my career that I hope that I can give back to the community about everything that I learned. I can keep learning and keep following things, going to the test guilds and other conferences, listening to guys that have top hedge things. But what I'm really noticing is we are not that advanced. We have guys that are really, really good and far fetched about knowledge. But the majority of the community is still struggling in having a development team separate from the quality team. I think they need to listen more and more about these things. Not to make the change now, but at least to start thinking about it. You know, placing the seed and then eventually when you do that and guys are intelligent, they will start thinking about it, start questioning things and the change will happen. It's my evangelist vein coming in.

Yeah, that's why I love that you post this on your blog. It may be old hat to you, but someone else sees this and maybe like oh my gosh, what is this? So spreading that knowledge is critical so I thank you for that. All right, Cristiano, before we go though, is there one piece of actual advice you would give to someone to help them with their performance testing efforts and what's the best way to find or contact you? Well, you can.

Well I'm going to start backwards. The best way to contact me is I'm present in LinkedIn and in Twitter. So you can find me there in my blog. I have there all of my contacts in my email also. So you can reach me, contact me, everything that you need and if I can help, I will be there. Regarding performance, I will say sometimes with everything, with trends, with performance, I say that don't look at performance as one tool or I want to achieve this or that. Look at what is your problem? Do you want to have a more performant application? Why do you have a problem now? You need to focus on what you really need to solve because challenges we all have Everywhere of course, if you can do that since the beginning to have this little bit of concern regarding performance, regarding security. Because at the end of the day it seems that now we are demanding more from the teams but it is the assumption that those things should already be there but they were never really addressed. So now we are addressing those. So performance security that were afterthoughts are now needed to be built in within the DNA of the team. So everyone in the team should be worried about how will if my my code that I'm producing here, how it will affect the performance over all of the applications application. I need to start answering these questions before even before defining performance tests. So heavy load performance test is in the code review in the pair programming. Look at what you are doing and we saw a presentation in Test Guild about the big O notation. Those small things are really really important when you are reviewing or big chunks of code with cycles and everything, everything that you can gain there will impact the performance over. Because if everyone is looking at their piece of code and questioning is this the best code, best performance code that I can do and will it interact with the other pieces of code that I have with the rest of my team and everything will work. It should be a good first step. Of course if you are already more advanced than that, start looking at those small tools that are out there, K6 Gatling and experiment for yourself. See how easy it is just to do a performance to an API, internal API for starters and then take it from here. Talk with your developers, the rest of the team. I say all the developers, but it's the rest of the team because we are all in the same team. So talk with the rest of the team. Say that oh, I'm looking at this tool. I'm going to try this. What do you guys think? You guys and I believe that all of that concern and collaboration is needed because if you are the only one concerned about performance, you will not go far. So we need to have everyone on board for that to change.

Thanks again for your performance testing awesomeness. Links to everything of value we covered in this episode. Head on over to testguild.com P87 and while you're there, make sure to click on the try them both today link under the exclusive sponsor section to learn all about SmartBear's two awesome performance test tool solutions, Load Ninja and Load UI Pro. And if the show has helped you in any way, why not rate and review it in itunes? Reviews really do matter in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Before Performance and Site Reliability Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack performance testing awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Performance and Site Reliability Podcast. Head on over to testguild.com for full show notes, amazing blog articles, and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.