Performance Testing Fitness Functions with Lukasz Nowanski

Welcome to the Test Guild Performance and Site Reliability Podcast where we all get together to learn more about performance testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Performance and Site Reliability Podcast. Today we'll be talking with Lucas all about real world examples of performance testing. Lucas has been designing and developing software for over 18 years. He started his career as a web developer with a strong focus on UI technologies, but in time his interest shifted towards back end technologies. At Dell emc, Lucas worked on big data projects, designing solutions where the performance of real time streaming applications was one of the key aspects that could determine the success or failure of the solution. Over time his focus has shifted to building event driven distributed systems within microservices patterns and backed by RESTful APIs. Testing has always been at the heart of everything he does. And in his opinion, only highly skilled software engineers can call themselves real professionals if they embrace testing as a must have skill that helps developers develop reliable and maintainable software. You don't want to miss this episode. Check it out.

Let's face it, performance testing is tough. Traditional load testing tools create scripts that are bogged down in data. It's hard to read and require a lot of work, even for simple playback. Load Ninja cuts out the dirty work by using AI to inspect and debug your code or almost instantly. No coding necessary. No changing of hands or relays between teams either. Give it a shot. It's free and easy to try. Head on over to testguild.com smartbear and click on the link to learn more.

Hey Lucas, welcome to the Guild.

Hey Joe. Thanks.

Awesome. Great to have you. Is there anything I missed in your bio that you want to share with the Guild, Lucas?

Not really. I think you covered it nicely.

Very nice. So I know that you gave a talk recently at Ministry of Testing, one of the webinars called Performance Testing A Journey from Manual Testing to Automated Architectural Fitness Function. So it seemed like an awesome topic for the show. That's what I like to dive into today at a high level. Can you explain why you created this presentation?

Yeah, sure. So basically one of the guys who is leading that meetup for testing, Conor Fitzgerald. Gerald is a colleague of mine from work. We work together. He's a test head of Testing in Populo where I work and he kind of when I joined the company a year ago. Exactly. I gave this presentation about why performance testing is important and that we are not doing it in our company and how things could be automated and integrated as part of the CI CD pipelines. So I gave this presentation and I started exploring more and building that testing features as part of our pipeline. He got very interested in it. He said that it would be, it sounds like a great material for those test meetups. Because he's facing with a lot of questions from people who are attending those meetups. You know, they were looking for stuff like, you know, how to performance tests, what tools should you use to perform performance tests and where to start basically. So he thought it would be something great to show to give some feedback to people how you can start this journey.

Absolutely. I see think of the testing community. A lot of things, some things that are often overlooked are performance related. So I guess if someone's just getting started, how would you explain what is performance testing to them?

Yeah, for someone who is starting a career in software development, I would say performance testing basically means that you're making sure that the application that you're designing has to meet certain criteria in terms of behavior. So how fast it can respond and react to certain actions. Depending what your application is doing. Is it a UI based application or API based application? There are specific, you know, like throughputs and actions and responses, so requests which you're firing a duplication to do something and then an expected response. So I suppose to say it very in an easy way. The performance testing is basically making sure that your application is performing under specific loads in a timeline timely fashion. So basically support your requirements.

Great. So in my experience that needs to actually be architected into the application. So at what point does a tester get involved in performance testing? Do you think it's part of the role to be involved from the beginning or is it actually just testing it at the end with a tool like say loadrunner or a JMeter?

I think that currently testers, they don't yet pay too much attention to performance testing and I think they should be involved from the very, very beginning when the application starts to evolve. So when you have it on the drawing board and you start building architecture, you should be thinking things. How are we going to formulas test this application? I am in the opinion that testers and everybody who are working on developing that application should be in a position to say exactly the same way as you know, how are we going to do the end to end testing of this application? Or how are we going to make sure that the features that we are building are meeting requirements of our customers? The performance testing should be looked at the very beginning as well because certain changes or architectural decisions could be made differently to accommodate performance testing as well. For example, I am of the opinion that in modern times, with times that currently software is developed, API first approach is crucial. And it also allows you to test your applications much easier when it comes to performance testing if you have very clean APIs.

Awesome. Now I want to get into API testing, but before we do, you do have a slide called understanding your application. And you list out the parts of a particular application. Is that a place where you would start with the team, maybe list out your different components or different things that make up your application, then say, okay, for this part of the application, what can we do to make sure it's more performant?

Yeah, exactly. So that slides. Understanding your application is something that I started myself when I joined the company and I started looking at. I knew there's a bottleneck in the service that we are developing. I didn't know where the bottleneck was. So in order to design the performance test, plan the strategy, how I'm going to do it, I had to understand exactly what are how the application is behaving. So what are the different components that are building this application? So it's a very good thing to start with. You draw out a very simple diagram, how it interacts with all systems. What are the external systems, what are the internal systems, what are the databases that are used, what's technology that's used to build the application, how the data is getting into it, how the data is getting out of it. And in this way you can kind of draw certain edges where the possible bottlenecks could be.

Now for bottleneck or to find out your performance goals, I think a lot of people just focus in on say application shell support 50 users, right. And that's their goal. But I would think it's much more complic. So when you do the understand your application, are you able to come up with better goals? Like maybe our database can handle x amount of transactions or other type of goals that may be beyond just how many users your application can support?

Yeah, it's an interesting question. So when I was doing my performance testing, I wasn't really thinking about database. How many concurrent users can execute different queries in parallel? That was not my goal. My goal was to say, okay, so we are building this application for the customer. Our customers expectations are that, you know, we should be able to handle an amount of of tags. A tag is basically a group of people that are grouped under a specific name. So let's say one tag can have maybe, I don't remember now, 8,000 or 9,000 people in it. So what I was trying to do when I was building this strategy, understand the typical use case. And what I did is I went basically to recapture statistics how our custom are using our product. And basically it's called something that's called, you know, most used or favored. I suppose nowadays a lot of different products have something very similar. So you basically go over and you check, okay, so most typical tag or safe search in our product used by our customers is this something that has around 10,000 people in it. So that's a very good, it's a very good example. What in my opinion should be taken into account as your performance goal. So when you're building your performance goal, you concentrate on the strict requirements. You shouldn't go over the edge trying to build it for 100,000 users. If this is not your use case, if your use case is 10,000, concentrate on 10,000 first, then see what you can do further.

So once you understand the architecture, you understand what your goals are. Another thing people trip up on is test strategy. So are there any common gutches that people may be missing this step that kind of breaks down the overall performance because they didn't really focus on the correct strategy?

Yeah. So in the test strategy, when I look at it, it's very important that when you're running a performance test, they going to happen against an environment which is very close to production environments. So that's why one of the key goals that I always repeat to everybody who wants to start performance testing. Performance testing against local environments and development environments. It is of course possible, but it doesn't give you the correct metrics and the behavior of the system in real time. So how exactly our customer is going to use it against the. That kind of replicates similar semantics of this environment. The identification of the common use case we talked about a second ago and then how you're going to run those tests. So when you're building test strategy, you have focus on, you have a couple of choices. You can go with full automation initially or you can go with something very simple like manual testing maybe just to, you know, get your tests right. Draw the baseline, of course, and then focus on automation. So for the test strategy, drawing a baseline is very important. In our example, when I was doing manual testing initially, we didn't have a baseline at all. All. So trying to fix or implement anything in our codes was very difficult in terms of measuring the performance because we didn't know what the performance was. So was one of the things that we had to do was to Draw that baseline first. And of course the last thing which is quite important was once you select the tools you're going to use and are you going to do it manually on or in an automated fashion. You should also pick up certain tool set for how you're going to monitor your performance test. So running them is one thing, but reading the output results of those tests is another thing. You have to be very careful because there is a lot of different tools out there that you could use. Some of them are more powerful, some of them are less powerful, but it doesn't mean that you can't use those less powerful ones as long as you have kind of very clear image in your head. What exactly do you expect from those tests? What you want to concentrate, what you want to capture. You should focus on a couple of metrics only and make sure that you can capture those. That's basically the test strategy, in my opinion.

Yeah, I love the idea of a performance baselining. I don't know why more people don't do it after a code change. But I'm also curious to know with your opinion. With. I think a lot of people make choices with architecture without doing like a baseline. Do you ever recommend that people say, okay, here are the three different type of architectures we might go with. Let's run baselines against all three and see which one performs better. Or is that like just too early for that type of information?

Yeah, it's an interesting thing. I'm not very keen in this approach on the architecture changes. So when I was doing this presentation, I mentioned on purpose the fitness function and the, you know, evolutionary architectures. Because I am in the opinion that currently, whatever architecture we choose as software developers, they are prone to change. It's not like it was 10 years ago where architecture was set in stone, it was drawn on a board shared with everybody and it wouldn't change for the next five, six, seven years. People, people change, the times change, tools change. Everything evolves very quickly now. And we have to be very careful when we are building our architectures, not to over engineer, but you know, make it in a way that they can evolve quickly in time because it's inevitable they will change. So when I say that drawing a baseline is important. So once you have some sort of architecture in place, it doesn't matter if you didn't have any performance testing before, but you have some sort of architecture, it is working. And make sure that baseline is pretty much made into a fitness function, which means it always runs in an automated fashion at least once a day. And it's measuring that your requirements for this architecture are still met. And I think this is the key goal going forward. If you decide to make a change in the architecture, you don't have to worry that much because now you have something that measures the behavior of architecture change. So you know exactly does adding an additional table or maybe changing the MySQL data store into PostgreSQL datastore. How does it impact my architecture? You can actually measure it because you have an automated way of testing it.

Well, I love this. Did you coin this? Is this your idea of fitness function? I've never heard of this. So is this something you made up or is this a common computer concept I had no idea about?

No, it's not my idea. I've heard about it. So it's becoming like an interesting trend, especially around the evolutionary architectures. And I think there is one book that I'm actually mentioning in my slides as well, Building Evolutionary Architectures from Tuft Works. So I watched a couple of interesting videos from two well known architects. I don't remember their names now. They just fly out over my head. I think Needed Ford is one of them. And they basically mention the evolutionary architectures and they also call out the fitness function. And they say, you know, as a definition of a fitness function is something that can measure in an automated way the behavior and characteristics of your architecture. So if it changes and it changes daily, how that change is being captured and quantified. So I took this concept and pushed it down into the performance testing because I thought to me, performance testing is something that measures the architecture in a quantified way and it should become one of the fitness functions. If you think about there's different other fitness functions that you can use, this one is well fitted for performance testing.

Very cool. So you also have a slide on bottlenecks. Do you have any tips on how to search for bottlenecks or locate bottlenecks? Because I find a lot of times people may say, well, performance got worse, but they can't necessarily tell you why.

Yeah, and again, the searching for bottlenecks, it is probably one of the most difficult parts of your performance testing because once you come up with some sort of test strategy and what are your performance testing goals that you want to achieve, you start running those and you have to make sense out of the data that your performance tests are outputting. So this is, in my opinion, searching for bottlenecks. Now there's a couple of things that are important when you do that. So as I mentioned the tooling is important. So in my case I'm using Grafana and Promet for visualizing different aspects of my application, Prometheus for capturing different metrics. There's other tools as well that you can use when you're searching for bottlenecks. But one of the very interesting things that I observed myself, and it's another trend that's becoming very popular nowadays, it's the observability. So currently developers are also forgetting about how important well structured logs are in application design. And it is very interesting. When I was running my manual tests, it's actually the logs but pointed out me to the bottleneck. So there was a custom code that I had to write that could basically measure different aspects of my application. So where does something start, when does something finish and what exactly was done, how long it took and it goes, it outputs into the log. So the well structured logs helped me to identify the bottom line because there is nothing clearer than saying publishing of 35,000 people finished in 77 minutes. Obviously that is a bottleneck. You can see it as, you know, written on paper by reading your logs. But you have to make sure that your logs are structured. You have to make sure that you do have a good logging mechanism in place in your application and everything else like the Grafana dashboards or whatever that you use for visualizing or capturing your metrics. It's just something that can basically, in my opinion, just drive you into this bottleneck. So you can rule out certain things if you see that your GVM metrics are behaving correctly. So there's no problem with ram, there's no problem with cpu, you to have to go and search farther. It is difficult, but it requires a good understanding of how your application and the technology that's used to build it. And then you can definitely come up with something.

Now you're right. I have been hearing more and more about observability, but for some reason I can't wrap my head around it. How would you explain observability? Is this just making designing your application so that when it does run in production that you're able to debug it quicker?

It's maybe not. I don't see that as a debugging it quicker. I see it more like I understand exactly what, what happens at any point of time and if something does happen, I'm being notified about it. So let's say I know that observability has a couple of different aspects. One of them could be, you know, notifications that are triggered if something happens. So for things like if there is an exception triggered somewhere in your application, you could use tools like Sentry, which automate notification handling for that. So if there is a problem exception thrown in the logs, it notifies, it sends a notification to Sentry. Essentially you can be configured to send you an email, an email or Slack notification that something unexpected that you didn't code for happens and that should be addressed quickly. So in my opinion, this, this is one of the parts of the observability. The other aspect is obviously logs. The logs have to be clear very often, especially nowadays when you have multiple microservices instances running on multiple nodes, you need some sort of way of aggregating those logs in one centralized place. And tools like Kibana for example, help a lot, can help a lot. But if you don't have logs that are being produced correctly with the correct format, that can be indexed by Kibana, then you, you don't really have observability because the observability is, is also to make sense out of those logs, to search for them based on specific patterns. So if your application is dealing with some sort of business, kind of, let's say my application is dealing with accounts. So make sure that when anything that happens around an account, an account, account identification or identifier, account ID is present in your logs everywhere. If you have other identifiers that could help you out, make sure that are present in your logs. Make sure that you're using tracing IDs, span IDs, etc, because you can then correlate multiple logs and multiple kind of events that are happening inside your application. And then it's much easier to identify where the problems are and you don't have to worry about things like oh, it happened on this node or in that node, tools like Kibana can aggregate those loops and you can make sense out of them. So that to me is observability.

Nice. So, you know, when I started performance testing 20 years ago, it really was just HTTP, make a request to a web page, get a response back, measure it. But I believe more and more applications like you said, are being created with microservices. So why are APIs more important or how can APIs help people with their performance tests? Does it open up other avenues that weren't, weren't available to them earlier, maybe, you know, 20 years ago?

Yeah, it is a very interesting question. APIs have the concept of their contract based. And in a test world, a contract based testing is very important because if you set up a contract for Your application, you can write test scripts even ahead of time before the application is developed because you have some sort of a contract. An API well formed API that's designed up front allows you to set up those, those tests easily. As I said before, the application is even built. Now why I think this is important, the performance testing is much easier if you can, if you have APIs, because you don't have to really. I know that there are tools like JMeter that allow you to do things, you know, you can go and query tables, go into the databases, you can go and you read JMS queues and stuff like that. But if you think about it, these are just underlying technologies that are part of your architecture that can change in time. If you couple yourself to those underlying technologies, your performance tests, whatever tests you're building, will have to change eventually because something in jmeter that accesses MySQL table is not exactly the same as the test that's accessing a postgres table. Or if you're using Kafka versus Rabbit and Q, if you couple yourself with underlying technologies, then performance testing becomes quite difficult. It's not very prone to changes. When you build API thr applications with the clean API and focus your performance testing by hitting those APIs, it's exactly the same concept as you have any UI that can change because UI frameworks change and people are very changing them very constantly with the new ones coming up. Very often if you concentrate on the API testing, you're not really worried about, you know, the UI is going to change in one year or 12 months or a new component going to come up. I still have my API and I still can test my API. Does it still meet my requirement? Performance testing is exactly the same in my opinion. You concentrate on API testing when you're performance testing. You make your APIs in a way that allow you to do the performance testing easily and then you don't have to worry about underlying technologies like databases or message buses.

So look, as I mentioned, you did give this presentation on a webinar recently. Did you get any interesting questions or the same question over and over again that you weren't expecting that you're like, oh, maybe I should have covered that in more detail or people are really struggling getting their heads around this particular concept.

I don't think I got any. I'm not saying I didn't get interesting questions. I don't remember. It was, it was 20th of May, so a couple of weeks back and I was expecting, to be honest, I didn't get Some questions that I expected to get, but maybe because the audience was different, I have to tailor my presentation accordingly to the audience. I, I did get some feedback that it was very technical. I really tried to make it less technical and more focused on the agenda on why the performance testing is important and why you should automate it. So I don't really think that I got a question that would really sidetrack me. One of the things that I was expecting to get is why I built performance testing on using a build tool like GitLab and not had and didn't have a dedicated performance testing in servers or even environments. Right. But that question didn't come and that would be one of the questions I would ask when I was making a presentation. Yeah. Why did I. Well, I did it because of the nature of my performance testing. So I know that performance testing can be broken down into very different, you know, states like you can have load testing, you can have stress testing. But what I did, I didn't really build that. By definition of a fitness function, you just want to make sure that your application still meets your requirements. So the performance testing in this, in this case, when converted into the fitness function, it doesn't have to be make it that stressful on your, on your servers, if you know what I mean. So it's not really a stress test, it's not really a load test. It's more like it takes 10 seconds to load 50,000 people. Every time I make a code change. I should measure that. This 10 seconds still is max, but it doesn't take 20 seconds, which is twice as long. It still takes 10 seconds. I don't need to have super powerful, you know, load testing machines from, I can, from where I can spin multiple virtual users or I don't need 10 threads or many CPUs, etc. That's not needed. That's why it's much easier and simpler to build it this way and make it to be part of your CI cd because it's running from your continuous integration anyway from your build server.

I think that's critical because a lot of people like you said, think of performance testing as like this big end thing you do at the end where a bunch of different users and what you're doing is almost, you're shifting it left. So developers, as they check in code, they're getting instant feedback. So you're able to do that using this approach is what it sounds like.

That's the idea of this is you get a very quick feedback. We know that performance tests can Load tests and the other tests of that similar kind can run for many, many minutes, even sometimes hours. And sometimes something can go wrong. Usually people run them maybe once a week, maybe at night builds maybe every two weeks, depending on how big your application is. This approach is completely different. It's shifting left. It's treating it exactly the same as your CI CD pipeline. When I make a code change, I want a quick feedback on my unit tests that my application is intact. Performance testing built as a fitness function in this automated way allow you to do that exactly similar way. I could run them on every code check in. I decided not to do that because environment is quite busy. We have, you know, a lot of teams using the same environment. So whenever the code commit happens, it would be not feasible. The results happening from those for those automated performance tests would not be, you know, you would have a lot of false positives. But having to run them at least once a day of peak hours against your master branch is good enough to make sure that your architecture characteristics are intact after the code change.

Awesome. So, Lucas, should everyone be doing performance testing? Because like, back in the day, it used to be like a group, just like security testing used to be a different group. So nowadays, what are your thoughts on that?

And I have a very interesting answer to the question. I think that question actually was brought up as part of a test meetup. How would you go and sell performance testing to, you know, to your managers, product owners? Well, I gave a very simple example example. Managers came to us. Sorry, yeah, managers came to us and said, we have to onboard a very big customer that will have about, you know, 60,000 users. Can we do this? And my answer was, let me tweak my performance tests and I will run them now and we'll let you know. So I did run them. I checked, and our application can support 60,000 users in kind of, you know, peak time. And the answer was, yes, you can onboard the customer that has 60,000 users. And I think if you think about it, that feedback loop was about, I don't know, one to two hours, maybe three hours. Because obviously I had to make a little bit changes to make sure that can happen. But that's, to me, that's a very clear answer to the question, should we perform this test? Of course you should, because how else can you know that you can support requirements that will come at some point in time? You don't until you perform this test and you have your baseline.

Awesome. Okay, Lucas, before we go, is there one piece of actual advice you can give to someone to help them with their performance testing efforts. What's the best way to find or contact you?

Yeah, so you can find me on LinkedIn. I think that's pretty much the only place that I usually check. Most of the time I'm not very active on Twitter or other different social networking that are available. LinkedIn Twitter from time to time. That's where you can find me.

Thank you Lucas for your Performance Testing Awesomeness. Links to everything of value we covered in this episode. Head on over to testguild.com P36 and while you're there, make sure to click on the Try It Today link under the Exclusive Sponsor section to learn all about SmartBear's two performance testing solutions, Load Ninja and Load UI Pro. So that's it for this episode of the Test Guild Performance and Site Reliability Podcast podcast. I'm Joe and my mission is to help you succeed with creating end to end full stack Testing Awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Performance and Site Reliability Podcast. Head on over to test guild.com for full show notes, amazing blog articles, and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.