Performance Testing Infrastructure Building with Nikolay Avramov

Welcome to the Test Guild Performance and Site Reliability Podcast where we all get together to learn more about performance testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Skill, Performance and Site Reliability Podcast. Today we'll be speaking with Nikolai all about performance testing, infrastructure building and his Automate the Planet course on performance testing. If you don't know, Nikolai is a team lead developer in test at Automate the Planet and lecturer at Telerec Academy Alpha QA. And he's been at this for over 10 years. He's worked as a Quality Assurance Engineer at Telerik, which you probably know was recently or not recently a few years ago acquired by Progress Software and he started his career in the field from the ground zero. So he's taken part in the Telerik Academy to become a principal QA engineer ninja that is at the heart of any innovation of automation initiatives that he's involved with. And during his tenure he's developed and integrated several custom frameworks and tools used throughout the company. To name a few, he's created a custom load and performance testing framework, a system testing framework, web service testing frameworks, visual validation frameworks, and many other productivity tools that really give them the highest quality of web and business assets for the company. So as you can tell, he knows a lot about different types of automation but focusing on performance testing. But we'll see what else comes out. In the past few years though he's been working at Automate the Planet, he started spending more time consulting different companies and helping them accomplish their test automation journey to quality, quality awesomeness. As I mentioned, he has a lot of experience in a lot of topics. You don't want to miss this episode. Check it out.

The Test Guild Automation Podcast is sponsored by the fantastic folks at Sauce Labs. Their cloud based test platform helps ensure your favorite mobile apps and websites work flawlessly on every browser, operating system and device. Get a free trial, visit testguild.com sourcelabs and click on the exclusive sponsor section to try it for free for 14 days. Check it out.

Hey Nick, welcome to the Guild.

Hey Joe, thanks for having me in your awesome podcast.

Thank you. Great to have you. Like I said in the pre show, I love what you're all doing at Automate the Planet. Before we get into it, Nick, is there anything I missed in your bio that you want the Guild to know more about?

I think you got it right. I just go through the different layers of software development and different products testing, so that gave me the opportunity to realize how different Perspectives are in terms of automation and performance testing and I love that.

So I'm just curious to know, you probably started more doing functional automation. How did you get involved in performance testing? And over the years have you seen more and more people wanted to learn more about performance engineering? Performance testing?

Yeah, actually performance testing was one of the first things I heard when starting in the company and I was very surprised to see not really quality assurance, saying the words load testing. I'll be doing load testing or performance testing. But it was actually the database developers that were implementing some new SQL code and wanted to throw some load against it to see how it performs. So even from the first days I got really interested into this and along the way I was also always looking into getting involved into such initiatives.

Awesome. Whenever I think of Telerec, I always think of Fiddler. So were you involved at all with Fiddler and did that help you at all with performance testing?

I would say I'm a power user of fidware and know very well the people behind it. It also got acquired like a few years ago. It was not initially written by by the team, but the new stuff around it are really exciting and we have been using it really heavily accomplish different tasks. So yeah, I was mostly on the marketing side where we were trying to handle all the traffic that the feedware webpage download page was having.

Awesome. And I'm just curious like this maybe a personal question. Why seems like a big company to I think automate the plan. I just assume it's a smaller company like in more consulting. Was there a reason why you made a move like that?

Yeah, actually that was a really hard decision and I think I took the most I can from my position there and just wanted to replay this for other companies and try to help them achieve the same high level of quality and automation they had. It was like one of the best processes I have seen and I'm trying to give that to more people.

Very cool. So as you're doing that, are there any concepts you find that most newbies or intermediate testers maybe get wrong about performance testing?

Yeah, there's something actually called the performance testing paradox, because when you start with automation testing, let's say you just pick the tool and start writing some scripts, stuff like that. But when a team really wants to start with performance optimizations and performance testing, I don't think that taking a performance testing tool right away and throwing some is the right first step. I can see how the regression testing suite that has to be in place is the most important thing before that because no one wants website that is very quick but is not working properly. So I can see that even over the few months we're working on similar project where the regression suite is taking key part in the performance optimizations through the company's work consulting. So this is I think something that gets often wrong. So let's go, let's go and throw some load on that website and see what happens. I think you should have like the whole team approach to aim for good performance and it should be the goal for everyone. So that makes it a bit harder. But when it's done right it can be done in the best possible way.

As you can tell, I'm a lot older, so when I started performance testing was always done like at the very end after the application was built, the infrastructure was in place. Have you seen a shift left or is that like just a standard now where people are incorporating performance testing earlier in the lifecycle than they did back in the day?

Yeah, I would say I was lucky to be involved in a process where that was a standard and I have never been in a place where like my word is not heard or someone just throws me some piece of code. Let's go and test that. We have always been involved early in the project and I think that's one of the main reasons of the success of the teams I was working in. This includes actually performance and performance testing. Like even on the first stages of planning we were considering that and thinking what needs to be done in order to achieve good performance and what needs to be done in order to test to its fullest the feature we are developing.

So along those lines, and I think you mentioned stress testing and load testing and the performance paradox you mentioned earlier. I think that's the first thing people think about performance testing. But nowadays I would think a lot of with the type of applications people are writing one user perspective on front end application like that's JavaScript heavy would come into play now. So when team focusing on performance do they automatically just go let's throw like slam it. With thousands of users and not even think of maybe a one user experience?

Yep, I think that's, that's really right. And during my training this is one of the main things I focus on and stress on is that we should first see how to optimize a single load on our web page and then think about bringing some more load and seeing how it performs then. And understanding the foundations of the technology you're working on is really key in this. Just to know how to reach the Performance and all these metrics we have available, how to use them the best possible way. So yeah, I think this is the other thing that's often taken mistakenly and this is something I'm trying to change with that training.

Yeah, what I love about the training is you cover a lot of open source tools. So once again, maybe it's just my generation, when I think of automation, I would assume people just think about like big open source or big paid solutions for performance testing. Are there any like it looks like there's a lot of simple tools available in a browser now that can help people with performance that they may not know about it. Am I correct with that?

Yeah, that's right. And we don't have to spend a huge budget on buying all those tools if we're just starting. Of course it's needed at some point, but the first thing we can do is just to explore what we have in our browser. It's a lot more than a browser nowadays. It has so many features and they are very useful really and help us, help us to analyze and act on in many cases and see what would be the result even before we apply the code. Stuff like that, which is very helpful in, in any testing, any performance as well.

So what are some of those tools? Is it just the Chrome dev tools or web debugging tools, things like that?

Yeah, yeah. DevTools has quite some functionalities like for example the local overrides you can do and you can apply some changes to fonts and CSS style sheets, resources that are downloaded which basically show you how the timeline of the loading will change. And that also is something that is funny because when we say we're doing performance optimization, everyone thinks about scaling up and bringing more resources over while just getting the fonts on the right, just with the markup of the page, getting all the images optimized. Stuff like this is also crucial to performance and to the user perception of the website.

Is it hard to convince people that they need to start there without them jumping to like let's just do a 100,000 user load test or something?

Yeah, soon they realize it with the good examples and good demos when they get their hands there. And I think they don't realize it's that simple. Like they think it should be something really complex involving like huge infrastructure, but it actually isn't great once they know.

About some simple tools that are built into the browser that can help them from a one user perspective. You did mention eventually they will need some sort of larger tool, some sort of commercial tool probably most likely. So is there anything that you recommend teams go through in order to select? Is there a process you use? Because I assume different teams have different needs. So any recommendations on how a team selects a tool?

I guess the question, yeah, really the question is what is important for the particular system, whether it's facing customers or not. This is really important into the direction you're going to take. But one of the tools I really like and really good to start with and to learn how to use it in all its functions is I think webpagetest. It's just something that is built around the real need of those features is not something for marketing or something else. Like all the different features you have, single point of error testing, those waterfalls that are drawn, the different metrics and the visual experience. This is all very useful for anyone to take a look and to track. So I really would recommend starting with that. It's not load testing, it's just capturing performance, but you can track that over time. And one other thing I like is that you can integrate it over with some dashboards and actually get this trend and at some point if there is an issue to drill down really to the bottom of the request and see what actually happened. So I think it's very useful in such cases.

Absolutely. So I love that tip. Also, I think with other open source tools there's two that always come to mind that I hear about. First one is JMeter and the second one lately I don't know for some reason is K6. So any suggestions on what tool fits, what type of team or certain applications that one may be better for than the other?

Yeah, actually I think JMeter is like kind of an industry standard to start with. At least it's been there. It's been there for a while, so I think anyone should get familiar with it. But I'm using K6 also as example because it's showing that not only quality assurance engineers that assumably can only click on buttons can, can write low performance tests, that this tool shows how any developer and anyone who knows how to script can use it and write meaningful tests. But still the key here is the perspective of a person who knows what's doing. So you need to set your goals first to know what you want to achieve with that tool and then to see how to achieve it with the concrete tool. So if you know this, if you are aware with your targets, you can do that with any tool out there. It's a matter of getting used to using it.

So let's talk about that then. I know in your course you go over a performance test plan and to me that sounds old school. Coming up, the performance test plan. So what would be in a performance test plan to have people know about and plan about before they actually jump to any type of tool?

Yeah, no one wants to write test plans and no one likes them, including me. It's just that this test plan, especially when I, over the last two years I'm teaching at the academy, so these things are, you know, the first things we teach people just to start with planning, it's good and it brings issues upfront and it gives you the possibility to ask questions early. So this is why I think it's important in performance testing also. And this is something that I realized in Telerik, that actually you can combine all the information in the company. You don't have to be the only person that is looking for information and finding it in, you know, IES logs or some data around the servers. We can go to the marketing team, we can go to the SEO team, we can ask them what data they have and to collect the requirements and expectations for our testing. And in performance testing this is crucial because what other way do we have to select the load with which to load test and what parts of the system we have to stress on? We cannot test everything and we don't have to. So the good test plan will give us those answers. What are we expecting? What are our most important parts of the system and what tools and what kind of testing do we need? And I'm including in that not only the performance and load testing tools, I'm including the monitoring of the web servers, I'm including also the real time monitoring of the users and basically tools like pingdom that give you notifications for functioning website or server. So all these combined gives us a good picture of the performance of our system. It's not just one tool and we'll do this and that. So with the whole, the whole spectrum of tools is what really matters. And if everything is combined and you can see the results at one place with good notifications, this is something that will work. Definitely.

Absolutely. So I actually started my career mostly doing a performance engineering performance testing. And one thing that tripped a lot of people up back then don't know if it's still an issue is the amount of load they put on a system. So I assume this would be addressed in the performance tests. In the performance test plan is transactions per second. Is that still an issue where people maybe just say, oh, let's slam a thousand users on it Doing things as quick as possible and then they say oh my gosh, the application crashed, we need to start over. Like it's a false alarm because they're putting transactions that would never happen in like any given timeframe. So any thoughts around that on how to create it more like a realistic performance test plan that really encapsulates what a real scenario would happen in production with your real users?

Yeah, and a really hard decision is nowadays is whether to include the third party requests that every website is making. If you just inspect the traffic of one website, especially the Martingale ones, you see requests like Google Tag Manager, which invokes tons of marketing tools for tracking and whatnot. So when we're recording traffic and designing our load test payload, we usually get rid of those so that we get only the load on the server. But, but actually the user is seeing those requests even though they are sync. They shouldn't block the loading or something. They still feel it, especially with if they have low bandwidth on their network. This is why I like this performance testing approach with WebpageTest because you can see all those, all that information and when we're throwing quote, we can still isolate and see how the server performs. But that's not all. We still need all that information. And also we should test what happens when some of those requests fail, what happens when Amazon servers fall and or we are getting just slow or inconsistent responses. And this is also something that should be considered important.

So that's a great point. That's something I didn't have to struggle with. Usually we own the code. Nowadays people are pulling down open, it's almost like they're creating a puzzle to create an application, all these different services to create a solution. So how do you handle that then? Do you do mocking? Is it like a service virtualization that you put into play? I don't even know how you would handle for a realistic performance test.

There's no good way to mock all those services. There are really too many. But what really helps is that the marketing team would know that everything they do affects the system and any, every, every new tool they add is going to affect the customers. So this like whole team approach, including all the different departments is very useful not to get too many of those dependent requests. But when we are testing for load, I don't see a good way to mock those. We just have to load our servers, see what they can handle and then see in the real life how they perform.

So that's a good point then. Do you do a lot of performance testing and production. Now back in the day that was kind of taboo. In order to be able to do that type of test, obviously you can't put on a huge stress or load. But is there any way you can actually do some sort of performance test in production that doesn't impact users, but does gate view that type of data?

Well, the performance tests you are doing nowadays are actually just taking a look at all the metrics you're collecting from the servers from Google Analytics or any other analytics tool to see how the users are, what response time they are getting. This is also performance testing like you are seeing actually what how the load is distributed across countries. You get all that information. So seeing that and acting depending on the results, I also consider that as performance testing because if you don't have eyes on production and what happens in real time, then you're in trouble. You have no idea how, what to do.

Yeah, you know, even I just forgot about it. I assume a lot of people forget about Google Analytics, right? Most pages have it, most web pages I can go in and see this page is performing poorly based on real activity that's going on in production. So I assume that's a good tip. People should be checking the Google Analytics for their web applications. At least I would assume.

Yep, many people don't know they have it. So this is my main point here. Whatever we have, we should use it especially for marketing. I know they have all kinds of tools. For example, they can draw a heat map on the page where the user focuses, what buttons they click, where they're looking. No idea how they do it. But this is very useful information for performance and load testing and we should take advantage of it, not just go blindly and decide that, you know, some of our product pages is the most important or the login. So all that data helps and it should be taken into consideration and iterate over frequently. We used to take from Google Analytics like the top 50 pages that were targeted and based on that information to change our tests frequently. So this is adapting to our users. For example, you acquire Feedware, you buy the tool the next day and you were so far you were testing the homepage only and the login and suddenly you get that huge traffic on those pages. So this is really challenging and you should adapt your testing.

So another thing I always found complicated was it was easy to create a script, it was hard to analyze the results. Now with all these other services, how do you any tips for analyzing the results? Like I used to have to go to a database person, a mainframe person, and get them all together and say, okay, what's going on? Can we pinpoint? This action's taking a long time. But like, how do I know where in that, that action is? The actual slowdown?

Yeah, this, this kind of debugging and drilling through the results is very hard, especially if you're not familiar with the domain. I'm always, always trying to do it myself and then asking for, for help from someone on the team, checking it together with them. And it's very hard actually, especially if you take a look at those. For example, look for memory leaks from on the brow on the client side code. This is a matter that requires lots of knowledge on how the browsers work, how the front end code works. And I think we're getting to the point where the quality assurance engineers, they start to understand all that and they have to, because this is the best way to addressing your testing properly.

So I know you're doing. I don't know. I think I saw something about a session for a conference for performance testing infrastructure, setting up the infrastructure. Now I think this would be a plus. Back like once again, I keep going back to back in the day, but we used to have to set up a staging environment and we had to spend a lot of money on servers and the raised like really hard metal in place servers to mimic production, which was always expensive and it was never realistic. We run a test in staging, we go to production, never behave the same way. I would think nowadays it'd be easier now with things like Docker and Kubernetes to create infrastructure. So when you talk about performance testing and infrastructure, what's that going to cover?

Yeah, actually this is a six hour workshop I'll be doing on the QA submit conference and it's basically the third day of the training combined with some of the theoretical part from the other days. So it will basically go through setting up the whole process of performance testing, which I mentioned, with all those tools and bringing them up into Docker on your local machine is a very good start for it. Just building a container and starting it over, seeing what we can do, including continuous execution of those tests and having those results into a single dashboard. This is what I'm trying to achieve there. And we can see that the building of the images and like all that technical stuff takes no longer than one or two hours in total to complete. If everything is fine, of course, but, but you know, knowing the tools and choosing why and which tool to use takes a little more time. So again, I'm not focusing that much on the tools themselves, on the setup, but on the way we use it together to achieve this full picture of performance testing process. And yeah, using Docker is helping quite a lot with that. And thanks God, everything now is open source and we can integrate it with all these cool tools that are available. Even though in the past we have done that like we were scaling servers, we were building virtual machines, thinking how to provision the code into each environment as part of the process. But as you said, it's not worth the effort and the money spent. I think just tracking and seeing the trend and analyzing the results is also something that works well and is much cheaper if you want to scale the load and to distribute it, yes, you have to spend some money, but still there are solutions that give you this option and you can just enable that.

Great. Okay Nick, before I go, is there one piece of actual advice you can give to someone to help them with their performance testing efforts and what's the best way to find or contact you?

Yeah, so I split that into two parts. One, if you're just starting with performance testing, I would say just do not jump straight into the commercial tools for load and performance testing. Do not throw this random load to your servers just because you want to find issues. Spend some time on to educate yourself how the system works, what the tools you have in hand, the web debugging tools, whether it's fidware or something else, the Chrome dev tools, Firefox, whatever you have, just see how we can use it before switching to all those other tools that simulate load. And for those who are looking for a tool to measure performance, I would say you should start small. Start from your machine iterate often. Look for the metrics that matter for your case, not the ones that everyone is using because everyone has a unique case. And we should consider that we should involve the whole team in the process and make sure that better performance is a goal for everyone in the team, not just us.

Awesome. And if people want to learn more about you or Automate the Planet, what can they do to get in touch?

They can head to the Automate the Planet website or find me on LinkedIn. I'll be happy to help anyone out there struggling with performance or just getting started.

Thanks again for your performance testing awesomeness. Links to everything of value we covered in this episode. Head on over to testguild.com P73 and while you're there, make sure to click on the Try them both Today link under the exclusive sponsor section to learn all about SmartBear's two awesome performance test tool solutions, Load Ninja and Load UiPro. And if the show has helped you in any way, why not rate and review it in itunes? Reviews really do matter in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Performance and Site Reliability Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack performance testing awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Performance and Site Reliability Podcast. Head on over to testguild.com for full show notes, amazing blog articles, and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.