Simplify Performance Engineering with Intelligence with Josef Mayrhofer

Foreign.

Get ready to discover some of the most actionable performance engineering, testing and site reliability device with some of the world's smartest engineers. Hey, I'm Joe Calantonio, host of the Test Guild and SRE podcast. And my goal is to help you succeed in creating application performance awesomeness.

Hey, it's Joe. And welcome to another episode of the Test Guild Performance and site Reliability podcast. And today we'll be talking with Joseph all about simplifying performance engineering with intelligence and much more. Joseph is the founder of Perfumetrics. He's worked in more than 20 years in the performance engineering and security industry, which is really cool. And together with his team, he is following a mission to make customers performance and security dreams come true. He has a lot, a lot of deep knowledges in this era. So if you want to learn more about about performance and how you can make your performance testing better, you want to listen all the way to the end. Check it out.

This episode is brought to you by the awesome folks at SmartBear. Listen. We know load testing is tough but necessary. So investing in the right tools to automate tests, identify bottlenecks and resolve issues fast saves your organization both time and money. And that's why SmartBear created Load Ninja, a SaaS load testing tool to help teams get full visibility into performance so you can release quality software that faster than ever, make testing effortless and give it a shot. It's free and easy to try. Head on over to loadninja.com and learn more. Hey Joseph. Welcome to the Guild.

Thank you so much, Joey, for having me. It's my pleasure.

Awesome to have you. I guess before we get into it, I think I really butchered your bio. So is there anything that you want the Guild to know more about that I left out? Yeah.

So actually it's important that we all share the knowledge, I believe so. You're doing an amazing job in this area. So I really appreciate your show and yeah, happy to be there today. Talk about performance engineering, AI powered performance analysis and all that stuff.

Absolutely. Love it, love it. And speaking about sharing the knowledge, that's one of the reasons why I came across you. You have a really great website and on the website you have a lot of resources. One of them being it was like calculators about different. Like it's like a performance toolbox basically. So I thought we'd dive in there and then maybe talk a little bit about other things based on that. So I guess what is the toolbox if people haven't seen this?

Yeah, it's Actually open source. So we created this as part of our job. It's always the challenge at the customer side projects. We were asked, oh, how many concurrent users do we need to simulate? And you can definitely easily calculate it, use a spreadsheet. But after being asked these questions a few thousands of times, we build a nice open source toolbox to calculate the session time, how many concurrent users you use and all that stuff. We use little slow, quite easy, straightforward, simple math, but very useful. And we see a lot of performance engineers around the world using this open source toolbox and yeah, it's a pleasure to provide it to the entire community.

Very nice. And right now I think there's like six options, but I'm sure you'll be adding to it. The first one is throughput, the second one is concurrent users, the third is session time. You have one on impact of improvements and you have another one. I think an expected availability and allowed downtime. So out of those six is there one that's more popular than another?

So definitely find out. The concurrent users is a big topic for many customers, many performance engineers and also what is the actual performance or maybe impact if you have a downtime, what is the business impact? How many dollars or euros or Swiss francs are you going to lose if you're running performance issues? That's very important. It's all about awareness these days. I believe as soon as business teams, decision makers realize the big dollar value of this slowness downtimes, they are happy to invest in performance engineering. I believe it's big eye opener for them as well.

Yeah, I mean always hard for some reason to get teams to think about performance. But I think having this one in there to show them, hey look, based on improvements to performance, you're going to get X amount of return on investment is pretty cool metric for sure. What's allowed downtime? I'm not sure. Is that for SREs?

Yeah, it's more for SRES because sometimes you have these SLAs. Maybe you need to work hand in hand with the business teams to specify proper SLAs, KPIs and then you can figure out let's say they want to have five nines or four nines or three nines. You can quickly calculate how many minutes or hours is your service allowed to be down in order to fulfill your KPI or high availability requirements. So it's quite handy. We are using it sometimes in our projects and we decided let's share it this entire community and we see also other engineers are picking it up, exploring IT and using it in their project.

Nice. So you know, I started in performance over 20 years ago and back then it was a lot of people didn't take it as seriously and it seems almost like there was like it went down, people kind of ignored it. Now it almost seems the past year or two, maybe just because I speak to a lot of people, it's back up again in popularity and it's almost like it's getting morphed with site reliability engineering. So I'm just curious, you run a company dedicated to performance, what are you seeing? What are some trends? Why maybe is there seem like an uptick in people have an interest in performance testing?

Yeah, it's funny, I noticed the same. So from my perspective there are more, more complex technology layer. We have Kubernetes, Docker, big movements, lift and shift to the cl. This is bringing additional complexity. So customers are more curious about how they can save some money, how they can optimize. I believe another trend is these days many customers realize that they can no longer simply add more hardware to optimize their systems. They realized we need to validate the systems, we need to simulate load, we need to go back to the design and code level. And this is a big trend. They realize hardware based tuning, it's no longer doable. So they need to simulate the load in their test environments. And we are doing that.

Yeah, once again I know one of the answers were always like, oh, we'll just add more hardware. Hardware is cheap, so like problem solved. So what do you say to someone that's older like me, that maybe hasn't done performance engineering in a while and now people are going cloud, native containerization. Would someone need to know what skills would they need to actually be successful in that type of world with that type of performance?

Yeah, it's definitely you need to. This is maybe also a big educational problem because it takes a while that you pick up all these different technologies. When we educate in our juniors and share all the stuff they need to know, we need to give them a quick overview about all the different layers, not only programming, but also network communication, all the different protocols, the challenges in Kubernetes databases. You need to be really a kind of full stack engineer these days to optimize and tune systems for speed. And it's more and more getting complex these days because we have now the cloud, we have Kubernetes, we have Docker each and every year, almost an additional layer. When I look back 20 years when I started as a performance engineer, it was easy. Most of the time. One application server, one database, tune it for speed, job done. But these days you have hundreds and thousands of servers, so it's a quite different challenge. They all need to work in a perfect meta to bring up a very good performance for a single application. So complexity is increasing. Definitely.

No great point. Yeah. I used to have like a web server and a database. Ooh. But you know, that's, you know, you'd be able to test it and you were good to go. But so, you know, talking about, you said you have to almost be like a full stack engineer right now for performance. You know, when I look at your resources, you have a lot of white papers and they touch pretty much every stack here. So I noticed database tuning that that was actually one of places where we always would find issues where maybe a SQL statement or something would go haywire because someone could just do like dynamic SQL statements and it would just drive everything nuts. Is that still an issue? I know what they have these other databases nowadays, not necessarily like a SQL database, but there are more. I don't know what the name is, it's slipping me. But database have evolved as well. So are there any new tuning techniques that you need to worry about as a performance engineer when you're dealing with databases?

Yeah, absolutely. The challenge always first of all have a meaningful comparable data lake compared to your production environment. There's a lot especially we do business for banks, data anonymization going on whenever you scramble your data. Is this still comparable to your production? Sometimes we need to deal with these questions as well. Based on our experience, you need to have good transaction tracing tool. There are tools available also for databases so that you can exactly see what SQL statements are getting slower and how to optimize them. Because without that for sure, Oracle DB2 and all these databases have some stuff built in. But sometimes if you simulate millions of transactions, it can be quite Ms. And challenging to figure out what execution plan and what SQL statement is slow and why. What is the particular root cause in that? To me, 20 years ago, a big proportion of the tuning was done in the database layer. These days we have a lot of layers on top of it. Kubernetes, the Java layer or maybe Docker as well. The cloud is absolutely also in multi cloud, different network zone segments can bring additional overhead. So I believe a big change in the last 20 years in the performance engineering space. And we need to learn and share the knowledge with the entire community because it's really hard from my perspective as a junior engineer to learn all these skills Take it up and then become a successful senior performance advisor. But it's important as well at the same site. So the demand is increasing. It's more and more difficult to learn all the knowledge needed to fulfill the customer needs.

Absolutely. It's always been a weird niche to even know someone that knows any performance testing was difficult. Now it seems like you just mentioned these other skill sets on top of it. So you know, a lot of teams are trying to push performance just like everything left in their software development life cycle. And developers, I mean they now they have performance on their definition of done and like how much performance can they really know? What have you seen work? Obviously you work for, you know, your company, does performance. Do a lot of people reach out to that to train their teams or is it, do they have a team like yourself that handles performance or what are the trends you've been seeing with that? Is it someone embedded in a sprint team? How does that work?

So we see actually a big mix. There are definitely microservice teams, lift and shift teams, transforming stuff to the cloud. They try to take care of everything. But even in such complex multi cloud projects, when they build microservices there comes usually a point where they need to do some use acceptance testing and end to end load and performance testing. So usually the developers are not taking care of end to end performance testing. They engage a separate testing performance engineering team like Perfumetrics and we assist them. Sometimes we reuse the existing scripts, sometimes we develop new end to end scripts to make sure that there's a proper final performance validation in place before the code is shift or move to production.

So I just want to follow up on, we were talking about databases, you mentioned security and in your bio you actually have security. So how much is that now involved in performance? Is it, is that a bigger piece than maybe we've seen in the past? How much do teams need to be involved when they're doing performance? That they're making sure that they're following all the laws, they're not using real data for performance tests?

Yeah, that's a good question. So actually it was more happening by mistake to me. I was a couple of years responsible for application security in a major bank in Switzerland and this is where I learned all the application specific security tricks. The reason why we are combining it, we performance engineers, we need to have full stack mind very often we need to investigate the code. And in that area we also come to know some security vulnerabilities these days. Also tracing tools, they support security checks as well. So it's more a win win. We assist security teams more bringing the application security focus, bring security to the code. And this is where I believe a performance engineer can also provide value if he has some security mindset or knowledge as well.

Absolutely. So I don't know why I keep going to the past. It feels like maybe because it's the holidays. I feel like the ghost of performance past. But I was just thinking also about tooling. Back in my day we had an awesome enterprise solution. It was able to handle everything. The monitoring, it could handle putting the load on the application. It had a dashboard, it had everything in it. Nowadays from what I've seen there's solutions out there, but it seems almost like anything with open source you need to mix and match to create a solution. What are you seeing and what tools seem to be popular nowadays? Do you see people struggling with finding what tools to even use now to do this type of testing?

Yeah, you definitely need to bring a lot of flexibility. Usually number one investigate what the customer has already installed using. Maybe they have some open source toolings like JMeter, Gatlink. We see big movements to Gatlinks especially in the finance service industry where they need to simulate massive data volumes. So this is one of the outstanding I believe trends. They are moving away sometimes for their commercial load testing tools going more to the open source toolings. Customers try to save some dollars for sure when it comes to load testing tools, there's the bigger source movement away. I believe it's eventually much easier in the load testing area use open source tools compared to the tracing area. When you have some APM observability use cases and you need to collect and trace all these millions of transactions and use AI to figure out what is going on in the load testing area. You could even use if you're a developer, reuse your junit scripts or whatever and then build on top of that meaningful load and performance tests. There's still some edge cases where you need to simulate load on maybe some Citrix environment, some SHP where you need to the powerful professional load testing solutions. But there's definitely a big trend. I would say more than 50% of our projects are fully on open source load testing tools these days.

How do you step to speed on all the things? Because I know like there's a lot of solutions out there like key six it's more developer centric locus. Do you try to say well let's stick to JMeter or something that's like a open source standard. I hate and I'm not gonna say hate JMeter but a lot of people aren't fans of JMeter. So is there any recommendations or rules of thumb you give them?

I prefer tools where I can touch the code. So also performance is code. I need scripts. I had made some mixed experience when I don't see the code and I can only do some wizard based scripting. This is not my kind of thing but yeah you need to learn all of these load testing tools on your full stack Engineer I believe it's absolutely doable to but everyone has his preferences for sure. If I see our juniors they prefer more scriptless coding but as they get more experienced they move to the script based load testing tool like Gatling where they can write complex scripts, powerful reusable scripts. I believe they learn it step by step and when they have the needed seniority they use the powerful script based load testing solutions.

Absolutely. I definitely agree. How about user experience Back in the day once again it used to be just all about load testing, stress testing. Now it seems a little more nuanced, a little more like how's it for one user and how's it look and how's the user experience with performance? How. How quick is the transaction? Is that more of a thing now that you have to engage with?

Yeah, especially when there is no load and performance testing solution in place. We usually start using some open source tooling there create browser plugins where you can check already single user performance. There are big rich client applications these days, single page applications where you can do a lot of use the browser Chrome developer tools and check what is the end to end user experience. Because a big proportion of these days is now spent on the browser layer. There's also a trend use the full browser, generate load on the full browser. We had mixed experience in that area. So there's a challenge regarding number of load injectors you need to provide because these virtual users are very heavy. So usually best practice from our side is use open source solution, generate the big load on the protocol level and maybe use something like Selenium or some full browser based solution on a single user base just to have also on top of it the end to end response time available.

Right. So not so different. Back in the day I used to use LoadRunner to put the load on it and I'd use a WinRunner script to check the front end. But now obviously it'd be in a loop so it wouldn't be accurate. But it was just for sanity's sake right?

This is the idea that's awesome. This still Works very good.

Still works. Love it. So you know you did mention AI. I always thought AI or machine learning would be perfect for performance testing or performance engineering because you have all this data and that's what's really made for where it seems like it's really been pushed more towards the functional testing. So can you talk a little bit? Do you think AI can help with performance engineering and maybe we'll talk a little bit more. I think you have a solution that actually implements AI and how that helps.

Yeah, it's a big trend I believe definitely we should use advanced toolings to simplify our job. This is the key message here. We see it already working in a perfect fashion in the APM area, observability example. Dynatrace has an awesome platform in that area I believe also it could be a big help for performance engineers who use it day by day investigating the tracing. But also when you need to share the knowledge and you need to learn what are the best practices. For sure you can go to Stack Overflow and whatever Google engine you wish and you can try to find something meaningful. But at the moment we are working on a performance engineering maturity model and we try to centralize a lot of knowledge best practices in there. A customer can do kind of self assessment, answer some key questions and then we will use AI to generate a kind of remediation plan. Actually we're trying to put the performance engineering knowledge into action because we see it's so difficult to speed up and to get this knowledge. Wouldn't it be much easier to use AI and share this knowledge that is available on the market with everyone who is interested in performance engineering?

Oh, all right. So it's not what I thought it was. It's. I love it's gonna benchmark. So using AI to create a benchmark to help you with performance, that's interesting, right? Have you learned anything from this? A lot of times when people create a solution and then it goes in the wild, you're like I didn't know people are going to use it like that. Or wow, that really works to do this.

So we created this performance engineering maturity model a long time back because we realized when we start initial conversation with the customer, we need to have some meaningful information how they are doing compared to the competitors. For instance, a major bank, they are doing almost nothing in the performance engineering area. The first question will be, oh, what is my. What are my peers doing? Why do I need to keep up? And we focus more on the build test and the production domain. We created meaningful Practices for each and every domain. And we can exactly, after a short assessment, five minutes, give performance engineering benchmark rating and show them how they are doing and use AI to generate a remediation plan for these guys. And it's not only our knowledge, we collect knowledge using share advice and credit back features. So even you as a performance engineering expert, you can go to our platform, give us your recommendations in certain areas, we would collect it. And whenever we generate a remediation plan going forward, if we use your advice, you will get some credits. You can kind of earn some money on our platform as well if you wish.

That's cool. Almost. You're probably collecting these over time. So you probably have a database of really interesting stats or something you could pull up. Is anything that pops up like most clients have issues with blah blah blah or are there any commonalities with that you see like over time like oh wait a minute, I never noticed that correlation before.

So the big challenge for many customers is still shift left. They do performance engineering by far too late in the life cycle. Spend a lot of money fixing these issues. And this is one of the biggest problems, production as well. This close the loop, give feedback to your development teams as well. In many companies this is not implemented at all. So there's a big. Maybe the siloed mindset is still there, big separations, DevOps, not really implemented in a good fashion. And then you will lose a lot because if you do not continuously check some production, give the feedback to your development teams, they cannot learn. Your technical depth will pile up and you run in more and more issues. So this is one of the two big challenges out there, I believe.

Yeah, absolutely. Struggling with shift left CI cd. It looks like you also I believe have a solution this year. A cyborg or is this a solution? I'm learning on the spot. So I just pulled up the site and I see CICD in cyborg. So what is this?

The cyborg stuff is more something where we need to do a full browser based end to end user experience. So let's say we use Gatlink, simulate 10,000 concurrent user on the protocol level and then we want to have a single user simulating the same use case, measuring the end to end response times. So we have created a cyborg based on Selenium. It's more a keyword driven approach where you can easily create a test case keyword driven without any program knowledges. And you can like in Loadrunner bring in some measure start measurestop commands to measure actually certain actions and you can run it in a loop, let's say one test case 500 times during one hour to see what is the end to end response times while you are executing a load test. So it was a long time a vision from us to reduce the scripting effort. If you look to your technical domains you have security tester, automation tester, performance tester, they're all writing somehow scripts and maybe we'll implement this in the future. But our vision was also cut down scripting effort. Have one kind of script framework that could help you in all the areas security, automation, performance as well.

No, it looks really cool and I love how it does incorporate security. Like you mentioned fuzzing OWASP a lot of things. So I think it's much needed. So this is really cool. So justify. I believe you're well known in performance but especially with finance and I know there's things like fintech coming out and like it's like the hot space. So is there anything that you see in finance that makes it more difficult or more demanding for a performance engineer?

Yeah, finance is a big topic. I believe there are standardized core banking systems. You need to optimize them for speed. And there's also one trend that me and you as normal customers, we no longer go to the banks. We do everything online. So we expect fast mobile applications, mobile banking applications. So banks realizing that they are fighting for this digital banking customer and they know they need to improve the end to end user experience response time. So I expect definitely more performance tests optimizations in that area as well. The other trend from a banking perspective, banks are lifting and modernizing. There are systems they want to get rid of mainframe. They introduce kind of standardized core banking systems and tuning these Systems like Taenos D24 is a challenge. You need to do a deep dive to get rid of the performance problems in there and but banks can definitely save a lot of money if they modernize their platform because the doors wide open for future improvements. And yeah from a performance engineering space, definitely an important topic to make sure that your engine in the background is running reliable and yeah, in a good fashion.

Well yeah, you know once again my past I worked for an insurance company and they had mainframes. People still probably surprised they still have mainframes but you know we always had to worry about mips and like using too many mips and that would cost a lot of money. So that's interesting how a lot of people trying to move away from that. Very cool. Okay Joseph, before we go, is there one piece of actionable advice you can give to someone to help them with their performance testing efforts and what's the best way to find or contact you?

Absolutely. So I'm happy to help everyone. Please come on our website or gobenchmark.com we try to share the knowledge with everyone. I believe knowledge sharing is really key, especially in our performance engineering domain. Talk about what works, what does not work for you. Use our platform and to me one of the best advices would be that you keep up learning new things. Don't get stuck in your current activity. There's a lot of fantastic stuff to explore out there and learning new things is really key these days. Try it out all this kubernetes cloud stuff, shift left CI CD stuff. You can pick it up and yeah it's exciting area in the performance engineering space and yeah I would be happy if you reach out to me anytime.

Thanks again for your performance testing awesomeness Links to everything of value we covered in this episode. Head on over to testguild.com P106 and while you're there, make sure to click on the Try them both Today link under the Exclusive Sponsor section to learn all about SmartBear's two awesome performance test tool solutions, Load Ninja and Load UI Pro. And if the show has helped you in any way, why not rate and review it in itunes? Reviews really do matter in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Performance and Site Reliability Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack performance testing awesomeness. As always, test everything and keep the good. Cheers. Hey, thanks again for listening. If you're not already part of our awesome community of 27,000 of the smartest testers, DevOps and automation professionals in the world, we'd love to have you join the fam@testguild.com and if you're in the DevOps automation software testing space or you're a test tool provider and want to offer real world value that can improve the skills or solve a problem for the Guild community, I'd love to hear from you. Head on over to test guild.info and let's make it happen.