Agile, Automated, Advanced: The New Age of Performance Testing with Dylan van Iersel

Foreign.

Get ready to discover the most actionable end to end automation advice for some of the smartest testers on the planet. Hey, I'm Joe Calantonio, host of the Test Guild Automation Podcast and my goal is to help you succeed with creating automation awesomeness.

Hey it's Joe and welcome to another episode of the Test Guild Automation Podcast and today you're in for a special treat because I'm going to speak with Dylan all about what I think is a super important topic and that is next level performance engineering and why it's so important. So you don't want to miss this episode if you don't know Dylan is an experienced IT consultant with nearly 25 years in the field, specializing in agile consulting, architecture consulting, general IT and project management. He's a pioneer in implementing SCRUM and agile practices and he's led initiatives in continuous integration, deployment and automated QA since the early 2000s. So he really knows his stuff. Dylan also focuses on automated testing and equipping teams with the most effective tools and methods that really come together. And him co founding of Profana, which seems like a really cool tech that I think you want to know about. He founded in 2019. It's a continuous performance engineering solution. If you don't know what that is, if you don't know why you need a solution like a continuous performance engineering solution, you want to stay all the way to the end. You don't want to miss it. Check it out. Are you ready to elevate your software's performance? Before we jump into today's awesome conversation with Dylan, let's take a moment to spotlight Profana, which is a trailblazing tool in performance engineering. If you've ever found performance testing to be a daunting task, you're not alone. Believe me, I started off as a performance engineer over 25 years ago and it was really difficult. But now tools like Grafana are here to change the game by automating the whole process and they integrate seamlessly into your CICD pipeline and serve as your quality gatekeepers. As dashboards, detailed traces and continuous profiling, Grafana streamlines your testing ensures that you're always in the know with automated result analysis. It distills complex data into actual performance insights without being a data science expert. So as we gear up our discussion with Dylan, one of the co founders of Profana, consider the impacts of proficient performance testing on your daily releases. Make your application more performant. Now check out the free trial offer for test skill listeners@testguild.com perftrial or in the link in the comment down below. Hey Dylan. Welcome to the guild.

Hey Joe. Nice to be here. Thank you.

Awesome to have you. I guess before we get into it, is there anything in your bio I may have missed that you want the guild to know more about?

No, I don't think so. I think you pretty much covered it all. Yeah, like you said, I'm very much focused always on developer productivity and providing developers with the right tools to be, you know, able to focus on the good stuff instead of the mundane and repetitive stuff, so to speak.

All right, so I guess that leads me to is that one of the reasons I always love speaking to founders, why they invested all their time and energy into us solution. You selected performance engineering. So I guess why performance engineering?

Well, it stems from one of my earlier jobs that I did where we used to have like really big performance issues. And also we were developing software with a large number of teams. So you can imagine that testing performance in such a large scale operation is pretty intense. And then I came across Daniel, who's my co founder and he has developed a solution exactly for this problem which I thought was really super interesting and it worked well. So we decided to turn it commercial.

Great. Yeah, I'd love to dive in. I guess the first thing we should probably tackle is there's a lot of times with performance testing, for some reason terminology seems to throw people off. So, you know, I think you mentioned a lot of things that I see online around performance engineering. So I'm just curious to get your take on performance what you see as performance engineering versus performance testing.

Yeah, that's a good question. And that's exactly what we have been ourselves struggling with quite a bit because initially we positioned ourselves as a performance testing software company. But actually, you know, it's a bit like you can test your system for performance and just say you pass the check marks. Does it actually handle the required load? Right. We think that leaves a lot on the table to be further investigated and to be further optimized, which has to do with what are the metrics that the actual system on the test is experiencing. So while you're doing your load test, so imagine like a real life situation because in the end we make software for, you know, production for live customers and your application can be performing really well. I mean, all your customers are of course are delighted with software. However, the way that you are able to achieve that is by scaling away out of problems. So that means that there's still an issue or multiple issues in your software, even though you are actually able to manage those issues by just simply scaling out. So that's the difference. So the difference between performance testing and performance engineering is in the first, you personally, I say you just simply check, does the system handle the load? Does it not throw like hundreds of errors? Does it keep running? Does it stay alive? The second one is actually diving deep into the system and further optimizing resource usage, scalability and stability issues, resilience, that kind of stuff.

Absolutely. So I guess, you know, when you talk about scaling, I guess nowadays people say, oh well, just scale. We have a containerized environment will just scale up more and more infrastructure to handle. So we'll have no, no performance impact whatsoever. So obviously I know that's not true. You probably know that's not true. So can you tell a little bit about how can you scale performance testing and why does it matter for optimization, especially as we live now in a cloud, native containerized world?

Well, it's not so much. Well, so there are two aspects to this. So one is scaling your way out of issues, but the other one is scaling the performance testing. So if you have like a modern, let's say E commerce setup with multiple, in modern architectures there are multiple microservices, databases, front ends, you name it, mobile applications. So there are a lot of components that contribute to the overall, let's say user experience and also to the cost. So the traditional way of doing force test is we release once a year. So we do a performance test once a year. The modern way is of course, yeah, but we don't release once a year. My microservice releases two times a week and the microservice of the team next to me is also releasing two times a week and the front end is releasing like three times a month and so on. So in that, let's say, I can't think of the word, but that high speed environment, that this agile environment performance testing really becomes an issue. It's really, you can't test all those systems together at once anymore. You need to do isolated regression testing. And so it becomes actually, so the testing itself becomes an issue of scale. How do you test applications, like dozens of applications or hundreds of application across tens of teams or hundreds of teams and still manage to go to production like every day or every week or so. That's why we think this is all about automation, this is all about automating, not just executing the test because that is, to be honest, rather easy. But the real work is actually in collecting and analyzing the data that comes from that load Test. So if you execute a load test or a performance test, like for across this ecosystem, like 10 every day, then collecting the data and analyzing that data becomes a real, real hassle.

All right, so is that what you mean by next level then? I know a lot of you talk about next level, taking it to the next level. How do you even. It's so hard for me, I always said writing a performance test is probably the easiest thing, but now we have all these other components and all these other dependencies. How do you even know how to create a performance test? Is the first thing. The second thing is the results were always hard to tell to Curly. Oh, this is because of this and that. So if you're in multiple agile teams and they don't necessarily know what the other team's working on, but these actually have dependencies on one another, how does that all work when it comes to performance? Doesn't it make it even harder to do?

In a way, yes, of course. And in another way that. So this is actually the reason we promote heavily promote, I still believe heavily in shift left. Right. So you want to get the feedback as early as possible. You want to know right away if you've introduced a performance issue. So in order to do that for just your team and just your, let's say, microservice, you have to, of course, design your test in a way that it's kind of isolated. And this also allows you to do it repeatedly, over and over again. So automation and small scale allow you to do it repeatedly, fast feedback, etc. There comes, of course, a point where they need to. So the way to do it is, for example, to set up stubs or mocking, so that you can simulate the external components external to your microservice. And there is, of course, another stage, if you will, where you combine a number of those microservices, you introduce, let's say, the database into it, so it becomes a mix of a number of components and you test those together as well, and you collect all that data and you put SLOs on those and you just simply validate the results and look for ways to optimize software.

Okay, so who's responsible for this? You have to have like a data scientist that actually looks at all this data and then knows, okay, this is because this and that, and make hypothesis to see if it's true. How does that all work?

No, that's actually what we do for the teams because we also believe. So developer experience is a really big topic. Right. So in order for developers to be productive on the topic that they like to be productive on, which is creative code development of code, you don't have the inclination or the desire to learn data science. So that's why actually you need automation. You need smart solutions that know how to collect that data, analyze that data, draw conclusions from that data so that the teams themselves, and possibly the occasional performance engineer that is supporting those teams, actually knows just works from the conclusions forward. Right. No data science involved. That's all in software. All right. We believe that should be in the software. So what Pavana does, for example, we provide common solutions for common stacks with known, let's say, patterns of, for the data that indicate or point towards possible errors or discrepancies. And our analysis software just executes the analysis and gives that feedback.

All right, so I assume a lot of teams you work with already have a lot of performance tests, but they're probably not very structured. So how does profana come in? Is it profana like a wrapper on top of it that gives them like a framework that they can then implement and point to and then Prafana then analyzes all the different inputs and creates outputs with some insights based on it?

Yeah, that is pretty much it. We work with a number of open source load test tools. So you can write your test in JMeter or Gatling or K6, that's all fine. And when you execute the test, when you send the metrics from both the system under test as well as the Lotus software itself, if you send that to for example, Influx or Prometheus, then Prana picks it up and analyzes it automatically. All that's required is to, in your, let's say when you start executing script and when you run the system and the test, that there are some tags put on it on the metrics data so that we can collect and automatically create the dashboards and the analysis for you.

How do you typically see people pushing performance testing left? Like, do you see them doing like 10,000 users? Is it more just like, how does it perform when it's interacting with this other feature? Is it like, how does that work? Like, how do you get. In order for people to be successful performance, like you mentioned, you need to shift to left. But a lot of people have problems even doing performance testing. So how, how do you recommend they shift to left so that they could benefit from a tool like Prafana? Do you have any good practices that you've seen to make it really work well for development teams?

Oh, that's really a good question. The way that we usually work is that we have an onboarding program which works really well. So when a team has a desire to start load testing, there's a little effort involved in, you know, setting the SLOs, writing the test script and so on, integrating it into the CICD pipeline, and that's basically it. From then onwards it's executed well whenever you want on every commit or every merge to a certain branch or whatever. And then that load test is executed which drives the feedback.

Maybe a weird question. Do the teams know they need this a solution or is it usually a tester that tells the development teams, hey, you know, application needs to be performant, probably need these tools in place because especially as we go towards microservices. How does that work?

That differs, of course, greatly. Yep, we've seen all kinds. It's not uncommon that people experience some drastic performance issues in production and then start looking for solutions, then start actually to become aware of the importance of performance or become aware that they have a problem. So that's one end. It's also very common that the more seasoned developers are very much already performance aware, then still very often it's considered very labor intensive, so difficult to set up in the regular sense, difficult to maintain it in the regular sense. So yeah, it's driven, it's usually driven either from the top down or from the teams themselves, depending on what kind of experience there is in those teams.

All right, so you're probably sick of hearing this, so AI.

AI, AI, AI, AI. But this is one area where I always said this makes sense to have AI, specifically machine learning, because you have all this data and I would think that's what machine learning was built for. So is that something profana embraces or something that you all use or utilize?

Yeah, we're definitely looking into it at the moment. We're experimenting with. Now in these days, you would say they are traditional data science methods. So we're experimenting with data science methods to automatically detect anomalies in the data. Because right now what people need to do is they need to determine their service level objectives for the software themselves. And if you don't know what you're looking for, then there's also nothing that is reported to you. Right. So the easy way to get started is to set a SLO on CPU usage or the number of requests or garbage collection durations or things like that. But you might be missing a whole lot of signals that are deviating from the norm and those, you might be missing those, and those might indicate an issue. So we're developing a feature that Automatically tunes detects these patterns and automatically detects large deviations from those patterns. That's more a traditional data science approach. So next step is of course to indeed employ more like machine learning approaches to actually pattern recognition and which might drive root cause analysis.

All right, so I also been reading a lot about Profana. I'm always, I love, I came across this probably in a news article and I said, oh, this seems like a cool solution. And I love how you have written everywhere software performance equals business performance. And I always find that a lot of times it's hard for developers or teams that I've spoken with to see the value of performance or be able to explain to the business why it's important. So can you explain a little bit more what you mean? Because I think this is really a really good killer analogy. Why does software performance equal business performance?

Yeah, well, I'm always also a little bit surprised that for me it's very evident and also something that I've known for a long time. But you know how for example Amazon and Google have done research on the value of performance. So one big conclusion from that is that value that performance, better performance actually directly impacts customer satisfaction and conversion. And, and that's of course very much, let's say everybody gets that right. And everybody also gets, I think that having very bad performance, nobody wants to go there, but having really bad performance means your site might not function at all. I think there was an incident with some ticket sales a couple of months back. It completely burned down the site. Yeah, that's a bit of a, that was a bit of a shocker. But there's a third aspect, and that's what I touched on earlier a little bit is that performance also directly impacts the cost that you're making of keeping those systems running. If your software is not performing adequately, you might be using two times the amount of resources. That's actually that would have been necessary if you had optimized the software. So that's to avoid unexpected high charges or unexpected, you know, low customer satisfaction. Performance is very, very important. Yeah.

So I think another barrier, maybe it's not the same because you mentioned a lot of developers you speak to are performant aware. Is that a lot of people when they hear performance testing they think, oh, it's like a dark arts, like they have no idea where to start. This is too much effort, I don't know how to do it. And it sounds like to me like a solution like Profana would make it almost more accessible for people. Is that something you would agree with.

Of course, yeah, absolutely. Because like I said, automation is the key and it encodes literally the knowledge of a performance engineer, how to deal with the metrics coming from both the system and the test, as well as the load test. So that's a whole lot of knowledge and experience and effort that is put into software and that makes allows the developers and the teams to just focus on developing software. Of course, if you have to be somewhat performance aware while developing it, and also if you by error introduce some performance defect, you of course have to go back to coding it, debugging the software, eliminating that performance bottleneck. But the system is in place to continuously guard you from introducing these very costly mistakes, potentially costly mistakes into production software.

How does it guard you? Does it bubble up best practices or make suggestions to help, you know, like how to do what you're trying to implement or if it sees something you're doing that's kind of wonky, does it say, hey, this doesn't look right?

One of the key aspects is of course, when you integrate it into your CI CD pipeline, of course you can use it as a quality gate, right? So the software tests the performance. If the performance has a large deviation from an earlier run or from a well known baseline run that you've put in production, let's say three days ago, then the software is flagged, the build is broken, you need to go back and look and dive into the details. So that's what PFANA then provides you. It provides you with the dashboards and the details of that particular test run as compared to the baseline and the previous run. And it also allows you to, for example, dive into the traces and the continuous profiling that is recorded during the performance test. So it gives you access, immediate access to all that data that is collected during the performance test, how it compared to the previous run, how it compared to the baseline run. And it really allows you to quickly dive deep into what the potential issue is.

So you're not just throwing up a dashboard with a bunch of information. It also sounds like you keep mentioning automated test run result analysis. So it sounds like it automatically does the analysis for you as well. Someone logs into, they automatically see maybe some key things to start looking into to make it easier to debug.

Well, in what, let's say the upcoming feature, any metric that has a deviation from some kind of baseline that was automatically created is flagged. But up until now, currently you are responsible for setting the service level objectives, which we can help with. So we have golden path profiles. Let's Say paved road profiles. If you have a spring boot stack, for example, with PostgreSQL database, then we provide some custom we ready to go dashboards ready to go SLOs and the system will detect deviations from on those SLOs. Right? Yeah. And in, in the near future we go a lot further than that. But that's not yet.

So it sounds like when someone installs it then if they, they you have some common templates, maybe that of standards. And once you have that, it's like our here the SLOs we find are going to be more beneficial for you. So it sets it up automatically for you in that way.

Yeah, exactly. Yeah.

Nice. So I don't know if you. How much you can reveal. You've revealed a little bit so far. What else? Is there anything else on your roadmap or where do you see the future of performance engineering and how you could see the needs of. What are the needs that are going to be, need to be filled, especially as we develop quicker and faster, more often even than what we're doing now?

Yeah, well, I think so. There's of course, let's say the AI route. So we came into existence to support development teams as much as we can by automating everything that is that you need to know about performance testing and performance engineering. And there's still lots and lots of steps that you can take there. So improvements in root cause analysis using AI, for example, to explain, let's say, the situation that you have uncovered. And also another big piece that's also so adoption by the teams. So you can make performance engineering as easy as you can. But indeed, if no one in the entire company is actually performance testing, there's nothing to check, there's nothing to analyze. Right. So adoption needs to be also facilitated, which also means that you have to have more insights on how you as a team are performing, how you are performing against other teams, how the company as a whole is performing. So it's performance on performance.

Nice. Yeah. So I wonder if you can not pit teams against teams, but you can see which one is performing better than another app with insights like, hey, this one's always checking in code and performance a little off. Maybe we need to give more training, like not to penalize them, but just to see opportunities, let's say.

Oh, so badges.

Yeah, right.

For example.

Yeah, right, right.

Those are absolutely topics that are on the roadmap.

Oh, very cool. Love that. All right. So you know, I always recommend people get their hands dirty and I think performance testing is needed and I think a Solution like this is very helpful. Do you have like a free trial someone can go to and actually just implement and say, hey, let's check this out and see how it works for us?

Absolutely. We have a profana starter that allows you to get up and running in like 15 minutes. And it's just, you know, a simple load test on a example system under test, but it allows you to see how it works, how it integrates and what it can do for you.

Love it. And we'll have a link to that in the show notes as well. And I highly recommend everyone check it out. Profana. And we will have a link for it. It's Profana IO. But I'll have a special link right to the trial. I highly recommend everyone checks it out.

Great.

All right, Dylan. I know one trend I've been seeing a lot about and I actually wrote about it in my 2024 trends is observability. How developers need to start about thinking about observability from the beginning. They need to bake observability into their systems. And that's one way you can make things more performant because you have open telemetry and all these solutions then that can help you with the tracing and everything. Are you seeing the same thing? Do you have any thoughts on observability? Is this something important that you see as a trend in 2024?

Yeah, absolutely, absolutely. It's maybe good to know. We, if it wasn't clear already, we built on observability tools. So one way to do, I mean if you want to do a solid performance test and especially performance engineering, you need proper observability tools to get the data and to be able to analyze the data. So the Pafana really builds on top of that. So what I also do see is people and companies relying more and more on observability alone. So that's a bit, for me, that's a bit like Max Verstappe, you know, the famous Dutch race driver.

Going out on the track without any testing and just expecting to win the race without doing, let's say, track testing and testing of the car. So you can have all the dials and metrics and whatever in place in production and you can have the fastest response time to those metrics. But if the so called dirt hits the fan, you're always lagging. So especially in the context, for example, of SRE and observability, I still think as good as those tools are and as needed as those observability platforms are, we rely on It a lot. It's not enough to rely on it alone. Because prevention still in the end I think is better than curing.

But this is a good point because a lot of people have said, oh, I don't have to worry about performance, I just use observability. I test in production. Why, why even spend time and effort in it? And it sounds like this would.

That's what I hear. Well, not a lot, but in some cases. And yeah, it's like I said, prevention is still better than curing. Curing is very costly.

Right? For sure. And when we say costly, it's a dollar value. Right. Because if you have bad performance in production, if you find it great. But I mean, you've already opened yourself up for lost revenue, downtime, probably bad reputation, all the things that.

Exactly.

Yeah, you caught it. But good for you. But Dan, you really hurt yourself waiting to the end, right?

Yeah, I would say it's a risk that you're taking consciously.

Great stuff. Okay, Dylan, before we go, is there one piece of actual advice you can give to someone to help them with their performance engineering efforts and what's the best way to find contact you or learn more about Prana?

Yeah. Okay, so. Well, the best way I think to get started with performance engineering efforts is first of course to just really realized that you need it and then actually I think it's not so difficult to get started. It looks maybe very daunting which is why we have set up the PFANA starter. Simply create an account, download the starter package and you're off to the races within 50 minutes. And it will show you, you know, it is a starting point for you to experiment with. So initially you just run it locally on your laptop. You have a local system under test and a local test to test it, which of course doesn't make sense at all because in a real life situation you wouldn't want use the same hardware to to actually do to generate the load and run the system on the test. But it's a nice starting point, you know. Next step is to use that same test, but run the system under test somewhere else. Next step is to integrate in the CI cd. It's step by step, should be fairly easy. So yeah, awesome.

And the best way to find or contact you or learn more about Prafana?

Well, like you said, you're putting a link in the show notes. Other than that Profana IO. You can also look me up on LinkedIn. Reach out.

Awesome stuff. Thank you Dylan for this awesome resource and highly recommend everyone check out that free trial and check it out for themselves.

Thank you.

Thanks again for your automation awesomeness. The links to everything of value we covered in this episode. Head on over to testguild.com a488 and if the show has helped you in any way, why not rate it and review it in itunes? Reviews really help in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack automation awesomeness. As always, test everything and keep the good. Cheers.

Hey, thanks again for listening. If you're not already part of our awesome community of 27,000 of the smartest testers, DevOps and automation professionals in the world, we'd love to have you join the fam@test guild.com and if you're in the DevOps automation software testing space or your test tool provider and want to offer real world value that can improve the skills or solve a problem for the Guild community, I'd love to hear from you. Head on over to test guild.info and let's make it happen.