APM in DevOps and SRE with Andreas Grabner

Five, four, three, two, one.

Drop that.

Welcome to the Test Guild Performance Podcast where we get together to learn more about performance testing and site reliability with your host, Joe Colantonio.

Here it is. Here it is. Here it is.

Hey, it's Joe. And welcome to another episode of the Test Guild Performance Podcast. Today we'll be talking with Andreas Grabner about APM and beyond for Performance Engineers. Andy has over 20 years of experience as a software developer, tester and architect and is an advocate for high performing cloud scale applications. He's also a regular contributor to the DevOps community, a frequent speaker at technology conferences, and publishes regularly at Dynatrace's blog. And he's also one of the co hosts of the podcast Pure Performance. So he has a lot of experience. Really excited to talk to him. You don't want to miss this episode. Check it out. Let's face it, performance testing is tough. Traditional load testing tools create scripts that are bogged down in data. It's hard to read and require a lot of work, even for simple playback. Load Ninja cuts out the dirty work by using AI to inspect and debug your code almost instantly. No coding necessary. No changing of hands or relays between teams either. Give it a shot. It's free and easy to try. Head on over to testguild.com smartbear and click on the link to learn more.

Hey Andy, welcome to the show.

Hey Joe, thank you so much. And I have to say this is a really cool entry music that you, that you have there. So I definitely need to talk with Brian Wilson who is the co host of our podcast that we run and to learn a couple of tricks from you. That's pretty nice.

Awesome. Very cool. So Andy, it's been a while. Is anything I missed in intro that you want to tell us a little bit more about yourself? No.

I mean, it's actually always interesting to hear somebody kind of reading your bio and then being reminded that it's been more than 20 years already that I've been in the industry and that you're right. So I run a podcast, it's called Pure Performance and you know, inspired by people like you and others like us, especially also in our case, inspired by Mark Tomlinson and James Pooley of perfbytes. So we've been running our podcast for a while. So maybe some of your listeners also want to later on see what we are doing there. But overall I think you're spot on with the intro. Just trying to help people build and deploy and run better software.

Awesome. So Andy, I'D like to start off maybe talking about APM, traditionally, how people think about it for performance testing, and then maybe go into how you see it maybe evolving over time, or how it's moved maybe in the past few years. So traditionally, what is APM to performance engineering or performance testers?

Yeah, I mean, APM stands for Application Performance Monitoring or Application Performance Management, depending on which marketing team you'll want to listen to. It's been around for many, many years. I mean, I'm a little biased, as you obviously know, and I want to be clear here. I'm working for a company called Dynatrace, been working with that company for 12 years now, and, and we've been in the APM space for 15 years now because that's when the company was founded. And from the early initial stages, when we built our product, we've always been very close to performance engineers because this was kind of the problem we wanted to solve in the beginning. It is you have performance engineers that run tests against black box applications. And then we had this technology that was basically loading an agent into a runtime where it's Java or. Net. And obviously over the years, many more runtimes came about, but it was basically about loading an agent into these runtimes and the agent is automatically doing code instrumentation, it's automatically pulling out key metrics, it's automatically understanding problems, let's say, in memory management, in CPU hogging, in massive IO, and basically giving performance engineers a look into the runtime. So that not only could they then say, well, under certain load, performance is very slow, or we're seeing high CPU spikes, but with APM tools, they were able to say, well, it is this particular method, it is this particular location of memory, it is this particular access to that particular file when this particular URL is triggered, which then allowed them to really say, well, we found the problem, but we now also have the root cause. And this, I think, was the big value of APM to performance engineers. Getting insight into this. What used to be a black box.

From your experience, has that evolved, for example, it used to be just seemed like more of just a performance engineer, really niche. But more people I speak to, the more they've been talking about shifting right, and monitoring and actually using more APM type solutions. Is that a trend you've seen?

Yeah, I think there's multiple trends we've seen. So first of all, you're right. While apm, at least from my perspective, we started in pre Prod and then we moved into production. It was like 10 years ago. Because in the end it's great to obviously understand performance in, let's say load testing environments. But in production, when there's something wrong and something slow, you want to have also more data so you can actually understand what is slow. But then the trend that we then saw is hey, if we also have APM in production and, and APM tells us what type of requests are executed, which type of users are actually using that software, how are they using it, can't we then just extract kind of the data and generate workloads? So that means instead of coming up with let's say artificial workload scenarios in preprod, we can actually leverage and harness production data so that we can create more accurate, realistic workflow scenarios. So I think that's one trend we saw. It's kind of using APM or monitoring tools across the delivery pipeline. The next trend we saw is not only being used in production and being used in, let's say, performance engineering, but then the whole aspect of automation came in. So with automation, I mean, how can we leverage this data in CI cd? How can we automatically, every time when we run a build and we run a little load against it, how can we then automatically tell or ask the APM product? So what have you seen? What are the performance characteristics? I like to talk about the performance signatures. What's the performance signature of this particular component that I'm just running through my CI and how has that changed from the previous code check in the previous build to this build? So I think we saw this so as a kind of recap from load testing into prod learning, from prod updating and automating the generation of workloads to then really integrating APM strategically into every step along the pipeline through automation. That's not the last thing. I think that the next thing we saw is that APM in its name, it has application. I think it also moved and kind of expanded. And that's true, I think for every vendor that is out there in the space. And what I mean with this is it expanded to other parts of the infrastructure where your applications run. So now if you look at, at any vendor that claims they have APM capabilities or observability capabilities, they typically talk about full stack monitoring, which means infrastructure, runtimes, containers, kubernetes, services, microservices, service meshes, applications, end users, and then also adding to that log. So it seems that apm, while that space has been around for a while and has been well defined and there's a lot of analysts out there I think it's expanding and every vendor depends is trying to build this or come up with a solution that provides full stack observability. And that means really just covering more ground, not just in an individual app or process, but everything surrounding and then also looking into different types of data sets. I mentioned logging earlier, I mentioned events that come from different runtimes, whether it's Kubernetes or your cloud platform. So I think that the market is evolving and is changing and all this data in the end benefits many, many people and obviously very much so the performance, the capacity engineers, these high reliability engineers. So there's a lot of people that benefit from this change in our space as well.

Awesome. So there's a lot of things I want to dive into there. First one is you mentioned Performance Signature and I think I've seen this in one of your sessions as well. Could you just explain that a little bit more? I think it's an important, important concept folks should know about.

So I don't want to take credit for it. So the word came from, I think from two different groups of people. One, Thomas Steinbauer, he's our performance architect here at Dynatrace, he gave me that term initially and then I think it was also picked up by one of our partners called T Systems mms. So it's a service department of T Systems in Germany and they built a plugin into Jenkins that is called the Performance Signature plugin. But the general idea is something that we now also see in the, in the realms of site reliability engineering when we talk about SLIs and SLOs level indicators and service level objectives. So the idea of the Performance Signature is typically you are looking at different metrics with different indicators. When you assess the quality of a piece of code, you may look at response time, but response time alone typically is not the only thing. You want to look at memory consumption, you want to look at IO, you want to look at all sorts of different metrics. And I think depending on the component that you're testing, a front end, a backend component, let's say an online banking system versus, I don't know, some batch processing system, there's different metrics that really you as a performance engineer look at to figure out is it good or not. So what the Performance Signature really is, it's basically a combination of key metrics that are important for you. And you basically then take these metrics from build to build to build and basically look at how do these key metrics really change over time. It's like a Signature, right. If you are typically signing documents, you always have the kind of same signature. So if you do have a good build and you know that these 10, 15, 20 metrics are what you expect and if you're validating the the same metrics, build to build to build, and you see a change in this pattern, this the change in the performance signature, you know something is wrong. And this is the idea here that we are allowing, we're giving performance engineers more metrics that they can then automatically have analyzed and compared with previous behavior with the previous signature and then alert in case something is out of norm and there's different implementations around it. I mentioned earlier, at least with the people that I work with, the guys from T Systems, I want to highlight Rafael Pianke. He did a great job here. He wrote this plugin that's available on Jenkins for free. You can download it then. Also we have a open source project called kept. So it's keptn sh or you can also find it on GitHub. This open source project is also implementing the same concept of performance signature. Repackaged it in a feature that's called a quality gate. So that means you can define a list of service level indicators. These are metrics that can come from different data sources from different tools. Then you can ask Captain, Hey Captain, analyze these metrics for me and then tell me how these metrics are doing either as compared to hard coded SLAs that you may have or compared to previous builds that you have where you have run tests against earlier. And I think this is a very key point here and whether you call the performance signature, whether you call it quality gates, or whether what I believe the SRE community now refers to as slis and SLO service level objectives. The idea is you automate this as much as possible and making sure that you're looking at metrics that really matter or indicators that really matter for you.

Awesome. So Andy, maybe just my imagination, it seems like site reliability SRE has really been driving some newer innovations in what seemed to have been a more stagnant performance area. Is my perception true or is it just the opposite? Maybe performance has been influencing other areas like site reliability engineering.

Well, I think it went into both directions, right. I believe the site reliability engineering teams and I mean you've been around longer than I have, but when I first encountered them was obviously Google made them big and famous, but I would assume that they obviously looked at what performance engineers have been doing in the past in pre production environments and they just try to figure out well, how can we come up with something that not only works in preprop, but how can we extend this to production? How can we learn from production? How can we then shift it left? How can we not only build systems that where we are a service organization that provides, let's say performance or capacity planning services back to the organization, but how can we also offer this as self service, fully integrated into the delivery pipelines? So.

I think SREs have definitely been influenced by performance engineering, but not only by them, but many other folks as well. And I also think that, let's say those performance engineers that have been around for a long, long time, like the two of us, we are now looking at these new roles and new type of team structures and try to learn from them as well. Right. So I think it's a give and take and we're all influencing each other. And the interesting thing is that.

With the new type of architectures we have now in the cloud native world where you have many more moving pieces and everything is much more dynamic, we just also need new approaches for doing performance engineering and capacity planning. That's why I think we can learn from these new SRE roles that I believe have initially focused on these new architectures and they try to find new ways of doing things, new ways of evaluating performance, making sure things are stable, able and reliable.

So yeah, absolutely. And you didn't mention Captain. It's the first time I actually heard of this tool. Could you just talk a little bit more about this? It's an open source solution and who's it ideal for? You think would really benefit from this tool?

Yeah, so Captain, so it's spelled K E P T N and you can find it on GitHub or on Captain Sh, that's the website. So what it really is, it is a control plane for continuous delivery and automated operations. Now what that means is it was actually heavily influenced by the teams that are delivering software here at my company where I'm working. So Dynatrace, but also some of our customers that came to us and say hey, how do we deploy and operate modern cloud native applications? Do we really? Is Jenkins still the tool that we need to use for continuous delivery? What are we doing for operations? How do we automate remediation? They ask questions like hey, we're building event driven microservice applications. But yet our tooling that we use for continuous delivery on for operations is still the same tooling we used when we deployed enterprise Java apps to be very, let's say static and everything was very static. But now everything is event driven and it seems it's slowing us down where it's costing us a lot of overhead of keeping all these Jenkins pipeline files up to date and investing a lot of time in automating scripts. So that's why what Captain solves on the one side, it's an event driven control plane. As I said, that is taking an artifact. An artifact can be a Docker container, it can be a lambda, or it can be a serverless function. And it's taking the artifact and it's basically managing the lifecycle of the artifact by pushing the artifact through. I call it an opinionated delivery pipeline through different stages all the way to production. And the opinionated, why is it opinionated? Because what we learned, at least what we've seen, what you typically do is you get a new artifact, you deploy, you test, you evaluate, and then you either promote or reject. And so in Captain, you don't have any pipeline files anymore. The only thing you say, how many stages do you have? Like let's say dev stage and prod. Then you say, what type of deployment strategy do you want to have? Do you want to have a direct deployment, a blue, green, a canary or dark deployment? What type of test strategy do you have? Do you want to run performance tests? Do you want to just run functional tests? And how do you evaluate it? This is where the SLO files come in. So what Captain really does, it takes an artifact and it then through event triggers, the right tools to do the right job at the right time. That's also what we learned when we built this open source project. We have seen internally with many, many development teams and everyone has the little flavor for the tools. So some may use JMeter, some may use Gatling, some may use NeoLoad, some may deploy on Kubernetes, some may on ECS, some may still deploy on some physical machine somewhere. So that's why Captain has an event driven approach, where an event is internally triggered to deploy something and then the tool of choice, which can be specified in what we call a uniform file, is then picking up the task. It is then doing the deployment and reporting back to an event. To Captain, hey, I have deployed. And the same goes true for testing and then evaluation. So the problem we really solve is.

We are bringing flexibility.

We bring in an event driven model to the cloud native world, basically leveraging the same architectural aspects and concepts that application developers have been using in the cloud native world, but also to your delivery and also to your operation tools. Which means, Captain, as a second major use Case is focusing on automating remediation. That means you can send events from your monitoring tool from production to Captain. Captain then understands what the problem is and is then triggering again events to trigger the correct remediation actions. All of this is controlled through.

Specification files that live in git. Whether it's GitLabs or whether it's GitHub or BitBucket doesn't matter. So we're following a GitOps approach. That means everything that captain is kind of orchestrating can be fully controlled and it's also version controlled in your git. So there's a lot of things that I was just kind of talking about. There's also a lot of material out there. So go to Captain Sh. You know, use your search engine of choice and you'll find a lot of videos that we've recorded as well. And for performance engineers in particular, I think the quality gate capability, we run some tests and then we evaluate the SLOs fully automatically and then we tell you what type of score you have achieved. Meaning the score is an indicator on how your performance signature has improved or, or degraded. And I think that's why we believe that Captain is going to be very interesting for performance engineers as well.

I love it because it seems like it's actually giving you feedback now where before you have to actually gather the data and then come up with the feedback. It sounds like it's more of an easy way to give you that feedback automatically.

Exactly, yeah. It's all about self service. Right. In the end what we wanted to achieve is an engineer that delivers, that writes new code.

I want to check my code in to Git. This is going to trigger a build drop. Out of the build comes an artifact and the artifact is then managed by Captain. That means it's deployed, then the tests are executed and then it's evaluated. What Captain also has, it has integrations with Slack, with MetaMost, with Ms. Teams, with all these different tools. So that means you immediately get feedback on what Captain has, what Captain is doing in the moment. Right. Where, where is the artifact right now? And also it, it gives you feedback on the quality gate evaluation. That's all self service. And I think that's that, that is the thing that is most important these days because this is the way you can scale organizations by providing these things as self service.

Yeah. As well as the auto remediation that seems like people have a lot of data but not necessarily are acting on it. So this sounds like maybe this can help facilitate that as well.

Yeah, exactly. So for Auto remediation One of the first use cases that we've heard from our operations teams here at Dynatrace, they said when we get a problem notification then we typically open up a couple of dashboards and depending on the problem we know where to look at. We need to grab these five metrics and these things and these logs. And this is stuff we just automate with Captain. So in Captain you can specify a so called remediation file which basically tells Captain, hey, when a problem like this comes in, here's the data that you need to gather and then you either give it to the ops person on call through Slack or whatever and then they need to decide what they want to do or we can even link it to automated remediation actions. So we can say if this problem happens, then here's the first thing you want to try. If this doesn't work, here's another thing you want to try and if all of these things don't work, then escalate it to a human.

So I definitely probably see this as a trend that would be picking up in 2020 for sure. You also mentioned earlier maybe a few times about automatic test generation and just curious to know is that something you see more teams using based on their APM data, is using that data then to generate tests early in the workflow after going into production?

Yeah, I think I see two trends here. On the one side, yes, the generation of better, I'd say closer to real life test scripts and workloads, that's one thing. And therefore definitely there's solutions out there that commercial solutions. I think also some open source solutions that are harnessing APM data and then generating scripts out of it. So that's one option. But I think what I looking further than just 2020.

I think what we will see more and more especially.

Thanks to the new deployment models that we have, you can just deploy a new build in production and then just let it either kind of duplicate production traffic to that new version. Right. So you basically do a traffic shadowing thanks to service meshes that's easily possible through, I mean it's also possible through your load balancers that you have these days. But the idea is instead of running a test in pre prod where you kind of fake production data, you actually deploy it in prod and you direct, let's say a certain percentage of you copy traffic and give copy traffic to that test environment and figure out how is this behaving versus to let's say the version that is currently also retrieving the same traffic. That's one option. The other option is that you also do like a canary deployment or you're turning certain new features on through feature flags, which means you go direct into production and really get the feedback from traffic that is the most accurate, which is the real user. So I think we see both directions. We see on the one side.

Using monitoring data to generate better test scripts and workload models. But I believe what we will see more so going further into the future is really leveraging these new deployment models so that we can start experimenting faster with production workload and actually getting new features out to end users quicker. And I believe if this is done right, if you only expose, let's say, a new build to a very small number of people or to a very small part of your infrastructure, then the impact is obviously also smaller and therefore it's safer. And I think that's what we're going to see.

So Also speaking about 2020 and beyond, Andy, is there any new functionality in Dynatrace that you're really excited about that may have just been released or you see coming out shortly that you think is going to help performance engineers and server liability engineers?

Yeah, there's many features and actually, I'm not sure when this is going to air, but.

We have our new annual conference coming up in Vegas. So the first week of February we'll all get at the Cosmopolitan and a lot of the new announcements will come out. I think the big things that, that we as Dynatrace are working on is better support for the environments that people are moving towards right now. And that's everything in the Kubernetes world. We have a long standing relationship with the vendors that also provide commercial support on top of Kubernetes or that provide, let's say open. We have very good relationship with red hat on OpenShift and also pivotal with cloud boundary. So that means we are strengthening our integration there because we see there's a lot of more data that these platforms give us. And so you will see announcements on feeding more data and especially events to Dynatrace's Davis AI engine. Then what you will also see is just we call it. And I think I did a performance clinic. These are one of my YouTube videos. I've done this in December. We call it calculated service Metrics. That means you can define and create new metrics on the fly that can leverage business data. So I can, for instance, say I want to get the performance of search queries with a particular search term or I want to get the number of database queries that are executed depending on a particular type of end user or behavior. So there's a lot of, a lot of interesting things that are coming out here so that we can get more domain specific metrics.

Without having the developers to actually put this in and then let's say calculate it in the app and then expose it via some interface. So there's a lot of these things are coming. Also there's going to be a lot of enhancements and advancements in our real user monitoring.

And yeah, I think many, many more that I think the best would be to just wait for the perform conference and all the announcements. I believe the sessions will be streamed. That's true for the keynotes and everything that happens on main stage and for the breakouts, they are also recorded and I think made available also later on for the attendees.

Okay Andy, before we go, is there one piece of actual advice you can give someone to help improve their performance testing efforts and let us know the best way to find or contact you.

Let's start with the easy answer. The best way for them to contact me is either through Twitter, so it's rabnerndy or you can also reach me on LinkedIn, Andreas Grabner and I'm sure you can find my blogs and some other material and then you find easy ways to contact me. I'm always happy to help, especially because I think I always learn a lot when talking with people. It's not just that I can give, you know, let's say some assistance, but I believe I always also learn a lot of things. The big piece of advice for me on how to help it comes back to what I mentioned earlier with self service. I believe if you are in the performance engineering space and you are giving feedback, performance feedback back to your engineers, then try to make this as automated and as self service as possible. For instance, to give an example, one of the people that I work with is a group at Citrix. They have developed chatbots and so now engineers can ask the chatbot about how is my build doing? What are the key performance indicators? So as an engineer you just open up your slack and you just ask your virtual assistant and get all the data. So I think that's an important piece. It's about automation, it's about self service. If you're not in performance engineering but you want to have an impact, then I would just say ask exactly for the same thing. So ask your colleagues how can I get performance feedback in a fully automated way?

And I believe that gives you more autonomy that reduces the, let's say, the dependency to others and makes you more autonomous. I think that's the the thing here and it's a big piece of advice.

Thank you Andy for your performance testing awesomeness. For links to everything of value we cover in this episode, head on over to test guild.comp plight performance19 and while you're there, make sure to click on the Try it Today link under the Exclusive Sponsor section to learn all about Smart Bears awesome product Low Ninja dedicated to helping you create faster performance test. So that's it for this episode of the Test Guild Performance Podcast. I'm Joe and my mission is to help you succeed with creating end to end full stack automation awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Performance Podcast. Head on over to testguild.com for full show notes, amazing blog articles and online testing conferences. Make sure to subscribe to join the Guild and continue your testing journey. This has been a Joe Calantonio production.