The Future of Application Performance Monitoring with Scott Moore

Five, four, three, two, one.

Drop that.

Welcome to the Test Guild Performance Podcast where we get together to learn more about performance testing and site reliability with your host, Joe Colantonio.

Here it is. Here it is.

Here it is.

Hey, it's Joe. And welcome to another episode of the Test Guild Performance and Site Reliability Podcast. Today we'll be talking with Scott Moore, all about application performance management. Scott is the founder of Scott Moore Consulting and has over 27 years of experience. He's also an active thought leader in the performance engineering space and has helped a ton of folks with his consulting services, his blog, and his appearances on podcasts like this, and also in person around performance testing, performance engineering and performance monitoring. So get ready to discover APM awesomeness in this episode. Let's face it, performance testing is tough. Traditional load testing tools create scripts that are bogged down in data. It's hard to read and require a lot of work, even for simple playback. Load Ninja cuts out the dirty work by using AI to inspect and debug your code almost instantly. No coding necessary. No changing of hands or relays between teams either. Give it a shot. It's free and easy to try. Head on over to testguild.com smartbear and click on the link to learn more.

Hey Scott, welcome to the Guild.

Hey, how are you? It's good to be with you.

Awesome. Great to have you on the show. So Scott, we've been talking on email and I thought a good topic to talk about today would be apm. But before we get into it, just tell us a little bit more about yourself.

I have been doing performance engineering since probably late 90s and have tested large systems, small systems, fortune companies all the way to startups. And I just have a ball doing it. So it's pretty much all I do. You might say I'm a little bit obsessed with performance and I talk about it all the time. My license plate is http503. So there you go.

That is awesome. And you are one of the thought leaders I've been following for years. So I really appreciate what you've been doing. So one quick question is before we jump into deeply, is what is APM.

Application Performance monitoring or Application Performance Management kind of interchangeable. Starting to hear a new term thrown out there called observability now. And that's basically saying we don't want to just monitor something, we want to actually be able to do something about it. Or we want systems that are going to tell us what to do when they find things that they're monitoring. So that's something that I think we should probably talk about today. But that's sort of observability is sort of the next phase of where API APM has existed in it, but it's really known as systems monitoring in the old school term.

You sent me over a cool infograph of what is APM that we dive into. It'll also be available on the Test Guild performance page that people can visit and they can follow along with. But at a high level after you know what apm, the next level it dives into is digital experience. And it's made up of synthetic timings and real user sessions. Can you just talk a little bit about this layer of digital experience?

Sure. And just for context, I created this slide from Gartner's report from back, I guess it was the 2018 report saying this is what APM really is and it's made up of this. You can sort of see it as a three layer cake. And that first layer is that digital experience that you're talking about. And that's made up of two pieces. The piece on the left is synthetic timings. There's robots there on purpose. So think of you've got some automated script, maybe it's a selenium script and it's grabbing page timings, full page renderings. It could be running even a performance testing product. But only one virtual user goes out there and automatically kicks off every 15 minutes throughout the day and monitors your website. That's a synthetic timing and you're getting the metric from the tool itself. On the right hand side you have real user sessions. Sometimes that's called rum, real user monitoring. So think of as the users are accessing your application, there is something that's monitoring the performance metrics from their actual navigation on your site. So in those kind of metrics, think of anything that the browser itself might report under the covers as how long each image took, style sheets, JavaScript, and how long the page took as well. So you have those two and doing a litmus between those two, you should be able to have a good understanding of. Here's what our synthetic scripts might show in a pristine lab situation with no bad network connectivity or anything like that. And then you've got your real users out there. So by having that, a use case for troubleshooting might be, hey, our synthetic timings are really, really good. But we've got one office in the United States that is reporting really bad numbers. But it's only that office geographically. So then you can kind of find out what the problem Might be by knowing what it's not, knowing that it's not everybody else. You can focus on just that office and find out why is that office having a problem today? Is our Internet provider having a problem? Is there a problem with the WI fi that day? And that can really help you narrow down the problem quickly.

That's a great point. Is there anything else that can go wrong in this layer, Scott, that sometimes people get wrong that could throw them off?

Oh, there's tons of things. Right. And this is the biggest problem is people think about one or the other. They think about either the user experience, but then they don't know why it's taking a long time. So they need the other pieces of that puzzle. And then there's the other piece, which is you try to dive as deep as you can to start with and you forget about the fact that end users don't care what's behind the veil, so to speak, what's happening on all the machine and the cogs that run it. They don't care what is causing the problem, they just know it's slow. So many times you run into a disconnect between it and the business.

If you start with a top down approach and you start with what the business sees or what they experience, I should say with regarding performance and dive down from there, you can always dive down and find sort of the haystack. That's the problem. I always use this analogy. Once you find the haystack, then you can go into that deeper level, work with subject matter experts and pick that needle out of the haystack. That part generally is pretty easy if you can get to that haystack quickly. And having both the top level, that digital experience and, and that middle tier is very important to mix them together, merge them together to see I guess, a 360 degree view of what's going on with your app.

Very cool. So how's that information provided? I assume this monitoring is done in production and then that's how you're capturing the synthetic timings and the real user session information.

Yes, and production is obviously not the only environment that you, you might want to monitor all your environments. In fact, I would recommend that you move that digital experience as early in the lifecycle as possible. Especially the real user monitoring. I don't see that happening a lot today. Think of developers just wrote some code, they throw it out there, but they have no clue as to what it looks like. You don't need to run a load test to find out if just one user is bad. Part of that, maybe that CI pipeline is not only a functional test, but, but a small performance test that provides that or have the user actually the developer themselves go into the browser and go to that function that they just wrote. I'm assuming it's, you know, it's a whole function that they can actually use a whole webpage to render and pull the information from Chrome or Internet Explorer, whatever they're using, and get that possibly in an automated, or not an automated fashion, but from a tool that reports it into some sort of a deliverable, some sort of. Maybe it's a database or maybe it's a spreadsheet or something like that. But as they're actually navigating through and they can find that out immediately before they even move it into the test environment.

You know, that's actually a great point. A lot of times we talk about shift left with other things in the software development life cycle and a lot of times with performance once again. It's almost like the waterfall days where you throw it in production, then you realize, oh, we have a performance issues. So it sounds like if you can implement APM solution earlier in the software development lifecycle, you probably could catch a lot of things before it gets to that point where it's really mission critical.

I am actually very surprised at how often I will go into a company I've never been into before. Doesn't matter how advanced they are in a lot of their development pipeline and they've got the most talented people, but when it comes to performance, they still have these very simple blind spots and they're. They bring me in and ask me the question. Well, we are performance testing, but we're still having problems and we don't know if things are getting better or worse between versions, between builds, between major releases. And now our customers are asking us for this. What do we do? I'm just surprised we haven't gone further to crack that nut as we are today. Just my experience though.

It's my experience as well. That's so true. All right, so once again, going back to this image, once we get by this layer, there's another layer you have that's called application discovery, tracing and diagnostics. So I thought maybe we. Yeah, no doubt. So all of those, each individual stages are they put together as one.

All right, so just think of whatever technology stack you have, it's made up of something, right? It starts off with the base level. So let's say now we're not talking about serverless right now. Let's just think a traditional on prem, we've got physical servers or virtual machines. We start off with an operating system, Windows, Linux, those things have resources associated with them and that's what's sitting in the middle. And I call these middle four, the four horses of the performance apocalypse. Because one of these will tend to get you in the end if you don't watch them all. And they are cpu, memory, disk and network. When these resources have become depleted in one of these areas, you will experience problems. When the CPU queue hits two, you can just watch the transaction times just go through the roof because it's getting depleted and it has to start figuring out what to do. The things that are on the left and the right are just the other different technologies. They're not base OS metrics, they're the other metrics that move you up into the stack. For example, you may be looking at CPU and it's not doing very much at all, it's not working very hard. But on the other hand you can move up into a different layer like say the database layer, right? And then you look at all of the database activity and you find out there's queries that are very inefficient. Has nothing to do with the cpu, it's just that the query itself is bad because you're doing select all on every single thing and you shouldn't be. It may not cause a CPU to work very hard, but it's really going to take a long time for that round trip to happen. So now you've got to figure out if you go and trace from the end user GUI all the way back to say the database, where is it spending its time. So if you know that the resources, the base resources are okay, there's all of these other metrics that you can look at. One thing that's not really listed here, but it's kind of implied as we go further into this is logs. That's a big thing that has become very popular in the last four or five years is you know, we've got Splunk, we've got elasticsearch. So we should be able to take real time metrics and metrics that have been, you know, kind of saved off. And also look at log files like say in the same area of the cake. We're looking at the web server itself. You've got Apache or nginx or Internet iis. Microsoft is there is on every web server that is compliant with W3C there should be a time taken field. One thing I've realized recently is Microsoft's the Only one that has it turned on by default, which was very surprising to me. You have to go in and turn these on, but once you turn them on, that's another source of information. So between synthetic timing, real user and then a web server log, you can kind of see, okay, how long did it actually take on that web server? How long did the web server take to give them the page and how long did the web server say that it took to give the user that page? You look at that versus the time that the real user monitor shows the time, then you can actually find out, well, how much time is that JavaScript on the front end really taken to render that full page? Now you've got that litmus between the two and it's real easy to realize, oh yeah, We've got this third party JavaScript we're calling and it's holding up, it's blocking the whole process. It becomes very easy once you have the information, but that's why you need all of that stuff. The problem here is there's a lot of information. There's tons of data that are coming across all of these products, these OSes, these stacks that you have. So it becomes very difficult to kind of wade through all the noise and pick out what you really need to see that's going to help you find the bottleneck within your application. And that in some cases becomes more of a dark art than a science.

So that was my next question. Like, how do you correlate all this information to find out? Slow performance, front end is caused by the SQL statement on the back end or it's getting lost in a queue or something.

Well, the old man in me is like, well, back in my day we had to go uphill and look at it. No, it's, it's actually, it ha. It's become so, so much of an overload that if you have to do it manually, if you have to do it old school. Let's say I was running a load test and I was looking at end user timings and I had to cross that with four or five other different major metrics. CPU for example. That's pretty easy. Start with the base stuff. Are the base CPU metrics Is the CPU Q, what percentage is the CPU utilized? How much memory do I still have? Enough. How much network? If I put this application out in the field at a remote office and I'm going through a really bad WAN connection or through a SAP, those parts are pretty easy to pick out. But then when you start adding everything else into play, you pretty Much have to talk to the subject matter expert. I don't know everything about an Oracle database or a SQL Server database or a postgres database. There's probably somebody in the company for that app that does. But as a performance engineer, there's no way I can know everything about everything. I mean, I wish, I wish I could say that, but there's no way that I can dive as deep on every technology that exists because I got to move from engagement to engagement and I have to be sort of a jack of all trades in that sense. So let's say that they've implemented Oracle and we do find that Oracle is the problem. I'm going to ask the database person what is the most important thing that you would want to see are the things that you would look at that you're probably already looking at when there's a potential outage or something of that nature. Usually the DBA knows that, will tell you that, and then they're surprised that you might have a product that you're using on say, the testing side that could marry that information to that top digital experience and those OS metrics and then create this wonderful nirvana graph that says at this number of users, this timing is horrible. This specific timing is horrible. And the reason why is because of this, because you have that, that correlation. Some tools allow you to do that almost automatically. Some of the commercial tools kind of guide you through it. There's some that are better at analysis than others. Some of them, if you're piecing together your own open source thing, that's where you're going to spend the time and the money getting to that root level, either by the resources you're going to spend and the person actually researching it down, or you're going to spend the money on a license for a tool that will guide you there. And if you don't mind, let me just expand on that from the bottom piece of this graphic because we've got AIOps, right? This is something that's the newest piece that Gartner started mentioning. They actually had renamed it from something else. What that is is the machine learning artificial intelligence. We hear about that in the IT realm applied to lots of different things right now. But this specifically would say all of that stuff in the top layers that we've just finished talking about, we know it's a lot of information. Why don't we take all of that data, put it into a big data storage engine, run it through some machine learning and hopefully create some kind of predictive way to find out if we can head off a problem and let the technology itself guide us. So If I'm monitoring 1000 Java virtual machines that are sitting out in my production thing, wouldn't it be nice if the AI engine could scan all 1000 of them and determine over a period of like say a day of monitoring them? There is a very slow memory leak and it can then calculate. You have two days before these JVM start running critically low and you're going to start having errors in the application and people are going to start placing tickets with the support center. That can only come when you a have enough data in that middle tier and you're feeding it into a decent AI engine because that's so new. There's very few products out there that will allow you to do that. But that's really where we're going into the future.

So Scott, I want to talk a little bit more about the AI piece, but you did mention earlier serverless that the four pieces of the I love how you said the performance apocalypse is we were talking specifically about on prem type solution, but I've been hearing more and more about serverless. So any tips around how does serverless impact performance from the way it used to be? Now, is it the same same deal? Or are there new complexities that introduce when you start using serverless?

Without trying to be very cynical, as some of my colleagues are today about this, we first have to really define that serverless does not mean that there is not a server there. It means that that layer has been removed from the concern of the developer so that the developer can focus on writing and deploying code and not having to worry about any of the layers below that. If you think about how Cloud is really developed, it used to be, hey, cloud is somebody else's computer. It will take the computers, we'll put them in our data center. You just put your virtual machines here. Let us worry about the bare hardware starts with that. Now we've moved into containers difference obviously between virtual machine and container. We've removed that hypervisor layer that you have to worry about emulating that machine. Now we just put the code on a machine and it runs no matter what operating system is sitting. So it kind of removes the whole operating system. So boot up a tier Serverless or functions as a service, any of that stuff comes code, we just deploy that code and all of that management of the stuff underneath just gets taken care of it for us, so to speak. This introduces a new paradigm in how we address performance because now for the first time, if you truly go serverless. Let's say you have a staff of 10 developers that are working on this project and each one is developing their own microservice and they just deploy it as a serverless function. You will be able to take that cloud bill from Azure, aws, Google Compute, and you will be able to see how much each one of those microservices cost you. What if you're the developer that who hasn't paid attention to writing performant code, you're going to be isolated as, hey, you're the guy that's costing us the most money. So it's actually a good thing in my opinion, because it takes a true financial impact and marries it to bad performing code. So it's going to force, in my opinion, it will force developers into paying attention to that as early as possible so they're not responsible for that high bill. So there's, there's still those elements there that are underneath. It is. It is.

It's to the benefit of the cloud provider that the development staff not worry about that. Hey, you don't have to worry about that. You just keep writing code. We don't care how fat it is, we don't care how much of a pig it is. You just keep writing it because we're billing you for every cycle on that, on the CPU that's underneath. They're just taking care of it for you. You see what I mean? So it's of their benefit for you to write really poor code and quote, unquote, not worry about it till the bill comes.

That's a great point. A lot of times people say they don't even have to worry about performance because they can scale up to infinity, but there's a cost associated with that. So that's a great point.

As long as everything's free, everybody's happy. When the bill comes in and the manager that pays the bill finds out, then there's questions. So there's a lot of ways that we can go in terms of improving that development pipeline by checking all that stuff earlier, earlier. And I think serverless as we go serverless.

We can do that. It's going to be a more visible way to do it, I guess, is what I'm saying. If you think about containers as well as we throw microservices in there and you just wrap everything in a container, those containers still need to be monitored as well. So you're starting to see new vendors pop up like Instana, Signal fx, that just got purchased by Splunk because they saw the Opportunity Cloud Native Monitoring datadog is now going for an ipo.

Cloudflare is playing a lot in the edge market where you're now writing and deploying JavaScript directly out on the edge. I mean, I've actually done this and it took me like 40 lines of code in about 30 minutes and I had a website running and all I've got is a domain. There's no server, there's nothing. There's 45 lines of JavaScript and it's out there. So that still needs to be monitored. And maybe there's not a CPU number that I'm measuring for that, but it's probably going to go back to user experience. How long is that taking? Somebody's going to have to provide instrumentation or guidance or a metric somewhere that says this is why it's taking so long.

Awesome. Scott. I just want to go back now to talking about being cynical AI. So AI has been a big, big push in the automation space. So called AI machine learning. You did mention it. So is this a real. It sounded like you really believe it. So what are your thoughts on the AI machine learning solutions you have, you've seen so far in the market to help you with performance testing? Just tell us a little bit more about that.

Well, the only one I know of that has gotten some traction in the market is Dynatrace. They have an engine called Davis and there's some pretty cool stuff with that. I'm not sure how mature it is, but I actually have the privilege of having some experience working at a client who does AI and machine learning. The difference is they do it on unstructured data. So the company I'm working with now, currently right now is doing natural language processing where they take communications, break up that communications, they tokenize sentences and emails, things like that, and they look for keywords and they're looking for things that will trigger alerts. And from those alerts companies can take action on that. So think of use case where you're a bank and you're trying to determine if there's fraud going on or you're the Nasdaq and you're trying, you're the SEC trying to figure out if somebody's doing something illegal. It's that type of thing. But it's built around unstructured data. This, what we're talking about is more structured data. You know, the information that you're going to get back, you're probably going to get it in some small parsed file, like a JSON file or xml, you know what's going to be in there. And it should be easier to implement that than it would be for something totally unstructured where you have no idea what's going to be in there. So when you have these known values, then you should be able to put it in some kind of an engine that over time learns, hey, you know, for the past 12 hours this number has been decrementing every hour. So much percentages. So it's not a far fetched to say do a calculation to see when it's going to hit zero. Well, there's your memory leak thing, so I think it can be done.

Like I said, other than Davis, I don't know very many other things that have applied it to specifically performance metrics.

My colleague in the faith, James Pulley, has mentioned to me that there's some work that has been done at NASA and it's a totally different way of approaching AI and he's probably more familiar with the different methods that you can use for machine learning than I am. But it's just beginning. Can it be done? Yes, I'm watching it be done in the unstructured world. So it's just a matter of time until we get really good at it in the performance space. I thought it would be good to share what Gartner is currently saying is full performance monitoring and that's what this slide kind of represents and then just talk about where we think we might be going. And I think the AIOps piece, that's where we're going and that's what you'll see the biggest need for in the next, say three to four years.

All right, Scott, before we go, is there one piece of actionable advice you can give someone to help them get started with the APM performance testing efforts and let us know the best way to find or contact you.

Well, the best way I'll do that in reverse. The best way to contact me is Scott Moore. That's my website and there's an email address on there. You can use HelpcottMore Consulting and that's an easy way to reach me as well. I've got a pretty big mouth and I'm on Twitter. Odetester is my Twitter account and I'm pretty active on LinkedIn. Those are the two primary social media platforms I use. In terms of getting started in apm, I would look into something that's easy to get and implement and install without costing you a lot of money, especially if you're just looking into it. And one of the pieces I'm currently looking into is the elastic Stack. So you may have heard of elasticsearch, the Loggly, the Kibana Dashboards, things of those nature. This is an open source solution that also has an APM piece. It has a real user moniker associated with it as well. I think there is licensing levels for support and things like that, but you can implement this stuff quickly. In fact, each of these pieces of the Elastic Stack have a container. So if you have Docker you can literally download three or four containers and do the wiring behind the scenes and in about 15 minutes you have a working APM solution with visualization. That's probably the easiest one that I know of that you could start with and start looking at. How do I get data from these systems turned into a graph that I can look at and go huh, that's really cool. I didn't know it was doing that. I would probably start with that and then if you got serious about it, look into some of the other commercial products out there, like a Dynatrace or I'm trying to think of a few other ones. AppDynamics I think is another one and there's tons of players out there. If you're specifically needing cloud native container type Kubernetes monitoring things of that nature, you're going to need to look at things like datadog SignalALFX thank you Scott.

For your performance APM testing awesomeness. The links to everything of value we covered in this episode. Head on over to testguild.com p likeperformance p05 and while you're there, make sure to click on the Try Today link under the exclusive Sponsor section to learn all about SmartBear's awesome software as well as their new Load Ninja performance testing tool. So that's it for this episode of the Test Guild Performance Podcast. I'm Joe and my mission is to help you succeed with creating end to end automation awesomeness. As always, test everything and keep the good keep. Cheers.

Thanks for listening to the Test Guild Performance Podcast. Head on over to testguild.com for full show notes, amazing blog articles and online testing conferences. Make sure to subscribe to join the Guild and continue your testing journey. This has been a Joe Calantonio production.