Splunk Real User Monitoring (RUM)

Welcome to the Test Guild Performance and Site Reliability Podcast, where we all get together to learn more about performance testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Performance and Site Reliability Podcast. And today we're in for a special treat. We're gonna have Shajwit on the show talking all about real user monitoring, past, present and future. And a little bit about Splunk. We'll see where it takes us, but Shajwit leads Real User Monitoring performance team at Splunk. As a former software engineer, he loves working with developers to find elegant solutions to the problems and has built several enterprise tech products in his career thus far. So you're in for a special treat. You don't want to miss this episode. Check it out. This episode is brought to you by SmartBear. Listen, load testing is tough. Investing in the right tools to automate tests, identify bottlenecks, and resolve issues quickly and could save your organization time and money. SmartBear offers a suite of performance tools like Load Ninja, which is a SaaS UI load testing tool, and Load UI Pro, an API load testing tool to help teams get full visibility into UI and API performance so you can release and recover faster than ever. Give it a shot. It's free and easy to try. Head on over to smartbeer.com solutions forward/performancetesting to learn more. Hey Sajwa, welcome to the Guild.

Nice to be here. Joe.

Awesome. Awesome to have you. This is a topic I've heard a lot about, especially when I started my career as a performance engineer. Real User monitoring. Before we get into it though, for the folks that don't know how would you explain what is real user monitoring?

Well, in short, real user monitoring, as the name suggests, it's identifying how end users of your application are experiencing the application. Do they find it fast? Do they find it slow? Do they find it bugg? Do they find it a smooth everything as a smooth experience? So what are your real users real end users experiencing on your website?

Nice. So I'm going to pull things out of the article you wrote. I'll have a link to this in the show Notes on Real User Monitoring, Past, Present and future and I thought we'd maybe walk through that and use it as a jumping point for other things. Where did real user monitoring start? Like what were the the beginning genesis of it and why did it start?

I guess so the entire real user monitoring space. In some sense it's as old as the web, but obviously as the technologies on the web, the browsers themselves have evolved over time, over the last 15, 20 years. That's meant that the real user monitoring tools have also evolved to keep pace with technology and the new ways websites are built. So broadly speaking, 20 years ago, right, some people call it the Web 1.0. It roughly corresponds to the time when websites were not very interactive. Developers used to just build something with text and images. It was very static, it was all rendered on the server and on the server side. And those rendered web pages were sent as is to the browsers, which were, you know, commonly either Internet Explorer or Mozilla at the time. And they used to just render them as is without too much of interactivity. So this was the web 1.0 phase, circa early 2000s or even late 90s. And over time, as JavaScript became popular, things started becoming more interactive, the web pages became more interactive. You know, there'd be many APIs, for example, a Yelp might call a Google Maps API just to make sure they can show things on a map which renders within JavaScript on a web page. So, you know, web pages generally became a lot more interactive. And this, some people call this the web 2.0, right? Effectively what this is is interactive pages, many of which were rendered client side or within the browser itself. And right now we are just seeing a continuation of this as websites become more and more interactive. We are seeing single page websites like Twitter and Facebook and you know, the experience where you just load something once and then you just play around with it as if it's running locally.

I remember the good old days when it was 1.0, I guess how you describe it. And it was really easy, it was easier back then because like you said, I could just monitor my server, I could put a load on a server and I get kind of idea how things were. So with these front end JavaScript heavy applications now, what kind of issues does that? Cause if someone started off their career in 2000 to now that maybe they need to change their mindset. So you know what, real user monitoring has moved beyond what I can conceive and I need to start moving towards this other way that people are going towards.

It's a great question. And in some sense this is the genesis of what we are building here at splunk. So 20 years ago, right, when the websites were still not interactive, it was still pretty much static content, it texts and images being rendered on the server side. If you think about it, the only thing that can go wrong is, well, the most important thing here that can go Wrong is something on the server side. So as long as you're monitoring the server uptime, you're more or less fine, right? Because other than that, the only other thing that can go wrong, possibly that could cause problems in user experience is just network slowness. And as an application developer, you don't really control the Internet, right? So if you're an application developer, the only thing you can literally control is your own server uptime. So ROM tools of this era, which were very much focused on tracking and measuring server uptime, right? And over time, as to your point, the applications have become very much front end intensive, JavaScript intensive. Lots of things happening within the browser in JavaScript. And what this means is the complexity of such applications has exploded. Things could be going wrong all over the place. There are so many browsers now, so many different browser versions. Maybe the app is not designed to work on a specific browser version, right? Maybe it's not compatible with a certain device type. Maybe the device is running out of memory. Maybe the slowness is just due to something within the device itself. Maybe because there are so many third party dependencies. A typical app may be calling a Google Maps API, it may be calling a Stripe Payments API. And those are just two of the most common examples. There could be dozens of APIs behind the scenes. And this is all of these are third party APIs. I haven't even started looking at first party microservices that might be, you know, responsible for some slowness on, on an application there are so many different things happening, it's very difficult to pinpoint what exactly is going wrong. So from the perspective of an application developer to troubleshoot such modern applications, by the way, the modern applications work really well. They give you a great user experience. But things could go wrong at so many different breakpoints. So the use case becomes how do I know which ones of these complex interactions are causing slowness? Right. And that this is what modern rum solutions that measure real user experience need to monitor for everything that an application is interacting with. Every interaction between a browser and a provider, which ones of these interactions is actually responsible for poor performance?

Wow. So there's a lot of unknowns there. So if someone doesn't really know, I could conceivably someone back in the day, you own the server was on a raised floor, you knew what was going on. Now like you said, it's on a cloud, it's using multiple services, who knows what third party you're consuming or from the packages you're bringing down into your application. So how do you even know what to monitor then at that point, are these tools now smart enough to know, hey, there's actually a dependency here that you may not know about that you want to look in on.

Exactly. And that is the root cause of the problem. There are so many things that could be going wrong. How do you even know where to begin? And this is where we believe realm solutions are headed. Measure everything end to end. Do not make opinionated judgments on these other things that are more important than the rest. Measure everything end to end. Measure every request the moment it originates from a browser to the moment it creates some it results in some microservice call on your backend or on a third party backend. Measure everything at full fidelity and show the developer how all of these complex interactions are cumulatively affecting end user performance.

Great advice. So this is kind of like a newbie question, someone that's been in the field for a long time, but I think my skills are getting older here. This almost seems to me like an older term, real user monitor. A lot of times I hear now observability and telemetry. Is this the same thing or is this something different?

It's a great question and short answer is yes and no. Right. As is the case with most such questions, observability is, for the lack of a better word, a very hot term these days. And it's also a very loaded term. People sometimes tend to equate it to monitoring or troubleshooting, but I'd say observability is really a level of abstraction higher than that. While it's true observability is most, sometimes, you know, the same as monitoring. But at the end of the day, what it comes down to is that the actual systems that customers and developers are responsible for are getting super complicated these days. Right. It's almost as if there is a 20 years ago there used to be a small garden which they were responsible for monitoring and suddenly now there is a tropical rainforest. Right? And I'd say the distinction here is that observability is best understood as knowing the state of very complex system and having all the tools and metrics and logs at your disposal so as to make sense of a very complex system which is a level deeper than just monitoring or troubleshooting.

So is monitoring then a piece of that you would have to have correct? Real user mantra is part of your overall plan.

Absolutely, absolutely. So real user monitoring is a subset of what you need to ensure that your system is truly observable. You need many other things. You need a suite that can help you make sense of everything. And real user monitoring will only tell you how your users are perceiving or perceiving your application. Right. You'll still need some kind of backend distributed tracing solution, right, to know how those end user requests are propagating through your your backend. Because that's one piece of it. Second piece of it is that because those backend requests typically sit on some infrastructure, you'll still need some infrastructure metrics to know how that infrastructure is behaving and whether or not infra is behind the slowness of applications. And last but not the least, you just need logs, right? Because once you figure out where the problem is, and once you figure out what the problem is, you still need to go deeper and look at the logs to understand what is the root cause of the problem. So in some sense people talk about this, but observability is like that holy trifecta of metrics, traces and logs which combine all the possible information that you need in order to truly understand the state of your system.

Great. And like you said, observability is kind of like a hot term right now. Another one is reliability engineering sre. Back in the day, the only people really concerned about real user monitoring was performance engineers or performance testers, a network team type deal. So who do you see concerned about or using real user monitoring now? I guess it would be in the site reliability space. Or is it? Who should be familiar with this?

Certainly site reliability engineers are one kind of users for a real user monitoring like solution. But I think the broader trend in the industry is that developers are increasingly responsible for not just deploying the code, but also for maintaining and securing it. Securing it and everything to do with the entire lifecycle of what they're shipping. Right. So I think there's a big trend in the industry that real user monitoring solutions will eventually be utilized by the developers themselves. I guess the most relevant folks to begin with would be the front end developers, the ones who are actually using React Angular and any other modern framework to go in and build all these fantastic experiences for end users.

So I guess that leads me to the question, how developer friendly is a tool like Splunk's real user monitor? Does it fit into their workflow or how does that fit in?

We definitely hope so.

All right, cool. So how does it work though? Like they're developing and does it plug into their IDE and they're able to get the metrics?

Yeah. So we have a SaaS service. The first step is getting the data into the real user monitoring tool. Right. So there is a small piece of JavaScript that needs to go into the source code of any web app. It's usually just placed in the header of any webpage that needs to be monitored. And that telemetry runs within the browser of every end user. And it collects all the performance metrics, all the error metrics, etc. And sends it over to the Splunk backend. Here we ingest everything, we make sense of all the data that's coming in, and we present insights. And all of those can be visualized as part of the. What we call as the Observability suite, which is a SaaS service that can be accessed obviously from any browser anywhere.

So another term we used a lot is synthetic monitoring. Is synthetic monitoring part of RUM as well? Still or digital experience?

Great question. So it's complementary to it. One way of looking at it is that at least in our view of the world, DEM is equal to RUM plus synthetics. So in some sense the two products are measuring the same thing, but differently. Both of them are measuring user performance. Rum is measuring performance for actual end users, whereas Synthetics is measuring performance for synthetic users or test traffic that you may be sending from several vantage points around the world.

So for site reliability engineers, are they usually familiar with synthetic monitoring as well, or is that also the same group?

It's a good question. Yeah. So for the most part, both of these tools are used by very similar set of people. And that's not a surprise because like I said, right? I mean, both of these are effectively measuring the same thing, but slightly differently. So in order to get a true picture of web app performance, ideally you need both of these data sources and you need to look at them in some kind of a complementary fashion to understand what is happening.

Great. Another term I've heard a lot of like, I don't know anything about server liability engineering. I started this podcast just to learn more about it. One of the reasons and a lot of things that people talk about is budgets or how to optimize performance. I know Splunk has many different areas, Dem is one of them. But Ron, that seems to be a piece of that. How does rum, if at all help with optimizing the performance for error budgets that people may be managing?

So one way to think about this is that one of the metrics that a RUM like solution measures is the error rate from a website. Or, you know, it could be things like how Many front end JavaScript errors is this website logging in the console, Right. Or it could be things like of all the third party APIs that you are using, how many of them are throwing an HTTP 500? How many of them are throwing an HTTP 400? Right. So the all of these different metrics, error rates, et cetera, are captured by a RUM like solution and they can be utilized by the developers to track their SLOs and eventually to improve their, you know, SLOs over time.

Great. So are there any pieces of rum, especially Monaday's rum, that a lot of developers or a lot of, say, reliability engineers aren't taking advantage of or don't know that could help them with something that they're dealing with?

So I'd say that the recent, there are three recent trends that developers should know about. The first one is.

This concept of core web vitals. So around sometime earlier this year, I believe it was in May of this year, but Google came out with this announcement that they are going to start using the core of web vitals metrics. There are three kinds of metrics. There's lcp, FID and cls. And Google said that they are going to start using these metrics as a key component of their page rank results. So what this means is developers naturally are incentivized to ensure that these metrics are optimized so that their apps will appear very highly in Google's page ranks or page rank results. So what this means is that modern RUM vendors, they are incentivized to support the calculation and display of these core web vitals metrics straight out of the box. So this is the first best practice that customers and developers need to be aware of. The second best practice I'd say is around this notion of an end to end trace. This is going back to the larger trends that I was talking about. The fact that apps are getting more and more complex. And what this means is because apps are getting complex, there is a big need on the part of SREs and developers to see the complete picture end to end. And by end to end, I mean a request can originate from the front end, from a browser, it could lead to an API call on your backend. And within the backend it might propagate as a series of microservices operations. So if you just look at your microservices in isolation, you'll definitely not understand how certain errors or certain, you know, high latency events are affecting end users, if at all. So another best practice is to use a solution that gives you the true picture end to end. It answers the question if any end user is experiencing Slowness on any particular operation on your web app? Is it down to any specific microservice operation within your backend or not? Right. Rather than a blanket, oh, it's the backend's fault, right? Sure, it might be the backend's fault, but, you know, let's get granular about it. What exactly is happening that's causing this end user to be seeing problems? And the third piece here, the third best practice, which I believe many developers ought to take full.

They ought to start taking advantage of, is, is that because websites are becoming very complicated, it's not merely important to just track page load times. Right. What is, if you think about it, what does page load even mean? Right. If you're on Twitter, do you know what your page load is?

You could just be scrolling on and on and on, and you'll be on the same page. Right. And suddenly scrolling will stop. But that won't mean your page load time has changed. Right. A document load time or a page load time, in some sense, it doesn't really mean anything. In many of these modern applications, which have lots of micro front ends and, you know, small, small things, small pieces of functionality which may be causing slowness. Right. So what this means is that developers should start thinking about how to instrument pieces of the application that they truly care about. You know, the pieces that represent core business logic. Right. And how do they do so themselves? Using an API that is flexible, lightweight, and allows them to quickly measure what they truly need and show them results.

Without spending a lot of overhead. So I'd say that's the third piece. How do you truly leverage the power of open instrumentation?

Love it. And the second point you made was almost like following a transaction path, knowing all the things that the touch points. And so when you have bad performance, you can actually pinpoint it. How well can you pinpoint things nowadays? Can you actually say, oh, it's actually the SQL statement that's being called that's causing slowness? I mean, how detailed can we get nowadays?

That's the end goal. I guess it really depends down to how much effort you have put down in your instrumentation. Right. In theory, sky's the limit. In practice, you generally tend to run into, you know, limitations with instrumentation, but there is no reason why you cannot trace every transaction end to end. You know, there are definitely things you can do all the way to the database layer straight out of the box, but probably not for all use cases.

Great. Now, are there any best practices you recommend when people are trying to choose a real user monitoring tool? What should a real user monitoring tool give you, you think nowadays that keep up with the modern demands? What should people be looking for that a tool offers them?

Yeah. So I'd say that the three best practices that I just mentioned, those are only going to become more and more important over time. Definitely look for a tool that supports this notion of end to end tracing without sampling anything. So you know, or in other words, end to end tracing at full fidelity so that you don't you have complete visibility into everything that could be going wrong in a very complicated application. Right. So that's I think key secondly, the ability to see the core web vitals metrics ideally out of the box without doing any, you know, fancy hocus pocus behind the scenes. And last but not the least, any kind of instrumentation that allows them to instrument things that they truly care about as opposed to an inflexible code which only provides you access to metrics that are available straight out of the box and nothing else.

Kishwa, before we go, is there one piece of actionable advice you can give to someone to help them with their Rum efforts? And what's the best way to find contact you or learn more about Splunk?

The best way to contact me is either email me or contact me on LinkedIn or contact me on Twitter on both those social media platforms. My handle is my full name Shashwat Sehgal and the piece of actionable advice that I would say is Sign up for Splunk Rum. It's very easy. All you need to do is go to the Splunk website, go to the section where it says Observability Cloud and sign up for a free trial of the Observability Cloud suite. And Rum is just one of the components of the Observability Cloud. And just to add, you can get started with Splunk Rum if you sign up for a trial in less than 15 minutes. So it's as easy as 1, 2, 3.

Thanks again for your performance Testing awesomeness links to everything of value we covered in this episode. Head on over to testguild.com P79 and while you're there, make sure to click on the Try them both Today link under the Exclusive Sponsor section to learn all about SmartBear two awesome performance test tool solutions, Load Ninja and Load UI Pro. And if the show has helped you in any way, why not rate and review it in itunes? Reviews really do matter in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Performance and Site Reliability Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack performance testing awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Performance and Site Reliability Podcast. Head on over to testguild.com for full show notes, amazing blog articles, and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.