Performance Monitoring using RayGun with Innocent Bindura

Welcome to the Test Guild Performance and Site Reliability Podcast where we all get together to learn more about performance testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Performance and Site Reliability Podcast. Today we'll be talking with Innocence all about performance monitoring and probably other performance topics that come up. Innocent is a seasoned software developer at Raygun with over 10 years of software development using the Microsoft stack of technologies. You know, he loves new technologies and from time to time he loves to.

Get his hands dirty and find out what works best.

So we have the best of both worlds.

Someone that actually works for a company.

Does profiling, but also actually does it on his own as well.

So that's going to be awesome.

Innocence Professional experience spans several industries ranging from online marketing, retail, financial, to entertainment. You don't want to miss this episode.

Check it out.

This episode is brought to you by SmartBear. Listen, load testing is tough. Investing in the right tools to automate tests, identify bottlenecks and resolve issues quickly could save your organization time and money. SmartBear offers a suite of performance tools like Load Ninja, which is a SAS UI load testing tool, and loaduipro, an API load testing tool to help teams get full visibility into UI and API performance so you can release and recover faster than ever. Give it a shot. It's free and easy to try. Head on over to smartbear.com solutions performancetesting to learn more.

Hey Innocent. Welcome to the Guild.

Thank you for having me, Joe. Happy to be here.

Awesome.

Great to have you. Before we get into it, is there anything I missed in your bio that.

You want the Guild to know more about?

I think you nailed it.

Oh good. Love to hear that.

So Innocent, what I've been seeing lately is a shift left in software development in general.

But now lately performance and security are starting to get pushed left as well.

But the problem is I see developers.

Have a lot on their definition of done.

So what do you see in this current environment?

You do work for a profiling type software.

Any thoughts around what developers and how.

Responsible they need to be around performance as they're developing their applications?

Yeah, absolutely. So having been a developer for the past 10 or so years myself, I have come to learn that your reputation as an organization is on the line and the perception that is out there is ultimately the responsibility of the developer themselves and how well their software performs out in the world. So without monitoring, I don't think it would be possible to know exactly how that of software is performing out there. And for your own peace of mind as a developer, you would want to know that the quality of code that you're pushing in is actually fit for purpose.

How does a developer know that there's.

So many things you could, I guess, monitor? So as the developing. Are there any key things you usually do as you're developing code to be aware of, to see, make sure it's as performant as possible?

Yes, there are various approaches that I would normally consider when doing my work. Firstly, there's a concern of is it a customer facing application or is it something internal? And if it's something internal, how fast do we want it to run and how critical is it? So an example would be developing something that is way internal and it sort of summarizes the data that we need to present to customers. So there is a time limit there in which I am supporting to process those things. So I tend to do a little bit of explorative monitoring where I plug in an APM tool during my test cycles and just explore some of those traces and dip into them and see if I'm having some bottlenecks. Quite often I find my bottlenecks around other pieces of technology that I don't quite have control over. And those kind of things will lead me to maybe changing the architecture of the application before we even hit the market.

Awesome. That to me is an ideal situation with the traces. Anything specifically you look for, because I know some of these can really pump out a lot of information in the log files.

Yes. So working for Raygun has enabled me to do what I would refer to as dog footing. So I tend to run the APM tool against my code and the data gets ingested through the Raygun pipeline. That saves me a lot of time to sift through logs and enables me to look at issues that the ingestion pipeline has identified for me. So most of those tend to be N1 queries caused by object relational mappers. So one of the key things that I've found is, you know, there are so many things that you have to keep in mind as a developer. You have to. One of the things is following software design patterns like domain driven development. And what that says to me is that type of architecture advocates for sort of a hierarchy of objects going down. So with object relational mappers, you'll find that loading one object will probably load the entire object graph when you just need a subsection of it. So your APM tool, rather the Raygun APM tool that I use, often points those things out to me and I Am able to go in there and either optimize my object model, optimize the queries.

Nice. Now what I've seen work for a large organizations is usually when you work for a performance team, it's sad, but a lot of people don't even test for one user. And when you're in the performance team, usually test for multiple users.

So where does a developer know where.

To focus their energy when they're writing the code? I think you mentioned, like, is it when you go to production, are there certain areas where customers complain about and then maybe if you're adding a new feature to that, that's where you should be focusing on performance wise.

So while I'm developing, I take a test driven approach to my work. And with that same mindset, I tend to monitor things as I go and I do what I call explorative monitoring. And not to rave about Raygun too much, there is another piece of technology that I use which is Datadog, which accepts arbitrary metrics that can be pumped out of my application. So I tend to develop dashboards and monitor my methods using those kind of things, mainly looking at how long it's taking to execute the method and how often that method is being called and compare that with design intention. So if, for example, if I expected a method to be called only once and on my dashboard, I see that with each execution that method gets called twice. That is something that calls for more attention and that's how I would go about determining where to focus the attention of my development efforts.

Great. So we probably need to dive into a little bit, what is Raygun for this next question.

So at a high level, before I.

Ask any more questions, let's just set the stage. You know, what is Raygun for the folks that don't know?

So Raygun is a company that produces three set of products mainly geared toward three types of monitoring. So the first one is crash reporting and that's our more mature product out there in the market. So you know, traditionally developers tend to collect logs and when something goes wrong, they have to maybe log physically onto the server and start sifting through those logs. And if you're on a web farm, for example, sifting through those logs becomes a cumbersome task because you have to go through all the servers. With crash reporting, you get your crash reports centralized and there are other features that actually help go through those errors. Crash report tends to aggregate the errors for you. We generate an identity hash for each error type and form groups that you can then look through it using the Raygun front end. And if you have integrations like GitHub, it also offers you the ability to go and point directly in your repository on the line of code where that exception is occurring. And then there is another product which is real user monitoring. Unlike Google Analytics, this is more geared toward the developer and the performance of their page. We collect metrics around First Contentful Paint, First Paint, load time, ssl, DNS lookup and things like that. And that sort of tends to paint the picture of how the code the JavaScript is performing on the client side. And along with that, there is a server portion which one can drill down into and that can either lead you into the other product, which is crash reporting, or it can take you to the APM trace. And that brings me to the third product, which is application performance monitoring. So, so this brings me to the other point where to the other product is application performance monitoring. APM consists of an agent that runs on the server, and that agent taps into your processes and sends the traces through to Raygun for aggregation and issue identification. So when you look at the whole suite of products, it actually offers you a holistic picture of how is my code performing on the client side with the customer, how is it performing on the server side in house, and what type of exceptions am I experiencing on a regular basis. So it also enables you to maybe take a more modernized workflow in working out and ironing out your issues with your application.

Awesome. So I'd probably like to touch on all three phases here, but.

So the first phase is a developer.

Is creating code and that you want them to create the most performant code possible. So the best way to do that before they check in is to actually catch issues before they check into the code base.

So a lot of times developers complain.

That they have to go to multiple locations to look at data. So does your solution or do these.

Type of solutions tie into their IDE.

So that they don't have to change context as they're developing, that they have this information right in front of them?

No, unfortunately not. It doesn't tie into the ide, but it does offer a centralized place where they can come to and have a look at the performance details of their application. And this is more geared toward a productionized environment, so that doesn't necessarily tie into your ide.

Gotcha.

So once they develop code, they're doing.

The monitoring locally, they find their issues, they fix them, they check it in, there may be another team working on another feature and they check it in and for some reason there's a clash that caus performance issues between something between the interaction between the two check ins. So does your solution or these type of solutions help catch those type of issues as well once they go into CICD type of environment?

So we are a little bit later than the integration of those two unrelated commits that come into your code base. But we do offer deployments tracking. So this sort of offers you the ability to see sort of a trend before the deployment and after you deploy you see there's a blip on your deployment graph that shows you when the deployment occurred. And from that point onwards if you see your crash rate increase or your load time increase, then you know that you've got a bad deployment and you are able to define or maybe roll back to deployment previously before the one that caused the issue.

Cool. So to give people an idea, this is audio, but is anyone you can walk through maybe an example of how a solution like this works in a lifecycle so people know like at what points it could help them along the way?

So there is no way I can point where there is something that is already in existence that customers can play around with. But we do offer a free trial over 14 days and we are more than happy to talk to people interested in learning more about Raygun, maybe talk about extending that trial so that they can properly evaluate it.

Like I said, once the developer is.

Coding they check it in, it goes to they actually deploy it.

At what point then do developers get.

More involved is when it crashes in production. Then you said you have a crash reporting feature, so would they be alerted and then they would go to that report and the report then would be able to does it do any sort of mapping like it's in this particular module or this particular method, anything like that?

Yeah, so we do get a full stack trace of those crashes as they happen. Even for minified JavaScript files, there is a function where source maps can be uploaded, we decode them and do line mapping. So once your bad code gets into production, there is some form of alerting that happens at five minute intervals. The first time we experience an exception, we send out a notification in the form of an email. Or if you've got Slack integration, we also send a Slack message to notify you of the crash. And if that crash continues to happen within five minutes, those notifications continue to roll out.

Very cool. Okay, so next question.

A lot of times people when they're.

Using monitoring solutions, they'll complain it adds overhead.

So does Raygun, how does it work?

Is it.

Do you have to Implement it within your code or is it outside?

Like, what's the overhead of using a solution like Ray Gun?

All right, so for the two products, namely crash reporting and real user monitoring, there's no real noticeable overhead. Once your crash occurs, there is a small overhead of an HTTP request that will come to Raygun with the payload of the factories and details of the crash. And then in terms of ram, with each page load, each page refresh that occurs, there's also a tiny payload that comes through to Raygun which is unobtrusive and doesn't affect your application in any way. However, application performance monitoring is a little bit different because it has to hook into your code and examine what's happening there. So generally what I have seen is there may be at most a 20% impact on your CPU. And then if we start refining fine tuning the integration, you notice that impact reduced to less than 10% and maybe operate consistently at round about 5 to 7% of your CPU impacted. And the techniques that we use to achieve that is through sampling and using an exclusion list of certain libraries that you may not have control over.

Great. But with crash reporting, I'm just thinking, I used to work for a large company and they had a lot of on prem solutions for healthcare providers. So when something crashed they'd have to go get the logs.

Does crash reporting work?

So it can be. So it lives in the cloud and when something happens on one of those local installations, it would report that log to the central location.

Yes, indeed. So all crash reports hosted by Raygun and we've got a 60 day retention period in which you can come back and look at those crashes. That's the standard offering. There is another offering for crash reporting that will extend up to about 180 days. So your crash reports are all centralized and we've got granular permissions depending on who you want to access those type of crashes. We can manage that using the Reagan offering.

Perfect. So another thing I see people struggle with with performance is a lot of times when people think of performance, they think of like a performance test or a load test with multiple users on the system. But now people are creating like single page applications that are really JavaScript heavy on the front end.

You talked about the real user monitor.

From the client side. Are there any key metrics that you recommend people use or is it more intuitive where Raygun just automatically knows what to monitor and then it'll alert you if it finds an anomaly?

So it is a bit of a mixture of both. There are certain things that are intuitive for Raygun to monitor. So if you are in the case of static pages, so every single application comes from static page load and then the spa specifics start happening after that initial load. So we intuitively track that initial load and then we offer custom metrics that you can tap into and send information that you are interested in about your single page application. Unfortunately, there is no way for us to figure out what's intuitive in that sort of space because of the different flavors that are out there for single page applications. There's Knockout, there's React, there's angular, just to mention a few. So we can't sort of look into all of those JavaScript single page application technologies and come up with one size fits all way of monitoring them. But what we do offer is collection of custom metrics and then we can then display that in our front end based on how you have collected that information.

Cool. So you did mention something about exploratory monitoring, I think earlier.

I think a lot of times if.

Something doesn't break, it's hard to know that there might be an opportunity to make things better still. Is there any suggestions? You have to use some sort of exploratory monitoring to be able to locate maybe improvement opportunities within your application, even though it's not officially broken.

Yeah. So that's where APM shines for me. To give you an example, I've got an object relational mapper that used to be a flagship product for Raygun before they became Ray Gun. I think back in the day they used to be called Mindscape and they had this object relational mapper called Lightspeed. So we still use it internally within Raygun. And one of the observations that I have gotten out of APM is these queries that we term n +1 queries. For example, an object which is probably representing your aggregate root at the top of your graph and then it loads another object and it loads another object to map out that full relationship. You end up with thousands of tiny queries going down to the database to get just one value that you want to update. So APM sort of surfaces those things up and instead of perhaps wanting to update this value synchronously, I can then maybe change the architecture and follow a command query responsibility segregation approach where the update is a fire and forget it will happen at some point and system is eventually consistent at some point. So that is what I tend to use APM for. It's mainly for explorative monitoring. I don't know what I don't know. So let me put this tool in order to identify issues. Even though things are working and not failing, there will be an opportunity for improvement somewhere. And one of the other things that APM tends to suffice for me is very hot code paths. So again, I'll come back to the object relational mappers where if we have implemented specific methods to manipulate objects before they are loaded into memory, you'll find that every time we call that ORM to load the object, they are those methods that we don't want, not necessarily want to be executed at the time of loading. They are not concerned with anything that we are going to be updating. But because they are part and parcel of the architecture that we have put in place, they always execute. So that sort of thing, you wouldn't be able to know about it unless you've got something that can offer you an explorative monitoring ability.

Awesome. I love that approach.

I know apm, there's a lot of.

Tools on the market and I'm a big believer in there is not one tool that everyone should use. It depends on your team.

So someone's listening. What would make a team say what team is the type of team that.

Would make Raygun the perfect solution for them?

So what would make Raygun the perfect solution is, I think when you look at the holistic picture, if you are experiencing an issue in one area, you will be probably led to exploring other areas closely related to it. And I think this is where the Raygun suite of products sort of shines for me. I will give you an example of an issue occurring in the real user monitoring. When you drill down into a page, we've got this histogram that shows you the distribution of users and we sort of put them in buckets of quantiles. So that P99 quantile, which will be on the long tail of that histogram on the far right, when you click into that, it will give you a list of sessions. And when you drill down into those sessions, you'll get sort of page instances with a server portion. When you hit that server portion, it then drills down into your APM trace. So there may not be a one size fits all. But if you don't have a set of tools that will paint the whole picture for you, it then becomes difficult to jump from one system into the other into the other trying to diagnose those issues. Reagan has it already mapped and I think in some cases as well, without the correlation that Reagan offers you, it is virtually impossible for you to have that holistic picture of this is what happened in the front end? It took 90 seconds for this particular page to load because the server portion had the lion's share there and what was actually happening on the server. So I would say Raygun really shines when you after that holistic picture of end to end and if there were any crashes during that page load, you will be able again to drill into the session and view the pages with crashes and you can drill down into the crash reports and actually see the payload that was associated with that HTTP request coming your way and see the very line of code that caused the crash based on the data form data or whatever data was included in your request coming through to your server.

Okay Innocent, before we go, is there one piece of actionable advice you can give to someone to help them with their performance monitoring efforts and what's the best way to find contact you or.

Learn more about Raygun?

Thanks Joe for that question. I think the one actionable thing is keep your customers in mind. User experience ultimately defines the perception of your brand out there. And as a developer I take pride in the quality of my work and that pride is born out of knowing exactly what that user experience was. And I think Raygun is one of those tools that can give you that ability to gain that pride, to gain control of that narrative you want out there in the world about your brand.

Awesome and is the best way to find or contact you.

I am available on LinkedIn. People can Google or search for Innocent Bindura or they can send an email to innocentregan.com if they do, I'll be happy to ask for some favors on their behalf. I can't say what because I don't know what the powers that I will authorize and yeah, and they can always send any form of information through the chatbot on raygun.com thank you incent for.

Your performance Testing awesomeness. The links to everything value we covered in this episode. Head on over to testguild.com p and while you're there, make sure to click on the Try them both Today link under the Exclusive Sponsor section to learn all about SmartBear's two great performance testing.

Solutions, Load Ninja and Load UI Pro.

So that's it for this episode of the Test Guild Performance and Site Reliability Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack performance Testing awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Performance and Site Reliability Podcast. Head on over to testguild.com for full show notes. Amazing blog articles and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.