How to Track Test Related Events (New Relic) with Rodrigo Martin

Get ready to discover some of the most actionable DevOps techniques and tooling, including performance and reliability with some of the world's smartest engineers. Hey, I'm Joe Colantonio, host of the DevOps Toolchain podcast, and my goal is to help you create DevOps toolchain awesomeness. Hey, do you want to know how to actually track test related events or do you even know what that is? Well, if not, you're in for a special treat because we have Rodrigo Martin joining us, who is a principal software engineer and test architect at New Relic, where he leverages his two decades of experience in software testing to enhance testing tools and processes. I'm really excited about this episode. Don't have a lot on information on this, so I think you're really going to learn a lot. You don't want to miss it. Check it out. Hey, Rodrigo. Welcome to the Guild.

Hi, Sho. Thanks for having me.

Great to have you. I guess before we get into it, it's a pretty short bio, so is there anything in your bio that I missed that you want the Guild to know more about?

No, I think that's pretty much it. I think it covers really well my background.

Awesome. Awesome. All right. I guess before we dive in, I came across an article you wrote about how to actually track test related events. I guess at a high level. How do you even explain what does that mean? How to track test related events?

Yes. So basically this lies in the world of observability. As you know, observability is being in the news for the last couple of years, long couple of years. And usually when we talk about observability, we think on tracking stuff that happens in production. Right. Having observability in our production systems that are usually really complex, a lot of microservices, micro front, et cetera. But sometimes we neglect the other aspect of the software delivery lifecycle and that includes testing. So basically the idea of this article is to let people know how can they track events related to their testing processes and tools and that part of the pipeline. So they can, they can have better observability and they can debug issues faster.

Nice. So obviously you need to have some sort of observability in place, I guess for this to work. What needs to be in place in order to actually get started though?

Yes. So basically you need to have a place where you can store these events. In the case of my article, I work at New Relic, so I use New Relic, but you could use any other tool where you. You can track your Events. Right. Where you can store your events related to testing or production, et cetera, you need to actually instrument your code to actually send those events and define exactly what you would like to track and which types of tests are you most interested in. What type of metrics do you really like to track, coverage, etc. Right. So it's kind of defining what you want to track. Having a place where you can send these events and eventually query that platform and get your answers.

Cool, I guess. What are some of the key challenges someone would is trying to solve? Like why use test related events? Like what is the benefit to them?

Yes. So one of the main benefits is that you will have a way to ask questions about the context of your system whenever those tests were executed. Right. Usually when you have hundreds of test cases, thousands of cases or even more, it's really hard to pinpoint the exact issue or where does those tests fail within the pipelines. So having this instrumented allow you to ask better questions and be able to pinpoint exactly where you need to debug those failures. But not only that, you can also track performance of your tests. That's really important because we want our pipelines to be as fast as possible. You can also track coverage or other metrics that are more related to governance. Let's say if you're working with multiple teams or multitude of teams and services and you would like to have a centralized way to see things, those are probably the most important use cases.

All right, so say I have a test suite of selenium tests, a playwright test. It doesn't matter, Cypress, whatever it is, do I have to instrument those tests or can I run them in staging or development? And how do I know that if this test fails, how do I correlate it with the test event?

Yeah. So basically you will need to have a way via a plugin or via your test runner or your pipeline to send the events that you are most interested. Let's say you work with playwright. That's what we work with. Right. Every time you run your tests, there's usually an opportunity to use an after hook. Right. So after each test is finished, you can collect your events. Right. And send this to your observability platform. That's one way of doing it. Maybe in the pipeline you are interested in sending events on your test coverage reports, then you will instrument at that point in the pipeline. Right. So there are several points where you can actually instrument. And you mentioned environments. Environment will be also one of the dimensions or metrics that you will track. Right. Because Maybe you have tests that run on staging, on development, even in production, that will be another dimension. So you can then filter out what happened when I run this test in staging versus maybe what happened when I run the same one in development. Right. Those are different dimensions that you can track with instrumentation.

All right, so if this is running in the background, it's collecting these test related events. If I'm running, like I say, a functional test, I go in, I validate, I enter a patient's name and it's able to do something. Say that's the test. Will this uncover things that maybe the test doesn't? I don't know if that makes sense. Test actually isn't testing like say I did that, the test passed, but underneath it triggered some sort of error that I wouldn't have seen in that particular flow. Yeah.

So basically I think you have two ways of measuring that. One is measuring coverage. Right. On the one hand, you can track not only your percentage of coverage, but also which lines are you covering and which lines are being missed. Right. So you have a way to correlate which files are being covered with those tests or not. But at the same time, for every test that you execute, one of the dimensions that can be stored is the pull request number. Let's say for when this, where this test was executed, in which environment, and then you can start correlating. Okay. These tests start failing in this pull request build. And then you can have with your platform, you can start asking questions around what happened during that time. Right. There was maybe change, was there was a deployment there, there was an alert fire in a database, let's say, right. You can start asking questions with your platform around that test execution.

So say tests have been running fine for a month. Someone pushes what they think is a small change, the test stop failing, you can correlate to say, hey, the change you just pushed is the cause of this failure. And you could roll it back to test it.

Yeah. And of course, if this is a change that your own team is doing, this is just a matter of a regular CI CD pipeline. This is my test, it's failing. That's good, right? But if something starts failing and it wasn't change that you introduced or your team introduced, if you have this data in your observability platform, like the latest deployment deployment markers, et cetera, then you can start correlating those things and you will be able to debug more easily those situations.

All right. Because I used to work with the team, it was eight teams, sprint teams, all checking in code. And if something broke, you never knew which one actually broke the build. So you didn't know which one to roll back. So it didn't make any sense. So this way it sounds like I was in that situation. I could say, team Sopranos, you checked in code, roll it back, that broke it.

Yeah, correct. Because you will have a time window as well. Right. And you can ask within this time window, what were the latest changes that were deployed to the environment? From a service perspective, you could also eventually think about what infrastructure changes happen during that time frame. Right. More on the. Any new version of a database, any certificate renewal, whatever you can ask. You can do this with observability because the idea is that you can investigate and understand or try to understand these unknown, unknown situations. Right, nice.

So, all right, so that's a good point. Say once again, performance for some reason spikes up, but you don't know why. You could say, okay, what changed in the system? And it may be something you didn't even think about this, the new hardware piece maybe and you could trace it to say, well, ever since that new hardware piece was added, the performance started spiking. So that is most likely the cause.

Yeah, that will be one, one use case. And if we go one step beyond, if you have events on of your testing tools and processes and executions, you can also have alerts. Right. And you can, even before a test starts failing, like actually failing, you could start seeing, okay, there's a degradation on test execution time. Or you could set alerts on these tests are consuming way more memory than before. Right. Maybe 2x memory than before. Right. So you can potentially act on those things before even you get a defect.

So that you can run this in production as well. Or is it just pre production?

We, we run it in pre production only but potentially it can be executed in production with any sort of monitoring, synthetic monitoring, et cetera. It will be exactly the same concept. Right. Like we are in the article, we are talking about instrumenting tests that are executed usually pre merge, but you can do exactly the same thing with your synthetic monitors and yeah, that would work as well.

Nice. So I assume there's multiple test related events you can track. Are there any like key metrics or key events that you always add? Is it up to the user to go understand what each one does? Like, what do you recommend?

Yeah, so again I like to split this in kind of two categories or the two main use cases that we are using right now are for tracking code coverage. And basically this is pretty straightforward. There's a Service, Right. So usually you will have, in the list of events, you will have the service name, you will have the release tag. That is the version of the release, the tag of the release that, that you are testing with this coverage. Right. The percentage of coverage measured in any way the team thinks it's better. We are measuring the amount by lines, right. By statements. So we are measuring the statements that were covered, which ones are missed, the basic and also the owning team. Right. Which team is the owner of the service and of these tests. Right. So basically with only that information, we can start getting trends and create dashboards for teams. I work on a platform team, so we have kind of templates of dashboards that we provide to teams so they can have at a glance a dashboard of their current coverage, not only per team, but also per group, because we have groups that are form of multiple teams and those multiple teams have multiple services. So you can start seeing how you can drill down and get like an aggregated set of coverage events. So that's on the one side, on the coverage, but now on the test execution side, basically the most important things that we track are talking about Playwright, let's say in which browser the tests were executed, in which environment the pull request, the outcome. This is really important because Playwright provides whenever you, Whenever you wrap up your test execution, you have an outcome field that basically says if the status was expected, unexpected or flaky. Right. And that's really important. So we track also the. If the test was flaky or not, of course, the status, any error messages that the test produce, any error stack that we can get from the, from the report as well, error values. And also the owning team, I think, and environment. I don't remember if I mentioned that one, but those are the main dimensions.

All right, so you mentioned flakiness. So can you measure flakiness over time? A lot of times people, oh, just rerun the test to pass, but they never check on the outlier, like, all right, why does it fail every now and then? Can this actually uncover those type of things that drive you nuts, like looking through log files, actually say, okay, for some reason every first of the month or something, it's failing because we have this job running in the background.

Yes, yes, totally. Because you can use whatever visualization you have to kind of make graphics of this data. And basically if for every test that you run, you know the status, if it's flaky or not, you can create sort of time series graph. Graph, right. And say, okay, in the last month I can see A trend here, everything was flat. But as you mentioned, maybe every 29th of February things start blowing up. Then you can pinpoint these sort of things. But not only issues, you can also. One thing that I like doing with this data is to measure if the actions that the teams are taking to reduce flakiness are actually working. Right. And you can see that really easily with the trend. If flakiness is slowly going down, then whatever actions the teams are taking are being effective. Right. It's a way to measure those efforts.

Oh, that's interesting. So you could measure maybe a team, their flakiness is going down, what are they doing differently and then using that to educate the other teams. Hey, you want to implement this because this is what's helping this other team.

Yeah, totally. And I think that's really, really helpful to see these things at the team level. Of course you have all the data, right? Potentially you can have all the data. So you can take more high level decisions on your testing processes and see average and media and stuff, et cetera. That's useful. But I think this is most useful for team specific efforts, right. Where you can pinpoint exactly your situation, your number of tests, your flakiness. You can even say what is my spec that is failing the most right now, which one is the most flaky, et cetera. Right. You can be as curious as you want and that usually goes gives you better results.

Right. So how hard is it to interact with the data? Is it like a gen prompt and you just ask it questions and it's like, how does that work?

Yeah. So basically the events that we have at New Relic are being sent to the New Relic database. It's NRDB and basically you use NerQL, which is new Relic query language, to kind of make your queries on the events and that would be it. We do have like a beta ON prompting with AI via NerQL, so you can use that as well. But it's pretty straightforward and in the article you can see that it's pretty similar to SQL. So if you have a basic understanding of SQL, you can start getting useful information.

Now you mentioned it alerts the right team if something goes wrong. How does it know that? Is it through tagging of the actual test script or is it annotating something within the your dashboard or the data?

Yeah. So basically when you create your alert, you create your alerts based in whatever events you have in your platform. Right. In this case we are populating test related events so each team can create their own alerts. Based on those test related events. And remember that we are also tracking the owning team on each test that is being executed or on each coverage event that we are tracking. So that's really easy to do. Right? So you have your team ID there so you can start crafting alerts on again, whatever you think it's worth it to alert.

So I've got to follow up. You mentioned something about code coverage, so like I said, I'm totally ignorant with this. So are you able to say like I ran this test suite and it only touched this amount of features, so I have a gap of 60% of my applications not being covered or 70% or the high risk areas are not being covered. Does it give that detail as well?

If you instrument it, you can get that detail. We are still not doing it right now. This is pretty straightforward. We are just measuring the percentage of a given measure in Master. Like what's the code coverage percentage from a unit testing perspective? What's the end to end testing coverage from? From an end to end perspective. But you have that data available, right? Because let's say in Playwright, if you see the report, you can see which are the files that are being executed with your tests, right? What's the coverage per file? Let's say. So you could potentially send those dimensions as well to your platform and then you could have that, that information as well.

A lot of times I worked at a company, they'd have a hotfix and you're like, all right, we have 2,000 tests. I don't know what test covers this hotfix. So in theory I could query and say, hey, which did this touch? The functions or the features. I thought that I know was checked in as coverage.

Yep, totally. This again, this all depends on you having the ability to read the report of the tool that you're using, extract the data that is important for you to track as events and send it over to that platform.

How hard is it to do that then? Does it all depend on the tooling that you're using?

Yeah, I think it's. It really depends. For instance, for Playwright, we are just parsing out the events on the JavaScript and TypeScript etc. And sending it over via our own CLI. But basically the workflow is you get the events, the data of the events that you need either by parsing reports or if you have an API or if this is something that can be handled with your test runner, there are multiple ways of getting that data. Then it's basically calling an API to send that event information to the platform that you're using and that will be it. Right. Then you need to send this payload in a compressed form, et cetera. Those are technical details. Details. But that's basically the workflow. So if you work in a platform team, this is way easier because potentially you have a way or a tooling that everyone uses. So you can implement these things at the platform level and you don't need to reinvent the wheel for every team that needs to instrument the data. And yeah, that's basically how we do it right. At the platform level.

Cool. So I could use this as like a final check or a final gate that if these test events occur, these errors, then don't put it into production.

Potentially you could, but we are not doing it right now. We are doing the, you know, the, the plain old cicd. Does it fail it or does not fail? Right. Even if a test is flaky, the test, the build passes. Right. It's okay, it's flaky, it failed, but then it passes. So that's it. So we are not doing this for as a gate, but rather as debugging tool, debugging information and for improving the performance of our pipelines.

Gotcha. How do you know what's meaningful though, like security? You obviously have a security scanner and it flags like a thousand things. You're like, how do you know when this actually fails or gives you the data or what's really important?

How do you know what's really important to check or to debug?

Yeah. Does it ever give multiple points of data? How do you know where to check? Like where, where to focus in on?

Well, I think it depends on the question that you're trying to answer. Right. That's always your indicator. Let's say for instance, we started to have some reports of teams that their end to end test started to execute way and way more slow. Right. They were slower and slower and slower. So we as a platform team had that question in mind on okay, we have all this data, how can we check exactly what's the reason of this sudden spike in performance related issues? So we had a chance to, okay, let's go and check in the, in the machines that we are executing these tests, what's the average duration of the test? I don't, I don't remember if I mentioned duration. But duration is another really important metric. Right. So you can have an average on per browser type or a medium per browser type and on which machine and then you correlate that with the actual build execution duration and Each of your pipeline stages, duration. Basically we did this, we have all this information and we found out that the machines that we were using to execute test actually were kind of old AWS machines. And we could upgrade those machines for the same cost to format to machines that are kind of the same cost but way better processors. We did that change. We measured that change again with the same events. Right. Did the change measure the test execution Duration? Saw a 30% improvement in performance and we were able to fix this situation. Right, so that's just an example. It depends on what type of problem you're trying to solve.

All right, so I hate this term roi, but it sounds like you can actually use it to justify purchases. Like hey, if we buy this new hardware, we'll get a savings of this because the performance will be that type deal.

Correct. And you can do it really easily because you have all the information aggregated and you can query right away and do all the time series, etc. Otherwise you could do potentially the same ROI analysis, but it will take you way more time.

Nice. So is there anything like teams get wrong with implementing test events tracking? Do you see like, like there any best practices for doing this or are there things that like you probably shouldn't use it for or shouldn't do?

No, basically I think the main things that should be tracked are the ones that we discussed. And then it's a matter of again which type of metrics are you most interested in? And as more context you add to these events is better. Right. It's kind of hard because you kind of need to predict the future. Like what type of information will I need in case I have a situation in the future. We discussed the most important generic ones, but as you mentioned, maybe having in the events the files that are being exercised by your test could be potentially something that can be implemented. So there are no best practices rather than what would be important for answering your questions.

Gotcha. Is this a new feature of New Relic or has this been around for a while?

Actually it's let's say a use case of using New Relic. Right. Because it's. If you abstract this outside of testing, it's just sending events to New Relic and using the platform. So basically it's a use case of hey, this is usually related to or focus always in production, but I'm trying to give this more like a testing flavor to it.

All right, so New Relic, I assume everyone knows what it is, but just in case folks that don't know what is New Relic.

Yeah, so New Relic is A platform for teams that enable observability basically is a way to instrument your applications, your infrastructure, your cloud services use single platform that will help you with observability.

Very cool. All right, so is there anything on the roadmap you see of you have all this data I assume, is there AI machine learning at play or things you could do to stop predicting things without you having to know what to even ask? Is that crazy?

It's not crazy. I don't have it right now in the roadmap for, for my group, for my team, but it's something that we are definitely researching and seeing. What are the what, what's the best way to to implement gen AI for for these kind of processes?

Okay Rodrigo, before we go, is there one piece of actual advice you can give to someone to help them with their DevOps observability efforts and what's the best way to find or contact you?

Yeah, sure. So my piece of advice is don't forget what happens before things are deployed to production. That's really. There are a lot of opportunities there to implement observability and make your CI CD pipelines and your developer experiences way richer and way way better. If you instrument observability there as well. If you use observability on those parts of the software delivery lifecycle, you can basically Find me on LinkedIn. My profile is Martin Rodrigo there? Does I have a GitHub profile? I don't remember the profile there, but I'm sure that you will get the link afterwards in your notes.

I have a link for that down below, so definitely check it out. Yeah. And for links of everything of value we covered in this DevOps Toolchain show, head on over to test guild.com forward/p169 so that's it for this episode of the DevOps Toolchain Show. I'm Joe. My mission is to help you succeed creating end to end full stack DevOps toolchain awesomeness. As always, test everything and keep the good. Cheers. Hey, thank you for tuning in. It's incredible to connect with close to 400,000 followers across all our platforms and over 40,000 email subscribers who are at the forefront of automation testing endeavor. If you haven't yet, join our vibrant community@testguild.com where you become part of our elite circle driving innovation, software testing and automation. And if you're a tool provider or have a service looking to empower our guild with solutions that elevate skills and tackle real world challenges, we're excited to Collaborate, visit testguild.info to explore how we can create transformative experiences together. Let's push the boundaries of what we can achieve.

With lutes and lyres, the bards began their song A tune of knowledge, a melody of code through the air it spread like wildfire through the land Guiding testers showing the secrets to behold.