More Than Just Noise Reduction with Guy Fighel

Welcome to the Test Guild Performance and Site Reliability Podcast where we all get together to learn more about performance testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Performance and Site Reliability Podcast. Today we'll be speaking with Guy Figgle and Michael Olson, all about more than just noise reduction using AIOps. Guy is the GM for AIOps and GVP of Product Engineering at New Relic. He was previously the CTO and co founder of Signify, a startup acquired by new relic in February 2019 that basically integrates machine learning to increase uptime, strengthen systems, and to make day to day operations of an SRE and DevOps team and make day to day life easier for SRE and DevOps teams. Guy has more than 20 years of experience innovating and leading the development of highly scalable global software solutions. Throughout the time, he's worked with international companies such as Tango and Vonage and oversees the development of more than 20 patents. We'll also be joined today with Michael Olson, who is the Director of Product marketing. So if you really want to get up to speed on the latest and greatest in monitoring, especially with AIOps, you don't want to miss this episode. Check it out.

Let's face it, performance testing is tough. Traditional load testing tools create scripts that are bogged down in data. It's hard to read and require a lot of work, even for simple playback. Load Ninja cuts out the dirty work by using AI to inspect and debug your code almost instantly. No coding necessary. No changing of hands or relays between teams either. Give it a shot. It's free and easy to try. Head on over to testguild.com smartbear and click on the link to learn more. Hey Guy, Michael, welcome to the Guild.

Thanks for having us Joe.

Great.

Thank you.

Awesome. Great to have both of you. I'm really excited about a new initiative I think New Relic came out with with AIOps. It's actually something I've been hearing more and more about. So I guess before we get into all that at a high level, how has DevOps changed, in your opinion, to get us to the point where we have to coin a new word called AIOps?

Yeah, to me, DevOps is really all about improving the way that teams work in order to be able to ship software faster, do so more frequently with greater reliability. And what that means, I think is being able to respond quickly when problems occur that may impact customer experience or jeopardize service level objectives to the rest of the business and particularly as we're seeing software teams modernizing their applications and infrastructure as they're adopting cloud native technology, there are just a lot more things that on call teams need to be able to monitor and react to. There's a wider surface area, there's more software changes happening, there's more operational data being emitted, there's more alerts, plus more pressure to be able to find and fix incidents faster as well as prevent them occurring from occurring. And I think that's really led to the genesis of AIOps technology over the course of the past few years.

Great. And I guess like I said, I just started hearing about the term AIOps. Michael or Guy, any thoughts around what is AIOps? Can you just explain it a little bit more?

Sure, yeah. I think one of the things that we've heard a consistent area of feedback is there's certainly a lot of hype around AI and machine learning more broadly in the software delivery space. And software teams, particularly DevOps and SRE teams, are really looking for clarity around what is AIOps, what are the actual use cases around how I can start to utilize this as part of my on call workflow. And over the course of the past few years we've seen this new category of technology that's emerged that is focused on putting AI and machine learning in the hands of DevOps and SRE teams so they can prevent more incidents as well as respond to them faster. I think Gartner originally coined the term AIOps to mean artificial intelligence for IT operations to describe this space. And AIOps technology is really all about using AI and ML to analyze the increasing amount of operational data that's being emitted across our software systems in order to be able to predict possible problems, determine root cause, and then ultimately drive action and automation to fix the problem problem.

I'll be honest, I'm a little older, so I been performance testing for a long time. We used to use a lot of monitoring solutions, one of them being New Relic. How is this different from apm? Is this a totally different set of functionality that a different team would use than say a performance engineer?

Yeah, I think they're very complementary. And at New Relic, one of the things we believe is that AIOps capabilities are a key requirement for observability. If you think about observability, it's all about providing that connected real time view of all of your data in one place so that teams can pinpoint issues faster so they can understand not just when an issue occurs, but what caused it and why, and then ultimately get context. That's needed to be able to quickly analyze and proactively take action on that data. And so we see AIOps really augmenting the value that software teams can get from traditional monitoring solutions by providing that layer of intelligence to be able to analyze and take action on operational data in service of being able to detect, diagnose and resolve production incidents a lot faster.

Yeah. And if I need to think about AIOps and how does it differ from an APM then or an infrastructure solution, think about APM as just another type of signal gets consumed by a higher level solution that implements different types of algorithms to make sense of that signal.

Perfect. Perfect. Guy, I know you had a startup earlier and you really did focus in on AI machine learning type of software, which is kind of right now it seems to be a buzzword, but it seems like you were here early on. So what got you involved in this in this direction? I guess.

Yeah, you know, I mean I start to tinker with those technologies. I mean, I think was six, seven years ago when, you know, I remember back in those days people have talked about Hadoop, you know, that's was kind of the big thing that will solve all our problems. But yeah, I think, you know, AI technologies have been with us for many, many, many years already and started to implement and read about IBM research that you know, were put back in the 60s. And so I think that the changes that we were seeing in the past few years is cloud compute. And it allowed us all to start using greater compute power just in our hands, using you know, spin up some instances and cloud resources and then taking the academia knowledge and articles and start to think about how we can implement some of those concepts in a very practical way. And the way to test it was really using cloud technologies to prove that ideas can actually work in the field and kind of being much more specific to particular use cases. And that's exactly what we've done back back in those days signify and even before that.

So it seems like this we'll get into the solution, but it seems aimed at the sre. So what makes it about an SRE role or activities that make it prime to be used AI machine learning type features? Is it because it has a lot of log files that need to be correlated and using machine learning helps you resolve that quicker and faster.

Again, it's just an acronym, you know, stands for Site Reliability Engineer and Google were the first one to be really good at that and to actually set that as a dedicated role or thinking about Site Reliability Engineer. It's an engineer, it's a software engineer that has a vast knowledge and background in systems and infrastructure and networking. And that engineer is capable of coding and just, you know, taking an end to end approach. So AIOP solutions or different types of observability solutions are pretty well defined for the SRE role. Just because during the job to be done as a site reliability engineer, you own reliability, you own uptime. So all of the things that can help you to understand and to act on maintaining that uptime and system level availability is going to be something that you as an SRE would want to use. So whether it's log files or metrics or events or incidents or any other type of information, you know, it's just a tool in the arsenal of tools that as a SAT reliability engineer you will, you would want to use. And the great thing about starting to incorporate AI and machine learning techniques is to just be able to do all of the analysis faster. And it's not to replace you, it's really to augment you. So we look at AIOps solution as a great augmentation to the site reliability engineer knowledge and capabilities just to become much faster and kind of have that superpower, if you want to think about it that way, to just operate.

Yeah, I love that. I think I was watching another interview with you guy and you mentioned applied intelligence as a term that you like better and I definitely agree for sure.

Yeah, I definitely like applied intelligence much, much more than AIOps. I think, you know, AIOps, again it's a very specific term industry now adopted but coined by Gartner. So you know, analyst and applied intelligence is really the essence of applying real day to day techniques to your specific area and having that understanding both on the area that you're trying to solve. Being like professional and expert user and you know, don't stay at the theoretical level and just take specific techniques that works. Doesn't need to be just neural network or deep learning, you know, and all the, all the hype, it can just be a pure good old statistical approach or a clustering model and apply those techniques to the data that you really understand.

Great. So I also have a functional testing background and the biggest issue there is a lot of times tests will fail, but they'll fail for an unknown reason. So they add a lot of noise and people start ignoring it. Also with the security scans you get a lot of false positives, a lot of noise, so people start ignoring it. I guess it's the same thing with sre. Could you just talk a little bit about is this an issue in this area as well. And maybe how a best practice of how to filter out that noise.

That's exactly how it all started, you know, and we call that alert fatigue, right? You start to get those incidents and those alerts, those pages over and over and over. Sometimes only the first one is relevant. Sometimes you get tens of them or 20 of them and it actually points to the same specific problem. But if you think about it, and when you have a cascading failure effect, then you know, your database might have a problem that can impact your middle tier application or a specific microservice layer that will impact and that will slow down your end user application. All of those services will start firing and screaming at you with problems. And when in fact there's only one specific problem you need to go in and fix, which is your database table or you know, your disk space or. And sometimes it's more complex than that, obviously, but definitely an area of focus for applied intelligence, which is being smarter around correlating and aggregating those alerts and really filtering out all of the noise. And noisy alert can be the same alert going over and over and over, which, you know, just doesn't make sense. You just need to know about it one time. Or you might want to know a specific Alert is happening 10 times. But instead of getting the actual alert 10 times, just getting a single alert and have a counter, hey, I've seen that thing happening in the past five minutes for 10 times. And that will be useful information. And sometimes you will see other slightly different alerts that can just be grouped as the same issue. You know, kind of, you have different types of symptoms and then the issue is going to be what is the scenario that you're handling with, right? Like you can have a cough, you can have a sneeze, you can have maybe a little bit of fever, and it will all point into the fact that you might have a flu or something like that. Right? So all of those techniques tend to do exactly that.

Great. So I think we're all familiar now with what the problem is and now what's the solution? And I believe New Relic just came out with a brand new solution. I believe it's been in beta for maybe six months called New Relic AI. Could you just tell us a little bit more about what is New Relic AI?

Yeah. So Joe, earlier this week we announced the general availability release of New Relic AI, which is a suite of AIOps capabilities available in New Relic that are all about enabling DevOps and SRE teams to reduce alert noise as well. As proactively detect and resolve incidents faster. So we're really, as Guy mentioned, focused on augmenting and arming on call engineers with AI and machine learning capabilities to simplify and accelerate their incident response process. I would say in three ways. One, by helping our customers find potential problems or detect anomalies in real time before they turn into full blown incidents. By then reducing alert noise and helping teams focus on issues that matter most by grouping and correlating related incidents. And then third, by giving our customers intelligence and enrichment to be able to quickly diagnose issues so they can understand the problem, get closer to probable root cause, and then ultimately take actions to solve the problem problem.

Great. And you actually have a really good I'll have this in the show notes, I believe it's like an ebook. And you have a framework on the three step framework for AIOps best practices. And you mentioned three areas already. I just want to dive in a little deeper. Maybe how AIOps or new relic AI can help with each one. So the first one was detect. So in the lifecycle, how does new relic AI help with the detect piece?

Yeah, so the way that we approach the initial detection phase, you know that's really the initial stage of the incident response process. Right. Is how do I detect a potential problem before customers notice or before it turns into a full blown incident. And so the way that we approach that is by making it really easy for DevOps SRE teams to be able to monitor their applications, their services, all of the time series metrics associated with those based on SRE golden signals like latency and traffic and errors and saturation and, and when a threshold is exceeded, if for example, my web services is having a latency issue that exceeds a given threshold or my database service is exceeding a particular saturation threshold, we can actually then trigger a real time failure warning and anomaly directly to the teams that are best equipped to diagnose and take action on that. One of the core design principles that we really believe in is around meeting our customers where they are. And we know that a lot of on call teams are collaborating on potential problems and on service reliability directly within collaboration tools like Slack. So one of the ways we're providing anomaly detection is by surfacing those real time failure notifications directly within Slack channels that our customers choose so they can quickly jump in, collaborate on incidents or potential problems, diagnose the root cause and take action to address it right away.

Nice. And so it helps you understand the issue as well. Does it keep historic data? So over time you could say this team or your team, you always make this, you cause this issue or there's something in your process that's always causing this particular issue and here's how to resolve it going forward. Don't know if that makes any sense, but does it have like proactive type of historical recommendation that bubbles up saying hey, over the past six months based on this data, we recommend you do this to avoid this in the future?

Yeah, we guy, I would love for you to jump in and speak to this in more detail, but we do give our customers some, you know, some analytics to be able to drill in and view past anomaly data that might be related to the current, you know, problem at hand. Really? To provide clues to help users diagnose more quickly.

Yes. And you know that that feature is something that we definitely offered back in the, in the signify days. As we integrate more and more capabilities into the new Relic experience, we'll bring even more. So you know, stay tuned for the next releases to come. But we are definitely have the technology to do exactly, exactly that and that's very helpful for, for our users. So. Absolutely.

So what does one need to get started, say using a solution like new Relic AI? Is there a heavy integration or how does that work?

Yeah, we've really focused on making it easy for our customers to be able to get up and running with AIOps capabilities with minimal configuration required. And having been in and around this space for a while, this is a source of frustration that we've heard from a lot of on call teams is some of the other approaches to AIOps can be pretty cumbersome to get up and running with. They can require a pretty significant amount of time for initial configuration, training, onboarding, as well as requiring weeks or oftentimes months for the system to learn from and study your data and train the models before it provides any value. And so that was one of the design patterns that we really looked at as we built this solution is how do we make it faster and easier for on call engineers to get up and running with AIOps without requiring a significant amount of time for configuration or training the system. So we make it pretty easy for customers to be able to connect data sources from either existing incident management tools that are emitting alerts or incidents or events like a PagerDuty or a ServiceNow or OpsGenie or VictorOps as well as monitoring tools that our customers might be using that are also emitting events and alert data like a new Relic or a Splunk or Prometheus or Graphics or Amazon Cloudwatch we make it pretty simple in a few clicks to be able to integrate those as data sources. We then normalize all that data. We identify relationships across alerts and events and correlate and group that data into one more actionable issue and then enrich those incidents with context. You know, things like which golden signals is the issue related to so I can get closer to root cause and understand why the issue occurred. We can tell you what components of your environment are related to to help isolate the problem faster. Was it related to a database service or web service that started failing, for example, in addition to suggesting responders or people and teams who may be best equipped to take action to go solve the problem. So we have customers that based on industry standard knowledge and global decisions built into new relic AI that are able to get up and running and start seeing as much as a 50% or greater reduction in alert noise, you know, right away without requiring weeks or months to train the system.

Great. That was actually my next question. What I love about vendors speaking with vendors is you usually deal with a lot of enterprise customers, a lot of crazy environments. So you may have assumptions of how something's going to work, but you don't know until you release it in the wild. So any other case studies or any feedback you've gotten since this has been in beta for a bit from customers like you mentioned, maybe 50% reduction. Any other success stories like that?

Yeah, and I want to just notice maybe that, you know, I think, I think it would be important to mention that those types of techniques and the technologies that we are using have been proven for the past four years already. Right. So yes, I know we've been in beta and then we've released in ga. But as mentioned before, this is a lot of the technologies are based on the acquisition as you mentioned, and we've worked with customers for a while. Right. So definitely have a lot of experience and it's a very mature technology that we're working with. I can definitely mention another interesting use case and that's where we have plugged proactive detection to one of the customers that we've worked with. We really surfaced some real problems in their production environment that they even know about. And you know, that customer is a pretty complex customer enterprise environment. They have many, many different microservices and not to all of those microservices. They had tuned alerting mechanisms and alerting systems, especially because they had multiple different types of notification systems and different workflows. And it is really hard to maintain all of those sets of Logics. If you think about it now, when we plug in an automated anomaly detection model with proactive detection to their time series data, and we were watching the entire type of time series metrics, then we've actually detected anomalies in a particular workflow which they didn't really know about. And when they got the notification they said, well yeah, that looks a really legit problem. They went in there and were able to change JVM tuning on one of their Java applications to tune the memory in that container and just result in a much faster performance to the end user eventually. So it's a small example that type of an enterprise, when it comes to a downtime to their user, it's like, you know, resulted they've estimated it in about $2 million to their revenue in one hour. So that's pretty, pretty massive.

So Guy, you bring up a great point. This technology you were part of Signify and this is you were acquired by New Relic. So this is a mature technology. So say your existing customers from Signify that are familiar with the technology, is there anything different? What's new on the New Relic AI piece that they may not know about then?

So we've actually changed completely the experience. We brought the technology and integrated it completely from scratch into New Relic 1, which is the New Relic latest observability platform. So now you have access to metrics and logs and traces and events as well as third party incident management providers all in the same latest look and feel experience. And it is all connected and programmable, meaning you have the exposure to all of the APIs and you can start using some of the data that the system detect and you can set your own dashboards on top of that or your own applications. It's a fully programmable platform, that's the NR1 observability platform. And then on top of that again, the combination of being able to correlate time series anomalies in a pretty easy way, that you don't have to configure machine learning models, you just need to specify to New Relic what are the applications that you care about. So from a data science problem, now it's a business problem, what are the things that I really care about? And then it's up to New Relic to detect what problems we find in those applications and bringing all of that into the ability to correlate to existing incidents, alerts, other events and reach those events. All of that is a completely new experience. As well as new capabilities, we've made a lot of improvements on some of our Machine learning models and the correlation techniques. We've exposed our correlation engines all the way to the end users, so it's easier for them to now set their own logic and infuse the system with their own logic again, you know, speaking directly to not replacing those SREs, but engaging with them and empowering them. So there are lots of improvements and with this release.

Great. So like I said, I like to keep this at 30 minutes. We're right up on 29 minutes. So I have one last question I always ask everyone, but is there anything you definitely wanted me to ask or you wanted me to touch on before I end it with the last question?

Not on my side. I mean, Micah, anything that we're missing?

Yeah, same here. Nothing that jumps to mind, Joe.

Okay, cool. So let me ask the last question. Okay, Guy and Michael, before we go, is there one piece of actual advice you can give someone to help them with the AIOps SRE efforts? And what's the best way to find contact you or learn more about New Relic AI?

Yeah, I'll start. I think having been around the DevOps space and the software delivery space for the last six, seven years, I think it's really interesting how the role of a traditional systems engineer has evolved and we've seen automation, for example, become the new normal. And I think if you look back six or seven years ago, there was a real fear around automation technology around does this replace what humans do? And I think what we found and what's been borne out over time is that automation doesn't replace human work. It takes away the mundane toil that enables humans to focus on higher value, more creative work. So Guy spoke to this earlier, but we've seen the role of the traditional systems engineer evolve into more of an SRE or DevOps engineering type of role, where the role becomes providing common platforms and tooling that enable the organization to build and deploy software faster and more frequently. And I think we're still in the early days of using AI and applied intelligence and machine learning technology as part of our our incident response process. And I think the takeaway I would leave on call teams with is AI and machine learning doesn't replace humans. It augments them with power tools to be able to more quickly analyze, take action on their data as part of a goal of being able to simplify and accelerate an incident response process and ultimately detect and understand and solve production issues faster. As far as your last question, if you're interested in learning more, feel free to go to newrelic.com and check us out and definitely happy to make ourselves available if we can answer any follow up questions.

Yeah, to me, you know my biggest suggestions for SREs or anyone who want to start with applied intelligence and start to look into what are the capabilities and how do I implement those into my day to day as an sre. You know, take take the first Coursera course on like generic machine learning. Not so much so you can implement those algorithms, but so you can start open your mind into what's even possible. You know that it's not science fiction, it's not like you know a virtual robot to take your job. It's about some, a lot of math and algorithms that can really help you to be much more effective. So start taking the first ML course using Coursera. Maybe think a little bit just with deep learning. Again, not so much to know how to implement those algorithms, that's not really important. But just try to start and understand what's even possible. And once you understand what's possible, like I'm sure that each one of you as SREs out there will know, you know and will have some great ideas how you can implement some of those third party vendors into your environments. So you know, don't be afraid of the math and the algorithms and you know, just try to think or with that, you know, that's how I started. I don't have any PhD in ML or AI and still so I encourage any one of you to just start with that.

Thank you guy and Michael for your SRE awesomeness. For links to everything of value we covered in this episode, head on over to testguild.com P31. And while you're there, make sure to click on the Try Today link under the Exclusive Sponsor section to learn all about SmartBear's awesome performance testing solution. Load Ninja and if you haven't already, make sure to head on over to perfguild.com and register for the online conference dedicated 100% to helping you succeed with creating Full Stack performance testing awesomeness. You don't want to miss this year's event. Head on over to perfguild.com and register today. So that's it for this episode of the Test Guild Performance and Site Reliability Podcast. I'm Joe and my mission is to help you succeed with creating full stat and testing awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Performance and Site Reliability Podcast. Head on over to testguild.com for full show notes, amazing blog articles and online testing conferences. Don't forget to subscribe to the guild to continue your testing journey.