Automated Observability with David Wynn

Get ready to discover some of the most actionable DevOps techniques and tooling, including performance and reliability for some of the world's smartest engineers. Hey, I'm Joe Colantonio, host of the DevOps Toolchain podcast, and my goal is to help you create DevOps toolchain awesomeness. Hey, it's Joe. And welcome to another episode of the Testkill DevOps toolchain. And today we'll be talking with Dave Nguyen, all about observability, AI and observability, data pipelines, gaming, and probably a whole bunch more. This guy knows a lot of things. So David is the principal solution architect at Edge Delta and an enthusiast for all things technology. With a career spanning 15 years, including a pivotal role at Google Cloud for games, which is really cool. David has built up a deep understanding of solution architecture, observability and the transformative power of cloud services in the gaming industry, and far beyond which you can definitely want to see all the way to the end to hear all this. Don't want to miss it. Stick around.

Hey, if your app is slow, it could be worse than an error. It could be frustrating. And one thing I've Learned over my 25 years in the industry is that frustrated users don't last long. But since slow performance isn't sudden, it's hard for standard error monitoring tools to catch. That's why I think you should check out bugsnag, an all in one observability solution that has a way to automatically watch for these issues. Real user monitoring. It detects and reports real user performance data in real time so you can quickly identify lags. Plus you can get the context of where the lags are and how to fix them. Don't rely on frustrated user feedback. Find out for yourself. Go to bugsnag.com and try it for free. No credit card required. Check, check it out. Let me know what you think.

Hey David, welcome to the Guild.

Hey Joe, how's it going?

Awesome. Great to see you. I'm really excited to see your name come my way and I said, oh, I have to get him on the show. As I mentioned, the pre show, this is really, I assume it's a lot of testers listening to this, not necessarily familiar with some things like observability. So I thought we'd start off really easy. How would you explain what is observability? Maybe how is it different than monitoring that I think a lot of performance testers and testers may be more familiar with?

Yeah, it's a great question and I wouldn't be doing my job if I didn't say it depends. But if you go to our old friends Google Search, if you ask Bing these days, which is pretty smart, if you go to ChatGPT, whoever, you'll get a number of different answers. As to what is observability exactly? The way I like to think of it is observability is about making sense of a system based on the outputs that it creates. That's the long and short of it. It's not the most popular definition. If you go and you search the Internet for the most standard one, you'll probably get. Observability is the practice of collecting logs, metrics and traces in order to understand an environment, which is true, but I find one level beneath what is useful. Doing observability, which is in a practice, is about helping you understand what in the world is going on with this thing that usually has ended up in your lap that you didn't build and somehow are now responsible for. And you've got to figure out when something doesn't go right, what is going to happen next. So using all the tools at your disposal, whether they be dashboards, search graphs, et cetera, we gotta piece all of that together into something that helps us actually move forward and make sure that everything works all right.

So how do we piece that all together? I've been hearing a lot about something like opentelemetry. Is that how you piece it together? Is that just us, just something? One of the ways you can do that.

I would better categorize it as one of the ways that you could do that. So if we think about what they call the pillars of observability. So I'd start with logs, which would be just messages that are written to yourself inside of code that you could swap in some text for. It is exactly as standardized as that sounds. So if you've ever seen task failed successfully, or if you've ever seen any of these other somewhat cryptic messages with error codes that don't make any sense, that might be all you get in the logs. Or you might get something that's really useful that said, hey, I tried to do this and it didn't work. And here's all that I tried. Excellent. You get anything and everything in between, but it's a bit like reading 18 different devs journals in order to understand how things are working. Then there's metrics, which is the second piece. Metrics are just numbers. They are exactly as useful as just numbers and they are exactly as not useful as just numbers. Hopefully you put the right numbers on the right things and are counting the right stuff. Particularly in games, this can be one of these finicky things because not everything you want to measure it translates very well into being metricized. So if you want to measure how happy a gamer is or how challenged they feel, or how successful they feel about the last match, all you really can count is, did they play again and how long did they play for? And, you know, maybe some proxy for smashing the keyboard, if the USB keyboard is still connected or not. But it's thin, it doesn't give you everything, even though what it gives you is very precise. And then the last step is tracing, where essentially it tries to tie the two together. So you'll have this concept of a trace, where things underneath it spans are all grouped together in one concept of a thing. If you hit checkout, for example, on an E commerce website, it. Checkout is not one action, if we're being fair. Checkout means you got to make sure you're logged in, you got to check the inventory, you got to make sure it lines up with shipping, you got to make sure we register the payments and so on and so forth. And so you can combine that all together in a smaller, coherent way into a trace, which is what that is known as opentelemetry. We're bringing this back full circle, I promise, Joe, I didn't just veer off in left field. OpenTelemetry has been one of the most exciting efforts in recent years to, to actually try and standardize what should traces look like. The schema was the first thing they really started with. So as we move this data across the wire, how does it look? Because prior to OpenTelemetry, everyone was using their own formats and they all thought they were all better, and they were all proprietary, and they just. OpenTelemetry says great, if we all cohere around this standard, which also comes with an agent collector, where you can collect all this information from as well as SDKs and things. But standards are great because then we can all build together and make new and useful things. So all of that comes together to try and make that work. And OTEL as a standard is something that, where I work right now, Edge Delta, for example, we make it a core part of the protocol of everything that comes out of Edge Delta is in the OTEL standard. And what that means is you can make use of all of those tools that are OTEL compliant, whether they be proprietary or open source or any of that other stuff, because hooray for interchangeable Parts.

So I love that term, metricize. That's a cool term. Sounds like a thrash band. We'll probably talk about metrics later. But before we do, you got me thinking. So if someone doesn't have observability yet and that they've been hearing about observability because it's everywhere now, where do they start? Do they have to start with OpenTelemetry or they just go with their logs and then build off of that? Do you have like a roadmap you recommend or if you're going to go with observability, I would do this before you actually go down this road.

It's a great question. So it depends a little bit on where your development practice is inside your organization. So if they are very trendy guys and gals and they are very tech forward and they'll probably jump in with tracing because that's the thing that people talk about. But it's also a little bit like git was like 15 years ago whenever it was like, oh yeah, we're looking at Git, but we're still at svn. Like it's not. We're not actually there yet, obviously. So there's a lot of openness to debate. Where I would personally start is get everything on every machine in one place. This could be as simple as just rsyncing the logs to one machine so that you can grab them all at once. Congratulations, you're doing observability. It's a little bit shoestring, but you can do that. The next step up might be to use either a observability tool to do this, and there are plenty of good ones in the space. Whether you're talking about Datadog or Splunk's a little bit on the pricier end, or elasticsearch, or you could use Edge Delta as a core observability platform as well. Put everything in one place so that you can only. You don't have to go and bounce between clusters and you don't have to bounce between VMs whenever anything goes wrong. So that's step one and then two, figure out what's most important. Because most of the time you aren't going to need to view every nook and cranny of the application when something goes wrong. Some people may know this because they've worked with the system long enough. They're like, oh yeah, the brokerage service or whatever it is, that thing always acts up. Da da da da da da da. The more you can prove that with numbers and the more that you can validate that with data you collect. You can bring that back to the application team and make sure you work together to bring those types of things down. Sometimes there's inherent flakiness you can't get away with. Let's say you're depending on, I don't know, the uptime of ChatGPT, for example. It's like, all right, well, that's totally out of our control. We can only do so much about that. But generally there are things we can do to be more proactive, to move things forward. So collect all that data and start moving that forward.

So do you see people making mistakes where they make assumptions, where they think they know what's important? Do you recommend starting with the floodgates open so you can make sure you're not missing any correlations? Or is it, I guess, in general, someone should know how their system should work and what is important to them, rather than just, let's monitor everything for now.

I don't think people should necessarily know how their system worked because that just leads to everyone feeling bad about themselves all the time. The whole point of the system is that it changes. If it didn't change, the jobs would be a lot easier. That's true, but it's going to change every moment. So if we take a step back from that, I think starting in a targeted way tends to be a little bit more effective because if you start with either a target application or a service or just one piece of the puzzle, you can grow out the context that you need from there and add in the pieces that make the most sense. You could even just start with whatever last postmortem was and drive in and say, okay, we know this was a problem last time. How do we decorate this with enough information that we can make sure it doesn't happen again? Or if it do that, we take the MTTR down from an hour to 15 minutes or whatever the goal may be. But start small just because boiling the ocean is really hard. It takes a lot of energy. And also, people don't talk about how boring it is because mostly the ocean doesn't change and you're just tired at the end of it. So start with a small thing that you can win at.

Love it. So why observability? I've been hearing more and more about it maybe the past year or two years, maybe it's been around longer, but it just seems to be like on an uptick. Is it because we're going more cloud native when we're going on services and we don't have control of our systems anymore. So when we go, production is just a crazy, crazy ride because we don't know what's really happening or why. Why the uptick in observability if there is one?

Yeah, I don't know if I'm just getting older and nothing makes sense anymore. So there's a need for observability. But I think when we think about the trends and forces that are in the market, it's probably driven by microservices because on the one hand, microservices make things much easier. As the pattern takes hold and people develop on it more and more. The small, the micro part gets easier to understand and is great. The macro part of them coming together becomes. Starts to look more and more like distributed systems. All I know is I didn't take that course on purpose. And so now I've got a real problem. But I think that's what's driving the effort. Because you see with a lot of the big players and the small players, the rise of AI for observability is not just a trendy thing. But it gets directly at that notion I mentioned earlier of how do we make the most sense of this as quickly as possible. Because errors in general are long tail phenomenon. If it was the same thing that broke every time, we could fix it and then it wouldn't be the same thing that broke every time. But I. I've given talks before where I talk about the weirdest database error that I ever came across. So I was working on a timekeeping software with this custom middleware ETL kind of transport thing. And basically we figured out, oh no, the time records aren't showing up in Salesforce. David, go fix it. All right, I can do it. First we check in. It's like, all right, Salesforce up. That was. That was the easy one. Okay, we'll go to the other side. Applications up. Okay, that makes sense. And then. Oh, that's weird. Like, if we drill in enough points, we see that there's a database error. And I look at the database error and the database error says, write failed. You haven't paid us call support. And I was like, boss, Jim, what. What is this? And he went, oh. And he went. And like he had to go call somebody and like give him a credit card and something. But the. The short answer was we used a database sync service to get things up to Salesforce and the credit card had lapsed and so we had to make sure it was up to speed. Point being, raise your hand if you're Listening right now. And you've run into that error before because the odds are slim that you have. So every error in observability is this tiny little edge case of stuff that we didn't expect, because that's the nature of what it is, is unexpected and undefined behavior. So I think the need for observability has risen over time because that kind of stuff only grows the more interaction effects we have as we try and make it easier to develop things faster and faster.

So did that ever happen in an environment where you had observability or did you have to do the old fashioned way?

This was. What a great question. So I was close enough to the metal at that time that we didn't have a formal observability platform and I knew where to go. But the story of the moment to moment of that is that I had to check each individual system and when I didn't understand what it was, had to go and check with a different person every single time. So I could have absolutely cut that time well down in terms of the investigation if I had been able to actually have that all in one place and somewhere that I could go from. But this proprietary middleware tool didn't necessarily get everything off of the device. And then. So we were not set up for success. I should also add, I wasn't DevOps or SRE at the time. I was actually just in professional services and they were just busy so they asked for my help.

Right, right. Cool. Yeah. So then it's a good question then, like who uses it then? Is observability just for SREs? Is it just for production or do people now use it in staging or development to find issues before they check in code? Or is that totally separate? Totally not.

It's a good question. We've gone so far without saying shift left and I'm going to just say it once, which is to say that what used to be a solely production level phenomenon has started to shift left. Because the feedback loop that we assume application developers have and need is when it's very tight and very focused and we can accomplish it maybe with like unit testing and a couple of integration tests, and more or less we can get the right behavior from there. But the observability makes no assumptions about what is actually going on. So as we expand the notion of coordinating across each of these different services in order to verify the correct behavior, we have to turn to higher and higher level abstraction tools in order to make sure that they all work. And that even assumes that our test Harness and everything else works. So while the meat of the value, I would say, is delivered to teams that own the platforms and own the production runtimes, we are seeing an expansion in all different directions.

So another dumb question. How do you know what to look for when you don't know what's happening? Like in that database example, if you did have observability and say you had Edge Delta, how would that have helped?

That is if you could answer that question, Joe, then I think you and I should start something up right now. Because that is the golden question in observability. It really is.

The known knowns, the known unknowns, and then the unknown unknowns, right? The answer is usually because of the way that humans experience time and the way that it passes linearly, we can usually crystallize a time window to when something was working and when something wasn't working. And we can bisect that time window over and over to the point where we drill into something that is suspicious. Usually errors and things crop up when things go wrong. You don't usually get a system that swallows everything in a way that just productively works all the time. I haven't worked on that many rlang programs, so that's the only one that I can think of that may or may not be true. Supervisors everywhere. But the typical application likes to fail and fail fast. So it doesn't usually fail quietly. If you're in Java, it fails pages at a time. If you're, you know, in Apache, then you just have to look at the 400 code and hope that you know what went wrong. Or the 500 code and then you push it forward from there. But usually where there is smoke, there is fire. So if you don't know what you're looking for, you start by looking for the common stuff and then you drill in to the things that more and more might be smoke, but might. You didn't realize it was like crystallized smoke that was being decorated on some kind of food dish.

Love it. So then what is automated observability then? Is that a way that makes it easier to identify these issues? When there is a suspicion around something, does it bubble up? Insights I just have in my notes automated observability, I think just curious to know what that means too and how that helps.

Definitely. So automated observability is, if we think about the gradient of what we just discussed, there's the very manual. Pull everything together, know what you need, search for something and hope that you know what is going on. Edge Delta exists and other Vendors, I should say, have been in the space have been doing this as well. But one of the things that we've been pushing forward with is solving this problem of what can the tools do to present more and more information to people so that they don't have to do it all themselves. And in that sense, we're looking to automate the observability practice that goes the full gamut from what are the things that we should be ingesting into the system? Because if you're only ingesting 4 out of 20 log files that you should be grabbing, if you're only capturing four out of 400 metrics, if you're only capturing some traces, that's a huge problem with, with traces. This is not a tracing episode. I'm going to go ahead and plug in here for Joe. There's going to be a tracing episode one day, I'm sure, follow up, and I'm not going to be on it. So that's going to be a fun one. But tracing has its own fascinating discipline around sampling and things. But the nature of that is that you don't catch most of the samples, or you build some very sophisticated infrastructure to only catch the things you need. Which is to say, the more you can automate, the less. The quicker you can get to making sense of things and the less that you have to actually fiddle with by hand. Again, referencing that changing piece that we talked about. Because if you set everything up once and you decide to set it and forget it, that is great. If nothing changes, which almost certainly won't happen. So you gotta walk the balance of how do we capture everything to get it into the system? And also if things are changing all the time, a very common way to set up alerts. I can't believe it's been this long since I've set alerts in this conversation. But oftentimes a core function of an observability platform is to alert you when something goes wrong. And maybe you're okay with the occasional 500 or the occasional thrown function or uncaught exception or what have you, that could be okay. But maybe if it jumps above a certain number in a minute or in 15 seconds or something, it'll send out a message. Historically, those have had to be very specific and very much known in advance. So for Apache, for the 400 errors, I'm comfortable with 20 per 20 seconds. Great. But if you don't know what that magic number is, then it would be nice if your tool could figure out what does normal look like? Because maybe Django and MYSQL just don't get along for this release cycle. Fine, that's a baseline that we can handle. But if there is a jump in bad things, then we should be notified about that. And the way that we do it at edge Delta is we actually use a dictionary of things that are errors, crashes, timeouts, failures, various other things that of course, you can edit. Because it turns out it doesn't matter what platform or OS or application framework that you're using. All of those things are bad. And if we can figure out what a spike in bad things is, we can be pretty certain there's something uncertain going on in the environment that needs attention. That way, it's automated and you don't have to write a magic number for every single alert you want and still get that same notification to dive in and make sure the system's going okay.

So is it learning over time, then and then based on anomalies, using some sort of algorithm, machine learning? It knows, okay, based on, you know, the system was running fine last month. Now you're seeing these anomalies. We could tell from last month what is different. Here's the segment you should look at to put all your efforts into right now.

Yep, we do a week, but the question is very much what is normal for, you know, this set of seasonality. Oftentimes Saturday and Sunday look very different than the rest of the week. So that's a good, a good baseline to go off of. But. And you'd be surprised if the application is written correctly. We wouldn't expect to see a huge surge in bad stuff, even on things like Black Friday and holiday weekends, just because for the most part, if it works, the ratios should all still be the same. Like that should be fine. But if the ratios get out of whack, and that could be because, you know, your queue got filled up and so stuff started getting dropped on the floor, or you need more load balancers spun up because there's traffic that's just being turned away. All that kind of stuff, that's usually what'll show up.

All right, so you mentioned alerts as an important function of an observability platform. I got me thinking, what are some other things you think if someone's looking for an observability platform, they should have these core functions to help you.

So probably the most basic one that people do is for search. So there is. Look, grep's always going to be with us. It's a great tool. Everyone should know it. Shout out to rip GREP also for making some amazing strides and improvements, which is a whole other thing. But generally those tools are devoted to one machine. And so if we get everything in one place, we're going to need to figure out how to search through it all, ideally in a way where we can just say with a click or something, hey, I want everything from this application space or this namespace or that has this pattern associated with it. Cool. So search is a core function that you see in a lot of both new and old observability platforms. The next thing you want to look at is metrics. Making sure that there are dashboards and metrics that the system supports so that you can graph the right stuff, throw it up on a wall in the office is the most common thing that people will do. But if you want to, you know, make sure that a monitor is devoted to that all that's all that's viable stuff. And then you got to figure out what your needs are from there. If tracing is something that team is going into, great. But if your team isn't using tracing, maybe you don't need a tool that necessarily supports that. If you are trying to work with specific applications, some of them might be better supported by some observability platforms than others. Most of them try to be general purpose, but like all competition in a general purpose place, some people have decided to lean more in one direction than the other. And you can also take a look and see if there are any other additions on top. So one of the things that Edge Delta does, for example, and I'm doing my best to make sure this is not a pitch, but just as an example. All right, as an example, we do something that we call patterns. Basically we take a giant block of log files and we will condense them into things that look kind of like glob syntax a little bit. So we'll star out the timestamp and port numbers and stuff that's highly variable but not necessarily instructive as to what it is. That way you end up with this essentially a pre sorted haystack of logs. So you've got the really common stuff at the top. Hey, you know, two hundreds are coming in at a million requests per 15 minutes or whatever. It is great. That was probably expected and I didn't need to see that a million times. But what that does is also highlight some of the things that are unusual and you can go diving into that unknown, unknown exploration a lot more easily. If the only tip off you may have is that some something ran out of memory or Some service had a higher than expected response time or someone just logged into a machine that should have been entirely terraform deployed all of those types of things. Even in the best case, you're not going to alert on every single thing, but that helps point you in the right direction. And that's just one of Edge Delta's differentiators. All of the others have their different strengths and weaknesses as well. I could list several, but I feel like it would be unfair if I listed any of them in particular. They all do good stuff. You should go do your homework and check them out.

Awesome. So you know I did mention in the intro that you did work for Google's Cloud for Games. I just want to squeeze that in because I think that's kind of interesting. Could you talk a little bit more about did you learn anything about maybe scalability and resilience? I know like these multiplayer games, they have to be really resilient and scalable and did observability play a role in that at all or how did that help at all with your experience there?

Oh absolutely. So. So I was at Sumo Logic before I joined Google and I had that observability perspective before I went in to the organization. The state of game architecture is really interesting from this basis because working for a cloud, hyperscaler, whatever you want to call it, game companies were doing cloud before the cloud was a thing. So a lot of their architectural patterns somewhat reflect this in the sense that they didn't bother with any observability patterns beyond just a couple of basic things that they could pull out and they assumed that the rest were good enough. So to give you a couple of examples, if you are playing a Most mobile games are set up exactly the same way as websites are. They've got a three tier architecture in the back end. And the reason that it seems kind of slow when you tap anything sometimes, especially if you're on mobile data, is that it's making requests and responses back to the server. But what that gives them is very granular understanding of how users are engaging with the game. What do they do, what are the actions they take. And almost like a clickstream they can understand what are the behavior flows that lead to different pieces. There are very sophisticated mechanisms for observability and for understanding behavior analytics inside of games. In that context, as you shift over to the ones that are cooler, I guess the ones that tend to involve a controller or maybe involve some sort of Fortnite or some sort of first person shooter, third person shooter, match based games, we like to call them that observability infrastructure actually leans much more on the more monitoring side. Where you asked this question earlier and I didn't quite get to it. My interpretation is that observability is about making sense of everything that's going on. Monitoring is much more. Is everything on in a much more basic way. So I think of like Nagios and some of the other tools that give you like those fleet diagrams and give you the green, yellow, reds and everything else like that. Great stuff for sure, but not necessarily insightful into the application all the time without a lot of bending and pushing into a different direction. That's sort of the line for me. And when you look at match games in the gaming industry, they typically just run a binary on a VM that they expose to the Internet and tell two clients to connect to which that's it. And so it's not producing most telemetry because it's got to re render the world or it's got to recalculate the state of the world 60 frames a second. And that's not a lot of time to even write to files for the most part. So what you would tend to see are metrics that they would occasionally pull out and in the thinnest way possible they would try and interpret those into good or bad. So it's a state where performance is such a demand that they don't quite have the headroom for where observability is right now. But I'm looking forward to the day where we continue to progress the technology and the state of the art to the point where that does become available to them in a way that would be much richer than we have today. Instead they're doing all these crazy hacks that almost like save the whole TCP stream and they'll try and play back all the button presses back and forth.

Oh cool. Right, right, right.

And do it that way. But that's. No one else needs to do that. They just.

Nice. So in the pre show you asked me what was my favorite game. I had a really lame answer for video games. I said I haven't played video games since Age of Empires and Duke Nukem. But I didn't find out what your favorite game is, Dave. What have you been playing lately? Or you're oh man. Obsessed over.

I have been playing a couple of things lately. But honestly the one that's really struck my heart lately is Octopath Traveler 2. That is they call a Japanese style role playing game where you get to collect a band of Merry people of all different skills and backgrounds and stuff and bring them together to fight. Usually you have to, you know, oh, no, like the town sheep has gotten away. Can you go bring it back? And by the end, you're fighting the concept of evil as an abstract purple cloud. And it's just glorious and beautiful. And I probably play covers of the music every morning while I'm getting my breakfast ready just because it's so pretty.

I can't believe I didn't ask this. AI, how is AI? AI is everywhere. How does AI play into observability? Do we no longer need to even know many things? Do we just have the AI run it and be able to bubble up? Hey, this is the issue.

Oh, man. I have very strong opinions about this, Joe, so I'm going to do my best to just stay off my soapbox, per se and answer the question.

Okay.

I should give a slight frame of reference, which is I've been in games for a long time and what we called AI back in the day was a series of if else statements and just what do you want to happen? If it was artificial and looked like it was intelligent enough that counted. So. So to crystallize what we're talking about here, I think we're talking about the current wave of LLMs and various other things that are going on. What I expect to see, everyone is trying to make the most use of the new technology, edge Delta included, to try and figure out how can we make the most of this and this happening. What is really interesting is that we don't, from a research theoretical perspective, we don't really understand how LLMs think or what exactly is the limit of what they know. And all we know is we stuffed the entirety of the Internet or various subsections of the Internet into it and we can sort of talk to it, which is amazing, given that it's basically a really fancy lookup table. What you're seeing integrated into a lot of different products is what is the small way that we can do AI to move this forward? And I think that's because there's not really a guarantee with the way that LLMs are laid out that we can just tell it like the Oracle, say, Oracle, something is wrong in the system. Please tell me what to do next. And sometimes the marketing material may make it sound like that, but it doesn't think.

Exactly.

That's not what it does. It will show you some amalgamation of the data it's seen in the past related to the data that you're presenting to it. So what you can do is in the small. You can have it make small recommendations. Edge Delta, we just released something we call on Call AI, where we are doing this for each atom of information. If something turns out to be anomalous, if it's a surge in bad things, we can give you a list of things that you might want to try. Powered by AI. This is kind of intended to solve a 3M problem. So if you get paged at 3am and something is broken and you're like, oh, my God, I've still got to figure out the brightness on this monitor, like, if that's the level of what's going on. Having a quick little checklist of things that may or may not be the root cause of the problem is very useful to walk through, because then you can just be like, no, yes. Oh, yeah. It's probably that. And it can kind of jog you forward. But I don't believe unless there is a significant surge in the technology moving forward, that we're going to trust all the thinking to AI. Instead, what I think you're going to do, what we're going to see is we're going to see it bubble up more and more. Useful suggestions. I mean, it's called generative AI. It's good at suggesting things. How do we make it easier and easier to perform the practice of observability? And that's generally by making things visible and trying to highlight the things that can be done next.

Awesome. Okay, David, before we go, is there one piece of actual advice you could give to someone to help them with their DevOps observability efforts? And what's the best way to find contact or learn more about Edge Delta?

Yes, I think the best way to level up that's most straightforward is go with your gut, listen to your feelings, find the thing that you don't want to deal with, and decorate it a little bit more so that it's a little bit less painful to drive forward with, because there's a reason that you don't want to deal with it. Either it's complicated or it involves people in a way that can be sometimes contentious or prioritization. That can be contentious. Decorate that with a little bit of extra information and start moving in that direction to where we understand more completely what's happening about this, both when it's good and when it's bad, because that will be the stepping stone that brings you more into a wider and wider scope. We can help you do that at Edge Delta. If you go to edgedelta.com or if you being edged delta whatever the kids do these days. Then you can install a free trial and take a look and see how things are going. Whether it's for kubernetes or on your VMs doesn't matter. Then we'd be glad to help you on that journey if you like.

Remember, latency is the silent killer of your app. Don't rely on frustrated user feedback. You can know exactly what's happening and how to fix it with bug snag from SmartBear. See it for yourself. Go to Bugsnag.com and try for free. No credit card required. Check it out, let me know what you think.

And for links of everything of value we covered in this demo DevOps toolchain show, head on over to testguild.com P13E and while you're there, make sure to click on the SmartBear link and learn all about SmartBear's awesome solutions to give you the visibility you need to deliver great software. That's smartbear.com so that's it for this episode of the DevOps Toolchain Show. I'm Joe. My mission is to help you succeed, creating end to end full stack DevOps toolchain awesomeness. As always, test everything and keep the good. Cheers. Hey, thanks again for listening. If you're not already part of our awesome community of 27,000 of the smartest testers, DevOps and automation professionals in the world, we'd love to have you join the fam@test guild.com and if you're in the DevOps automation software testing space or your test tool provider and want to offer real world value that can improve the skills or solve a problem for the Guild community, I'd love to hear from you. Head on over to test guild.info and let's make it happen.