Software Testing and Observability with Abby Bangser

Welcome to the Test Guild Automation Podcast where we all get together to learn more about automation and software testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Automation Podcast. And today I'm going to be sharing with you an episode I actually did on my performance podcast about a month back. A lot of people actually don't know I have a performance and site reliability podcast, so I wanted to share this episode on this podcast as well because I think this is a hot topic in all areas of testing, especially with performance testing. But it also is definitely going to apply to you if you're doing any sort of type of automation that is observability and testing. So grab your headphones because today you're in for a special treat. We speak with Abby all about observability and testing. How I keep hearing about observability. I saw a session she did for Future of testing and that's the only time I ever understood what the heck observability was. So you definitely want to hear this whole episode if you don't know. Abby is an engineer with a keen interest in working on products where colleagues are the users. Abby brings the techniques of analyzing and testing customer facing products to tools like delivery pipelines and logging. That's to generate really clear feedback and greater value to the end user which you'll hear in this episode for sure. I'm sure. Currently Abby is a site reliability engineer at Duffel where they are redefining travel. You don't want to miss this episode. Check it out. The Test Guild Automation Podcast is sponsored by the fantastic folks at Sauce Labs. Their cloud based test platform helps ensure your favorite mobile apps and websites work flawlessly on every browser, operating system and device. Get a free trial. Visit testguild.com.

And click on the exclusive sponsor section to try it for free for 14 days. Check it out.

Hey Abby, welcome to the Guild.

Hey Joe, thanks for having me.

Awesome to have you. You've been on my wish list for a while. So great to have time with you here today. Before we get into though, Abby, is there anything I missed in your bio that you want the Guild to know more about?

No, I think you covered a lot I guess given the still remote nature of the world. Calling out. I also help run CoffeeOps, the London edition if you're interested in joining a meetup that is more interactive than the average. It's not about a speaker. We do lean coffee discussions about everything. Kind of dev and ops. Therefore DevOps and yeah, covering all that it's really nice. It's run on London time, the one I do, but it's actually global and there's lots of sessions around the world. Yeah. And it's a great thing to get involved with.

How does someone get involved with that? Is it just meetup.com or something?

Yeah, if you go to coffeeops.comorg I might have to check that for you. But yeah, if you just go to the website for CoffeeOps, you will see where all the different sessions are. And a lot of them are meetup.com some are just email addresses that you can kind of ping the host for, but mine is on meetup. And yeah, just sign up and join.

Nice. And we'll have a link to that in the show notes. So if anyone wants to know, you definitely go to the show notes of this episode and you'll be able to get there as well. All right. So the whole show's about observability. But I mean, I've heard so many definitions of what observability is, and my eyes kind of like start going in back of my head. I don't understand it. You had a really great definition. Like I said, you did a session at future of testing for aplitools. So I guess to start off with how do you define what is observability?

Yeah, I think observability is another kind of word that people can get so caught up in the new hot phrase that they lose the intention behind it. And so with the intention of observability, it's about being able to understand your systems in particular, without making changes to the systems at the time that you need that information. So you want to be able to ask new questions of the data of the telemetry on your system without needing to go and add new logger lines to your application in the middle of it being in production and things like that. So, yeah, just getting insights into how your users are interacting with your application and how it's behaving in the real world.

Great. And I think sometimes people listening to the show might be a little bit new to site reliability engineering, trying to get more people involved. So you use a term called telemetry. What is telemetry?

That's a great point. So telemetry is basically just data that comes from your system. So the most common types of telemetry that you would run into is probably logging at number one. Almost everybody has seen a log line at some point if they've written code or tested code. The next most common or popular is Metrics. And this is where you start seeing those graphs of numbers over time. And another, the third most popular is tracing. And that's where you get an idea of what is doing the work in your system. So the easiest way to describe it is the waterfall graph in like a devtools in your browser that shows the loading of all the different pieces of that website, the HTML, the css, the different images. It shows you what took time while that page was loading. That's a trace. And you can do that for your back end systems as well. And those are the most common types of telemetry.

Awesome. We're actually going to dive into those three types of telemetry in a minute. But before we get into that, a little more high level is I've been hearing more and more about observability maybe the past two years. I don't know why. So why observability? Is it because more and more people actually testing in production or doing a shift right where they're monitoring production now they're like hey, wait a minute, we have all this cool customer real data we can work on to make our process even better. I'm just assuming that's where it came from. But why we hear more about observability? I guess the question, I mean the.

Snarky answer is that there's more tools selling products that have observability. If I'm honest, that's a part of it, right? It's a snowball effect of when it starts to become popular. There's ways to monetize it and there are products that are doing that. So you're going to hear more about it from those product teams and from those engineers. But why it's a snowball that builds from something like why is it that it's become popular enough to have a market to have products that people want to sell associated with it? And I think it's because systems are just getting more and more complex. There was a move a few now probably over 10 years ago, but to see that the complexity of a monolith application, so an application that's written all as one piece and deployed onto a single server, then replicated to multiple servers to serve the amount of traffic that you need, was starting to get really complex to build developer teams around trying to have 100, 200 developers all working on the same code base was getting hard. So one way to look at this was to break that up into multiple services, microservices being one example, but just multiple services of whatever size. But all of A sudden you think that things are going to be better, you think that complexity of the monolith is going to go away. But in reality, you just move the complexity and you can no longer have a situation where you can put debugger lines into your code and find any problem you run into in the past in a single application, whatever size it might be. You can put a breakpoint, you can run your application and you can see what's going on. Now what's happening is that we're making those requests across the wire between services. The hardest part is trying to figure out where the problem is, because once you've identified which service is having an issue now you can use your breakpoints, you can use all of our usual debugging techniques, but with the distributed systems, you don't even know where that problem's coming from yet. And that's where observability can step in and help give you a bit of insight into where you should be looking for your problems and what's causing bottlenecks, because it may not be the most obvious place in your system.

Nice. So someone may be asking, I think you mentioned debugging, but what techniques are people familiar with that actually rely on observability or could be enhanced by observability?

Yeah, I think I mentioned three popular types of telemetry with logging and monitoring, or metrics and monitor tracing. And all of those are pieces of the observability puzzle. But observability is more than just doing more logging or doing more metrics or doing more traces. It's thinking about the shape of your data and thinking about the user experience of needing to understand the big picture, all the way down to the nitty gritty details, and do so in a, a very kind of user friendly way. Because I think traditionally our tools can give us all these answers. I think the majority of engineers that have a hard time with the buzzword observability are frustrated because they're like, I already do this, I already have this data, I already get this information. Why are people making money off of this? Why are there. There's all this talk about it. And the difference is that the peak engineers, the engineers that are already doing this all the time, they've gotten so used to the kind of rough edges of the tooling that they think this is normal, this is what we can do. But onboarding new people into their system, experienced people or not, and especially onboarding new engineers who haven't used these tools before, watching someone try and navigate from the metric tool which tells you you've had a spike in 500s. Figuring out how to align that within your logging tool, which tells you which service is sending those 500s and the details of the requ and the IP addresses that are making those requests and all those things, and then connecting that data to your tracing tool. To be able to dive deep into a single example. Watching someone try and piece that puzzle together across tools is really painful. And so observability is about shaping data and in such a way that you can use tools that allow you to have those big conversations of, hey, we've seen a spike in 500 errors occurring, or hey, we've seen an increase in latency overall, and immediately start digging down within the same tool, within the same mind space. And when you realize that your kind of path of investigation is not quite right, this isn't what you're looking for. Being able to come back up to the start and have all that happen smoothly, that's what kind of your ideal state is for observability. But it's always a journey there. It's not something you can just do and stop.

Awesome. So I don't know if this is the same thing. I'm an old time performance engineer and Loadrunner came out with this plugin where not only would it say, hey, this response took 30 seconds, but it could actually drill down and say, it's the SQL statement that's taken up, you know, 30 milliseconds of this, whatever it was, and you could say, oh, okay, it was able to pinpoint the issue. It almost sounds like the same. Is that the same when you're talking about all this data? It's not just data, but it's also, it helps you know at a high level what's going on. But you can drill down to see the cause of what caused the initial bad point. A bad observability point.

Yeah, I think that's a really good example. Right. Because you had this tool that would tell you where latency issues were occurring. You'd have this tool that would say, this request is having a hard time at scale. This request is having a hard time period. And traditionally, before this new addition to Loadrunner came out, you had to know how to line that up with another tool in your toolbox to try and figure out why or what or where should you invest your energy to improve it. LoadRunner comes out and says, that's not nice. That's not a great experience. We need to do better. And now they've given you, within their tool a way to do that deep dive, but without removing the benefit of that high level conversation. Because you need both. You need to be able to abstract away some of that complexity to get a big picture view, but you also need to be able to investigate when you do find things, because that's your mean time to recovery. That's how you make things better for your users.

Absolutely. Now I'm trying to think. I saw a slide in your presentation that I saw once again from applitools on something about exploratory testing or exploring observability. Does that help you keep it at a high level to see things? Maybe they're not so close in the weeds that you can come up and go, wait a minute, I can see this other picture going on at a high level if I take it up a notch.

Yeah. I think the analogy there that I was making is that with monitoring, it's a little bit more like test automation. Hugely valuable, hugely valuable to be able to say, I know that there is something that is expected of my system or something, if something happens, that it would be a bad thing. Right. So either in the good way or in the negative way, I know what to expect and I would like to know that that continues to behave that way. I don't want to have to check this myself every day or every minute or whatever. I want to automate that. Monitoring is like that. You can set thresholds, you can set expectations, and when things occur that shouldn't, or when things become, you know, cross a threshold that they shouldn't, you get alerted to it. That's fantastic. But what monitoring doesn't do so great a job of is allow you to kind of quickly.

Ask new questions and quickly dive into new things. And so I compare monitoring to automated testing and I compare observability to exploratory testing. Both are hugely valuable. Both are going to continue to be important parts of our toolbox, important parts of our experience as engineers. But we have to value both of them in the right space. And so with observability, it's a. What you want from the data and you want from the tools is a way to be able to ask questions without feeling like it's a big investment to ask a new question. If you can imagine as a test engineer, when you're working in a system that takes ages to set up the test data ages, you're like, all right, what is my. What is it that I can get done? And once that test data gets dirty or that environment breaks, you're just like, table flip. I'M out until the next time. And so everything feels expensive. You have to kind of batch up your questions in such a way that they feel worth the energy of setting up that test environment, setting up that data. And we don't want that. We want it to be that you had an inkling, you had a wondering, and you want to just go into your data and ask that question right now with no overhead. And that's what observability is trying to build into your system. And we can do that with a lot of our existing tools. But some of the new tools add some sugar on top to make that a bit nicer for you.

We've mentioned this three times already.

You mentioned logs, metrics and traces. So one question you get probably asked a lot is where do I get data from observability? And I think the answer might be logs, metrics and traces. Am I wrong? Any other sources to get information or data for observability?

I think this is a yes and situation. So you're not wrong. Logs, metrics and traces are great telemetry sources, great ways to get observability into your system. But the and part of this is realizing that when you think about these three individual types of telemetry, you often can very quickly jump to which tools are which logs. Oh, I use Splunk and Cabana Metrics. Oh, I use Prometheus Traces. Oh, I use Jaeger. Whatever the tools are. Datadog, Whatever the tools are, right, you can kind of quickly map that type of telemetry to that tool that you use it for. And what you'll probably notice is at least two of the three, if not all three, are using different tools or the tools you're using are great for one, but not another. And so what we're hoping to do with observability conversation is we're hoping to discuss reshaping what we expect from telemetry so that we can get the outputs, the behaviors of those three unique tools and those three unique types of telemetry, but shaped in a way that you can get all three from one data type. And so the data type that you'll hear a lot about with observability is it's called events. And events are at their simplest, they're just structured logs. They're just JSON logs that have keys and value pairs. So they have a key of ID and a value of 1, 2, 3, a key of what server it was on, and the name of the server, we used to have it named after music. So, you know, doom was one of our servers, you know, those kinds of things. So, yeah, you'd have this key value set of information, message error code, all the things you think about from a log line in a JSON format, which is quite common for logging today. The thing that makes an event really interesting and unique in comparison to logging is that it also includes a few additional fields that come from the tracing world. So it includes a field called duration, which indicates that this is instead of a point in time log line, it's actually kind of a bucketed, you know, contextual piece of information. So it's something that actually can time how long it took in your system. It also, coming from the tracing world, brings in a trace ID and other IDs that help you piece together that puzzle of like, where did this occur in comparison to other requests in your system before and after and parent of this or child of that. And so by taking those pieces of data from the tracing world and taking the structured aspect of the logging world, you now are able to run visualizations over the top of these events that mimic all three telemetry types. You can query it like you can query logs. It has all that detail, it has all those sentences of messages, it can even have a stack trace in it, right, Like a log line. You can run aggregates over it. You can see your error rate, you can see your latency rate, things like that over time, because you've got how long it took right there in the event. You can also create that waterfall graph because it's got that locational awareness to other log lines, to other events to understand if it's a child, parent, you know, how that all pieces together. And so, yeah, so by focusing on. So you asked originally, is the telemetry logs, metrics and traces, is that how you get observability? And the answer is definitely yes. But also, can you start to shift your awareness of those three different types into a single type, which would give you that ability to explore across different visualizations a bit easier. And that would be events.

All right, so this is a very newbie question. I am a newbie here. So how do you get to events, though? Is observability a technique? Is it a tool that takes all your data types and then it does this data massage, puts it into JSON, puts it in some sort of dashboard where you're then able to query to get whatever you're looking for for your correlations to see what's causing issues. How does it work?

Yeah, no, it's a great point. So honestly, it's using a lot of the same techniques you do with logging. I said it's like logging, and it really is. So there are logging tools out there that can help you produce JSON based logs instead of string based logs. So historically, a log line is kind of like space deliminated and you'll have like a timestamp and then a space, and then you'll have some magical incarnation of letters and numbers that mean something. They'll mean, you know, an error code or something. And then there's another space and then another inclination about something else like a server ID or something, and then another space. And in order to understand the pieces of that log line, you have to run some sort of a regex over the top of it and start to break that apart into its pieces. That is, I would say, widely understood as a less than productive way of doing logging and that most people at this point are trying to use structured logs, JSON where possible. And so there's lots of tools you can use that are language specific, that are just libraries you add to your code base and you can then do like logger add field, and you can just add the field with a key and a value set rather than needing to worry about that regex. Same thing goes for these events you can use. The most kind of popular one to use right now is open telemetry. It's growing rapidly and I think it's hit 1.0 release just this year. And that is going to be the library that's getting investment moving forward. So if you're looking at using events in the future, I would suggest looking at that library first for your specific language.

Nice. So who typically uses all this data? If I'm a software tester and I get an issue on the front end, will this allow me then to say, a lot of times you have to look at server logs, you have to look at all this other data. Does this also allow me to give the developer, hey, I got this front end issue. And not only did I get this, but it seems like I, when I did my observability, I found this key event that occurred at the same time.

Yeah, this would be the same thing. And it actually is a really good example of where that relationship between the test engineers and the software engineers can make things better for everyone. So if the test engineers have identified a problem and they're unable to track it down through the observability data they have through the telemetry data that they have, that might indicate that we aren't exposing data that's useful to us. And so actually the software engineer, the first thing they do might be to start adding information to the system output before they ever even try and fix it so that they could see what the problem is. And that would indicate to you that on the spectrum of how observable your system is, you are not at the far most observable that you could be. And you're going to improve it now. And that's a part of the process. Right. But in the initial description of observability I gave at the beginning, the point is you should be able to ask questions of your system without having to add more telemetry in. And so it's a good indicator of how observable your system is with how frequently are you needing to add those additional events, additional fields to your logs, additional metrics to investigate things, and how frequently are you able to investigate just with what you already have that can tell you kind of where you are on the spectrum.

Nice. I guess another question someone could ask on this is how is observability different than monitoring?

Yeah, so it's in the same ballpark. So it's a part of knowing what's going on in our systems. As I was saying a bit before, I think monitoring is more about known known. It's more about how do you track what you already know is good or bad about your system and confirm that it stays put, stays as it is. Observability is about unknowns. It's about how do you find out when there's. That person just keeps calling your customer service and keeps telling you that there's a problem and you have no idea why, because you don't see it showing up anywhere else in your monitoring, anywhere else in your metrics. How do you dive in to find out that it's very specific because Brazil doesn't have daylight savings at the same time as everyone else. And so all of a sudden they're off by an hour and that's causing the problem. Which is a real example from something I've run into before. Right. So how do you find out that it has to do with a very specific phone type? How do you find out it has these really kind of unique identifiers that require you to be able to break down your data by these really unique bits easily. Can't do that with monitoring. You can do that with some forms of logging and even more so with events and observability.

So when someone's developing software, do they have to make their application observable like they have to make their application testable.

Exactly. Yep. They have to think about what are the things that matter here? What are the unique. The key things that we talk about is what are the unique bits that could help us debug this in the future. So are there any unique user IDs that are unique to this specific area of the code base that you want to make sure are available to you if something were to go wrong? Are there any.

Flags that you're using? So feature flags is a really good example if you want to know which feature flags are being used and being hit when you're a part of debugging systems. And so, yeah, I think you have to think about what is it that's important to you to be able and unique about this, to be able to debug later.

Great. And also what I thought was interesting about your session was that you had something about observability even applies to work away from code. I think a lot of people think it's just a coding activity. I don't know if you remember this session, but do you remember what you meant by work away from code? It works away from code observability.

I'm going to pretend like I remember everything I've ever said.

No, I think what I was meaning, what that comes to mind now is all I can say for sure is that having observability into your processes also provides a ton of benefit. So having observability into the delivery process that you have, having observability into that feedback loop of. You start to develop a story, you go to some testing phase, you're getting kind of feedback from the testing phase. You're going through the pipeline, your continuous integration and continuous delivery pipelines. How do those work? How long do they take? Where are the bottlenecks? Where are you finding problems? All of those things can give you observability into your process, and that is the power of being able to improve it. And so I think that's what I was probably talking about.

Awesome. So I think a question that came up during that event as well was, how do you know when you have enough observability has occurred?

I think the annoying answer is you never have enough. It's just. It's a journey always. Right. But I think that you know where you're able to debug things without immediately thinking, oh, I'll go add a logger line. I think when your default action is, oh, let me go put a logger line in that bit of the code to go find out what's happening, which I've been on teams recently where that is the first reaction to an issue in a certain part of the code base that tells us that that part of the code base is a bit of a black box to us right now. And so that would indicate to me that we have a ways to go to improve our observability into it. In contrast, when you're in a situation where you can ask really deep questions and the first reaction is to actually ask that of your data and not just muse about it in a room, I think that that's where you, where you're really good on the observability front. You trust that the information will be there for you to ask that question. And you no longer are needing to assume you're no longer in a situation where you go, most likely what happened was instead you're like, look at this right here. This is the thing that, you know, created that latency or created that error case. And I think that's you're tending trending towards a happier place with observability.

Okay, Abby, before we go, is there one piece of actionable advice you can give to someone to help them with their SRE observability efforts and what's the best way to find and contact you?

Yeah, so actionable advice, I think that be okay with small steps forward. I think being understanding that this isn't a flick the switch kind of a journey. This is something that you will be working on forevermore. As depressing as that sounds, that should also be liberating because it means you don't have to succeed in it. Day one. Start giving yourself the groundwork to be successful. If you don't already have structured logs, get structured logs. If you have structured logs, but you don't have timing data or tracing data associated with your logs, associate that data with your logs. If you don't already have a user interface that allows you to query that in unique ways, that you need to have a think about what the tool is that you need for that build from wherever you are because it is just going to be a journey and there's a lot of actions that you can take. I think your point about building observability in was just so key that I might just reiterate it in that often testers feel like this is very out of their realm and it's something they can't really get involved with. But I think it's actually the opposite as being someone who's not writing the code. You are asking questions of it and if you can't ask them for yourself, that's a really good indicator of an opportunity to add more visibility into your systems. And so I think as software engineers, we need to make sure to welcome questions from people that are not software engineers, testers, product owners, all sorts of roles, and we want to encourage those questions and we want to be able to answer them from our data and from the other role's perspective. Keep asking those questions and don't settle for, well, probably maybe look for ways to actually find the data for it. I think this is a team effort and it's across all the roles and it's really important. And contact information, yes, if you'd like to get in touch with me, the easiest way is Twitter. My DMs are still open and it's a bangzer. My last name and I think that's going to be the easiest way Links.

To everything of value we covered in this episode. Head on over to testguild.com a364 and while you're there, make sure to click on the Try for Free Today link under the Exclusive Sponsor section to learn all about Sauce Lab's awesome products and services. And if the show has helped you in any way, why not rate it and review it in itunes? Reviews really help in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe. My mission is to help you succeed with creating end to end full Stack Automation awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Automation Podcast. Head on over to testguild.com for full show notes, amazing blog articles and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.