Software Telemetry with Jamie Riedesel

Welcome to the Test Guild Performance and Site Reliability Podcast where we all get together to learn more about performance testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Performance and Site Reliability Podcast. And today you're in for a special treat. We'll be speaking with Jamie all about her book Self with Telemetry for Manning, which is all about the system that tells you what production has been up to. It's a really hot topic that I've been hearing more and more about from different guests. So really, it's gonna be a deep dive. If you don't know. Jamie is a staff engineer with Dropbox and has over 25 years of experience. She spent most of her career in software engineering but focusing on the production side. So really excited to have her on the show. You can tell by all that experience. Who knows where this conversation is gonna take us, but really excited about this boat. You don't want to miss this episode. Check it out.

This episode is brought to you by SmartBear. Listen, load testing is tough. Investing in the right tools to automate tests, identify bottlenecks, and resolve issues quickly could save your organization time and money. SmartBear offers a suite of performance tools like Load Ninja, which is a SAS UI load testing tool, and LoadUiPro and an API load testing tool to help teams get full visibility into UI and API performance so you can release and recover faster than ever. Give it a shot. It's free and easy to try. Head on over to smartbear.com solutions performancetesting to learn more.

Hey Jamie, welcome to the Guild.

Glad to be here.

Awesome to have you. Before we get into it though, is there anything I missed in your bio that you want the Guild to know more about?

Sounds pretty comprehensive. I just want to underline that I actually haven't been doing a whole lot of software engineering in my career, much as it sounds strange coming from a staff engineer. Which is why I think I bring some unique perspectives to a conversation such as this.

Now why is that? Because I think in your bio I messed it up. You said avoiding software engineering a little bit.

Yeah. While I actually do have a computer science degree. The thing is, is that when I left college and moved into the industry, I moved into more systems administration roles instead of code focused. I graduated 96 thereabouts and that was in the middle of the first.com boom when I had friends who were skipping college entirely and going into 80 hour week coding jobs. And I missed a few things back then. But you know the industry's come around a few times since then and I've managed to get onto those rises. But yeah, early on I focused more on the systems administration side, working with the hardware, operating systems, some of the internal stuff about how you keep systems up and running I focus my efforts on. It's proven to be very useful later on in my career as I've moved into working for software producing companies. This deep history with keeping things running and how systems work and how software interacts with everything else has given me a kind of deep background into making sure everything works. Bit of SRE, bit of DevOps. It's kind of both.

Awesome. That's what I was going to get. I'm already off topic. But you know, SRE seems to be a hot topic lately. But it seems like you've been doing this for a long time. So is your title sre? Is that something that your company has morphed into? Is it just a new title for what you've been doing all along?

It's more of a title of what I've been doing all along in a lot of ways. How the Geez, it's a little bit of history. I'll try to keep this brief is like the first phrase really broke out was operations. I mean we started shifting into that word, I want to say early 80s and we kind of stuck around until probably the mid 2000s when you started seeing transition to DevOps which was an attempt to break the logjam that had developed between development and operations. Sort of. There's been walls built between them and whenever you have walls like that you get communication problems. So operations was charged with making sure that everything stays up. Meanwhile, the people who have the biggest impact for keeping things up weren't actually respons, also known as development. So DevOps is all about building the communication paths back to where it used to be. Honestly in the 70s when it was all very all very together. I mean the computer room was just a bunch of people and they're much smaller infrastructures but we talk to each other and that's something that DevOps brought forward. SRE came out of more of the biggest companies on the planet is all focused on reliability engineering as a formal engineering discipline kind of top down in the industry. So you get these people whose entire jobs it is. Okay. You are responsible for making sure we as a company have high reliability and you are given a lot of power to make sure this happens. Such as you are allowed to enforce quality guidelines for software before it goes into prod. And a bunch of other things there. It sounds kind of different in little ways, but they kind of go the same direction, both DevOps and SRE. And if you've done one, you often can do the other style. It really does depend on, frankly, company cultures in a lot of ways. Some companies that were big in the sort of mid size, I would say, in the mid 2000s probably picked up the DevOps. So you may have some actual DevOps engineers titles in your tree somewhere. Whereas SRE really started picking up as a thing outside of Google and company around 2012 or thereabouts. Yes, the Bay Area's tech industry is very big about sre, and it's kind of diffused into the rest of the industry from there. But yeah, it's the way it works at Dropbox is, for the most part, it's SRE driven. But they've made a few acquisitions along the way. Like my company I'm with, I came in with the HelloSign merger. We were mostly DevOps, and that's kind of how it works. We're a lot smaller than they are, so our kind of scope of responsibilities is a lot broader as a result.

So because you were acquired or merged, does the company that acquired you, does it force you to go in an SRE mode though, or do they allow you to be autonomous?

Really depends on the company. Dropbox let us be what we needed to do, because that's actually one of the big reasons why the Dropbox merger wasn't scary is that culturally they were very similar to the company that was already in, which was generative. It's all about being a high performance company that delivers software quickly with high reliability. And how you do that is you get out of people's way in a lot of ways, encouraged the feedback loops. And they decided very early on in our acquisition that it was better for us to deliver than it is to do it their way. And there have been some reconciliations and efficiencies happening since then. Obviously you have to, but we didn't have an early mandate. Okay, this is how it's going to be. Shape up or ship out.

That's awesome. Cool. All right, so I want to dive into the book a little.

Yes.

I know this sounds cheesy, but I asked this question to everyone and it's why did you write a book on software telemetry, especially when you invested 560 pages? I'm just always curious to know. It's not a simple effort, in my opinion. So why this book?

I started this book at the prompting of one of my co workers in that I've been maintaining the telemetry systems at hello Sine since I started in 2015.

Kind of my baby in a lot of ways. And people realize I have a lot to say about it. But what I noticed is that while there may be may have been a lot of books and articles and newspapers and everything else in the industry about topics of observability and monitoring and the sort of metrics you need to keep track of as a software organization, there wasn't a whole lot out there about the systems themselves that support the telemetry basically where I've been living. Because yes, you can do a lot of things about instrumenting your code and what you do once you have your nice dashboards and how you write organ like software development managers use these systems to figure out what to focus their engineering effort on for the next few sprints and all that stuff. But there's not much in there about okay, so how do you write that? How do you write one of these things? Okay, so maybe you're not actually in a Bay Area company, maybe you're in some sort of long standing mid sized company from the American Midwest. No one really thinks about you being tech industry, but you've been having, you've had this job for 15 years. How do people like them hop onto this telemetry train? And I was kind of writing for people like that who already have existing stuff. They may have some syslog server somewhere they're trying to do a little something better with. And I wanted to give them a guide, if you will, to try to modernize those things and kickstart the process of getting their own internal software development practices bootstrapped to a more modern era.

Awesome. And I guess for folks that may not even be. You've been doing it since 2015, you said. But telemetry is something I've just been hearing more about with observability the past two years or so. So I guess what is telemetry?

Telemetry is at its very basic. It's what tells you what's going on inside the black box that we write. Because even from the very first program a human ever wrote, it was what we need. What we as a human need is feedback that the system did what it was supposed to or that it did it in the right way. In the early days it may have been the right wheels spinning in the mechanical computer or a light flashing in the right pattern that it's actually accessing memory in the right ways. You know, we're Talking ton of computing stuff. And it's only gotten more complex from there. I mean, since then. Like in the mainframe era, you often had a printer in the machine room spitting out the system log as various batch jobs ran. And someone in that computer room 24 hours a day watching that printer to make sure that it does everything it's supposed to. These days we've gotten a lot better about it. There's three major pillars of telemetry right now. Your centralized logging systems, which is simply a stream of events. It's the oldest style of computing going back to that 1970s era printer. The next step up kind of emerged as a discrete form, 2008, 2010 era called metrics, which is low cardinality. It's just a matter of a key couple of key values that don't change all that much on a number rates, latency, stuff like that. And the newcomer, which is distributed tracing, which is far more complex and really leverages modern databases to make it work. And that's the thing that can give you something that looks like a stack trace, but actually smeared across 15 different microservices instead of looking at a stack trace from a single one, you can see it. Spent this long in this service and this long in this service and watched that execution flow all on a single page. It's incredibly powerful. You get tools like Honeycomb IO which try to be kind of an all in one solution. And it's a way to get all of your events and metrics in one surface. Not everybody can do that. That is definitely a pay for solution, but that's just one example of what some of this stuff does.

Now we say not everyone could do that. Not everyone could use a paid solution or not everyone is able to get telemetry in place at a certain level of granularity.

There's a price for getting all the fun stuff. That's all I'm trying to say.

All right, cool.

Absolutely, absolutely. And that's part of why I wrote the book. Because there's a lot of way, there's various ways to get this stuff. And everybody is writing software starting from somewhere. And unless you're starting at an empty browser, empty code window, you have something in place already. You have to, because that's how you get told how it actually works. Maybe it's a bunch of print statements, maybe you've got a whole bunch of new relics stuff, whatever it is, getting it to something more modern takes time, effort, transition process. You can't always jump to the newest, hottest, shiny immediately. There's often A transition process, definitely budget involved, especially if you're involving commercial software.

Absolutely. So can everyone benefit from telemetry very much?

It actually does. It's incredibly broad. I mean if you've spent any time in companies, you know just how, let's just say greedy if you will. The business analysts or business intelligence people in your company are analyzing the click streams of all the users crossing across your site, trying to cross reference site user flows to figure out ways to increase revenue and all that stuff. That sort of deep analysis that the business side does. Software telemetry is beginning to realize that sort of deep analysis applies to software as well. So yeah, I mean the engineer looking to figure out why a support ticket failed, what happened behind a support ticket, absolutely will use that. But the thing is, is that the support person who took that ticket from a user can also use these telemetry systems to try to drill down to figure out where it happened and will set up the engineer to make it easier for the engineer for kind of at a higher level, you're talking engineering managers, they can use these systems to try to track quality over sprints and cycles and quarters and try to figure out how we're improving and how we're not. And for even at the director level, these systems feed into things like service level agreement and objective systems, more SRE type concepts keying into reliability engineering and all that stuff. It very much is the lifeblood of a modern software engineering organization. And that applies even to very large companies like say Mine or Dropbox or Google and even very small ones. We may have 15 engineers total and it's just kind of one big team. But the thing is that there's enough going on when you have 15 engineers that you have little islands of knowledge and everybody uses it in a slightly different way. So you need a flexible solution as well.

So you know, in the book you mention pretty much everything we talked about so far, but you have a section called what groups benefit from Telemetry. So sometimes think, someone may think, listen, I'm not an sre, I'm not a sysadmin, do I care? Should I care? You mentioned a bunch of things about quality and things. So I guess if I'm a tester, a software tester, a performance engineer, I mean, would telemetry help me with certain aspects of my job?

Absolutely. And it really does depend on the deployment styles of your company. If for example, you're an organization that does a lot of pre release testing, maybe you're actually writing shrink wrapped software where an online update isn't necessarily viable. You're doing a lot of pre release testing and that stuff absolutely does feed into telemetry systems. So when you're running your test runs, your selenium runs, whatever else you're doing to generate data, you can run that. All that data sits in databases that you can then run analysis on. These are the systems that tell you if you have a performance regression and you need to make sure it gets some engineering attention on a problem. Or if you're doing validation tests and like in a load test, you can get again that regression data or at least you can quantify the speed up that new changes come in so you can better pitch it to your higher level customers. So that feeds into the sales flow. Even so, yeah, this data can go very far, especially if you want to make a lot of use out of it.

Awesome. So it looks like the book's broken into three main pages. You have the introduction, you have use cases, which is cool, and you have techniques for actually handling telemetry. So I guess like in the first part you have a chapter on stages or emitting stages, creating and submitting telemetry. If someone's to follow this book from part one all the way to part three, would they be able to implement something right away?

Two points. I actually wrote this book more for implementers who already have something. And so what I wanted to do was to give you the tooling and a knowledge framework to look at what you already have, analyze how it works, and then reason about how you can improve what you have. Because dang near every one of us is working on something that already exists. And when we're starting something from absolute scratch. Greenfield. Yes. You have the world in front of you. And it's not. This book isn't great for that sort of greenfield experience, but it is much more powerful if you're trying to analyze something you're already working on. So yeah, that the emitting stage where everything gets into the telemetry system in the first place. And if you're a software engineer, this is probably your logging libraries or metrics libraries or if you've instrumented through like new relic or something else, that's kind of where your emitting stages, shipping and the presentation stages, not everybody has those. For example, if you're a new relic or some other software as a service system for your telemetry, that's kind of their problem. So yeah, you just make sure that they get the stream of events coming out of your system and then the cloud handles all the folding spindle mutilating and presentation and you get this really fancy dashboard. But if you're in a different kind of company that's been kind of in housing a lot of this stuff and you actually have like maybe an old school Elk Stack, which is elasticsearch, Logstash and Kibana, very famous telemetry stack that kind of grew to fame around 2012, 13, 14 era. You have one of those inside. This will help you reason about the various stages, how telemetry moves through, gets parsed, analyzed, ingested and then presented. So once you have that, it makes it a lot easier to propose improvements that can help you get better jobs.

So the example you gave of an older system was Elk Stack, but I worked for an enterprise company and literally this Software was like 30 years old. It was healthcare.

Yeah.

So could they use telemetry? Because obviously they must have logs, but I don't know what format they were in.

Oh yes. I love regulated industries. Just because you have some outside forces doing some of this stuff for you. You absolutely do. Because the thing about healthcare, especially if you've got hipaa, is that you have very stringent auditing guidelines. Who accessed it, when did they access it and did they access it in the right way? I mean, that's baked into the actual standards in a lot of ways. So in these systems you often have kind of a parallel telemetry system, which I actually cover in the book a little bit, which is more the security side. So this is something that tells. That's what I was talking about. It's like, who accessed it? That's telemetry. So if you log into a medical record system, a log is being emitted somewhere saying that you accessed it, when you accessed it and where you did it from. So that gets persisted for however long it happens. And that means the security or compliance office later on can review these systems to make sure that or to ensure to if regulators, if it comes up, yes. In fact, I can prove that this person accessed this, they did these sorts of things and then they logged out at this time. And I can prove it because I have the traces. But at the same time, if you're writing software in an organization such as this, it's a different world. If you've never worked in regulated software, having these regulators looking over your shoulder feels like agility killing Killjoys. But at the same time they're there for safety because you're handling one of the highest priority data types the world has right now, which is private health information. So you're going to be paying attention to Some things kind of I talk about later on in the techniques chapter, such as safe handling of private information. What happens when you accidentally leak some of this stuff into your telemetry systems, which means you might need to treat it like a HIPAA compliant system, which you don't want. But you do need to make sure you have ways to handle spills, redact them over time. Like I have a chapter on that one.

Absolutely. Are there any roadblocks that people, when they have the data like you said, and they already have something in place that may get in the way of them really succeeding? With telemetry, honestly, a lot of it.

Comes down to agency, to be completely frank. I mean so much of the ability to make change is cultural inside the organization itself. It's not technical, it's amazing. As a staff engineer, I'm kind of working at that interface between sort of the senior engineering and engineering management and actual directors. And from this altitude they call it, I can actually see a lot of the pathologies that show up over time. And I start having the leverage here to be able to do that. Now as someone who's just kind of picking up my book as an early career person looking to make make some waves, it comes down to a lot of buy in and just sometimes just doing something because no one's told you you couldn't do it. And that's a great way to kind of seize initiative, make a few pull requests to improve parsing of a field type. For example, it's some regex engineering. I mean that's kind of basic programming, honestly. But that kind of thing absolutely plays a role inside some of these telemetry systems. Or maybe you take the metric system that you already have and try to shave some of the barnacles off and make it easier to use for people, update the documentation so people realize it's there. You can actually leverage some of the power you already have. I mean these sorts of things can be done pretty early on in a person's career. But yeah, it's all about building leverage and understanding where you can make changes. Because the thing is like the telemetry systems are one of those systems that has a lot of leverage on it because if you make a little bit of improvement can have a lot of impact downstream, which looks really good on your performance reviews later on, let me tell you.

Nice. So like I said, really begin a question. How does telemetry relate to observability? Are they like a one to one relationship? Are they. How does that.

A little bit. There's actually A little confusion on this one. I was kind of, there was a certain period of time, oh I want to say 2007, 17 to 19, kind of in there when observability was kind of a standalone word and it was sort of, sort of implying observability as these distributed tracing systems. But these terms evolve over time. Over the last few years. Observability is now a bit more firmly in place as a practice. Observability is a practice. Telemetry is a technique that supports the practice. So like I said, observability is a whole suite of practices. It takes the telemetry data that you have, but it's also the procedures in place. You put procedures in place to review your telemetry and learn from it and ways that people building something can use the existing telemetry to reason about what needs, what you can work on next and how to fix existing problems.

Perfect. And so I think we went over some of the most of the basics when you go into part two, use cases and I like how you have like a section for cloud based startups and then you have non software businesses and then long established business it is it that different that each one needed like its own section, you know what I'm saying?

I honestly felt that it was needed. I mean to be. Interestingly enough that particular section had a lot of feedback from my reviewers early on.

Oh nice.

Because like I, it was very mixed. Some people absolutely hated it, they didn't believe it belonged there at all. And others were like this is amazing. I understood how it applies to me now. And that last question is really what does it. The reason I put that section in there is to provide a better roadmap for people reading the book at different stages of their career and different parts of the industry to try to figure out how these frankly somewhat abstract concepts collapse down into their day to day work. Because each of those use cases like the cloud based startup which grows up to a multinational megacorp, the non software based system and the old school legacy computers. Each one I look at at a different size of company starting at small, going up to the large internationals. Because the telemetry story in each is very different. Because if you look at like the smaller ones it's a lot of the same stuff in a lot of ways. Just because if you're small you have the same kind of problems but when you look at the very big things you see a lot of the same techniques being used but the details are very different and the decisions you make along the. Because I go from small to large and describe how they change over time will help people understand how telemetry does change. Because if you're starting at a startup with three people with a big idea and by the time you get to the place where, like where Dropbox is right now, post ipo, we have an international presence. They've gone through several radical changes in how they handle telemetry over time simply because what they did in the very first days, just sort of dumping stuff to a log file that you tail after sshing into a server, doesn't scale when you get very large. And so all these scaling decisions have to be made, the trade offs and frankly people you pick up along the way, because if you have fewer than 50 engineers, you probably don't have someone like me on staff who can actually run these things for you. So you're probably using mostly SaaS. It's when you get to about 100 or so engineers and a little bit past that, when you start getting professional ops type people in there that you start getting the sort of in house talent to be able to run these things internally and you get that entirely. Budgetary question, do we keep paying this software as a service provider that does all this stuff for us, or do we try to save some money and try to insource it? And sometimes that comes down to yes, let's try to insource it. And sometimes it's no, let's keep with the software as a service providers. And sometimes you're in a company that went into business in 1782 and has been slowly upgrading their technology ever since and you've got computers, since computers began to be a thing and started on paper and you have this deep, deep legacy. Companies like that feel very different than say a company that was only six years old. So go back to original point. The reason I have that section in there is because the industry is huge and people tend to lose sight to just how honestly vast our industry is. Mainframes are still there. I actually cover it in chapter two, which is something you don't get very much in tech books these days.

Yeah, I mean, I love that you cover mainframes. Once again, I worked for an insurance company as well. Oh yeah, they still had mainframes.

So it's like you get a lot of that.

I love it. Yeah. So this book's for pretty much everyone at every stage. It sounds like I tried to.

Yeah, it's very hard to write that way. Which is again how it ended up being such a chonker as it is. This Thing makes a lot of noise and you drop it on the table.

Yeah.

Nice. So, silly question, but if someone started at the cloud startup phase and they followed the steps that you've laid out and they became a unicorn company and they grew massively, would the steps they put in place following your advice at the startup help them grow? Or is it always like a, like a massive change when you skip from a startup to a mega unicorn company.

Using that particular use case, the startup going into a unicorn is. You have to look at it from the point of view of what you're building, the three people with the big idea, which is a phrase I use a lot. Yeah. You have probably what counts as a microservice in a larger company at the very beginning, it has a front end, it has maybe a mobile application, it has a back end. But the thing is, once that thing goes up to unicorn status, it's still an application, it still has front end, it still probably has a mobile X spray. It's just that there's way more code involved and that just the decisions you make going from like a small, like maybe two microservices talking to each other to something based on kubernetes, when you've got maybe 286 different services trying to interoperate with each other, you're going to run into scaling problems. Whenever you're growing like that, it's always waiting for the next ceiling and trying to build past it. And sometimes that ceiling is your telemetry system and sometimes that ceiling is, oh, crap, we're overloading a single server redis, and now we have to go cluster Redis, for example, and our other sorts of systems. So, yeah, I think it definitely does help. And the advice I give in there helps you predict what sort of problems will come in. Because if you're in one of these companies that's growing a lot, I give you some caution and cautions in there about what to look for when you're falling behind and trying to get ahead of the change curve. Because you always try to get ahead of that and try not to be reactive, because if you're being reactive, it's probably because people are unhappy. They're considering your stuff doesn't work very well. And developer experience is a total thing in this company. And right now it's kind of a seller's market for talent. So you can always get a job somewhere else in a lot of ways. So, yeah, that sort of thing, you need definitely to pay attention to it.

Great. So we covered part one, part two. It's a huge book, like you said, 560 pages. So part three, you have techniques for handling telemetry. Is there one of these techniques in here you think that people don't know enough about, maybe when they're dealing with telemetry?

Oh, yes. My favorite one in there was actually the last chapter in the book, which is chapter 18, Surviving Legal Processes. No one talks about this. And it's something that in the healthcare industry, you probably know more about than others, simply because you have higher auditing concerns, you have higher compliance. And if someone comes in and there's an actual records leak that might actually declare it a breach, there's a whole investigative process that follows after that. And sometimes these are depending on the jurisdiction you're in, sometimes these are driven by health regulators as opposed to the actual company itself. So I put chapter 18 in there, the Surviving Legal Processes, and try to take some of the scary away, because working with lawyers is scary unless you do it a lot. They make pronouncements and they say things and they're a lawyer and they're the one who know the legal stuff. And does that mean I have to do it exactly the way they say it, in the way they say it? Or can I talk back? How does this work? Help, help, help. So I wrote that chapter to try to take away some of the stigma and describe how it works. Honestly, the ediscovery process, which is what I talk about, has defined phases, and there's entire support structures out there for people like us who would be called data custodians because we're the ones who manage the systems of record that have this data. How to work with people whose job it is is to access their sources of record and come up with productions and everything else that's useful for investigations or actual lawsuits and case law.

I love this because a lot of times people, I think, do stop like a greenfield, kind of unrealistic. And I love how this is like, people were scared to talk to auditors, and they're reading this part, I think would help them because the company would be like, don't say anything unless you're asked. So it seems like a really helpful chapter, for sure.

The other thing I definitely wanted to show in there is how to do. It's a couple of questions to ask when you're trying to make big changes to your telemetry systems. Because, yeah, like, if you've done the rest of the book, you've got some big ideas you want to bring to your system, and people have agreed and we've got a project spun up and are trying to figure out what the scope is. There are some questions you can ask early on in your project to try to make it easier to deal with the legal processes should they show up now. Different industries have different risk profiles. Like healthcare is famous for being vulnerable to this, but other industries famously aren't. So yeah, it really does depend on but you can definitely take some concrete steps in your telemetry system design to deal with things like legal hold requests which require you to not delete things on the schedule you planned, and other sorts of requests.

Well, thank you for writing this book, Jamie. I think anyone that's doing telemetry definitely needs to get their hand on this book. But before we go, is there one piece of actual advice you can give to someone to help them with their telemetry efforts? And what's the best way to find, contact or get our hands on this telemetry software book?

The best thing I can say is look at where your pain points are. What's annoying because every software system is annoying. And when you're looking at the systems you're working with, where does it feel like you're having to work against the system? I mean, those are the best places you can focus on to improve these things. And you can do that as a software engineer, you can do that as an SRE, you can do that as DevOps. Heck, you can even do that as QA and even engineering management as well. All these people can. But as for reaching me, the most reliable way to reach me, frankly, is on Twitter. It's sysadm1138 from my old blog handle sysadmin 1138, which is still there. And I had a blog going back as far as 2004. Doesn't get much traffic these days, but it's there.

Thanks again for your performance testing awesomeness. The links to everything of value we covered in this episode. Head on over to testguild.com P86 and while you're there, make sure to click on the Try them both Today link under the Exclusive Sponsor section to learn all about SmartBear's two awesome performance test tool solutions, Load Ninja and Load UI Pro. And if the show has helped you in any way, why not rate and review it in itunes? Reviews really do matter in the rankings of the show, and I read each and every one of them. So that's it for this episode of the Test Guild Performance and Site Reliability Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack Performance testing Awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Performance and Site Reliability Podcast. Head on over to testguild.com for full show notes, amazing blog articles, and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.