Grokking Continuous Delivery Book Review with Christie Wilson

Welcome to the Test Guild Automation Podcast where we all get together to learn more about automation and software testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Automation Podcast. Today we're talking with Christy Wilson all about her new book Grokking Continuous Delivery as well as other topics like continuous testing and a cool open source project I just heard about. You want to learn more about called. I think it's called Tecton. So Christy is a software engineer at Google and a co creator of the Tecton Project and author of the Grokkin Continuous Delivery book. She loves quality software as you'll tell by this interview in cat pictures. Who doesn't love cat pictures? Really excited to have her on the show. If you want to know about continuous delivery, this is the place to learn from an expert. So you don't want to miss this episode. Check it out.

The Test Guild Automation Podcast is sponsored by the fantastic folks at Source Labs, the cloud based test platform that helps ensure you can develop with confidence at every step from code to deployment for Every framework, browser, OS, mobile device and API. Get a free trial. Visit testguild.comSourceLabs and and click on the exclusive sponsors section to try it for free today. Check it out.

Hey Christy, welcome to the Guild.

Hey, thanks for having me.

All right. I really botched a bio. So is there anything in your bio that I missed that you want the Guild to know more about?

No, I think it's pretty accurate. Especially love of cat pictures.

Absolutely. So I am really impressed. You work for Google, you're working on an open source project and you're writing a book. So I guess the first question is where do you get the time and why write this book? Because obviously you seem very busy.

It is definitely a lot of work. I guess. Where do I get the time? That's a good question. On the weekends. Yes. Or whenever I can. A lot. During the recent holidays actually we went to Whistler and then I was. My husband was skiing and I was just writing back at the Airbnb. Yeah. But then I guess as far as why write the book? Continuous Delivery I think is something that impacts pretty much every software engineer on a day to day basis. To some extent you're usually building something or running some tests and I feel like it's such an interesting space that where it can cause so much pain for people potentially, but it can also make their lives way easier. And I really like the idea of having a very accessible book that would help people who maybe I think Often you have to learn this stuff the hard way. And I really like the idea of having a book that would help people kind of get up to speed without having to go through five years of grueling pain to learn about it.

Awesome. Yeah. So thank you for that because I know there's a lot of people listening that are probably about to go through that grueling pain or have already. So I guess the first question is sometimes people say, what's the difference between continuous integration and continuous delivery? So I guess, you know, what is continuous delivery and how is it. How is it different from continuous integration? Or does it matter?

It's a good question. I've chosen to take a particularly pedantic definition of continuous delivery. I guess the book that I'm writing was actually originally called Grokking cicd, because what I'm writing about is actually what people often call cicd. But when I went and looked into these terms, it turns out that's something that people just kind of started saying organically, I guess, when that. Because continuous integration came first and then continuous delivery came and then people kind of mushed them together. And then there's also continuous deployment in the entire mix as well, which is also called cd. But basically continuous integration is all about the process of making sure that your software does what you want it to do. And then continuous delivery was this idea that came along later that kind of took it from the point where you've successfully built it and you know it does what you want it to do and took it further into how do we actually get that to where it needs to be running or where the users can access it and the processes around that. So in my mind, continuous delivery is sort of the umbrella, kind of the umbrella term that encompasses continuous integration and the deployment and releasing and all the stuff you do after that.

Nice. So if you work for a large enterprise, how realistic is continuous delivery? I think that's why people use the term continuous integration, because we're going to pretend like we're going to deliver it, but we're really not.

Right. I think if you want to look at the most idealized form of the processes, it's probably not realistic for anyone, or it's only going to be realistic for the most cutting edge company that is. But even those companies, all the processes are going to change over time. So even whoever's doing the cool cutting edge thing now, like in five years, are they going to be able to keep up with that? Probably not. I think. I like to think of it more as I've seen some definitions that refer to it as a discipline or a practice. And I like that idea of just. It's something. It's like if you're doing yoga or something like that, you're never. You're most likely never going to be the most amazing yogi ever, but you practice it every day and get a little bit better and some days are better and some days are worse. So I like to think of it like that as something where kind of any improvement is good and you're always going to be evolving. And maybe you don't need to get to the ideal. Maybe it's good enough for you to automate those tests that you were running manually. Like that might just be enough to kind of improve where you're at.

Absolutely. So the book's broken out into, I believe, what's it four main sections. You have intro, keeping software in a delivery state at all times, Making delivery easier, and pipeline design. So I like how you ease into. The first part is the intro and you start talking about testing almost right away. Like dealing with noisy tests and speeding up slow test suites. Stupid question. But how important is automated test to being successful continuous delivery?

I think it's extremely important. Another way of looking at how I organize the book is that that first section about keeping things releasable is all about the continuous integration piece of things. Because I think anything you want to add onto that later, like let's say you want to. You want to automatically deploy on every commit. If you, if you don't have any confidence that things are going to actually work, that's probably a really, really bad idea. You don't want to be. You don't want to just be slamming stuff out there constantly when you don't know what state it's in. So I kind of think the testing piece is the most important piece. And if you're not going to do anything, anything else, I think that's the thing to really focus on because it kind of lays the foundation for anything more advanced that you might want to do.

So I get this question all the time. So where does manual testing come into it? People say continuous delivery is not realistic because you need to have someone actually with eyeballs on it. But then you can't really deliver continuously automatically if you need that. So like, do manual tests still ideally exist or where does it come within the pipeline?

I guess it's a really good question. I think it depends on what you're doing. I think that like you said, if you're going to do so, continuous deployment, the Idea of, you know, every change that you make automatically goes out. I think that is realistically not attainable. If you do require manual testing to be. To be happening at some point in the process, I still think that you can get to a point where your processes are quite automated. And the manual testing bit, you know, there's just a button, something that someone hits where they approve after they've done the manual testing. Everything can be automated before and after that. So I think you can still have a lot of the elements of continuous delivery in place. And I think you could even say, I'm doing continuous delivery. I'm not doing continuous deployment because it doesn't go out on every commit, but I'm doing. I'm doing continuous delivery. I think it's important to examine why you're doing the manual testing. I think for some projects you need it and it'll be for some compliance reason or something you absolutely have to do that requires human intervention or, you know, people's lives are at stake. Like the slightest mistake is going to actually, you know, kill someone. And when you have those kinds of stakes, it's really important not to make a single mistake. And I think having kind of that manual phase where things are a bit slowed down makes perfect sense for a lot of the software that people make. I think that they have phases like that because they think maybe they need it and they can't think of another way to handle it. I think also a lot of the manual testing that people are doing is things that could be automated if you could potentially automate it. I think it's best to invest in automating the testing instead of making people basically do what a machine does. You know, going through a checklist of like this, this, and this. I think where manual testing really shines is in the exploratory testing. Like trying to think of a novel way to use the thing, something that you couldn't think of in advance and write a test for. And that kind of testing, if you can, you could potentially do that kind of out of bounds. It doesn't necessarily have to be a gate on your release. It's something that you have people focused on and doing all the time, and they're identifying new completely novel things that you can take back to your software and fix. But again, it totally depends on your project. And I think, I think you can still say you're doing continuous delivery if you're required to do manual testing, if it doesn't have to be an either or thing, but you might not be able to do continuous deployment.

So am I allowed to say Google or ask your questions around Google?

I think so. I don't know how much I love to answer.

It won't be crazy, just sometimes companies legal department will say can you edit this out? Can you edit that out? So just curious to know, when we talk about automated tests, what tooling are we talking about or what kind of automated tests are we. Do you use at Google, Is it like end to end tests? Is there a secret to the type of testing that you think more successful in? Most teams I use continuous delivery that have been successful.

That's a good question. I mean the funny thing is I'm probably incapable of leaking any Google secrets because I mostly work on open source projects.

Okay, nice.

So I don't see that much internally, but what from what I have seen, I would say that there's, there's not a lot that Google does that's different from what other companies are doing though I would say everything is automated. Anything that could possibly be automated is definitely automated. But still I think the same principles apply as far as, you know, like the classic test pyramid with the unit tests, something that's kind of integration test or end to end system test. That whole structure that, that applies within Google. And I don't think that, I think that that's a classic like that's, that's the same basis for every project. It doesn't really matter what you're working on. And I still feel very strongly that, you know, unit tests as the bulk of your tests, that's the most important thing. And then kind of a smattering or sprinkling of the other kinds of tests to fill in the gaps is the way to go.

Absolutely. So in the basic pipeline, is there anything that you think people get wrong within a basic pipeline or maybe they overlook like a certain stage? I think, oh really neglected. If only people did this more in the basic pipeline or understood better, they'd be better off.

I think that one thing that people neglect is how they approach the parts of their pipelines that cause them pain, I guess. And I think a lot of this comes back to testing. But I think when you're starting out a project it can be pretty easy to get things initially up and running and that will work for a while and then at some point things stop working as well because the test suite that previously took five minutes to run now takes more than an hour, something like that. And then I think people start reaching for ways to kind of automate that pain away. Like you know, the test that always fails. We're going to retry that test or the tests that take a really long time. We're going to split them up and we're going to run them on a whole bunch of machines. And look, it went from 45 minutes down to five. We're fine now. But I think a lot of those approaches will work in the short term, but they're just going to come back and bite you again at some point later. And maybe that's fine for your needs. Maybe you. But if you want your project to stand the test of time and continue to be maintainable for the next five, 10, 15 years, then I think it's better to kind of stop and really like kind of root cause what that pain is and why your tests are suddenly taking 45 minutes and is that actually what you want and kind of address those things at the time versus just kind of band aiding over them until they become so much of a problem that suddenly, you know, nobody can even work on the project anymore and you can't actually even add new features. And you have to start talking about the thing just becomes this legacy project where you can only add the odd extra flag here and there, but there's nothing more you can really do because it's just too hard to work on it.

Absolutely. And you have sections on dealing with noisy tests and how to speed up slow tests. So when dealing with noisy tests, how do you make sure that people don't ignore it so that it becomes like, I'm not even touching this because the test is so flaky that I'm not getting any value, I don't know what's real and what's not. Now.

It'S funny that you asked that because we also suffer from that problem in the open source project that I work on, Tecton, where some of our tests are kind of flaky and we've gotten to a point at the moment where I think people are actually a little bit desensitized to it because the same test is failing over and over again and you see another test fail and you think, oh, it's just the same thing. I think the only thing you really can do is to try to focus on keeping the signal good as much as possible. Basically, the second you see one of these tests start to flake, you have to do something. And that's something. It could be fixing it. Even, I would say even disabling it at that point is better than leaving it in a state where it is constantly failing and making it so people are no Longer paying attention. I think desensitizing people to the signals that you've put into your project is the most dangerous thing. So I think it's important to make sure that that signal stays good. Otherwise people will ignore it. Just like a car alarm on the street that goes off every couple of days. Like maybe you paid attention, maybe you looked the first time, but then later you're just going to ignore it.

So how important is linting to continuous delivery? You actually have a chapter on it and I'm just curious to know why you felt a whole chapter on linting was necessary.

That's a good question.

It's probably one of the shorter chapters to be honest. I think it's kind of part of just the whole kind of picture, I guess. There's also a chapter that I was working on recently where I was trying to describe sort of if you're starting from a greenfield project versus a legacy project, like what would you add? And I think that if you're starting from a new project, adding linting right away is, is, is a really good step because it sets you up to have just so much consistency in the project going forward. However, if you're looking at something that's been around for a while and you're trying to figure out how do I improve the state of this. Probably linting, depending on your needs, may or may not be the first thing you want to reach for. Especially because you'd be dealing with so much existing code that would be. Would not be passing your linting rules at all. I think though that it is, it is important to not only to have consistency but also linting does catch. It catches real bugs, things that, you know, like common thing mistakes that people make that it's kind of arduous to expect people to, I guess, you know, memorize and always catch themselves or rely on human reviewers to catch. I think a big thing to me is, is just again kind of the theme of if you can make it so that some sort of tool or automation can take care of something for you, then it's way better investment to rely on that than to rely on people to catch things or people to notice or people to remember.

Absolutely. I've spoken with a bunch of automation engineers that sometimes try to use linting for their functional automated tests like Selenium. I'm surprised it never takes off because like you said, it could catch a lot of low hanging fruit that doesn't require someone to manually go in and do like a code review on it. So that's a great Point.

I guess the next step after you have your basic pipeline in place is you go to part three, which is making delivery easier. So what's the biggest part of this, this, this section you think that makes that people focus in on that can take their continuous delivery to the next level? I guess that make it, to make it easier.

That's a good question. I think, I guess that just thinking back on the chapters that are in there, I kind of approached it from several different angles. I mean one of them, one of the big themes in that section is you might be familiar with the DORA metrics already from the DevOps Research Institute. But these, these metrics that have been identified by this research group that are kind of established, you know, the, I think the way that they identify pieces, I'm not sure if they've categorized them as elite performers, high, medium and low performers. I'm not 100% sure what goes into that categorization. But anyway, stepping back from that, these metrics that you can use to kind of evaluate your project and see, I think there are, let's see, lead time for changes. Oh man, I shouldn't have started listing them. I'm going to fall apart.

Deployment frequency and I think measure velocity.

Yes. Yeah. And overall they're about basically stability is kind of one category and then velocity is the other one. And I think that those are kind of two very interesting ways to look at your project. And the thing is that things are so intertwined that it's very hard to separate them. The first chapter in that section is all about kind of avoiding long lived feature branches and committing back to your main branch as frequently as possible. Because if you're not doing something like that, then it doesn't really matter how frequently you're deploying. If you're keeping everything in these giant branches that you're growing over months, then it doesn't matter if you deploy on every commit because suddenly you're going to get that thing merged in and then suddenly you're going to be dropping like six months worth of work into production, which is a lot more risky than if you're actually able to take things incrementally. But it's tough because it actually requires everybody who's working on to kind of adjust the way that they're working. And that kind of like cultural change I think can be challenging.

I mean, yeah, speaking of cultural changes, I worked for a large enterprise. They had like six or seven sprint teams that worked for two weeks, but they all coded in their local branch. And then at the end of the Sprint, they will all upload and then we can never roll back because we didn't know what was checked in that caused the issue. Is that common or is that like just a completely dysfunctional culture that I've worked in?

I don't know if I've ever heard it quite as coordinated as what you described that you would all, in a coordinated basis, put these potentially conflicting changes in. But I would say that that's very common. I think that's most of my experience in software engineering is some kind of pattern like that to varying degrees, because I think it's really hard to. Even if you try to think about what is the tiniest change that I could possibly merge in and how could I commit every day or every other day, I think it's extremely hard to try to think about some big chunk of work and then break it down such that there's actually like a meaningful chunk that you could bring in that frequently. I think it's. I think it's a very difficult thing to do. So I don't. I don't really blame anybody for, for avoiding that, but I think it does make things way better when you kind of go to the effort of doing it.

Absolutely. Does the book cover anything about infrastructure? I would think with continuous delivery, when you're running all these tests, you need to run fast and Lord knows what else for development and mocking. Do environments or infrastructure come into place at all?

Not too much. I was trying as much as possible to make the book fairly agnostic to a particular technology, especially because I work on a project, Tecton, which is meant to be CICD building blocks. And at the same time I wanted to write something that would be useful even regardless of how long Tecton is around for. I ended up using GitHub Actions actually quite a lot though, to do examples in the book because. Because it's something that's so. It's so easy for basically anybody that's reading the book to, you know, sign up for GitHub account, make your repo, try some of these things out, they can just go straight to that immediately. But other than that, I don't really go into much detail about any particular infrastructure or technologies.

So, you know, we've mentioned TechTown a few times. You give it a brief explanation of it. So what is it? Who would really appreciate or really, you know, really, it really would help them with what they're doing right now?

So Tekton is cicd or if I want to be very pedantic, just CD continuous delivery building blocks. That are built by extending Kubernetes specifically. So at the moment the easiest way, the group that this would appeal the most to is someone who's already using Kubernetes to some extent. Because you could, if you're used to using Kubernetes, it's not a very big stretch to install and run Tekton on top of that. At the same time, there's various cloud providers, including Google who are involved in the project, who are working on, like Red Hat and IBM I believe. Both have their own platform on top of Tekton. So getting up and running with that I think would be significantly easier than just installing the open source version of Tekton. And the goal of the project is fairly lofty and it's to establish a de facto standard for some of these reusable pieces that you might want to use across whatever CICD provider you might be interacting with. So if you could imagine every single CI CD project will have some kind of Docker build task or some kind of git clone task. And so one of the goals with the project was that if you could have one git clone task and then that would work with any Tecton compatible project, so you could use it with Red Hat's offering, you could use it with Google's offering, then that would give you a bit more freedom because you wouldn't be quite as locked into committing to one particular CICD provider. But that's more of the long term goal. And I wouldn't say that that's where it's at currently. So currently if I wanted to recommend it to someone, I would recommend it specifically to anyone who's working in a, in a situation where they basically have, they want to be able to have a huge amount of control over how exactly their, their CICD pipelines are executed. Like for example, especially like a team within a company who's providing those things as a service to the rest of the company. I think if we're talking about just some small project with a couple people working on it, this is probably not the route you want to go right off the bat. But we're talking about some larger organization where you want to kind of start enforcing policies around. You know, everyone builds in the same way. This is our standardized way of building that kind of thing. And you have the people available to put the work into kind of piecing this together because it really is building blocks. It's not, it's not like out of the box, you get the integration task, you kind of have to piece it together yourself. So it's more people who need something a bit more bespoke. Or if you want to build a CICD platform and provide it as a service, that's also a great option.

Nice. So how long would it be for someone to piece things together? I guess it depends on how big the organization is, but I'm just thinking of a large enterprise, like you said, that has different teams all across different silos. They don't talk to one another. Sounds like using this type of solution would kind of codify. This is how we do across the whole organization. So you could skip from silo to silo and probably help each other out and still speak the same language, I'm guessing. I don't know.

Yeah, that's exactly, exactly the goal. As far as how long it would take, I mean, you could get something very basic up and running pretty quickly. Not like GitHub Actions fast, but like a day or two I would hope would be enough to get like a basic clone, test build, deploy, sort of pipeline going. But then the rest of it would be more about establishing like, well, where are you going to, you know, are you going to have one repo where you put all of these prototypical tasks and pipelines in? How are you going to share them? It's probably a lot more about like the processes that you define around it. And then the other thing is that you need some, you need some visibility into what's happening. Tecton provides a dashboard that you can use out of the box. But if you need that to integrate with anything else that you're doing, like your, maybe you're using some specific deployment system, then I think that what a lot of companies end up doing is building some kind of layer on top, like a CLI tool or even, even a UI where they're kind of plugging together the individual bits that they need. And that kind of thing would take significantly longer.

Very nice. So before we leave part three, you also talk about building safely and reliably. Is there a certain point in here that maybe dependencies come up that people don't think about that could cause issues?

Yeah, I think I'm trying to remember what I cover in that. The, the content of that chapter is actually a very small subset of a larger effort that's popped up within the last year or so? I want to say that's so supply chain security in general is very, very exciting right now. And there's a particular standard that Google has published that is kind of mirroring some of Google's own internal policies called slsa. Or salsa. And salsa is this. It attempts to define these four levels of kind of the security of your supply chain, where the idea is that you can kind of start at level one and things are fairly basic and then you can kind of gradually make things more sophisticated, but you're increasing the security as you go. That chapter focuses specifically on some of the build related requirements that they're kind of drawn specifically from Salsa around how you build things and dependencies definitely come in. I don't go into a lot of detail, but I do talk a bit about how you can kind of accidentally get bitten by your dependencies, especially around version management and how if you don't, if you aren't careful about what version of your dependencies you're pulling in, then you can, you can accidentally break things in a way where, you know, it seems like everything's. It seems like you've pulled things together solidly, it seems like you've tested them thoroughly, but then one of your dependencies goes and changes and then suddenly something's broken.

Yeah, there's kind of a lot of. I guess there's a lot of. There's a lot of kind of small sort of edge cases that you can get into, especially around dependency management that can, that can bite you. Maybe not, you know, not every week, not, not every month, but maybe every once in a while something, when something goes wrong, it can be pretty catastrophic.

Absolutely. So I don't know how far along the book is. It looks like part four isn't. Is it? I don't know how up to date the site is. Have you worked on part four? Yeah. Which is a pipeline design. Is it still. That's the part that's still in process.

That's where I'm currently working right now. There's section is currently called pipeline design, but it is not. It's actually more like just kind of. It's like three random topics, one of which is pipeline design. So there's basically three chapters in that section. The one that I just finished working on is. I kind of, I think I referred to it very vaguely earlier, which is trying to give some guidance for like basically what can you do depending on where you're starting from, whether you're starting from a totally greenfield project or a legacy project. And then it has kind of some example pipelines of, you know, here's the ideal pipeline and here's how you might want to incrementally get there. If you're starting from scratch or if you're not starting from scratch. Here are some directions that you can go and kind of some trade offs to consider. And there is a chapter about graph design and there's also a chapter about. What I'm about to start working on is actually about scripts and specifically about BASH scripts, which is something that I think is really interesting because I feel like, I think I've seen some jokes before which I can't remember the wording of. It's like if you boil down what CICD is at its core, it's really like a series of BASH scripts that are kind of just like all plugged together. It's like Bash kind of runs the whole thing. But I feel like we don't talk about that a lot. So I wanted to say just a bit about. The BASH scripts you're writing are also code. So it makes sense to treat them, treat them like code. And maybe sometimes when the scripts get really long and complicated, you have multiple libraries of BASH scripts you're importing. Like maybe consider moving to a language that's designed for that instead of using Bash.

So as you're writing this book, it seems like when you're doing a brain dump, is there anything you look and go, wow, I didn't even think about this. Let me research it that you learned yourself as you're writing it. I don't know if that makes sense, but like you're teaching yourself like, oh my gosh, I totally didn't know about that or I need to explore that more before I can write about it.

Yeah, no, I think that happened quite a lot. I would say I'm much more comfortable, I think, talking about tests in general because I think that's where I focus a lot of my attention. Not that I didn't. I certainly was doing research around that as well. But then when it gets more into, into the deployment, for example, I was, I was confused about the differences between different deployment mechanisms. Like canary deployment, blue green deployment. I had a very kind of like vague and convoluted idea of how all those fit together. So I to spend some time kind of teasing it out. And I was, I was sort of surprised when, when I finally got, it was like, oh, I thought of that. I thought it was completely inverted. Like I thought of, I thought of blue green deployment as the most sophisticated thing you could possibly do. And I thought canary deployment was something that was, you know, like maybe you start with that but you don't, you don't end there. But it was actually the other way around where the canary deployment was kind of the more complicated option and actually has more. Well Maybe not complicated, sophisticated, has more potential for doing some interesting things if you really want to keep running with it. I definitely learned a lot about that.

Now, I'll be honest. I've known. Heard about Canary testing last week. For real. This is the first time I heard about blue green. I don't even know what is blue green? Because I didn't ask for some reason last week deployment.

I don't know why it's called blue green. It's also called red black, I believe. But the idea is that you have these like that if you're deploying a new version of your software that you completely deploy, basically instead of in place updating whatever infrastructure you have, you set up a completely additional set of that infrastructure. And you kind of. One's blue and one's green. I can't remember which one is which. And then basically you can try directing traffic over to 1. I think you could do it incrementally if you wanted. Like you didn't. You don't have to do it all at once. And then you just observe what happens. And one of the big benefits is that it makes it so easy to roll back. Because if you want, if something is going horribly wrong with your new deployment, you switch back to the original deployment and no harm done, you're safe to go figure out what went wrong and try again.

Sounds like a feature flag. Is that the same thing?

So the feature flag would be a way that you could control access to a feature within the software. But this is more like a version of the software is completely. So you've gone from version 1.3.1 to version 1.4.1, and you've installed the new binaries or the new images are up and running and you have those in your. In your new green or blue cluster. And then separately from that, you could be using something like feature flags to control who's accessing what or do AB testing. I don't, I don't get into any of that in this book. I just talk about the deployment. Deployment techniques.

Awesome. Okay, Christy, before we go, is there one piece of actual advice you can give to someone to help them with their continuous delivery efforts? And what's the best way to find contact or get our hands on your soon to be released book.

Let's see. I think the biggest piece of advice I would give, I'll try to give something more concrete than this, but the biggest thing I think is to. Is to not panic, I guess, not be intimidated by it at all. I think again, you don't have. There's really no ideal to get to. Any change you make in the way you're doing things is likely to be positive. I would just view it as, you know, something where you can, you can go and pick and choose and figure out what's best for your project and what, what change you could make tomorrow that might, that might make everything better. It might be that that's all that you need. You don't have to look at the end goal and then kind of get intimidated and like, I'll never get there. I'm not even going to start. And then one, I guess let's see if I could think of something that was actually more specific that you could do. I really like the idea of. I think it's in the the book by. I think it's Jez Humble and David Farley on Continuous Delivery. It says something about if something hur hurts, bring it forward or something to that extent. Like I would, I would just go and look at your processes and figure out like, what, what is the most painful thing, which is almost definitely going to be something that people are avoiding. Like what is the thing that everyone avoids doing or that we only do, you know, once a year because it's so awful to do. And I think that inside of that there's probably something that you could, you know, tackle again incrementally. You don't have to solve the whole thing. But that's where I think there's the most potential to, to really not only make your processes better, but also to make everyone's lives easier and kind of like reduce the stress level overall, which is important. And then as far as getting your hands on the book, I mean, you can go to. It's being published by Manning, so there's an early access version of the book that has all the chapters that have been, that have been reviewed embedded so far available and they become available as, as I write them. And most of them are available now, actually. I think there's just a few left, but that's the easiest way to get your hands on it.

Thanks again for your automation awesomeness. The links to everything of value we covered in this episode. Head on over to testguild.com a387 and while you're there, make sure to click on the Try for free Today link under the exclusive sponsor section to learn all about Sauce Lab's awesome products and services. And if the show has helped you in any way, why not rate it and review it in itunes? Reviews really help in the rankings of the show. And I read each and every one of them. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack automation awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Automation Podcast. Head on over to testguild.com for full show notes, amazing blog articles, and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.