Chaos Engineering Controlled Disruption with Mikolaj Pawlikowski

Welcome to the Test Guild Performance and Site Reliability Podcast, where we all get together to learn more about performance testing with your host, Joe Calantonio.

Hey, it's Joe.

And welcome to another episode of the Test Guild Performance and Site Reliability Podcast. Today we'll be talking with Miko all about kiosk engineering site reliability through controlled disruption. Mikko is a recognized authority on chaos engineering. He is also the creator of Kubernetes Chaos Engineering Tool, Powerful seal, and the network visibility tool Gold Pinger. You don't want to miss this episode. Check it out.

This episode is brought to you by Smart Bear. Listen, Load testing is tough. Investing in the right tools to automate tests, identify bottlenecks, and resolve issues quickly could save your organization time and money. SmartBear offers a suite of performance tools like Load Ninja, which is a SaaS UI load testing tool, and Load UI Pro, an API load testing tool to help teams get full visibility into UI and API performance so you can release.

And recover faster than ever.

Give it a shot. It's free and easy to try. Head on over to smartbeer.com solutions performance.

Testing to learn more.

Hey, Mikko. Welcome to the guild.

Thanks for having me.

Awesome. Okay, Miko, before we get into it, is there anything that I missed in your bio that you want the guild to know more about?

No. Apart from the fact that it's powerful seal and people typically go by Mikko and Maiko will do just fine.

I know it's a cheesy question to start off with.

I always say this, but I always curious to know why someone writes a book because it doesn't seem like it's an easy thing to do. So why did you write this book on chaos engineering?

Yeah, well, I'm not gonna lie. It was painful. And if I knew it would be this painful, I probably wouldn't have written it. Well, you know, as every story starts with a phone call. They gave me a call from Manning and they were like, hey, this Chaos Engineering thing, we saw you do this. And this powerful seal tool, this is great. We should write a book about that. And they didn't mention anything about, oh, that's gonna take like a of years and you're gonna have to wake up at 5am to squeeze it with your schedule at work. But there's something satisfying about having written it. So I guess, you know, after a little bit of time when I forget how much of a pain it was, I'll probably feel good about it. It's also interesting because it was during the pandemic, so I Actually had the extra time. I didn't commit to work. So the time that I would normally spend on the train, I would just squeeze it in and kind of push it before work every day. So. So, yeah, a little bit of coincidence, I guess, but the rest is history after that.

How'd you get into Chaos Engineering? I know it hasn't. I mean, I don't know how long it's been around. I've been hearing about it for maybe two years, two, three years. I know Netflix started with Chaos Monkey, but how did you get involved with Chaos Engineering?

Yeah, sure. So I think it's been a bit longer now than a couple of years. But basically what happened is that I ended up working on a project. When we started, we built a new microservices platform. And at the time, Kubernetes was this brand new thing. It was version like 1.2, and I was one of the people who would get cold at night when it broke. So in order to ensure that, A, for the things that happened already, we would continuously simulate them and kind of ensure that the platform can handle them now, and B, try to proactively look at things that might be broken and fix them or at least gain some confidence that what we're building can survive that. We started writing scripts, and the scripts later turned into powerful seal. Just this tool that, apart from a funky name, can simulate, can do a bunch of things. It can simulate different scenarios. So give it like a YAML file and say, okay, break this or add latency here, or copy this or modify that and then verify for me that what I care about still is satisfied. And later we open sourced that and I started getting more involved with the community. There was a conference or two that started around the topic.

You know, it's kind of like a subset of sre. The way I see it, it's a nice tool that more and more of the SREs that I know just add to their arsenal. And wrote the book, and then I had to talk about it at a bunch of conferences and podcasts like yours.

Awesome. Yeah, the book's broken out. I think it's three main parts. You have the first one, which is Chaos Engineering Fundamentals. And as I mentioned, maybe it's just been a buzzword the past few years, Chaos Engineering. So I don't know if there's any misconceptions you hear. So just curious to know what you see, what you think Chaos Engineering is and what it's not. I don't know if you've heard any companies hype it up to be something you're like, that's really not what it is. I'm just curious to get your opinion on that.

Yeah, absolutely. So, you know, probably the worst thing about chaos engineering is the word chaos in it because it gives people so many ideas and it makes it a harder sell. So at the base of it, chaos engineering is basically this practice of experimenting on a piece of software, on a system to verify that what you think is going to happen during turbulent conditions is actually going to happen. And there's a bunch of good reasons to do that. In particular with the cloud and different, more and more complicated architectures that we're rolling out, microservices, all of that, you have things like emerging properties where all of your bricks might work very well in isolation. The sub components, and then when you put them together, they still work, but then something bad happens in the system. There's some kind of thundering herd here or some slowness there, and all of a sudden you have unexpected properties that you didn't account for. But it can also be much more rudimentary than this. A lot of people still focus on getting like 100% test unit test coverage, which sounds great, but in practice this is like the most basic stuff that you pick up. Then a lot of stuff happens at the border between the different components. And in a distributed context, we have all classes of things that typically go wrong. Like the worst thing in the world, slowness when it kind of works, but you're not too sure why. So, yeah, chaos engineering is basically a tool in your toolbox that helps you take a bit more of a scientific approach to that and try to simulate these things before they actually happen. Or like I mentioned before, a lot of people accuse of it when something bad already happened and they just want to prevent this from happening again. What it definitely isn't. And a lot of the headlines initially the communications were like, oh, breaking things in production. And if you're not breaking things in production on purpose, then all of a sudden that's not the case. Engineering. And I'm trying to kind of disprove that part view of the book and wherever I talk about it, that you don't need to be jumping to production. Sure, the holy grail is that you've extracted all the value from your testing stages and pre production stages and then you're so confident that you simulate failures in your production. But in practice, in reality, there's a lot of value to be had before you even get. So there's that the other thing that kind of refuses to be dying is the randomness aspect. The initial chaos monkey that kind of started the entire thing was random. It was more or less just taking VMs up and down in your AWS tenancy. But we've kind of moved from that and got into a much more controlled kind of environment where you can have much higher control over what you're actually breaking. And the randomness is still useful. It's still something that when you don't know where to start, you have a system, you might not understand it that well or you might not have a good idea and be super confident about it. The randomness is kind of like fuzzing, you know, in fuzzing testing where you generate the random inputs and verify that it continues working the way you expect it. This also works in the context of chaos engineering that you do random things like break random pod in kubernetes or add random latency to this thing in there, but it's that anymore. So randomly breaking things in production is probably where it all started years back, but we've kind of moved past that now. Say I'd put it in the myths and magic kind of category at this stage.

Nice. Like you mentioned, chaos may be a little misleading. That's why I like your subtitle, Site Reliability through Controlled Disruption. Controlled Disruption sounds better to me than Chaos. I think it's what people are trying to do. But great subtitle.

Yeah, exactly. Have no idea how many back and forth on the subtitle there were. But yeah, we settled on something that has the site reliability in it and the controls disruption in it. So pretty happy with how it turned out.

Was the control disruption your idea or was it the publisher's?

I don't want to take credit. No, it might have been one of my editors.

So you mentioned a few interesting things there. The first one is randomness. So if someone's just getting started, like you said, a lot of times they just go for, I guess is the low hanging fruit is randomness. But where would someone stop on it for? Would it be risk based? Like we know this is a critical area of our application, so let's start there when we do an experiment rather than just doing something random.

Yeah, I think that an important kind of aspect of we try to do with case engineering is changing the mindset from oh yeah, testing is for exceptional circumstances. Well, testing for exceptional circumstances and make the errors and the kind of bad things that are going to happen. Something that you think about as normal rather than exceptional. So a lot of the low hanging fruit comes from Very simple things. And this is a nice thing about Chaos engineering because if you look at the return on investment ratio, this being super cheap makes for a good investment when you get value out of it. So if you are not doing anything to your system right now, probably just looking at spending 15 minutes and looking at, okay, this is a weak point. This used to be some kind of single point of failure. We try to make it higher variability. Let's check that it's actually high availability by simulating taking down some of those things gonna already give you quite a bit of value. And in the book I have that nice example that is like a common gotcha. When you have a systemd, a service that is set to always restart and then turns out that on the default values it's not actually going to always restart despite it reads like that if it restarts a certain number of times within a certain time window, it will stop and all of a sudden your high availability thing goes down. So this is what I usually start when I actually show people, it's like, oh yeah, look, this looks very basic, has like five lines in it and you would expect it to just restart. In practice when you actually simulate that, you might have a surprise. So a lot of low hanging fruit can be had like this. Then the entire class of things is typically the networking layer. And the good thing is that now we have the tools that are easy to use. So easy that there is really no good excuse no matter what you're doing with your application right now. Going to be talking to some kind of database, some kind of external system, some kind of API something and just playing with what happens when I add slowness to it. Or do my timeouts actually work the way I want them to or is my backup and retry logic working? Super easy to do. There are different options. You can do it at the kernel level and introduce latency to the networking level. You can stick a proxy in between and with something like toxyproxy and simulate different behaviors. And I guarantee you that probably every single application that you are going to play with just a couple of hours here and there like that, you're going to find some things that are not exactly spectacular. Right? So there's that. And then like I mentioned before, a common kind of reason to go for Chaos Engineering is something broke. Oh, we didn't expect that. Let's maybe have some kind of like regression testing style part of simulations. And this is something that I have on every team that I interact with right now.

So do you see people taking that regression Test and making it part of their pipeline. When people check in code, it runs quick Chaos test, I guess.

Yeah, basically. Exactly. Yeah. There are tools now that make it super easy to write up as a quick test that you can integrate in your CI pipeline.

So another thing you mentioned that was pretty interesting was a lot of people, when they think of Chaos Engineering, they think of production. But some of the examples you gave to me sounds like it could almost be used when you try and decide on architecture. I don't know, can you actually shift Chaos Engineering left? I know that's another buzzword, but do you see that rather than focus on unit testing, which is important, maybe Chaos Engineering in this type of world, when we're going cloud native or relying on third party services, that Chaos Engineering should maybe be done a little bit earlier.

Yeah, so I usually preface that question or answer to that question with yeah, still do your unit test. This is the like magic silver bullet that's going to solve all your problems. But yeah, the behavioral stuff is probably, you know, I don't want to say might be more valuable, but it's definitely a very valuable addition to do it as early as possible because this is what you care about at the end of the day, right? The behavior of the system as a whole. And it's also nice because sometimes you don't really need to understand what's under the hood that, well, you know legacy systems. So working with them this way is actually pleasant, as in, okay, so this is what it's supposed to be doing. So let's work on the assumption of the behavior rather than looking under the hood to see how it's implemented. You know, obviously your mileage might vary in different systems, different requirements, but it's an option and it can be very comfortable.

So once again, this part one of the book, Chaos Engineering Fundamentals, you also talk about observability. Before someone could start Chaos Engineering, do they need observability and telemetry in place or does it matter? Like when you know what comes first?

Yeah, observability, another kind of supercharged word at this time. So what I try to do with the first part of the book is to distill the practice into something scientific enough. Kind of come up with a template of an experiment. And what you're very quickly going to notice is that you can't do science if you can't measure things reliably. So the observability in the book, the way I speak of it, is about being able to observe a variable in a reliable manner. Reliable as in, you know, it's reproducible. And also your measurement doesn't affect the outcome, you know, unless you're doing quantum physics. So there are obviously tools in the observability, that sphere that work on different levels. But it can be as simple as looking at the law log line or some binary output are going all the way to your advanced Prometheus metrics and doing all kind of funky stuff with that. The kind of bottom line is that you need to know, you need to have a reliable way of measuring whether your experiments succeeded or not. Otherwise there's no point doing that.

Absolutely cool. So that's part one. Like I said, it goes over the fundamentals. I think there's four chapters in there. And then you get into part two, which is Chaos engineering in action. I guess my favorite chapter that I saw was there's a monkey in my browser. I didn't know you can inject chaos into the browser the way you, you mentioned it.

Can you talk a little bit about that? Because I'm an automation engineer and when.

I see that, when I see like a browser involved, I'm like, this is something I as a tester probably can get involved in.

Absolutely. So when I was writing the book, the only existing ones were very high level. They were like, oh yeah, I should be doing this because it's good for you. And look, this company did that. It was good for them. But in practice, in order to be able to do that, you need skills and you need to understand certain technicalities to be able to implement this experiments reliably. I say reliably way more often than average, personally. But so basically I try to make this part about different kind of layers of the stack. And you're going to see that I start with like Ciscos and very basics of what the system does and how you can interfere with that and how you can kind of introduce failures and verify what happens with the application as a whole. And then I try to kind of give different popular situations a go and illustrate how you can easily implement this kind of thing. So it goes through building it into the application itself, which is a little bit unorthodox because you risk implementing it in a bad way, but can also be the only option if you only have access to the application and you're not like a platform engineer. And then things like Docker and Linux containers and Kubernetes and they're kind of trying to show that it works at every single level. Then I did this illustration of what you can do with JavaScript and JavaScript is great for hacking things around because it's super permissive, so it's actually super easy to see. And with the growing amount of front end code that is executed in the browser and the logic that goes there, it's super fun to see that some popular applications you can kind of simulate failed request, the backend, like a very basic thing, and you're going to end up with some inconsistent data and probably sometimes errors that not necessarily tell you that it's inconsistent data. And that data can be further sent into the stack. It can cause all kinds of weird things. So a lot of assumptions in the UI is made on a series of requests succeeding and if you fail one of them, bad things can happen. So this was primarily to just illustrate that you shouldn't be thinking of it as just, oh yeah, if I run a platform or I work for Google and I have 5 million users, the same kind of mindset and principles apply wherever you are in the stack. And having a little bit fun of hacking things in JavaScript kind of illustrates that well, I think.

Yeah, chaos at every layer like you mentioned, which is cool. Nice. So is there any misunderstandings people get when they're trying to implement chaos in action that maybe like you said, maybe they don't focus in on all the layers that are available to them and they just hone in on one or anything like that?

Definitely. I mean there is a few of those misconceptions I mentioned a few of the randomness, but I think a lot about this is this chaos in the name is a very double edged sword because it made for a nice block headlines in the initial phase, but then no manager wants to be here about adding chaos to their system. Right. So this misconception is that when we talk about the chaos here, we actually want to reduce the amount of it in the system. We want to kind of, you know, unlock the map a little bit in a gamer sense to know more about how it behaves rather than increasing it by adding factors that you can't control. So just kind of changing the chaos engineering to maybe resiliency engineering initially or just talking about, you know, site reliability typically helps with that kind of stuff. And then the second thing is that people seem to be wanting to really go for like the top level immediately because they like, oh yeah, Netflix doing this in production, we're gonna do it in production too. But like I probably have mentioned already twice in the last 20 minutes, getting there is a process and it might not always make sense to be doing that to begin with. Right. If You, I know if you run some medical equipment, you might prefer to choose the risk profile where you potentially shift with a bag, but you don't accidentally kill someone because you introduce a failure that you didn't account for. So you're gonna focus way more on the testing before. But if you're running a website with funny cats and people showing their kids to strangers, the profile is very. And you're probably going to be much more aggressive with that because then finding these things early is much cheaper than fixing them late. So, you know, with a big asterisk. It all depends. And if you are building a medical equipment on Sumi, there is value to be extracted at every stage of this of you shipping the code. And there is no reason to not apply the same kind of principles, common sense principles. I'd like to say that you do for shipping any kind of code. It's all code at the end.

Absolutely. So part three of the book is Chaos Engineering in Kubernetes. Why a whole chapter or a whole part just on Kubernetes? Is that the one most people use and that's why? Or is there more misconceptions around Kubernetes? That's why you thought you need to dedicate a whole part of the book to it, right?

Yeah. So I've been maybe a little bit biased because I kind of experienced the Kubernetes story from day one and kind of seen how it exploded. And this entire deployment world seems to be just consumed by Kubernetes at this day. But I wanted to do a deep dive into how Kubernetes actually works under the hood so that people can understand a little bit better. Because, you know, it's one thing to go and install Kubernetes following tutorial, and it's a very different story to actually run that in production. Some SLAs at this SLOs, so I've had experience with the latter and I just wanted to kind of give an extra chapter. I could have just done one chapter on working with Kubernetes to implement Chaos Engineering, which is nice because you have this APIs and you can. You get some of the observability built in and. And you can do a lot of things automatically much easier than you would otherwise. But there's also this under the hood of all these moving parts and the bits that implement the behavior of Kubernetes behind the scenes. And I wanted to also do a chapter on playing with this and verifying that your Kubernetes setup is reliable and this highly available and so I ended up writing this long chapter on that and trying to kind of connect the dots between containers and then Kubernetes and what to do to make it reliable if you are the one responsible for Kubernetes itself. And besides, it's a fun thing to work with. Brand new. A lot of people work with it daily. It's open source, so it was a fun topic to write about. So I guess that was a factor too.

Yeah. Awesome. And I think this, maybe I'm wrong. This is what I saw. It's the first time you introduced Powerful Seal. Powerful Seal, just not just for. But is Powerful Seal made for specifically Kubernetes and Chaos Engineering.

Yeah. So Powerful Seal focuses primarily on Kubernetes, but it also talks to the clouds. So this is so that you can implement the scenarios for the software running on Kubernetes, but you don't always want to trust kubernetes to do things for you. So a common kind of scenario is, okay, so what happens if I take down a certain percentage of the nodes that implement my Kubernetes cluster? So Powerful Seal can talk to Azure and Google Cloud and AWS and OpenStack and a bunch of others to be able to implement this kind of thing. And that typically gives people quite a bit of mileage. But yeah, the Powerful Seal story kind of precedes the book and this is kind of where the adventure started really for me, at least for the Kubernetes. Chaos Engineering.

Where does Goldfinger Gold Pinger, not Goldfinger Gold Pinger come into play?

Yeah, so Goldpinger is such a fun story. So when you, I don't know if you're personally familiar with Kubernetes, but for me, Kubernetes basically requires an overlay network so that your pods can have their made up addresses and something needs to route that thing. So when we first implemented that microservices platform, we ended up with a brand new piece of software called Lano and it was written by a few folks I remember well at the core OS ecosystem and Kubernetes people are like, yeah, yeah, you should just run that for all of your networking. Like, okay, so great. We have this new piece of software that we don't know much about. We've never debugged it, so how are we gonna do with that? So in order to kind of gain a little bit more confidence that this is not gonna blow up while I'm asleep, I had this idea to basically write this little thing that would run a little process that would run on each of the boxes and it would query kubernetes to see all the other boxes or a subset if you can't afford all of them and basically periodically ping each other, measure the latency and then generate nice dashboard debugging and metrics so that you can alert on these things. And I don't think much of that because it was literally, you know, a couple hundred lines of code to get that off the ground. But then I ended up, my company ended up open sourcing then and the next thing I knew they had like a million downloads from Docker Hub and turned out that all kinds of people were like oh yeah, we're using Gold Pinger, such a great idea. Oh great. And then it turned out that some people from the public clouds privately were like oh yeah, we're using this tool too. Such a great thing. I don't know how we didn't think of that. So this is now like kind of add on a must have for every single Kubernetes cluster that I even touch because otherwise it's difficult to get insight into how well your network is doing and and spoiler alert, it's often doing much less well than you think.

Yeah, absolutely. So I think it's a must read book. I'm going to leave people with a little thing like a little cliffhanger so they have to get their hands on. The last chapter is Chaos Engineering for people where you walk through what is the mindset or mind shift required for effective Chaos Engineering as well as how to get buy in from the team and management. So I think people definitely just for that chapter alone should grab this book. So I think as we move in cloud native, I think Chaos Engineering is going to become more and more important. But just my opinion.

But anyway. Okay Miko, before we go, is there.

One piece of actual advice you can give to someone to help them with their Chaos Engineering testing efforts and what's the best way to find contact you or get our hands on your book Chaos Engineering site Reliability through controlled disruption.

Sure. If you want to contact me, mikolaibalikovsky.com is my homepage. Probably LinkedIn is the easiest way to reach out. If you want to get my book, it's available on Amazon or mining.com just put chaos Engineering in, it should show up and if you have any feedback on the book let me know. I love hearing back about how people liked it.

Thanks again for your performance testing awesomeness links to everything of value we covered in this episode. Head on over to testguild.com P88. And while you're there, make sure to click on the Try them both Today link under the exclusive Sponsor section to learn all about SmartBear's two awesome performance test tool solutions, Load Ninja and Load UI Pro. And if the show has helped you in any way, why not rate and review it in itunes? Reviews really do matter in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Performance and Site Reliability Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack performance testing awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Performance and Site Reliability Podcast. Head on over to testguild.com for full show notes, amazing blog articles, and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.