How to Practice Chaos Engineering with Michael Sage

Welcome to the Test Guild Performance and Site Reliability Podcast where we all get together to learn more about performance testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Performance and Site Reliability Podcast. Today we'll be talking with Michael Sage, all about how to practice chaos engineering in a real world example of how it can be done in the banking industry. If you don't know, Michael is a solutions architect for Gremlin. He has over 20 years of experience from system operations, software delivery, performance management, and prior to joining Gremlin, he's been part of all the big performance companies that you've probably heard about like Mercury Interactive, Hewlett Packard, as well as groundbreaking startups like New Relic, blazemeter and Sauce Labs. So he really knows his stuff. Really excited to catch up with them. It's been a while. You don't want to miss this episode. If you want to learn about the hottest topic in performance engineering, Chaos engineering, check it out.

This episode is brought to you by SmartBear. Listen, load testing is tough. Investing in the right tools to automate tests, identify bottlenecks and resolve issues quickly could save your organization time and money. SmartBear offers a suite of performance tools like Load Ninja, which is a SAS UI load testing tool, and LoadUI Pro, an API load testing tool to help teams get faster full visibility into UI and API performance so you can release and recover faster than ever. Give it a shot. It's free and easy to try. Head on over to smartbear.com solutions performancetesting to learn more. Hey, Michael, welcome back to the Guild.

Thank you very much, Joe. It's great to be here. It's been a while.

Yeah, it's been over six years last time I checked. I think you were in episode 75, which was like 2014, 2015, so it's crazy.

So.

So you've been all over the place. What's new? Is there anything I missed in your bio or. Before we jump into it?

Yeah, no, yeah, just a lot of ground has been covered over the last few years. I mean, the whole industry moves so fast these days, right? I mean, Kubernetes is everywhere. I don't even think that was much of our discussion last time we met, but yeah, no, it's really exciting. Chaos engineering, definitely one of the cool new topics. I'm super thrilled to be at Gremlin. It's new for a lot of folks, so what we're doing out there is, you know, it's partly educational, which I really enjoy doing. Showing People, how we do this, talking about strategies for chaos, how it integrates with observability and performance and the other things that I've done in the last 10 years or so. So it's an awesome time.

Awesome. So it seems like say reliability engineering is now the hot topic. You've been in pretty much every phase of performance engineering, it seems like through actually load and stress testing, monitoring, even been at some of these vendors, cloud vendors, where you run everything in parallel in the cloud. So why, why Chaos Engineering? Why, why Gremlin?

That's a great question. Something that occurred to me recently in general, sort of our relationship to failure or performance degradation. For so many years, I mean really two decades, it's been something to avoid for me.

Right.

Just be afraid of failure, be nervous about failure, be anxious about failure. It's this kind of constant push to protect ourselves and it's almost always feels defensive. Right. And something is always going to fail and then we go try to figure out why. And it's interesting to think about chaos engineering now because it's the opposite approach. It's embracing failure, going toward failure, creating failure on purpose, which it's kind of interesting. It's sort of like we just realized at a certain point that, okay, we're not going to be able to stop these outages from happening no matter how hard we try. You know, we have shorter release cycles with Agile, we've containerized everything. You know, we have auto scaling and redundancies and failovers throughout our architectures and all these things that we put into place to try to protect the app, to try to keep our availability meeting our service level objectives and still things fail. And it was sort of, we finally got to this place where the best to get in front of this is to make it happen on purpose in a controlled way and then see how we can deal with it. What can we do to overcome it? How do we, how do we manage these outages and these incidents? What do we learn through practicing this on purpose? Which is a tremendous way to understand your applications and your systems before they go wrong.

Absolutely. So just, you know, you being a longtime performance engineer, we've both been in this industry for a while. You know, back in the day we used to do things like soak test and stress tests to try to break the system. Is chaos engineering just a cool word to get people more involved in performance testing or is it something completely different in your mind?

It is different. It is a different approach. It actually works quite closely, you know, hand in hand with load testing, different Styles of load testing. So, you know, with load testing, I think we wanted to kind of do two things. Like we want to. We want to ensure that our traffic profiles work, that our application responds to what we expect the load to be. Right? And so we have our different styles of load testing and as you say, like a soak test or a stress test where we're kind of strategizing how we throw that traffic at the application and then seeing how the application responds. We're adding something to the mix with Chaos Engineering because we're still gonna do that, right? We're still gonna send the traffic, we're still gonna load tests, we're still gonna do these different styles of load tests, but then we're gonna unplug something or then we're going to take, you know, introduce network latency on purpose. Then we're gonna introduce process dying or taking a pod offline or something like that. Right? So different kinds of fault injection that we do on purpose during the load test and then gauge how that affects the application, what happens to downstream and upst, you know, what happens to our service levels and the different indicators that we're watching as we perform these tests. So we're still kind of doing it in a low testing environment. We're still sending that same traffic from our load generators and styled however we designed it. But then we're injecting faults because we know they're going to happen. What happens then? What do we learn? What do we expect?

Absolutely. So why do you see more companies embracing Chaos Engineering? Has technology changed since last time we talked that makes this easier? Like you mentioned, kubernetes taking off. Is it because we're able to create more realistic environments now and actually play around with turning things on and off, which makes this type of testing actually more feasible for companies?

Yeah, I think. I think, you know, it's another interesting tension that's happened is we've tried to reduce the failure surface, you know, the surface area of things that can fail by containerizing them or by releasing, you know, short iterative releases with small changes. But at the same time, with the adoption of cloud and with the adoption of orchestration like Kubernetes, we've increased the complexity tremendously. So what we've done is in our individual projects or services, you know, if I'm on a team that's responsible for one microservice, we understand that very well, you know, and we can test it really robustly and we make small changes and we know what breaks it. But once we put that out into the ecosystem where there are, you know, dozens or hundreds of other services interacting with it, where it's running on complicated infrastructure in AWS or Google Cloud or Azure, what have you, and all these different kubernetes, orchestration components, schedulers and APIs and all these things that all become places where it can fail. And so while to us in our work we understand it pretty well, but once it's out in the wild, in production, the complexity is just monumentally larger than it's ever been. And any failure in that environment can have a ripple effect, right? So if you know, some service, third party service goes down, what happens to our application? And, and you're not going to know until it happens unless you do Chaos Engineering. In Chaos Engineering, we can model that failure so we get ahead of it, right? So instead of waiting until the outage happens, when an outage happens, everybody panics, right? Where all the adrenaline, you know, shoot to through the roof. We're all trying to get the service back online or reverting their deployment or something like that. And it's hard in that frame to digest what's happening clearly, to understand it, to reason about it, to prepare for it next time and so forth. So we end up doing a post mortem or something like that where we try to do that after the fact, but so much has been lost because maybe we weren't taking good notes or maybe we were too frantically just trying to get things back online. With Chaos Engineering, this is a great use case for Chaos Engineering. Reproduce that outage on purpose, right? So last week something went down. You know, we had this chaos or sort of natural chaos, uncontrolled chaos introduced into our environment and we all panicked and, you know, we got it back online. We feel pretty good about it, but we're not really sure what we learned this week. Let's set aside some time, reproduce that outage and using a tool like Gremlin to do that, we can create those same conditions. Then we can do what we call a game day or a fire drill, where we practice our incident response, right? Create the conditions, get the right people involved, make sure we're using slack or teams or what have you for collaboration. Check to see that our observability solution is giving us what we need. The right alerts, the right telemetry, do we have any gaps in our monitoring? Is this something we would have known about in advance if we had only had the right telemetry for it? So Chaos Engineering helps us work with this explosive complexity that's happened over the last five or 10 years.

Absolutely. So speaking of this complexity, then how does someone even know where to focus in on? I know you mentioned like known issues or that that may have happened, production, trying to replicate them when you're trying to create the conditions. But is it truly chaos that you just. It's. That's what chaos engineering is for. It's able to find these things while you're really knowing where to focus your energy in on?

There are a couple of ways to answer that, I think. So I think one of the things that we like to keep reminding everyone or educating folks about is that, you know, in the early days and a lot of people first heard about chaos with Chaos Monkey from Netflix, right? And our founder was a part of that, part of Netflix, also part of Amazon. Our founders, both of them come from practicing chaos in its early days and kind learning a lot of lessons from the initial approaches that were taken at these large companies. And so a lot of people think of Chaos Monkey, which would, you know, back in the day would just randomly shut down EC2 instances. And you just kind of never knew when that was going to happen. And you kind of had to, you were forced to learn as a team, forced to learn how to adapt and how to, you know, architect your application to better handle those unpredictable outages. We don't really do that anymore. I mean, you certainly could. But what we suggest is controlled chaos, you know, think about what you're going to do. And so we gauge it, sort of guide ourselves using a kind of scientific process where we try to understand the system as it is right now, kind of the steady state, observing the system, you know, understanding the baselines. Then we hypothesize about what we think will happen if a dependency goes down or if latency increases or if a process dies or whatever kind of system failure we're talking about. We'll hypothesize about that. I expect that, you know, if this non critical, let's say we have something like a banking application and we have a user profile service, it's a place where a user can go and update their address or something like that. Let's say that that's not as critical of a service as the balance reader service that shows the user how much money they have. These have different criticalities for the user experience. The user might be able to tolerate not being able to change their delivery address or something, whereas if the balance doesn't show up on their account page, they're going to freak out. Right. So we would reason about those things like we might say, if user profile service goes down, I expect that the rest of the application will behave as normal as it usually does. I expect that that profile service going down will not interfere with the balance reader service or the payment service or the mortgage application service or what have you. So we think about that and then we go to a user tool like Gremlin to actually create that outreach. So I would go into Gremlin and I would select that service, my user profile service. I would then take it offline using Gremlin, and then I would observe my application and look at the impact. Can I still make payments? Can I still apply for a loan? You know, do I see my balance and do I see all my transactions? And if I do, then my hypothesis was correct. But I might not. There may be some underlying relationship between that profile service and the other services in my bank that causes chaos in a way that we didn't know about. The unknown unknowns. Right. And so that's the whole point of doing the experiments. You have a hypothesis, you run the experiment, create that outage or inject that failure, then you observe what's really happening, take great notes, save those notes. You can do all this in Gremlin, by the way, and then, you know, make the changes you need to make and do it again. And of course, over time we want to automate these things as we get more experiments that we want to run reliably every time we do a release or deployment.

Now I love that approach. You did a demo that's actually called the five steps of Looks like the scientific method for chaos engineering.

Yeah, that's right.

One of the steps that seems the hardest to me, and it's always seemed been the hardest for performance testing for me was not necessarily creating the test, but actually analyzing the data. And that was when I knew I had a server, a database server. You know, I knew the components that made up the system. Now, like you said, there's so many unknown unknowns. So if I had problem analyzing where if issue was when I knew the known knowns, how do I know where to analyze issue if I don't know the known knowns?

Yeah, that's a great question too. So like there are so many like chaos engineering on one hand. What we're providing are ways to eject this failure, right? Consume some resources, cpu, memory, disk, what have you, or interfere with the network, either black hole something, just drop all network traffic, or increase jitter and packet loss or latency, these kinds of things that we're actually going to do. But from a higher level, we're strategizing. We, we have a reason for doing these things. And so these.

The lens through which we approach these experiments. And one really great use case, like we talked about recreating an outage or understanding the blast radius around dependency failure, another really good one is validating our monitoring and observability. Right? So I actually just the other day in, in the demo environment had an actual legitimate outage and I had to go figure it out. This is like not even me doing chaos. It was just, you know, the, the application started throwing errors and there were all sorts of problems with the front end and I had to go inspect it and happened to be using a leading observability tool and realized that I had not added the integration for postgres. And I could see to a degree, I could see in my logs that the database was the issue and I could see when it started, but then I lost visibility. I couldn't get into the database metrics and actually understand the problem, which showed me a gap that I had in my visibility. And that's actually a really great use case for doing chaos engineering is, you know, inject these faults, then go inspect and see if you can troubleshoot the issue. We do these game days and game days, that's one of the real reasons to do a game day. We want to think about the experiments that we're going to run, but we also want to think about the people that are involved and the processes that we're practicing, the incident management processes that we have. And this is the kind of thing that could come up because we inject this outage, the team gets together, we're all collaborating on Slack, and someone is designated to go figure it out, right? Someone's designated to actually be heads down in the code or in the logs, try to figure out what happened to that outage. As part of this experiment, that person would have discovered that we have some visibility gaps or telemetry gaps in our database and now we can fill those, we can add that integration, get those metrics, and when it happens for real production, we should be in a better spot. Of course, we can go validate it again with another exercise.

So would you say chaos engineering then is like a process that should only be done in a more mature organization that actually has some observability and telemetry in place? Or do you see the opposite, where it actually exposes where you're immature and where you then need to, like you said, maybe beef up your observability?

Yeah, I Think absolutely. It's something that you can adopt wherever you are, and you're going to learn different things because of it. You know, if you're a team that's not done Chaos Engineering before, maybe you're still even just starting to adopt SRE practices. Right? You might not have even defined all your SLOs or your error budgets or things like that. It's a nascent team and a lot of organizations are in this position. Maybe they're a newer company or maybe a newer team coming together and starting to think about these more modern organizational practices. You might not know what you expect. And so a perfectly valid use case of Chaos Engineering is simply learning how your applications will break. So you might still formulate some hypotheses, but you have nothing to go on yet. You're just starting from scratch. So it's a learning experience in that sense. You start to internalize how your systems interact with each other, where the ripple effects are felt simply by injecting 10, 100 milliseconds of latency, let's say, into some service and realizing, wow, that actually breaks this other service totally unexpectedly. So that learning process is a very important part of Chaos Engineering is to kind of discover the problems. Then let's say you're a very mature organ and what you want to do is actually test your disaster recovery planning. Right. You already have, you know, your application might already be architected, let's say for regional failover, geo failover. You're running in, you know, production in AWS and US West 1, but you have failover to US East 2. You can actually cause that to happen and test your disaster recovery, your business continuity, planning around that. And that would be, I think, a more mature use of the same kind of technologies.

Nice. So who owns Chaos Engineering that you've seen? Where does it typically lie? Who is it the developers like, who like. To me, it's still, I don't think it, the word's out, even though it has become more and more popular. But like, who do you then say, well, this team should be doing, or does it depend on what organization you go to?

The teams that I'm thinking about kind of relate to the use cases I was just using. I think we talk to a lot of SREs. Absolutely. We talk to a lot of DevOps teams from both a developer standpoint and from an operations standpoint that I would say is definitely where a lot of the interest is right now. People are learning about this. They understand the importance of doing it it and why it makes sense, the value of doing it. But we also get a lot of interest from business continuity, right. From larger enterprises that need to test these disaster recovery plans and they want to be able to actually run that kind of game day, which is much more impactful for the organization and the business than it would be for a DevOps engineer. Simply trying to make sure that the database is scaling properly or something like that.

I used to work for an insurance company. They used to do this doomsday scenario every year. It was really hard where they take down all the servers and they'd have to recreate it and then another site, it seems like. And it took a lot of resources and time. It seems like chaos engineering for modern practices is what they could probably utilize for those type of scenarios. For sure.

Yeah, absolutely. What we're doing is creating that real disaster. I mean, obviously we're not creating a hurricane taking out a data center, but we're mimicking that in a way. Right? And so shutting down all traffic from your application to, you know, an entire AWS region should trigger that failover plan that automated failover, redirect the traffic and bring those other servers online or that other site online. And that's what we would do. We would actually call cause that failover. They'll trigger the conditions that would cause that failover to happen. And then you can verify and validate, just like we would with any other test, validate that the what we expected to happen did happen.

Nice. Now, you did mention earlier something called a black hole. And actually I'm not familiar with this term. This is a new term. What is black hole? Is this a gremlin thing or is this something that you coined or what.

Is it we might have? I think it's a lot of these. Chaos engineering is kind of growing in popularity and it, it's a little bit new. And so there's some new terminology that's emerging. Blackholing, the way we think about it, is shutting down all the network traffic from a service to and from a service. So we're basically making that service appear to be offline from a sort of Linux internals point of view. We're working with a traffic control which allows you to shape the network behavior the way you want. You know, you can introduce packet loss or you can, you know, have everything go out through one interface. One of the things we can do is, is just shut down all ingress and egress to that nic, basically. And so it appears to the rest of the world that that service or that server, what have you, is offline Right. So if we talk again about the bank, if that balance reader service, if we wanted to see what happens when that goes offline, we just select that pod, it's running kubernetes, and we just shut down all the network traffic. We block all network traffic to and from that pod. So to the rest of the application, it appears to be offline. Now, one of the things that these days, if you're running in Kubernet and you have a liveness probe set up, the Kubernetes should automatically schedule a new pod and roll that out. So one of the reasons again for doing chaos engineering, one of the ways you might think about it is to test that Kubernetes or whatever you're using for scheduling or auto scaling is doing what it should do. But we also will sometimes do things like disable that liveness probe and see how a sustained outage will impact the rest of the application. But a black hole is basically just what it sounds like. Everything just goes into a black hole. No communication gets through.

So you actually have a really great article on this. I'll have it in the show notes where people can actually div in a little bit deeper along this type of thing. When you're talking about black holes, you talk about different sizes of black holes. And to me, I would think people would automatically go to like doomsday type black hole supermassive. Is there value in all chains or do you usually recommend they start small or does it matter?

Yeah, it does matter. I think we do recommend that we start small. Right. I mean, again, the theme for doing chaos engineering is controlled chaos, thoughtful experiments. And so we want to know what we're doing each step of the way. Way we call the impact that an experiment has. We call that the blast radius. Right. Which is somewhat common terminology for any kind of outage or exploit or whatever. Like how big was its impact? So we think about starting targeting a small blast radius first. We want to internalize our understanding of what happens both with the chaos tool and with the impact to our apps. I don't know that, you know, unless you're talking about like disaster recovery like we just were. That would be a good use case for black holing the entire system. System. But, you know, that's kind of a use case. It's more interesting to black hole parts of the system, I think, and see how the other services and other dependencies respond. Sometimes you'll inadvertently break something else. Like the front end service needs to talk to that balance reader service and you break the balance reader service what does the front end do? You know, does it just slow down? Does it time out? Does something else break? So black holing to me can be more useful in a surgical way. However, a broader attack, a broader chaos experiment that I think is useful is a DNS outage. Because we see those in the real world. You know, I think fastly and I think Akamai both have had big DNS outages in the past six, six or eight months or something. And that's the kind of thing that you can do with Gremlin. You can just simply select all of the components in your application with one little toggle and then just break DNS for 5 minutes and see what happens. So those are, those tend to be kind of fun because a lot of that impact, you're unclear, like, which services are even using domain names, host names to look things up, or which ones are going to IP address and won't be affected by that. So you can learn a lot by doing that. Reproducing that kind of outage.

Nice. Now, kind of off topic or like kind of changing the topic a little bit is just came to my mind. I'm just thinking of people that came up as performance testers and it seems like not it's becoming rarer, but that they're morphing into things. If someone's listening, I always get asked, like, I'm a performance tester, what should I be focusing in on? Do you think they make excellent chaos engineers, or do you think that's something they should include in their toolkit? Or as you said, it should it just be like a DevOps SRE type of thing and performance testing never overlap?

Absolutely. It's an evolution. It's an evolution of the performance engineering toolkit, without a doubt. I think that that is there are two sort of tools that are sort of hand in glove for chaos engineering or hand in hand. And one of them is load testing, performance testing, and the other is observability. And so these two elements are really important. And I think as a load tester, as a performance tester, I would be thinking more probably in terms of resilience. Right. Rather than chaos. Right. And we, you know, we talk about that a lot right now in terms of the jargon of the industry street, because chaos has these connotations of being, you know, just taking a sledgehammer and randomly breaking things, or the monkey or the simian army and those kinds of things. Whereas, you know, resilience has a little bit more of a sophisticated sound to it maybe. But the idea Is that. Is that's what we're doing. We're actually as performance testers, you know, we're traditionally used to exercising our application based on the traffic that we're sending to it. So we're seeing how the application responds to a particular load profile or maybe as we said earlier, a soak test over many days or stress test, you know, just maximize everything at once. What we can do now is introduce those faults or inject those failure modes to further understand how our applications respond to that kind of stress. So it's not just about requests and response timings or, you know, the metrics that we're collecting when, you know, there are a hundred concurrent users or something like that. It's more about understanding that when a condition in the application changes and we do that on purpose. And so, you know, okay, my load seems good right now, and in my staging environment, I'm running whatever it is, a thousand concurrent users, things are responding, the seas are calm. Let's inject 500 milliseconds of latency. Then what happens? Then what kind of response times are we seeing? Where are we seeing errors? Are things queuing up at the database? You know, we create conditions that take us further into to that process of understanding how stress and load affect our application.

Nice. So, Marlo, you've had a long career. It always seems like you're always at a hot company. I'm always looking at 2022. I do like a blog post on trends for 2022 and beyond. Seems like kiosk engineering is definitely was one of the trends I had for this year. Any other trends you see that performance engineers or SREs should know about, that you're hearing more and more about as you're in the field working with different companies?

That's a good question. I. I think building on what I said a few minutes ago, I think resilience.

Is kind of a new way of thinking about all of this. It's not a new idea, but I think we're starting to see that term used more and we've been kind of building up to it for years with all sorts of fault tolerance that we've tried to build into these applications. But it's getting kind of its own place on stage, or it's getting its own chance to stand out out as a topic that we think about. Which I think just using the term resilience causes us to think about it in a slightly different way. We've been saying reliability for a long time, right? Reliability is really about our SLOs, right? It's really about are we hitting the goals or not? Are we exceeding those goals? You know, we know how to track that. And then we, we think about our reliability based off that. Right. Reliability is a state we're either reliable or not. Whereas resilience is more about how our systems respond to outages and where that tolerance is greater or smaller and how we then, you know, this is going to be a feedback loop back into how we develop systems, you know, how we manage outages, what practices we adopt as SRES or as DevOps engineers. So I think that that's an emerging space and I think there are going to be a lot of interesting developments of patterns, ideas, products, tools that are going to come in that silo, if you will, in that kind of under that banner headline of resilience millions. That's how I think about it right now anyway. You know, being at Gremlin, I think about it every day, so.

Very cool. Cool stuff. All right, Michael, before we go, is there one piece of actual advice you can give to someone to help them with their Chaos Engineering SRE efforts and what's the best way to find or contact?

Well, I'm easy to find. Michael Sage, gremlin.com and anybody's welcome to reach out. I think I would suggest that folks look into a couple things that we do and one of them is boot camps. Chaos Engineering bootcamps. I think, you know, this is new for a lot of folks and we do boot camps pretty regularly, a couple different levels of them. So newcomers. There's a Bootcamp 101, I think we call it, which is a place where you can come and there are hands on exercises. We provide all the tools and environments that you need. So you literally show up. We'll teach you how we do Chaos Engineering. Hands on keyboard most of the time. Great way to learn about the concepts, but also working with other folks, books, you know, we do little breakouts and things like that. The boot camps are phenomenal. We also offer certification, right? So Gremlin certification. You go to gremlin.com certification and you'll see our program there. It's. It's a great way to kind of get that badge as well as learn about Chaos Engineering and kind of take your skills up to the next level. And the boot camps and the certifications work really well together. Go to a boot camp, you should be ready to get certified. And then the third thing I would say is start thinking about game days. If you're not thinking doing them. And game days are really important. It's really important to get the muscle memory about how your teams work, how your applications work. Sort of like start working on those things. If you don't have a good incident management process, develop one and then practice it and use chaos engineering to trigger the actual conditions that are going to help you exercise all those components. Make sure folks have the right roles and responsibilities, and when the outage really happens, you'll have that muscle memory to help you through it and help you feel confident about solving those problems.

Thanks again for your Performance Testing Awesomeness Links to everything of value we covered in this episode. Head on over to testguild.com P75 and while you're there, make sure to click on the Try them both Today link under the Exclusive Sponsor section to learn all about SmartBear's two awesome performance test tool solutions, Load Ninja and Load UI Pro. And if the show has helped you in any way, why not rate and review it in itunes? Reviews really do matter in the rankings of the show and I read each and every one them of so that's it for this episode of the Test Guild Performance and Site Reliability Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack Performance Testing Awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Performance and Site Reliability Podcast. Head on over to testguild.com for full show notes, amazing blog articles and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.