Chaos Engineering and Site Reliability Testing with Tammy Bütow

Five, four, three, two, one.

Drop that.

Welcome to the Test Guild Performance Podcast where we get together to learn more about performance testing and site reliability with your host, Joe Colantonio. Here it is. Here it is. Here it is.

Hey, it's Joe. And welcome to another episode of the Test Guild Performance and Site Reliability Podcast. One of the more popular episodes from my other podcasts in 2019 was the episode around Chaos Engineering. And it's actually a trend I've heard more and more about as the year went on. And I think it's really going to help you with your performance and site reliability efforts in 2020. So that's why I want to re share this content with you so that you start off the new year right with some new techniques that could probably help you make your application more performant and and more reliable. So today we're going to be talking all about Chaos Engineering with Tammy Buteau, a principal site reliability engineer at Gremlin. Tammy's going to explain all the different ways you can test your system to respond to stress so you can identify and fix failures before they impact your customers, saving you and your company the embarrassment of software downtime, bad publicity and lost revenue. And if you're a tester, you probably have heard a lot about how Netflix does testing using a Chaos Monkey approach. Tammy touches on this as well, so you don't want to miss this episode. Listen up. Let's face it, performance testing is tough. Traditional load testing tools create scripts that are bogged down in data that's hard to read and require a lot of work, even for simple playback. Load Ninja cuts out the dirty work by using AI to inspect and debug your code almost instantly. No coding necessary. No changing of hands or relays between teams either. Give it a shot. It's free and easy to try. Head on over to testguild.com smartbear and click on the link to learn more.

Hey, Tammy. Welcome to the Test Guild Performance Podcast.

Hey, Joe. Really great to be here today. I'm excited to talk to you about Chaos Engineering.

Very cool. Yeah, I just heard about Chaos Engineering and bought the company Gremlin recently and I got really excited and then I did some online searching and I came along a bunch of presentations you did on Chaos Engineering. So that'd be awesome to get you on the show. But before we get into it, Tim, you could just tell us a little bit more about yourself.

Sure. I work at Gremlin right now. I'm a principal SRE and I've actually been at Gremlin. I was the ninth employee. So I've been there for over a year, since before we did our Series A and launched the company publicly. And we actually just last week launched Gremlin Free, which is our first free product which everyone can sign up for online to do Chaos Engineering. And that's been a really amazing journey because before that I worked at Dropbox for a few years where I was the SRE manager for the storage team, which meant I was responsible for reliability and durability for over 500 million customers. Data, everything that's stored on Dropbox. So big responsibility, really important. And yeah, I was successful during my entire tenure, which is a big thing. You know, customers really trust you and you need to do a great job and like their data is precious. Without your customers data, without storing it safely, without making it available to them and you really don't have a business when you're working on something like that. And then before that, I also did a lot of Chaos engineering at the National Australia bank and I actually got to start doing Chaos engineering back in 2009 and that was where we would take down entire data centers. So doing region failover. And I've been doing it for a long time, it's pretty new. But you know, at a bank you actually have to do it to keep your banking license. But that's just a little bit of a background about me and I'm Australian too. That's my accent.

Very cool. So how did you learn about Chaos engineering? Like in 2009, did you use a tool for this or was it just all manual?

Yeah, so back then I actually learned about it when I joined my team. I was asked to work on a team that had actually been acquired. It was a mortgage broking business and when I worked on it at the very beginning, it was not reliable. It had a lot of issues. And I was asked to go on that team to help fix things, to help make sure that we can improve uptime, to make sure that mortgages got processed quickly. And just some background on the mortgage broking business because, you know, a lot of people have never worked in that field. And on the back end, the importance of the systems there and everything has to be processed really fast. Because if you're a mortgage broking business, then what you care about is you have a customer come to you, they say, hey, I want to buy this house. They go to the broker, the broker then puts the mortgage into a number of different providers, banks, all these different places where the mortgage might get picked up. And then as a bank, what you want to do is process it faster than all of the other banks and give them a really good offer. So the person chooses you and then the broker comes back to the customer that wants to buy the house and they say, hey, we were able to get you these different offers in the last 24 hours. And we all know if you want to buy a house, especially somewhere like Australia, it's very competitive, you can often lose out if you don't put an offer in within an hour or two. So you really want to be able to process those mortgages fast. And, you know, taking longer than 24 hours to process a mortgage is just not on. That's really bad. You want to be able to do them within hours, which is actually really hard because it needs to come in with all the data, needs to pass through the crediting team. A lot of automation needs to happen. It needs to flow through all of your banking systems. And back when I was working there, you know, some types of mortgages, I think the longest mortgage I've ever seen had taken weeks to process, which is obviously really bad. So that's just not good enough. And the other thing is, if you don't have reliable systems in that industry, then you can actually get fined by the ombudsman. So a person can come to you and say, I tried to get this home, I lost it because they didn't process the mortgage on time. And then I actually got. They can get a fine if they say, go to an auction and then say that they want to win the house, they want to buy the house and then they don't actually get it. That can cause a lot of problems for people. And I actually do have a fine against my name. So that's why I know about all these things. I mean, it's actually, it's very serious industry. You need to do a lot of work to be able to keep your banking license. And that means that it's actually the responsibility of production engineers, reliability engineers, to keep the bank up and running, otherwise everyone in the whole bank will lose their jobs. And, you know, that means that you've got 40,000 people, say, working at a bank and you're responsible for making sure that everyone has a job. And you do get reminded of that pressure very often because, you know, the regulators are always checking on what you're doing and you get assessed all the time. And the reason I started to do chaos engineering was because I first was introduced to the idea of disaster recovery testing, which a lot of people know about. If you work in big Enterprises, but it's not very common in small startups or even in the Bay Area. Like, the idea of doctor or disaster recovery retesting isn't really common. A lot of people don't build their systems from the beginning to be able to fail over to a different region. And we've all seen a lot of issues. Like when S3 went down recently, I was on call at Dropbox when that happened. That was really bad. A lot of the Internet went down. And it's because a lot of people are just in that one region and they don't fail over. When I was doing Chaos Engineering at the bank at the beginning, we actually did do a lot of manual work to be able to initiate those tests. So we might have some scripts, but we never had really nice automated tooling or like a web UI where you could say, this is the Chaos engineering experiment I want to run right now. I'm going to fail over these machines from this data center to another region. But that's what we built at Gremlin, so it's really exciting. And we're the first company to actually build something like that that has a web UI that has an API that allows you to programmatically do all of your Chaos engineering experiments, including disaster recovery testing, which we call region failover in the chaos engineering world. So, yeah, that's. It's come a long way, but I actually think, you know, the idea of doing manual tests back when we were doing it in 2009, obviously that's really bad. And just testing in a way as an industry has come a long way. Now we all want to do automated testing, and we want it to be done regularly. We want to measure it, we want to get results. I mean, this is totally your world too. So, yeah, I'm excited that it's come a long way, but I still feel like we're just scratching the surface in the world of Chaos Engineering.

Absolutely. And I think to some people listening, Chaos engineering might be a new term. They may be familiar with the term chaos Monkey from Netflix, but I think people think of that as more as a, like an automated test that's checking an application, not necessarily like a site reliability. So at a high level, for the folks that don't know, how would you explain, you know, what is Chaos Engineering?

Sure. So, to me, I like to say that chaos engineering is preventative medicine, and it's a disciplined approach to identifying failures before they become outages. So what we do is we proactively test how a system responds under stress, and then you can identify and fix the failures before they say, end up in the news, end up in the press. Because a lot of these big outages do end up in the press. And your customers will then be coming to you and saying, hey, why are you down? When are you going to be back up? And the idea of Chaos Engineering is that it actually compares what you think will happen to what actually happens in your systems. And we like to say that you're breaking things on purpose. That's what we always explain it as. Because the idea, the goal is that you want to learn how to build more resilient systems. And you can always sit in a room and draw diagrams on a whiteboard and have a hypothesis about how things might break or how things might be handled if you injected some type of failure. But you never really know until you do it. That's the big thing. Another way to think of Chaos Engineering is the vaccine analogy. So it's injecting a little bit of harm, but it's for the overall good. So that's what our founder, Colton Andrus, he previously worked at Netflix and then founded Gremlin with our, our cto Forney, who also worked at Amazon with Colton doing Chaos engineering and failure injection. And he also worked at Salesforce doing it as well. So we really like that idea of the vaccine analogy because people seem to understand that.

Very cool. So my background, my first job really was performance testing. So we'd put a load on the system, then we would monitor it. It almost seems like this takes it to a different level where it's actually injecting, like turning servers off. Am I understanding it correctly?

Yep, that's exactly right. So back in 2010, the Netflix team created something called Chaos Monkey, which you mentioned. And Chaos Monkey became really popular because people like it was right when Netflix was moving to the cloud. So they did their massive migration with during cockroach to the cloud from bare metal. And what they wanted to do is be able to make sure that whenever engineers were building something on aws, they didn't just think that all of those machines will be there all the time. And this is also makes me think of like cattle versus pet pets. You know, for servers. Like, you shouldn't think of your servers as pets. You shouldn't give them unique names and expect that they're always going to be there. It's more that it's okay for them to just disappear and go away, like that's normal. So Netflix created that idea and it was just to be able to shut down an instance on aws. And the thing there is when you're a software developer, you need to make sure you're not hard coding things. And a lot of people do that. Like when I've. When we started to do Chaos Engineering at Dropbox, when we announced that we were going to do it, some of the engineers were like, oh, like my service isn't ready yet, because if you take down my server, then that will impact my service. And it really shouldn't be like that because we all know if you're an sre, if you're a production engineer, then you know that AWS or Amazon can take down a server at any time. They might have to do maintenance, they might have a power failure at their own data center, they might have some type of hardware issue, There might be a kernel bug or something like that on the server. There are so many things that can go wrong, so you need to assume that the server will disappear. Um, but that's actually a change in mentality. So it was good in 2010 when people started to understand that because of the great work that the Netflix team did to like, really promote that idea, that at any time, not just during disaster recovery testing, once a quarter at any time, you should be ready for a server to go down. You shouldn't have to get paged because one server went down. And then you need to go jump online and bring that server back up. Whether it is like a database server or something, something like that, you know, you shouldn't have to be paged to then do kick off some manual steps to create a new database machine. So that's really where the idea was born. And over the years it's advanced. So it started off with just shutdown and that was just shutting down an instance. But now there's also shutting down containers because we have Docker, we have Kubernetes, they're shutting down specific Kubernetes, nodes. And beyond that, there's also the injection of different types of failures. So there's also different types, like time travel. You know, we're coming up to daylight savings right now, so there will be the change in clock time, and you need to be able to handle that. So we actually created a Gremlin attack, which is called time travel, where you change the clock time and ensure that your system can handle that. And daylight savings is one reason to do that. Another reason is because of security certificates or other types of certificates that have expiration dates. I mean, I know every SRE I've ever worked with. If you say to them, hey, have you ever seen an issue because of certificates expiring and then causing an outage? Everyone will put your hand up in a room, they'll all put their hand up. So yeah, that's another one. And some others are injecting, latency, packet loss, packet corruption, process killing. There are lots of different types of failure that happen, but the thing is you don't want them to happen in the middle of the night, 2am, 3am, when you're really tired and then you're trying to debug an issue. What you can do is you can inject the failure proactively, you can kill the process, make sure that everything is handled through your automation correctly and understand how long it takes, you can actually measure it, see how long it takes you to create a new server, make sure that everything comes back correct and healthy. So yeah, that's really the idea. It's come a long way, but we're still at the early stages and an exciting time.

Cool. Yeah. The certificate is what happens in our test system all the time. All of a sudden our tests will stop failing. We have no idea why. Turns out always it's the certificate. It's crazy.

Yep, exactly. Yeah, everyone will always. Everyone knows that example. It's just something that always happens. I've heard some very interesting stories like, oh yeah, we didn't know that the certificate was going to expire because the emails were being sent to someone that no longer works at the company because they set up the certificate originally. So there's all types of things like that where you know, it happens in the middle of the night. You're like, what's going on? What certificate is this? How do we get access to it? Oh no, what are we going to do that's so much worse? And you can just figure that out if you change the clock time.

It sounds to me like it's more for a tool to inject faults at the system or infrastructure level. But does it can also help you in terms of maybe types of orders or timing of actual application transactions?

Yes. So we did actually recently build a new product called alphie, which is application level fault injection. And that's really a new concept, it's much more advanced. So usually what people do is they start with infrastructure level. So we call that ILFI infrastructure level fault injection where you're injecting failure on the actual servers or the containers or network related failure. But there is the idea too of actually injecting failure into your application. And right now alphys supports Java applications. So we've recently written a blog post about that. But that's a really important thing to do as well. And the idea here is you can do very precise fault injection so you can actually match against any attribute you're already using. You can really do precise scoping of attacks to things like for example, custom IDs, locations, device types after you integrate alpha into your application. And that means that you can create really small blast radius chaos engineering experiments. And an example that Colton likes to say is a good thing to do is say, for example, when he was working at Netflix, they built something like Alfie. And what you could do is you could say, I want to inject failure on just my Netflix account. So Colton's Netflix account on his PlayStation device at his house. So you can see that's a very small blast radius. And it's much better to be able to do that. Instead of the idea of just doing disaster recovery where you fail over an entire data center and it impacts every single customer. What you can do is you can say, I just want to impact me and I just want to impact my one device. And you can do much more granular types of tests. You can say, how does this type of failure impact a PlayStation? Or how does this type of failure impact an iPhone or an Android device or on the web? And we all know like actually it's going to be probably very different and you assume that it might be the same sometimes when you're starting out in this world, but in reality things are actually really different. Like the amount of time things take, the different types of error handling that will occur on a PlayStation versus an iPhone, it's really different versus the web. So yeah, I'm sure you knew that world totally so well as well. And I think that's a really good thing to be able to do. But it is definitely more advanced. The other thing is you can use Alphie for serverless applications. So that's like Lambda. So if you're not running servers and you don't want to do infrastructure level fault injection, you can actually do it, yeah, completely with Alphie for Lambda serverless apps or Azure functions or Google Cloud functions. So that's another really new step. And yeah, if you're doing that type of chaos engineering, you really are a pine because that's totally brand new.

So this sounds very complicated. So you know how realistic is someone's hearing this? Like, yes, we need to do this, but are there any prerequisites someone needs to have in order to start with chaos engineering?

That's a great Question will. One of our big things that we've been focusing on over the last year is really helping people learn about how to get started doing Chaos Engineering. Because technically, it can be quite complicated and a little bit scary before you start doing it. Once you've been doing it for a long time, you'll become very good at it and you will no longer be scared to run Chaos Engineering attacks. You'll actually know that it really is a scientific process. You can have a halt button to stop your attacks at any time. Like, you know, you want to be able to have a big red button that says stop all attacks or stop this specific attack, which we have built that into Gremlin. So that's really important. But that gives you. That gives you a lot more confidence. And in terms of prerequisites, I like to say first, what you can do is you can start off by using Gremlin free. So you can just sign up if you go to gremlin.com free. The idea there is it's totally free, but you can just try it out. But don't try it out on production, just try it out locally on your own computer. What you can do is you can run Docker for Mac. If you download that off the Internet, you can run a Chaos Engineering attack where you shut down a Docker container. And that will help you just understand, okay, how do I monitor that the container actually got shut down? Do I have some type of monitoring service like say, Datadog or New Relic? And then what type of observability do I have? Am I using something like Sentry or Honeycomb to be able to understand what's going wrong or how that maybe impacts an application that I'm running, or how it impacts certain users. So that's an important thing to think about and never be worried about, you know, having to start in production first. It's totally okay to start on a local development environment. It's also good to do Chaos Engineering on staging or on a test environment before you do it in production. But ultimately, like, you really do want to be able to do these types of experiments in production, but it's okay to start not there. That's fine. It's actually much safer and we always recommend it. I think it's something to me that I like to say to people is it's very important to me is that Chaos Engineering is a set of skills that you really want to practice and master as an engineer, because the more you do it, the better you get and the more confidence you'll have in your skills to be able to do it. And also you need a really good set of tools where you can do those fine grained attacks. And I think about it a lot like doing like heart surgery, you know, like Alfie really is heart surgery. Like if you're going to be doing this type of failure injection, you want to make sure that you practice, you're ready for it and you do it carefully. And we've tried to write some tutorials as well to help people get started and we've traveled around last year to give some talks. We I gave a was teaching a Chaos Engineering bootcamp to explain how to set up your monitoring for Chaos Engineering in advance, how to measure downtime using your monitoring tools and also how to set an SLA so you know what your customers expect of you. You don't need to, you know, say that externally and have it in contract as an agreement. But it's good to have those SLAs and SLOs. So that's like your service level agreements and your service level objectives before you do these Chaos Engineering experiments. So you know what you expect to see. But that's all that the prerequisites are. And the other thing is paging and alerting. You do want to be using some type of service like that. Like for example PagerDuty or Victor Ops or Jira Ops is a new one. But you know, you don't need to have all these things in place before you do the Chaos Engineering. They are prerequisites. But the thing is Chaos Engineering can help you make sure that you've set up your monitoring correctly. Because the thing is, who monitors the monitoring? That's always a common issue. How do you actually know if your monitoring is correct? Do you test your monitoring? Do you test your audit? Do you audit your alerts? Do you test the thresholds are correct? A lot of companies are going to say no. I know at LinkedIn they actually do this, which is awesome. I've got some friends that are SREs there and they do that. So yeah, it is actually super important and it will help you reduce downtime. But yeah, those are the prerequisites. But the good news is Chaos Engineering can help you by making sure that they're correct. For example, you could run a Chaos Engineering experiment to ensure that when that failure occurs, you do see it pop up into your monitoring dashboard. You see it in your logs and then you also get a page for it. And the page page is the right person and actually pages their phone and they get it, they acknowledge it they know how to fix the issue and then they can resolve it. I've heard so many stories where page GD was set up incorrectly. People didn't get paid for hours when everyone was sleeping and they were down for hours and nobody knew. And then they woke up to tons of emails from customers.

You did mention monitoring. And I lifted this question from one of your YouTube slides. You had a four golden signals for monitoring. I don't know if you remember them, but you said if there was only four, you would tell them use these four for monitoring. Do you remember what those are?

Yeah, so that's actually, that's not something that I created. That's from Google. So that is the four golden signals. They have a book called the SRE book and yeah, so you can check that out. And it's a really, really great book. It's actually free to read. And that the four golden signals comes from the monitoring distributed systems section in the book and there that you need to monitor latency, traffic errors and saturation. But yeah, those are really good things to start with. And I think that's a great book because actually yeah, the Google site reliability engineering book will teach you what you need to set up. What are some of the basics, how to get started. Because it's, it's not easy. But the thing is when you use a service like Datadog or New Relic, they actually come with monitoring dashboards out of the box. So you don't just start with nothing. Like they tell you how to install the agent on your servers, which I think is awesome. There's really nice little guided tutorial, very simple. And you just install the agent, then it automatically starts to report up to your dashboards. And then straight away you, you will be able to use those out of the box dashboards to understand what's going on with your system. And then they also have integrations. Like for example, if you're using Redis, then you can just turn on the Redis integration and you'll get additional data from because you're running Redis and through the agent. So that's really great too. It's just a. You know, usually big companies have a lot of people that work on monitoring and they have monitoring teams, like dedicated monitoring teams. I think pretty much all of the medium to large tech companies in the Bay Area have monitoring teams. At Dropbox, the monitoring team was over 10 people and 10 engineers, mostly site reliability engineers. And then they actually teach the rest of the company how to do monitoring. Because the really important thing I think is that not only your SREs or production engineers or DevOps engineers should have to access to monitoring dashboards. What you want to do is you want to give monitoring tools to the whole company, all of the engineers at the company, and then say, hey, this is what you should be monitoring for your application. This is the types of alerts that you'll see if something goes wrong on the back end or something goes wrong with the infrastructure. Because you want people to know if you see a MySQL gone away error, your application is receiving those errors and your application is no longer working. You want to know how to handle that, you want to know how to fix that. You want to know how to make your application more resilient so it can handle when issues are happening. And you also want teams to coordinate and work together and reduce those problems. So that to me is a big thing. Monitoring to me is a great way to unlock the gates between teams and get teams communicating. Because we all know in distributed systems, all the systems have to talk to each other. So then your teams have to talk to each other.

Okay, Tammy, before we go, is there one piece of actual advice you can give someone to help them in their chaos? Engineering, testing efforts and let us know the best way to find or contact you or learn more about Gremlin.

This is one of my big pieces of advice and it is going to sound a little scary, but I like to say that it's important to focus on critical systems, not low hanging fruit. And that's my personal advice. And the reason is because if you focus on a critical system, you inject failure for a critical system, then you're able to show people why it's important and you can actually show them the positive impact that you can have because you're more likely to be able to identify important things that you need to fix, like bugs like infrastructure as code issues, configuration errors, paging problems, monitoring issues. And a critical service might be, for example, your storage layer, your database, your traffic layer like NGINX or Apache, a proxy, anything that's really critical. Caching layer like memcache for example. That to me is a critical place to start. And you know, start first by just injecting failure, running MySQL locally on a machine on your own desktop. That's a great way to start before you move into production. The best way to find me is definitely probably through Twitter. You can find me as tammybuto T A M M Y B U T O W and also, yeah, you can feel free to just reach out to me. My DM'S are open. Ask me any Chaos Engineering questions. And then we also have a Slack that we created for Chaos Engineering and Site Reliability Engineering, which you can find@gremlin.com Slack and that's really great to join because There are over 2,500 people in there who are like talking about Chaos Engineering, who are sharing the different experiments they're doing, who are sharing what they're learning with each other. So I think that's a great way to learn about Chaos Engineering and also meet other people who are using Gremlin for Chaos Engineering experiments.

Thank you Tammy for your Chaos Engineering awesomeness. For links to everything valid we covered in this episode, head on over to testguild.com p likeperformancep15 and while you're there, make sure to click on the Try It Today link under the Exclusive Sponsor section to learn all about SmartBear's awesome performance test Tool Load Ninja. So that's it for this episode of the Test Guild Performance and Site Reliability Podcast. I'm Joe and my mission is to help you succeed with creating end to end full stack Automation awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Performance Podcast. Head on over to testguild.com for full show notes, amazing blog articles and online testing conferences. Make sure to subscribe to join the Guild and continue your testing journey. This has been a Joe Calantonio production.