The Evolution of Chaos Engineering with Benjamin Wilms

Welcome to the Test Guild Performance and Site Reliability Podcast where we all get together to learn more about performance testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Performance and Site Reliability Podcast. And today we'll be talking with Benjamin all about the evolution of Chaos Engineering. I've been hearing about Chaos Engineering all over the place. I. I think he has a unique take on it. If you don't know, his journey began in April 2018 with the chaos Monkey for Spring boot, which Benjamin published as an open source project. And prompted by really great feedback he received, the idea was born to actually create a company based on the topic of Chaos engineering. So in the fall of 2019, Benjamin founded Steadybit together with some co founders and during their time together as a German based consultant company called Codecentric ag, they gained a lot of enterprise and gained a lot of expertise in Chaos engineering and the implementation of distributed highly available systems. So already early on after finding Steady Bit, it became quickly clear that Chaos Engineering is only the beginning. So I'm really excited to see where they go, how they solve these problems of the future and that's some of the things we'll be diving into today. You don't want to miss this episode. Check it out.

This episode is brought to you by SmartBear. Listen, load testing is tough. Investing in the right tools to automate tests, identify bottlenecks and resolve issues quickly could save your organization time and money. SmartBear offers a suite of performance tools like Load Ninja, which is a SaaS UI load testing tool, and Load UI Pro, an API load testing tool to help teams get full visibility into UI and API performance so you can release and recover faster than ever. Give it a shot. It's free and easy to try. Head on over to smartbear.com solutions performancetesting to learn more.

Hey Benjamin, welcome to the Guild.

Thanks a lot for this very great intro and yeah, it's a great pleasure to be here and nice to chat with you about this very important topic.

Awesome. So you actually reached out to me on Twitter after I interviewed one of the people that works on Test container, Sergey. So just curious to know, how do you know Sergey? Does Test Container test do anything with your company or just curious to know what that relationship is.

Yeah, I was able to met Sega a couple of times on conferences around Europe and yeah there was a short time where we are able to talk about Chaos Engineering and what they are doing at test containers and there are some maybe parts in the future where we could work more closely together.

Nice. Yeah, it sounded like this cool use of cases with test teams where you could turn off certain, like, databases and services and see how the application performs under that type of scenario. So I assume that's the type of thing you're talking about most likely.

Exactly. Exactly. One big difference. We are doing all the stuff where we are injecting failures or bad scenarios into working systems and not like test containers, doing it to spin up big integration tests. So we are doing it in real systems because that's the point where you are more close to production as you can and you should do it in a distributed environment.

Nice. So I guess other stages where you would do it, obviously you do it in integration for us, you'd mess around. But really, like you said, the only way to really to get confidence is to do it in production. Right? To see how it actually will perform finally.

Yes, finally. But don't please get started in production first. It's a long way until you are ready to do it in production like Netflix is doing it or other big companies are doing it. You need to get started as early as possible. So we are trying to enable people that they are able to do it during the development cycle, that they are able to understand what's going on in production in a very early stage. And later on, of course, you should do it also in production.

Nice. So you wrote a really cool blog article on the evolution of Chaos Engineering. So I'd like to dive in there and see where it takes us, I guess. Where did Chaos Engineering come from? It's like I've been hearing more and more about it. I know where it came from came from Netflix. But like, what's your take on, like maybe the past of what was the genesis of where we are now with the Chaos Engineering original?

It's started already a little bit earlier in the hands of some people from ops departments in Amazon and other companies. And then they are getting started with jumps, some shell scripts. They would like to turn off some servers, they would like to turn off some instances. And that was the very early beginning. And then there was a big momentum when Netflix was also starting with the topic of Chaos Engineering and they released the first official tool called Chaos Monkey. That was a high momentum for Chaos Engineering in general because, yeah, everybody now in the company from Netflix needs to understand what's going on. If my resources are going down or dying or maybe my instances I'm running is dying and how should my service deal with this kind of outage?

Cool. So how did you get Involved Obviously you released Chaos Monkey for spring boot, so why do that? What was your involvement to get involved with Chaos Engineering?

Five to six years ago I started with the topic of Chaos Engineering doing my old employment at a company called Codecentric. And because I was sitting in front of very complex systems I was not able to get this trust and confidence in the systems we need to build for customers. And also I need to find a way how I can can hand over this new implementation of the new systems to people that are now needs to understand what's going on and also to understand what's going on in. Yeah, in an unstable environment. So it's point in time where you are not able to test everything with unit tests or integration tests. You need to find a way to break down your system to learn from it. Because you need to understand how all this auto scaling events or maybe queues are filling up, you would like to trigger some fallbacks and that's something you need to check and verify in a running system and not in a unit test or integration scenario. That was the beginning.

Nice. So it seems like I said I speak to a lot of people now. A lot of people bring up Chaos Engineering. Why do you think the growth is the present with Chaos Engineering? Is it because most people are going like cloud native and their services before that they are like third party where before we all had control over our servers and our data centers. Now it seems like it could. We're just interacting with a bunch of different services now. Why the growth of Chaos Engineering is the question in the present situation we find ourselves in.

Yeah, there's a very close connection to containers. So everything is now packed into containers. And you need to be able to understand how your microservice architectures are now working and how they are interacting between each other. Some of the customers are doing it with cloud providers, but all other, maybe not everybody is doing it in a cloud environment. Some are also doing it in their own data centers, but in a container way. Containerized. What is the word? I'm not a native speaker, I'm sorry with containers.

So here's a dumb question. I'm kind of naive. I know containers, I know Dart, I know Kubernetes. For some reason I've been hearing more about Kubernetes. Is there a reason why Kubernetes is more popular or seems to be more popular than Docker?

Yeah, because Kubernetes is doing a lot of the stuff you normally should do by your own. So everything is managed. For example, you are creating a service and the service needs to be scaled with multiple instances and Kubernetes is taking care about the scaling. So if one instance is dying, Kubernetes will get started a new instance for you automatically and you don't need to take care about. And you are able to run your images you have already built, doing your deployment development lifecycle and you now can run them in your Kubernetes cluster. It's a big advantage because you are able to get more faster into production and you can release much faster if you're doing it this way.

Cool. So one thing I heard about Chaos Monkey is it's very difficult. So would a steady bit. Is steady bit make it easier? I know there's other companies like Gremlin that makes it more user friendly. Is this kind of the same type of like what does steady bit do then? How does they come into Chaos Monkey?

We also started with the topic Chaos Engineering and during the last two years we learned a lot about from our customers that they are not able to do Chaos Engineering on top of all. So for example, you have SREs, you have people from the Ops department, you have developers and they are all more hunted by their own systems. They are all more hunted by new features. They need to get out as fast as possible into production. Customers are sitting in front of the systems and they are not able to use them. And now somebody is coming up and telling them you need to do Chaos Engineering on top of all. No, that's not working. So you need to find a way how people can use this. Very good principles and methods from Chaos Engineering but not like spending countless hours of setting up big pipelines or taking care about security issues. Because if you're doing Chaos Engineering once wrong, you are not allowed to do it anymore and you are killing your system from internal. So it's not a good approach.

So where do people start then if they're hearing this like okay, I've been hearing about Chaos Engineering. I know this is the thing now say reliability engineering, a thing my company's asking me to do it do. Like where would someone start then to implement Chaos Engineering effectively?

Yeah, you can get started with open source tools out of the market. It's a very good getting started point to understand what is possible. And also yeah, you can, you are still able to learn something from that. But there's a point in time where you need to take care about. Not everybody in the company should do a shutdown attack on your Kubernetes cluster. Or maybe there are some products, services that should not be attacked by all teams. So you need to take care about the orchestration. And that's the point where you, you need to bring up some other tools that can help you to get started as fast as possible. And the trick is that you need to be able to focus on your product, on your business, and not on. We need to understand all tiny tools. We, we need to do chaos engineering. It's more like focus on your daily business and use some tools that are helping you to, to find the right spots in your distributed systems where you should investigate some time and to improve your system.

Yeah, I can see how a lot of people get overwhelmed really quickly, but it sounds like people should focus in on the risk areas of their applications first before they get into, like I said, all the tooling and everything else that they can do.

Exactly. So, for example, if you are in the role of a developer, start from your application, from your service, get a list of all remote services. Your service needs to be alive and then you can check what's going on, if this service is dying or not responding as fast as I'm expecting it. And then you can increase the radius. So you can now do something like, let's verify what's going on. If one central database is dying, how many services are affected. And that's something you can increase over time. But first get started on a small radius.

So is this a thing regression testing or regression chaos testing, where something happens in production and they go, hmm, this was bad, we want to avoid this in the future. Let's create a regression chaos engineering test for this. So before we release now we have a pipeline that runs all these past known issues. Is that a thing?

Absolutely. And we are building something like this because people in very big companies like Netflix, Amazon, Salesforce, whatever, they are sitting in front of the systems and there was a very big incident in the past and now they need to understand, doing the postmortem analysis what's going wrong in my system. And they would like to do it in a regression way that they are able to replay a specific incident every time because they are not allowed or they are not able to run the same incident again and again. And yeah, it's like you can, you should use all your data from your monitoring solutions, from your APM solutions to create a very complex scenario, experiment, regression test, and you can replay it every Monday, for example, and you can trigger it after every new deployment. New features are coming up in production and you would like to check it like a quality gate. Are we still able to survive this big Outage from last month or is there something coming back again and that's something you would like to know up front and not in production?

So do you see this happening then? At every check in? Every time a developer checks in code, it would go to the pipeline and would kick off a Chaos Engineering regression test.

It should be triggered after every deployment. So for example, in the Q and A stage, there's a new deployment coming up. Then you should run a big catalog of tests of experiments and you need to get out a list report. Are we still able to survive an outage of 25% of remote service or are we still able to work even even if our services are not able to communicate over DNS? That's something you would like to know every time.

Cool. So who's responsible for Chaos Engineering?

There are some champions in all the organizations. So mainly it's driven by the people that are sitting in front of this incident of production, like OPS and SRE people. They are more the champions and they are fighting against the system and to survive all outages that are coming up. But they need to be able to transport their knowledge, their experience into the hands of developers so that you are able to get started very fast doing your life cycle. And this also means that SRE and OPS are later on more able to focus on their daily work and they are not more hunted by their own systems, hunted by their incidents and so on. So we need to find a way that we can transport all the outages, incidents and knowledge from SRES and OPS into the hands of developers in an easy and nice way. A big problem about this is they are not talking the same language. So for example, SRE OPS related person is more focused on platform level, on cloud provider level. They are able to see the whole system as infrastructure. But the developer is absolutely focused on his service, on his application and maybe on some services around. Yeah, his service needs to survive or to be alive. And so we need to find a way that they are able to talks the same language and that the developer is able to understand. What does it mean if an AWS zone is going down? I don't care. I'm just responsible for my Java application.

How do they then handle that if it's part of the definition of done? You also need to handle performance, security and also now you need to be able to make your application, I guess your code resilient. I mean how do they can they do little experiments as they're writing code just to make sure that it's resilient. So when it Gets to Chaos Engineering to have some sort of. They're able to handle it somehow.

Yeah, but there's one problem today. It's very hard to understand all the systems when they are deployed and you are. It's hard to understand how a distributed system is working and where should I focus, where should I spend time, where should I get started? So you need to be guided. You need to be pointed to a specific area in your system, like a big map where you should invest time. Because this service is not able to withstand an outage from a remote service. So you need to be. You need to find a way how you could skill yourself up. It's like you should be able to do Chaos Engineering but without to be an expert, you don't need anything about. You don't need to know anything about Chaos Engineering. It's more like, you know, sonarqube static code analyze. That's something we are building for, for distributed systems. Like a continuously check about your system, a continuous verification about the current state of your system and where are areas where you should improve and where are areas where you are already quite good.

Absolutely. So I definitely see this as a growing area getting developers more involved for sure. So any other predictions for the future of Chaos Engineering that you see?

Everybody is now getting started with Chaos Engineering but there's one point like they are getting out using a tool to do some chaos attacks and they are using it like a hammer. They are hitting the system at one point, but they are not able to understand what's going on in the whole system. They are just verifying that the hammer is working at at the point where they are hitting the system. So it's not a big advantage that you're able to. Yeah, it's more like you are verifying that your tool is working like the hammer.

So I guess we're always talking about building a culture of security, building a culture of automation, building a culture of testing. Now I'm hearing building a culture around of resilience, like how to handle resilience. How do you get developers aware? So it's not just a hammer, it is more nuanced than that.

Absolutely. We need to find a way to get people from SRE Ops, Q& A and developer more closer and they need to be able to improve their daily work with the knowledge from all the people from sreops and Q and A and dev and it's more like we need to find a way how we could collaborate hardware for German. We need to find a way how people could work more effectively together. And to improve their daily workflow. So it's. I'm a developer by heart and if somebody is telling me you need to do this and you need to use this tool and we have bought a big license now, you need to use this APM solution. No, please. It's my daily workflow and I will optimize myself and I would like to improve myself over time. So it's something that needs to be not like a hard quality gate or not something that is blocking your daily work. It needs to be feeling more like a gamification. You can improve yourself, you can learn from it, you can talk about with your colleagues. Hey now able to survive an outage from 80% of AWS and I don't care. So that's something people would like to talk about and that's something where they are able to build a culture to share their knowledge, their improvements, maybe later share their experiments or big, big scenarios they have created. Yeah, that's something I'm looking for.

Absolutely. I know another team that I guess there was a big AWS outage a few months ago and their application didn't go down and only because they were doing Chaos Engineering before so the team could celebrate. Go look, look what we did to management say look, AWS went down, we to handle it just because you know what we implemented with Chaos Engineering.

Absolutely. That's the point where you need to be prepared for. And that's a very special moment for this team. Yeah, absolutely. Also, companies are not working with only one cloud provider. They are building complex systems on Azure, Google Cloud and maybe also on aws. And now you need to find a way how you can shift all your traffic into the other cloud provider area. How you could survive an outage in one specific region of the world. And now you need to move all your deployments into another cloud provider into another region. And that's something. Yeah, you need to test every time. It's not like once in a year.

No, that's a great, great point. The failover test almost where if one site goes down, how do you direct everything to the next site? Almost like a load balancer.

Absolutely. And also something from the Q and A department. They're doing it in a very different way. You would like to replay an incident from an SRE Ops perspective, it's one way. But a Q and A engineer is more focused on verifications about the architecture. So for example, we have created and designed a fallback in the way of a queue. So this queue now needs to be filled up because of latency in one specific area of the system. And that's something they would like to verify in a running system. And it's more like a testing approach from the QNA perspective.

Oh, so interesting. It seems like everyone has their place to play, then developers would do one piece. QA actually isn't just, oh, SREs handle that. Developers handle it. They have the little lane that they could help with as well to make it more resilient.

Absolutely. And finally, you need to get something like a score. How resilient is my whole stack? So you need to get some metrics and you need to be able to understand where you should improve in your system.

How do you get that score? Is that just a baseline you do yourself?

It's a mix between how fast my containers are coming up, how fast is auto scaling event triggered, how fast can I react in my monitoring solution about an outage. So for example, if there's an incident in production, how fast is my team able to identify identify this incident and how fast they are to get rid of this incident. Or maybe they are spinning up a new cluster and now are moving the traffic into this area. It's a combination of multiple metrics.

Nice. Yeah, probably every company should have that. Then something you measure after every sprint or every major release to see how well you're doing.

Normally the SRE team is able to talk about these metrics. Sometimes they are quite easy. Like normally we are expecting about 100 errors per hour. If this number is increasing, okay, something is going wrong. Also, after a new deployment, they are taking just a look at the error count and if the new deployment is producing more errors than the deployment before, they need to roll back it. That's quite easy sometimes.

Nice. So like I said at the beginning, some people may be overwhelmed by this. Are there any like major steps people need to focus in on when doing chaos engineering? Maybe if they have these steps in place, they know, okay, this is the first step, here's the second, here's the third. To do an experiment or something with chaos.

Yeah. First of all, your system needs to be able to handle some outages. So for example, if you are using, let's say spring boot application, and this spring boot application is using a database. But if you are just turning out database off that database, the spring boot service is not able to respond anymore. And that's something you already know before you have created an experiment, you shouldn't run an experiment. So if you already know that your experiment will fail, makes absolutely no sense to run it because there is already a Learning and you should improve your system. If you have, for example, created a fallback like, okay, database is not working and now I need to be able to understand if this fallback has now been triggered. That's a very good first experiment. You can get started. Just turn off your postgresdb which is running locally on your dev environment.

How hardcore should you do an experiment in production? You said you shouldn't do it right away, but eventually you probably want to do it. At what point would you say, okay, you've done your due diligence, you've tested it all along the pipeline, now let's try to do it in production.

You need to be absolutely sure that you are able to see outages as fast as possible. If you are blind in production or your monitoring is not able to see it in a couple of seconds, don't do it. And also you need to be able to stop every experiment out of the box like an emergency break. So if something is going wrong, the system you are using to do chaos experiments, or to do experiments in general needs to be able to do rollbacks automatically, as fast as possible and hopefully triggered by, let's say, state checks. So for example, you're executing an experiment, there are multiple steps doing execution where the resilience engineering platform needs to verify if your system is still in a stable state. If not, roll back all experiments and inform your team that something is really going wrong. And that's something you need to be able to. That's something you should do before you do it in production.

So before someone even starts with chaos engineering, should they make sure they have observability and telemetry in place for us so they do get alerted when issues happen? Or can they still do them at the same time if they still don't have observability or good telemetry in place?

Perfect time to get started with both topics because you could then use experiments to verify if you're on the right track with your observer monitoring to test if your monitoring is working as designed. One customer is doing it like this. They have created a prometoys monitoring solution, but they are not quite sure if everything is working like designed. And so they are using experiments to verify if Prometos is able to see all outages in kubernetes. And there's also a quality gate, let's say until 30 seconds we need to be able to see incidents or errors in our monitoring solution. That's a gate they have chosen. So it's perfect to get started with both.

Nice. So once Again, Steadybit. How can Steady Bit help? Say someone hears this, they say okay, I want to get into Chaos Engineering. I know Benjamin works for Steadybit. He has a company called Steadybit. This is their specialty. So where would Steadybit come in or for someone that's listening.

So we are Steady Bit able to. So we are doing it with an agent based approach. So you are, you are able to install our agent in your system and then we are doing a lot of discovery so we are able to understand how your system is working and how your architecture is looking like and then we are helping you to get started with. Okay, we have done a lot of analyze about all targets you should attack later on in experiment and here's the list of potential entry points where you can create an experiment and then we are helping you with with a nice guided way or you can pick up some templates we have already chosen and designed for you because you should take care on your daily business and not on how should I create an experiment? Where should I get started? It's more like please point me to areas in my system where I can get started quite easily and in a very safe and secure way. That's something we are working on and we could provide right now.

Awesome. All right, Bhajan, before we go, is there one piece of actionable advice you can give to someone to help them with their Chaos Engineering efforts and what's the best way to find or contact you?

The best way to find me is on steadybits.com or on Twitter and maybe on LinkedIn. And the best advice is it's something very typical for a German guy normally. But don't build everything by your own. They are for all kind of areas, all kind of products. They are very good products on the market. You should take a look at this product before you create the next monitoring solution or the next open source project for one specific specific use case. Sometimes they are quite good products like test containers or whatever out there that you should use and get started. Yeah, that's the biggest advice.

Thanks again for your performance testing awesomeness links to everything of value we covered in this episode. Head on over to testguild.com P77 and while you're there, make sure to click on the try them both today link under the exclusive sponsor section to learn all about SmartBear's two awesome performance test tool solutions, Load Ninja and Load UI Pro. And if the show has helped you in any way, why not rate and review it in itunes? Reviews really do matter in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Performance and Site Reliability Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack performance testing awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Performance and Site Reliability Podcast. Head on over to test guild.com for full show notes, amazing blog articles, and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.