Resilience Testing (Failures are your Friends) with Benjamin Wilms

Foreign.

Get ready to discover some of the most actionable performance engineering, testing and site reliability device with some of the world's smartest engineers. Hey, I'm Joe Calantonio, host of the Test Field and SRE podcast and my goal is to help you succeed in creating application performance awesomeness.

Hey, it's Joe and welcome to another episode of the Test Guild Performance and Site Reliability Podcast. And today we'll be talking with Benjamin all about resilience as a team sport. Resilience testing in general. If you don't know, Benjamin is the co founder of Steady Bit and a product minded engineer with over 20 years of experience. Benjamin spreads his enthusiasm and new knowledge as a speaker and author, especially in reliability chaos and resilience engineering. He's going to join us once again. He joined us last year. Well actually 2021 and episode 77. So really want to catch up on a lot of new things that I think you're going to find a lot of interest in. You don't want to miss this episode. Check it out.

This episode is brought to you by the awesome folks at SmartBear. Listen, we know load testing is tough but necessary. So investing in the right tools to automate tests, identify bottlenecks and resolve issues fast saves your organization both time and money. And that's why SmartBear created Load Ninja, a SaaS load testing tool to help teams get full visibility into performance. So you quality software faster than ever. Make testing effortless and give it a shot. It's free and easy to try. Head on over to loadninja.com and learn more. Hey Benjamin, welcome back to the Guild.

Hello and yeah, thanks a lot for having me again.

Great to have you.

What a pleasure.

Yeah, great, great. I think it's a really growing area at the IU and especially as I mentioned in the beginning, you know, we had you on September 2021, episode 77 and then I noticed in my Test Skill news show, I have a Test Skill news show that I look at the biggest news articles and where people are investing money and I saw something in September of 2022 that you raised $7.8 million for your platform.

Oh yes.

So I'm curious to know like within that year, like I don't think you've been around for that long. So what? Why do you think I see that as a key indicator Anytime I see investors investing money in it, it seems it's a technology that people believe in it because they're putting that money in. $7 million worth. So why do you think there was so much interest, I guess in your solution.

Yeah, it's easy to answer because it's all about failures and failures is something really normal in the daily life. And failures are. You are not able to get rid of them, you have to deal with them, you have to learn from them and you have to be able to handle such incidents and failures. And it's a hot category because failures will always occur. Our complex systems are growing faster, as maybe some of us can imagine. And that's why it is so important to invest not only money but also like your time to improve your systems in a proactive way.

Nice. So like I said, I'm completely off script. So I'm just always curious to know as a founder, a company invests, you know, investor invests money in you. How does that help functionality of your product? Does it allow you to invest in certain areas that you weren't able to before? Or like how does that help your company or help people with resilience testing? I guess overall.

So in general as a startup it's important that you focus on specific parts of the product. You have to create a very strong product and with that you are able to convince new customers you are able to get more traction. It's like you need to find the right partners in the early days to be very product first related, be focused on your first product and then later it's like okay, there's something repeatable, there's something where people are picking up your product and they get immediately some aha moments in the journey and that's something you have to prepare for and that's something how you should start.

So I've read the press release of it and you mentioned, I think it was like resilience testing or resilience as a team sport because resilience is a shared responsibility and that you develop steady bit as a kiosk engineering platform that is equally beneficial to site reliability engineering, DevOps and developers. I guess what's kind of struck me is maybe I'm wrong. It seems like chaos engineering maybe a few years ago was really a trending kind of keyword. But some companies astound to use resilience and resilience testing more. Did you make a pivot or is it are these two different types of testing techniques? Just elaborate a little bit more of what you meant by resilience or why you seem to be focusing more on resilience testing or maybe it's just my imagination.

It's the next logical step because you as a company, as your organization, you have to Prepare yourself. You have to enable your people so well. Let me describe it in a way how our customers are doing it. So there's an expert group. Most of the time it's dedicated to SRE departments, DevOps teams, platform engineering teams and they need to roll methods like Chaos Engineering into the organization but how to make it addressable or how to. They need to convince people from other departments like product engineers, developers to also take care about resilience aspects in a very early stage of their dev cycle. And that's why it is a team sport. That's why we are talking about resilience testing or reliability testing where it's more getting into a natural fit like we are already doing with testing like unit testing, integration testing, end to end testing. That's something we are already doing as a development process of our software and it's sometimes we are stuck in this process of testing like happy path testing. It's more like that's not production, that's not how production has been represented or doing the job. There are bad conditions in production and that's something where we have to train ourselves, our organization and our. Yeah. Team.

Nice. So is resilience or reliability testing different than Chaos Engineering? I guess. How would you explain what is reliability testing then?

Reliability testing is this continuously approach where you would like to handle failures in a very efficient way. Where you would like to train your organization, where you would like to improve continuously your software stack your services to be able to handle under bad conditions. It's like the unhappy past testing and you would like to increase reliability of your organization of your whole system.

So it seems like there's also been a shift to cloud native so we no longer have like systems under our control like we used to. And so I'm just trying to tie this concept together. When you say failures are normal, is that why failures. You say failures are normal because as we move to like cloud first it's there's more and more things that out of our control because we're relying more on like third party services that aren't necessarily in house.

Yeah, it's the systems are getting complexer every day. We need to handle big load then we have to auto scale. We need to reschedule everything again because the load is not continuously running into the system. So it's like we need to optimize for costs. We are building on Kubernetes. Kubernetes has been a managed kubernetes by our cloud provider. So there are so many layers involved and the systems are sometimes maybe out of control. Hopefully not. But it seems so. It feels so because there are so many moving parts or maybe like you have created a very complex setup with Kubernetes. There's a load balancer in front of this kubernetes and this all needs to be connected. But the load balancer is not part of your Kubernetes cluster. It's a component delivered by your cloud provider. So you have, you have multiple departments, multiple layers of services involved and that's something you have to train control and know when things will go wrong or not.

Absolutely. So you know once again going back to unhappy path test or happy path. They're easy to run because people know, they know what they want to expect. How do people come up with realistic unhappy path testing and not just like I'm going to throw a billion users on this and I'm going to unplug the server. Something that would maybe never happen. How do you create I guess realistic unhappy path testing that's actually going to add value to your organization.

Yeah. Thanks to our strong or your strong base of failures. Like inside of your incident management system there is a lot of inspiration. You have already maybe some big incidents in the past where you can take a look at where you can learn from it, where you can maybe let's recreate a specific incident from last week. That's a good starting point. Or a nice example from one of our customers was okay. They have created a big setup Kubernetes based load balancer in front and the job of the team was okay, here's your base load test. Please execute. You need to be able to run this load test successfully. Check. We are able to do next step inside of the requirement catalog. You need to be able to do a routing update. Okay, done. Check. Perfect. But they have never tested a rolling update under load and this was like okay, but now we are in production and now we need to do a rolling update and we already figured out in a pre production environment we are not able to survive. Damn it. That's something where they are like we have to test it earlier in pre production, not in production. Because yeah, that was like requirements they have to fulfill but they were not able to.

So it almost sounds like it's like a loop. So you test, you go into production, you learn from production, you take those learnings and then you bake it into your once again from development to try to fix whatever you found or whatever experiments you found from production. Does that sound realistic what companies are doing?

Yeah, that sounds realistic. It's not like on every commit. It's not like on every build process. No, it's more like, okay, there was a deployment in a pre production environment like staging or Q and A. And then you can execute a list of reliability tests and you can use those recreated incidents reliability tests as a regression test to make sure that you are not running in the same incident again and again as a quality improvement.

So if someone's trying to get started with resilience testing, then it seems like start with your incidences first and make that a regression test. Is that where you normally see customers starting? As a starting point?

As a starting point. That's how our customers from the SRE departments are doing it. Like they would like to enable their teams. Okay, what is a good source of inspiration? Okay, here's the list of incidents they are taking a look at. Okay, prior one incidents. What was the biggest bad impact for us as a company? That's something we would like to check continuously. Then this SRE people are recreating an incident. They are getting in front of their dev teams. Okay, here's something prepared. Let's execute, let's learn from it as a team, as an organization, we only need to execute this and then you can improve your system.

So absolutely. So that's a good starting point for an sre. Once again you said this. Also your Solution works with DevOps and developers. Do DevOps have a different take on this or when would they use a solution? Like steady bit if you're using, if you're say a DevOps engineer or a platform engineer.

It's not really a very big difference between those departments, but in the culture of DevOps let's call it that, it's more of the daily doing for everyone involved. It's not that you need like a starting point from SRE or platform engineers who are enabling your dev people. It's more like it's part of the daily doing more deeply involved.

How are people discovering they need your solution? What are they looking in Google? They're looking for like how do they know? Like, oh my gosh, I need, I need something to help me with resilience testing. It doesn't seem like something that everyone would know, like, oh, that's something I need.

Yeah. There's one of my favorite quote by one of our customers is holy shit. We realize we have created a monster and that's like, okay, this monster is running in production. It's called their system and they are providing the services for their customers with a monster and now they need to provide a new feature. And there's so much fear like oh, don't touch it, it's running okay. Don't make it angry. Don't wake up the monster. And that's where the system is getting is too complex. And now they we cannot adjust something, we cannot roll out a new update and so on. Another big element or big event is like there was a major outage because based on an outage we are losing not only customers, not only revenue more important or more worse, we are losing customer trust. And that's something we are not getting back immediately. That's something you have to. Yeah. Hopefully you are able to get it back over time. But if it once it's gone, it's very hard for you because today's customers, there are so many other solutions out there. They are just moving on. They're ordering on a different shopping system from your competitors. It's very hard to get them back.

Absolutely. That's a great point. A lot of people don't realize this actually affects bottom line. It's not just a nice to have nowadays. It's almost like you need to do resilience type testing because just like you said, if you there's so many solutions out there, people just will drop and go to the next one. So that's a good point.

Yeah, it's a very good point. It's like you can imagine like on your smartphone it's just an app is gone in a couple of seconds and normally they will not select you again from the app store or whatever.

100%. So for the people that missed episode 77 now we talked about steady bit. A little bit. What does steady bit do? Because we talked about different, we talked about load testing as well. So what is steady bit focusing on as your core competency?

I guess so the core competence is we are more and more getting an orchestration platform to enable you to roll out reliability testing Chaos engineering in a safe way into your organizations. And it's very hard if you, if you're not an expert for chaos engineering or maybe you have never done it before. So how should you start? Where should you start? That's where we came in. And we are able to analyze your system, to discover your system and to point you to specific so called weak spots in your system where you can already find some potential improvements you can do. And we are also helping you to invest the time in the most efficient way because okay, there are, let's say 100 weak spots discovered, but maybe just two or three of them are very important for you and that's how we can guide you to specific areas of your system. And also we are preparing the perfect first experiment for you. Because normally the next question is how should I check this bad behavior? How could I simulate this bad behavior? There's already. Let's call it receipt. We are providing you to get started.

Nice. So how do people usually implement then or integrate with the orchestration platform? Is it part of the CI CD pipeline? And it's like a stage or a gate within the pipeline that. Okay, let's. We did a functional test, smoke test. Let's now run our resilience testing on this Kubernetes environment that we spin up and spin down using your platform to orchestrate it all.

So normal flow is like people are recreating or creating their experiments. They are executing the experiment. And normally if everything is well done, this experiment will fail because we have discovered something which is not working. The next iteration is okay, let's fix it. It's like in testing, test case is red. Fix it until it's green. And that's the same which applies to steady bit. Okay, first experiment run failed. Now you are able to identify okay, what should I improve Right now let's get out a new version of our service. Rerun the same experiment. Okay, it's not green. Okay, next step, let's automate it. Now let's automate this execution or this experiment. Every time we are doing a new deployment in a staging environment and that's how people are doing it. So that's the point where you can use our cli, you can use our API to just trigger the execution of your already created experiments test cases.

So you mentioned you provide the first experiment. Is that a generic experiment you give to everyone or is it based on someone's specific history or their environment?

It's based on what we have discovered in your system. So we analyzing your system like in Kubernetes we are able to see all services, all ports, how they are connected in between. And then we are providing you experiments based on your infrastructure, based on your service catalog. And then you can just pick up like okay, I would like to improve reliability. I would like to handle rolling update scenario. Just enable it and stalevit will check if you are able or not.

So does it give you a dashboard of like okay, we found maybe 10, 10 issues or 10 not issues or 10 areas you could focus in on. We recommend you start with number three because that's the highest priority or that's the most dangerous one. Do you do anything like that?

So it's it's always about handling the risk. And it's like okay, you are selecting one of your services in your landscape and then you can click on the service and apply. Okay, here's a list of 10 receipts experiments provided by statimit. Let's enable the first and the second and then just execute. And it's like a good starting point where you can start from your expectations, the capabilities your system needs to have. Instead of like okay, start with your tech and see what's surviving or not.

So yeah, it's based on then what's really happened in production or the interactions between your environment. You're not necessarily guessing. It's almost. It's giving you informed information of things to focus. Does that make sense?

That makes sense. It's about the capabilities like reliability, availability, rolling update and so on. Yeah, that's like the entry the category.

I guess would also help someone discover because say you have a new developer that may not know your environment that well. This your platform can let them know while it touches. You may not know, but in the background it touches this, this and that. So you need to watch out for these as well.

Yeah, so we are able on low level to see, okay, this service is calling a backend component and there's a connection in between and that's like okay, if this service is going down, you can see okay, who else has been part of the party if the service is not working or not.

It's really powerful. So how do you discovering all this? I thought the buzzword is machine learning, AI using AI machine learning technology. Or is this just very sophisticated algorithm that you have running?

It's a combination on multiple levels on where we are able to discover your system. So it's like we are able to call the API from your cloud provider like AWS to discover, okay, you are using EC2 instances, you're running in specific zones, you're using RDS Rational Data Service by AWS. And then also we are able to get on API level on Kubernetes to discover your Kubernetes cluster which is maybe managed by AWS Azure and so on. And then we are also able to get on a very deep level like on machine level, on virtual machine level, on container level, we can take a look inside of a pod to see okay, how many containers are executed inside of this pod. And then we can take a look inside of the container to see okay, what's going on inside of all containers, which processes are executed. So it's a very deep analyze of your system.

So does it do any type of alerting or does someone come in and they say okay, I need to do an experiment and it has like a dashboard. How does that process work?

So you are installing the agents, multiple types of agents and they are discovering a system and then it's like okay, here's a map we have created for you. Now you can select specific services inside of this map to apply specific capabilities you would like to see and then you can create your first experiment. It's not at that point in time where we are able to create a dashboard. Dashboard like okay, here are 100 alerts. You have to take care that's not at this point inside of the product.

Gotcha. So I also been seeing a lot of articles of people integrating with steady bit. So I saw something I did about Postman have an integration.

Yep, K6, postman and also case six.

So how does that work? Is say like I need to do a performance test. So do you have preferred providers or is it just you have like an open API that all these tools are just able to integrate with because you have a CLI tool that can do all the things.

Very good point. Because that was also like the biggest improvement we have done in the past year. So we started with integrate into existing monitoring solutions, APM solutions, load testing tools like K6, Gatling, JMeter. So there's a deep integration where you can reuse your existing load tests. But then we realized and thanks to our awesome customer base, they told us there are so many other parts where we would like to integrate and then we are at the point at steady bit. Okay, we are not the experts for all the list of tools. Let's build something where people can extend steady bit by their own to their own needs. So that's why we have created our so called extensions. And extensions are open source. You can extend steady bit whatever you would like with whatever you would like to like. Maybe you can create the next load test integration. Maybe you would like to integrate with another monitoring solution. Maybe you have an internal legacy system where you would like to discover parts of that system and also would like to attack parts of that system. That's possible with our extensions concept which is you can implement in any kind of language you would like to in go Python, whatever Java, it's up to you. And it's API based approach. So there's like an API and your extension can just interact with steady bit with orchestration platform via API.

Cool. Is there anything people or your customers seem to struggle most with with resilience testing or chaos Engineering maybe not that they do wrong, but maybe that they're focusing on the wrong thing or they're not getting the most value from it because they're focusing on an area that may not be as, as risk based as you mentioned earlier.

Yeah, most of the time it's, they are starting too big. It's like okay, we are they injecting let's say latency or package corruption in a with a big plus radius. And then it's hard to learn from it because you need to improve step by step. You need to start small, service by service or maybe instance to see if your system is able to survive or not. If it's surviving okay, let's increase the radius and that's most of the time there's a starting point where it's where they are starting too complex, too big.

So I just want to go back to failure is normal because you mentioned it a few times. You know I spoke to someone else and they told me dashboarding is bad because people just want to see green. They don't want to see red which brings up a bad practice because you want things to fail because you want to be able to handle it if it does. So how do you get people into a mindset that like red is good, like you found a failure, that's normal, that's good, now let's have a remediation thing in place to handle it.

That's the biggest switch or the biggest shift in your mindset. Normally you are just looking for like you mentioned, everything needs to be green. But if you would like to improve your system, first of all you need to start with failure, you need to start red because then you can create this confidence, you can learn from it, you can improve your system and you are able to say later on if it's green now okay, we're able to survive this bad situation we have figured out and that's like this different approach but it's not really something new. It's also like test driven development. It's also like starting from the red.

Great advice. All right, so Benjamin, I don't know are there any resources people can go to to learn more about studybit or are there any webinars coming up that people should attend that learn more and actually see this all in action?

Yeah. So there are two elements or two resources I would like to mention. The first one is we launched so called reliability Hub which is linked on our website where you can get like a list of receipts. You can see how you could use steady bit, could extend steady bit. You get a list of all extensions which are provided by us and our community and you get a very deep good list about discovery components and attacks. The second resource, it's a webinar. It's a very big and it's our first webinar as a company. It's been on the 26th of January this month. This year it's been a joint webinar with our friends from Datadog, also with one of our customers, Manu Mano, and it's been moderated by Kelsey Hightower as well. So it's a very, very good resource when you would like to get to know more about mobility testing.

Sounds awesome and I'll have a link for that in the show notes so people should definitely register for that event for sure. Okay Benjamin, before we go, is there one piece of actionable advice you can give to someone to help them with their chaos resilience, testing efforts? And what's the best way to find contact you or learn more about steadybit?

So if you would like to learn more about Steadybit, go to website steadybit.com or contact me directly benjamin steadybit.com via mail. You can find me on Twitter and LinkedIn. And yeah, be aware of failures. Learn from failures and you can see a failure. More like it's something where the system you have created with your colleagues, it's something where your system would like to talk to you to help you. It's like piece of information from your system and it's not something bad. It's something where you can yeah, get inspiration, can improve your system and can learn from it.

Thanks again for your performance testing awesomeness links to everything of value we covered in this episode. Head on over to testguild.com P1.07 and while you're there, make sure to click on the Try them both Today link under the exclusive Sponsor section to learn all about SmartBear's two awesome performance test tool solutions, Load Ninja and Load UI Pro. And if the show has helped you in any way, why not rate and review it in itunes? Reviews really do matter in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Performance and Site Reliability Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack performance testing awesomeness. As always, test everything and keep the good. Cheers. Hey, thanks again for listening. If you're not already part of our awesome community of 27,000 of the smartest testers, DevOps and automation professionals in the world. We'd love to have you join the fam@testguild.com and and if you're in the DevOps automation software testing space or you're a test tool provider and want to offer real world value that can improve the skills or solve a problem for the Guild community, I'd love to hear from you. Head ON over to testguild.info and let's make it happen.