Gremlin Certification, Reliability Testing and Chaos Engineering with Darrell Dunn

Get ready to discover some of the most actionable DevOps techniques and tooling, including performance and reliability with some of the world's smartest engineers. Hey, I'm Joe Colantonio, host of the DevOps Toolchain podcast, and my goal is to help you create DevOps toolchain awesomeness. Hey, it's Joe. And welcome to another episode of the Test Skill DevOps Toolchain podcast. And today we're going to be talking with Daryl Dunn, all about Gremlin certification, reliability testing, and chaos engineering. All the cool things that you need to know about nowadays, if you don't know. Daryl is a senior solution architect at Gremlin. I first heard about the new Gremlin certification, I think, on his LinkedIn feed. So really excited to dive into that. So if you really want to know how to upskill your career with one of the hottest, I think, topics in testing, chaos engineering, then you don't want to miss this episode. Just check it out.

Hey, if your app is slow, it could be worse than an error. It could be frustrating, and in my experience, frustrated users don't last long. But since slow performance is a sudden, it's hard for standard error monitoring tools to catch. That's why bugsnag, one of the best production visibility solutions in the industry, has a way to automatically watch for these issues. Real user monitoring. It detects and reports real user performance data in real time so you can quickly identify lags. Plus get the context of where the lags are and how to fix them. Don't rely on frustrated user feedback. Find out for yourself. Go to Bugsnag.com and click on the free trial button. No credit card required. Support the show and check them out. Hey Darryl, welcome to the Guild.

Hey, thank you. Thanks for having me. Big fan.

Awesome, appreciate that. Great to have you. So, Darrell, I didn't have much of a bio, so there's anything in your bio that I missed that you want the guild to know more about.

So I've been an OG testing guy. My first stop in testing was at Deloitte where they had a very large customized SAP instance. And so we did lots of scripting, lots of testing, lots of time saving with that. Automation was just kind of at the beginning, so started there. I eventually went to startups and got to make the trip without Dynamics. Cisco. Really interesting this week that was purchased by Cisco, so that kind of brings back a few memories for me. So it was great getting acquired by Cisco, but you know, again, all about observability. Observability doesn't work unless you have, you know, your system's being exercised and, you know, automation fits in there very well. So came to Gremlin. One thing that drew me to Gremlin was that there's a lot of original gangsters at Gremlin. The original, you know, Colton is still here. There's some other folks that are here that are engineers that came from Netflix and aws. I thought it would be a great place to learn. And so, yeah, that's me in a nutshell. But I have always been, you know, I'm glad they pay me. I would do this for free. So.

Wow, nice. So, yeah, you said, for example, there's a lot of OGs there and you'd want to go there to learn. Did you learn anything about Chaos Engineering that you didn't know about before you joined? Like, why? I didn't think about that.

Absolutely. So I came from the Gene Kim School, DevOps, so, you know, he talked about the story of doing testing in production. And so my initial view of testing in production was Chaos Engineering, that we would let the chaos monkey go and run through the run production. And you had to be resilient all the time, you know, to make that work. But in reality, you know, reliability management is kind of the watchwords we're shifting to, is about scientific method. It's about, hey, what happens when we lose a region? Let's now let's go out and create a game day, create a scenario that encapsulates that. You know, I thought it would just be like, hey, let's just go gangbusters and randomly attack systems. But that's not the case at all.

Absolutely. I guess I should have started off for people that don't know, you know, what is Chaos Engineering?

Yes, that's a great, great question. So for us, it's for Gremlin, it's a very controlled process, but the world would kind of define it as more of, hey, chaos happens. I'm going to release chaos into my environments and make sure that I have everything set up and I am resilient and I can last that because you can do functional testing and your site is functioning and it's the best, it's the most accurate application in the world. But if you don't have one configuration set in your YAML, you won't fail over when the east zone goes down. And so functional testing won't help you. And again, performance testing will help you because it can have to be the most performant app in the world. Only in the east zone. In the east, availability Zone.

So yeah, no, that's a great point. Nowadays, the way things are going, cloud native, like you said, you could have the best functional test in the world, you can have the best unit test in the world. But nowadays, like back when I started, everything was under our control. All the bare metal was in our raised floors and we knew what everything was doing. But nowadays, like you mentioned, you can't do, you can't test for what you don't know because when you get in the wild, there are things that could occur. Like you said, certain instances could go down. So is that something you've been seeing more of than companies think? Oh, we have this great testing process in place, so we should be okay. We don't have to worry about Chaos engineering.

Absolutely. You know, they, we do see that and you know, for us it's not, it can be a very, what's the word? It can be a very expensive exercise for a company to not plan for, you know, resilience issues because they're going to happen. I mean, if you don't introduce them with automation or like, you know, infrastructure is code. I mean, that's huge now. And how do you test that? When you test that through Chaos Engineering or liability management, to make sure what I automated, what I built, what I rose from the ashes, you know, into your infrastructure is resilient, you know. So that's the only one you're going to know.

Absolutely. And you mentioned something how, you know, when people hear chaos, they think it's just chaos. Like you just do whatever you want. And so, and so that's unfortunate. They shouldn't have named it resilience testing because then it's not.

Yes.

So can you talk a little bit more about how the chaos is almost like, I guess, designed chaos or almost experimental chaos?

So there's a few different caveats to explore here. One is so you can have chaos testing and do training. So this is a really. Some of our more advanced customers are, you know, taking support folks and they're putting them in more of like a flight simulator and they're doing chaos testing with them. And the support person, support engineer doesn't know what's happening. They just all of a sudden see the logs. They see, they start seeing this, what's occurring on their systems and how do they adjust that? And what is our runbook good enough? You know, are we, are we able to react fast enough? You know, and, and that goes with. And then also. And that reduces your mttr. How do we get back online quickly? Because those kind of things are going to happen. But you know, also you have to read, you know, a lot of times you read through a lot of engineering issues where we went down the wrong path for like four hours. You know, for some businesses four hours is an eternity, you know, to be down and to be looking in the wrong places, you know. So that helps you with that. I have performed several game days with really big customers and what always comes out is there's some change to observability. They always find something they're not. They're either, they're either looking in the wrong place or one team has perfect observability and the other team doesn't have observability at all. Or some, you know, and I won't say the customer, but we had one customer doing a game day with and one of their teams was like one of their services was down and we had taken it down Gremlin and they said have you started yet? And the other team was. And the other team was like, oh no, we're down hard right now. We're seeing all the exceptions come back.

So it's another good point, I guess. What do people need to have in place before they even get to Chaos Engineering? Like it's Chaos Engineering a more advanced technique. Like do they need telemetry and observability baked in before they go all in on like chaos type activities.

That really helps. But there are, you know, and yes, we completely integrate with a lot of the larger APM tools and even smaller ones kind of when we cause these effects to occur on your system. You know, we want those observability tools to be your, your visibility into how do I tell what the full impact is for sure. So that really helps, you know, and it's not like it was, you know, years ago where App Dynamics you had AppDynamics Dynamic Relic and those were expensive propositions. You can get, you know, you can have Open source Jaeger OpenTelemetry, you know, is everywhere to pull in. But we definitely recommend some level observability. But we suggest start out with the big rocks. I lose an Availability Zone. Everyone is in the cloud somewhere and there is the concept of an Availability zone. And I am present in both east and the west and so I should be able to fill over one to the other. Really basic stuff.

Nice. So who basically uses Chaos engineering? Is it SREs? I've heard some use cases where an SRE may have a down, something goes down and then they make it part of like a regression test with Chaos Engineering to make sure that they're able to fail over and be more resilient next time. Is that a common use case and who, I guess, who's the common user of Chaos Engineering and I guess other different activities at different places along your DevOps pipeline.

Yeah, great question. SREES developers. Imagine you get your develop, you know, again it's the old ship left for find it earlier. Then you won't have, you know, it won't be more as more expensive bug to fix down the road. But yeah, SREs are our main users. We're seeing a lot more platform engineers. You're getting a lot more, you know, customers that have, are forming a PE team that's, that's creating all their infrastructure and then it gets passed off to the developers to deploy their code. And then the SREs are kind of managing the whole, all from, you know, nose to tail, the whole environment. But yeah, SREs are our main users that we find that kind of regulate. Say hey, we need to see this on every test, you know, through, through the ecosystem.

Great. So do you always do Chaos engineering in production? Like do you ever get resistance like, whoa, hold up, hold up, don't turn that off. Or is it like always in a staging and is it more realistic nowadays with staging? Because you can pretty much mimic production with the containers and things like that.

Yeah, we talk to customers all the time about that and that is their first. We kind of put that myth to bed right off the bat. You, we actually recommend you don't start in production. We recommend you start in your upper environments. And that way when you're just about to go to, you know, your production environment. You know, in my staging, in my integration, those environments are resilient and if I do those same exercises things into production, then you'll, you know, but everybody wants to keep track with the, the unicorns, you know, the aws. AWS does Netflix and the big boys, they do it in production. And a lot of our, you know, I won't say a lot. Some of our customers also do testing in production, you know, as part of their builds.

How hard is it to maybe make Chaos Engineering part of. Before you check in code to run some basic tests to say if the service is down, I'm getting the right code back saying it's down or I'm able to interact. Is that a common use case? Hey, let's run some common scenarios of what could happen when the service down. Just make sure that our, our negative testing is working before we check in into the main branch.

Absolutely. With Gremlin, it's easy. I mean, it's literally, you know, 30 seconds to do a black hole test where I take a whole region down, I take a whole node down. You know, the different levels of failure. Of course, a lot of our customers are moving to Kubernetes or already there. And so sometimes you have some engineers that are new to that system and so they love it. They love the fact that they can go ahead and do some testing, some scaling up, scaling out, you know, to see are we, you know, are we reacting to that, are we reacting the right way to that kind of stuff. But you know, developers love to do that kind of stuff early because that keeps them, you know, it's like the old Gene Kim from Phoenix project was like when we started taking, giving developers pagers, bugs, started getting fixed faster.

Right, right.

When you're eating into their gaming time or vacation.

That's a great point. So, you know, so one of the myths is a lot of people think you have to do it in production. Are there any other common myths you hear about chaos engineering that people, you constantly hear and you're like, well, wait a minute, that's not necessarily true.

I think the other thing we hear is the random. They think it should be a random attack, that it should. Gremlins should just be sitting there throwing gremlins into your system randomly. And that can be done, that can be accomplished. Not without, with ease, but that really does. What we have found is it's much better coming up with a hypothesis saying, hey, you know, this is what we think happens and this. And we're seeing something else happen and you know, proving it, it just goes back to the old adage, show me, show me this is working. Show me a survival code, you know.

Yeah. So I guess companies, they almost treated like a magic bullet. Oh, we'll just do some chaos engineering. We don't need to know what the heck we're doing. And you'll tell us stuff that that's going to help us.

That's right.

You know, when I first heard about monkey testing or chaos Monkey, it's very cool. It started with Netflix. It was very difficult to implement. I think there's an open source version and it's like, you look at it, you're like, yeah, that's not happening.

Yeah, the semi army.

Yeah. Yeah. So can you tell a little bit how Gremlin may make the process a lot easier than maybe starting off that way?

Absolutely. And Phil, I'd like to, you know, welcome your, your users to go out to our website and check out A demo, we'll do a demo for them. But the thing is, Colton and team have productionized it to where it's easy to use. There's a great, we have a great front end where you can set up these tests. But don't be fooled. We're all, we're API driven. We have customers that run Gremlin completely headless and run, you know, via the API and we can handle those executions at, you know. Another thing that you can be rest assured of Gremlin 2 is we're SOC 2 compliant. Security from the beginning. And it's a big concern for customers with Chaos Testing, you know, they want to make sure that, you know, you're cleaning up after yourself and that you're not doing anything that we would, you know, not condone those kind of things. The thing that with Gremlin is a lot of your other vendors will use custom, allow custom actions, custom tests. And then how do you determine what, what is Gremlin, you know, what is Chaos Testing and what is not, you know, so that's high on hot, high mind for customers as well.

I assume with the API it makes it easy to integrate with your CI CD systems. It almost sounds like, oh, it's gonna be a, a manual process we're going to have to do manually before we do things. But using that API, I assume, are you able to make things a little more automated then with these Chaos checks?

Maybe, oh, 100%. Anything that we do inside the GUI inside of Gremlin you can do with the API. Create a scenario. We have a large customer out of West Coast. Again, they're completely automated. They use the API and they use Jenkins and they, their pipelines have Chaos built into them just like they do, you know, J Meter test and you know, functional tests. They automate Chaos Testing along with, and usually they do the Chaos testing first. And if the Chaos testing fails, you know, they, they have something to fall back. One of the things that we're revolutionizing is we're giving, we're giving the services microservices a reliability score and we're, we're, we want to be preemptive. You know, a lot of these tests, a lot of your testing is reactive. And so, you know, we run our tests, we find an exception. So we've introduced a suite of tests that are kind of curated, pre automated for you. You just tell us where the service is and we give you a score on it. So think of it more like, you know, this gives me a proxy of risk. Hey, I know right now that my service is reliable and I didn't have to know anything about chaos engineering. I just kind of pulled pointed Gremlin to my service. So that's one of the things that we can automate as well.

I assume once you have that score, you can see over time as teams work through sprints is again better or is it getting worse? It's something you can actually say, hey, look, it is getting worse. See, we had a scrub before. Now over time it's progressively getting worse.

And maybe that dictates and maybe, you know, some customers use it till they send out a slack message to the team and say reliability has fallen in this microservice to 68. Well, you know, 70 is our minimum. So what's happening here? And you know, we look at different, we look at resource exhaustion, we look at dependency, and then we also look at failover. So we do that. We know I mentioned earlier that we talk about the, you know, the real basic things. Do we, can we fail over in a node? Do we fail over in a zone? And then, you know, we mentioned earlier about Kubernetes. You know folks new to Kubernetes, a lot of orgs are, you know, throwing engineers into kubernetes and they're like kubernetes, you know, they're trying to get their arms around it. We're also doing some things where we're going to give you some detected risks like hey, we. And once we ingest the service, we say, hey, you don't have a health aliveness probe set up. You don't have health, you don't have enough CPU resources in our, in our mind to run this app. We go ahead and warn you about those things so you can take action on those as well.

That's awesome because I know a lot of people have issues configuring their kubernetes for their specific needs. So it's almost proactively checking that for you and bubbling up.

Yeah, exactly.

So how does it alert people? I know a lot of times coming from a performance background, running the test was easy. But analyzing the results, I'm like, gosh, I don't know what the heck's correlationing to what this is occurring. How does Gremlin report then? Is it automatic? Is it, does it give reports like how does that work?

We do we actually show you with the reliability scoring, we'll show you trending. You know, you have a one stop shop to get there. Plus all everything that we're discussing too can be dropped into a splunk or Dropped into, you know, a lot of larger customers have some sort of log aggregation strategy. And we, you know, we can of course put our logs in there for when a test fails or, you know, to a dashboard. Lots of customers use Grafana. And so, you know, we output our results to those dashboards. And then also the integrations are rich. We have integrations with JIRA and integrations with Slack. So if an RM test fails, we automatically create a JIRA ticket for you if you want us to, with all the details in there. What, what fail? Where did it fail? Did we find a detected risk? All those details get dropped into your jury tickets.

Nice. So I want to get to certifications before I do one. One other question, though. I know we're almost, we're 20 minutes in, but AI, AI is everywhere. Generative AI. I've been seeing a lot with Observability, a lot of companies coming out. Is there anything AI specific you have on the roadmap or maybe be baked into Gremlin since last time I spoke to y'? All? Maybe a year or two ago.

Yeah, absolutely. So coming soon, you know that we're in discussions to talk about those, you know, expanding on those detected risks. And, you know, hey, by the way, we're always seeing this detected risk in your environment. What, you know, you take action on this as a whole. So we've seen this in 100 services. You know, you may want to think about, you know, what you're doing to give you more holistic kind of use of things. A lot of that data is getting collated and pulled in. But, you know, we are definitely intrigued and looking at a lot of those AI avenues to build in. I can see us doing some, you know, some, hey, we, you know, scenario suggestions. Hey, you know, we've seen this problem with your environments several times. We suggest you run this scenario. You know, we do some suggestions there. And then also we're doing some things around fixes. You know, by the way, if you run this, this kubectl command, you will increase. You'll be able to scale horizontally. And then here it is in the bottom. And this is how you would run it. This is what it would look like. So that kind of stuff is. So we're being, again, we're taking that preemptive information and we're going to build on that in the future. But yeah, it's really exciting times.

Absolutely. And speaking about exciting times, you caught my attention again because you came out with a Gremlin certification. So a little bit. What is the Gremlin certification program.

Yeah, absolutely. So updated with the reliability, scoring and detective risk. You definitely can, you know, it is free, it's self paced, you know, and we give you the little trinkets and the gifts to add to your LinkedIn profile. And you know, again, it comes from, you know, kind of the, the OG's original gangsters in chaos engineering. With Colton and his and the team here. This is kind of the real, real and you know, as far as chaos engineering, it will introduce you to it, walk you through it. And it's not a cakewalk either. You know, you have to really concentrate. But again, it's self paced and it would walk you through all the tenets and give you a little exam at the end. But we prepare you for all that through the course and anyone can register it.

Awesome. So it's a course format. Is it videos? Is it like written? What's the format?

It's both. Both written and videos. And again, we'll walk you through some of those courses and it's really well done. And again it is updated because we had a 1.0 of this. This is the 2.0 that will definitely help folks get more familiar and good sea legs from the Chaos Engineering standpoint.

Nice. So if, why get Gremlin certified? If someone's Gremlin certified, does that mean they're Chaos Engineering testing certified regardless of the technology?

It's more. Yeah, it's more. It's both basically. So we kind of show you how to get there, but it's more Chaos Engineering to get your, you know, to get your foundation set in that arena. Game days, you know, we talk about that and you know, how to build good scenarios, but yet you will get good Chaos Engineering education along with introduction to Gremlin. It's kind of like, here's Chaos Engineering and then this is how we do it.

Awesome. And I would think this is great for onboarding too. A lot of times people get a tool and like that go at it and you're like, what? But it seems like maybe someone purchased Gremlin, they can take the certificate, have the team take the certification. At least they have some, at least a baseline and what the heck they're trying to achieve with it.

Absolutely. And we've also always offered boot camps. We will give you a boot camp where we spin up all the infrastructure. You don't have to, you know, you don't have to commit any of your infrastructure to the walkthrough. We'll do the boot camp with you and give you a place, a playground to do your cast testing in and to see it in action.

Awesome. And where can people go to get that certification for Gremlin?

I think I was just looking at that. It's gremlin.com certification.

Yeah, it is gremlin.com certification.

Yep. Get certified and walk through there and you'll be able to do it. It's really, really a great course and they've done a great job.

Nice. And once again, you did mention it is free, so anyone can take it, regardless if they use a Gremlin or not.

Yes, absolutely.

Nice. So what's the easiest way to get started with Gremlin? Is it like you said, is it a. Do you have a demo? Is there a free trial? Is there? What do people typically do?

Yeah, all that's available to you we definitely can do spin up a trial. You can of course do the certification as well. And then also we'll be glad to do a demo for you and show you, walk you through Grillon. But yeah, and on our website there's customer testimonials, all kinds of things to walk you through how to get started.

Very cool. Okay, Daryl, before we go, is there one piece of actionable advice you can give to someone to help them with their chaos engineering DevOps testing efforts? And what's the best way to find contact you and learn more about Gremlin?

Yeah, absolutely. So you can always go to gremlin.com and you can reach out to me direct darrellum.com d a r r e l l@gremlin.com and be glad to put you in touch with the right folks. The one thing I would just leave people with is yes, that thing lurking in the back of your mind that's happened to your competitor can definitely happen to you. So let us put that nightmare out of your mind and put that to bed.

And for links of everything of value we covered in this DevOps toolchain show, head on over to Test Guild.

And while you're there, make sure to click on the Smart Bear link and learn all about SmartBear's awesome solutions to give you the visibility you need to deliver great software. That's smartbear.com so that's it for this episode of the DevOps Toolchain Show. I'm Joe. My mission is to help you succeed, creating end to end full stack DevOps toolchain awesomeness. As always, test everything and keep the good. Cheers. Hey, thanks again for listening. If you're not already part of our awesome community of 27,000 of the smartest testers, DevOps and automation professionals in the world. We'd love to have you join the fam@testguild.com and if you're in the DevOps automation software testing space or you're a test tool provider and want to offer real world value that can improve the skills or solve a problem for the Guild community, I'd love to hear from you. Head on over to test guild.info and let's make it happen.