Keptn to help with DevOps and SRE with Andreas Grabner

Welcome to the Test Guild Performance and Site Reliability Podcast where we all get together to learn more about performance testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Performance and Site Reliability Podcast. Today we'll be speaking with Andreas Grabner All About Khepton to help you with all your DevOps and SRE efforts. If you don't know, andy is a DevOps activist at Dynatrace and a Dev Rel for CNCF captain and the Pure Performance Podcast host, another podcast you definitely should be listening to as well, which I'll have a link for in the show notes after this episode. I'm really excited to have Andy back on the show. It's been a while and really going to dive into what is Kaptain, how it can help you and some improvements that may have been added to it along the way over the past few years. So really excited to have Andy on the show. You don't want to miss this episode. Check it out.

This episode is brought to you by the awesome folks at SmartBear. Listen, we know load testing is tough but necessary. So invest testing the right tools to automate tests, identify bottlenecks and resolve issues fast saves your organization both time and money. And that's why SmartBear created Load Ninja, a SaaS load testing tool to help teams get full visibility into performance so you can release quality software faster than ever. Make testing effortless and give it a shot. It's free and easy to try. Head on over to loadninja.com and learn more.

Hey Andy, welcome back to the Guild.

Thank you so much.

Absolutely.

So Andy, it's been a while. I think it's been two years since we spoke and I do a new show now and one of the news items that came up was something about how you at Dynatrace what is the.

Dynatrace Conference perform was earlier this year. Yeah, exactly. Yeah.

Yeah. So there was an article that talked about how you revealed some options about Captain. You did a presentation on it, so that's what jogged my memory and said hey, I don't think I ever had Andy on the show to talk really full blown about Captain. So I guess before we get into that though, is anything I missed in your bio that you want the Guild to know more about?

Nah, I think it's all good, right? I mean, if people want to know more about me, they can just find me on LinkedIn or Twitter and see what I've been doing. Maybe one thing to mention that we Both are, I think, enthusiasts when it comes to performance engineering. So before my years at Dynatrace and I've been here for 14 years now, it was April 1, was my 14th year anniversary. Before that I did eight years with Segway Software that later became Borland and I think now Micro Focus or whoever bought them. But I was always the performance engineer there. And so performance is also in my heart and really glad to see it evolving. And yeah, that's all about me.

Nice. Yeah, Segway brings me back. I think it was Boston based company, right?

Exactly. So the fun thing is, so I'm here in Linz, Austria, so we like to call ourselves living in the center of Europe. Obviously it depends always how you look at the map, but back then Segway was a Boston based company. However, they acquired a smaller company in Linz who initially built load testing for databases and that later became Silk Performer. So you could also kind of say that Silk Performer also was born in Linz.

Nice.

Cool.

So you mentioned you were a performance engineer before you got involved with Dynatrace. We talked a little bit in the pre show. I just want to get your opinion on what's happening in the field here. There's a lot of people I speak with now that were like hardcore performance engineers are kind of being relabeled as SREs. Is that a trend you're seeing as well or is that just an anomaly, maybe that I'm seeing?

No, it's not an anomaly, but I think depending on who you talk to, you may find different people moving into the SRE role with different backgrounds. Just as you. If I speak with performance engineers, I see performance engineers kind of moving towards using chaos engineering, starting to understand site reliability engineering, and really focusing on resilience engineering in general. But on the other side, I also, when I talk with people, the sysadmins of the world, the IT ops, the cloud ops, I see also a lot of them kind of moving into sre. It seems there's like two trajectories into sre, but in the end it's about building reliable systems. And so who would know better than performance engineers on how to validate if systems are reliable? And then obviously people that have been operating systems for many years on what it actually means to run a resilient system. So this is why I think the two parts. It makes perfect sense that they are becoming SREs these days.

Absolutely. And I also think that if someone sees themselves just as a performance engineer, they have a lot of skills that can transfer over to help an SRE team, do you see morphing or do you actually see still like a dedicated performance team and a dedicated SRE team or other two teams kind of morphing together?

I think it depends a little bit on the size of the organization because, you know, if you have a very large organization, I typically see still kind of services teams, internal services teams that provide certain capabilities to their different development teams, DevOps teams, and maybe operational teams. I think it's hard, again, depending on the size of the company. You cannot be everything for everybody and therefore you're specializing. Right. And I think performance engineers are definitely not just, no longer just recording scripts, maintaining them and running load tests and then analyze the results. I see a lot of them really getting involved in defining good SLOs so that they can test against them. Using Chaos Engineering to bring systems into uncomfortable situations and then feel how resilient the systems are. You know, it really depends. The bigger the organization, typically the bigger the SRE team. And within the SRE team, you then have different special roles or special people that are focusing on a certain area.

Absolutely.

And you know, you mentioned Chaos Engineering. It's something I've been hearing more and more about, and especially resilience testing. So where does that fall within SRE? Is that something that all SREs should know, or is that something, once again that they would reach out to someone with a performance background to be able to perform a really good Chaos Engineering test? Or do they just do that for experiments and SREs? That's what they do out of the box anyway. So it's not a stretch for them.

Yeah, I mean, if you ask a, a performance engineer that has done performance engineer 20 years, then some would say, well, Chaos Engineering, what we're doing all the time, because with our load testing, we are bringing chaos to an environment and we see how resilient the system behaves under certain load. So I think overall, resilience engineering, it is a really cool, fancy term, but it's also a. It's not something completely new. I think we get to see new tools and new approaches to it. Overall, I think everyone that wants to build architect, run and test modern applications needs to understand what resiliency really mean. It starts with building the right architecture, making individual components resilient to the fact if a depending component is not available. The classical what if I need access to the database but the database is no longer there, or if it slows down, if there's a network problem, how do I react to it? Am I resilient on the Other side. I think there's also a big importance of observability. We talk a lot about observability these days. And I think performance engineers, SREs, whatever you call yourself, there's no way around getting insights into the system that you're responsible for. And observability is exactly the way to go. And for me, what I like, and this is something I've kind of talked with Anna Medina. She was an SRE at Uber, and then she worked for Gremlin, one of the chaos engineering tools. And I think now she's starting a new gig somewhere. But the two of us, we did a show together and then we coined the term test driven operations. Because as an sre, I think what you want to do is you want to use the mechanics of a load test and of a chaos test to really figure out how does your system behave under chaotic situations. Does your observability work? Do you get the right metrics, do you get the alerts? And if you get the alerts, are the right people notified? Or hopefully your system, you know, can heal itself if you have done it completely right. So I really like also that the thing of test driven operations as one of the new methodologies and principles that SREs bring to the table.

Nice. Yeah, I know chaos engineering sounds cooler, but back in my day, I think we would call it capacity planning, which is kind of similar. But so, you know, we talk about performance, we're talking about sre, and I think that's where Kaptain was born maybe two years ago. So I guess for the folks that don't know what is Captain?

What is Captain? Let me answer the question first by saying, why did we come up with Captain? Because what we tried to do, it was actually three years ago, we sat down, it was pre pandemic. We sat down in the room and we tried to use the tools that we had available to build an end to end delivery pipeline with automated deployment, automated testing, automated promotion. And then as we go into production and as we saw problems trying to use tools that could bring the system back to a healthy state. So we used tools like Jenkins, which was a given for us. We used tools like JMETER for testing, we used Slack for notifications, we used Ansible for making configuration changes. And as we built these pipelines, it turned out it was really hard because we built a lot of kind of glue code to make one tool talk to the other tool, and when this tool is done, call yet another tool, and then how do we react to problems? And so we said you know what, we need a better way to orchestrate all of these tools that allow us to bring new changes safe into production. And we need a new way to also react to problems in production. And this is where Captain was born. So on the one side, Captain tried to solve the tool interoperability problem. So we have an event driven system. Captain is an event driven system where we are connecting tools through open standard events. We're using cloud events, another CNCF project. So we are using, we're kind of decoupling process from tooling in the old world. You wrote a bash script or whatever scripting tool you used and you call tool A, trying to figure out what's the API waiting for the result, then calling tool B. With the proprietary API, we are now standardizing the way we are calling tools. We're doing it through an event model and tools can subscribe to events. That's the first problem we solve. And the second thing we do is like a human being, we make decisions based on data we look at. So data can come from the observability platforms, your metrics, your logs, your traces. So at the core of Captain, we have the capability to pull data from your observability platform, whether this is Prometheus or whether this is Dynatrace, whether this is Datadog, new relics, Splunk, you name it. And Captain at its core can make data driven decisions as it orchestrates your different tools to either move an artifact into production or when it automates the remediation of a production problem. So Captain, in a nutshell, it's a data driven and an event driven orchestrator that helps DevOps teams to push new changes into production and that helps SRE teams to, to keep systems resilient in production.

Great. And it is open source.

Correct, Captain is an open source part. So three years ago we started implementing it at Dynatrace, but then two years ago we donated the project to the cncf. It is currently, as of the time of the recording, a so called CNCF sandbox project. But we are in the late stages to become an incubator in the late stages means I think we're open now for public commenting and it's just a matter of days now for us to become a so called incubator project. Because in the CNCF landscape you start with sandbox, then incubator, then graduating, and once we are incubated we are on the level of tools like projects like Argo, which is a very prominent tool as well. And CNCF in General, that's the Cloud Native Computing Foundation. It was born from the folks from Google who kind of built Kubernetes and then they founded the cncf, the Cloud Native Computing Foundation. And now the ecosystem is really, it's growing, it's big. Everybody starts as a sandbox and this is why we're now seeing thanks to the adoption that we have, that we actually reach now incubation state, which means it's a stable system, it's a healthy community, it is real user adopters. And we have contributors not only from the original founders Dynatrace, but but we also have a lot of external contributors. For instance, right. From a Dynatrace perspective, datadog is one of our competitors. Yet we had the community built the datadog integration, which is a healthy sign in that community.

So yeah, I don't know if we'll keep this in, but I'm just curious, what does Dynatrace have to benefit from creating an open source tool like Captain, like you said, because you're creating an implementation for one of your competitors per se. But is there an issue they saw in the field that they thought if this was here that would help not only them but a lot of other people?

Well, I think the biggest problem we had to solve anyway also as Dynatrace is the challenge of tool interoperability. As Dynatrace, we integrate with a lot of different tools. We have integrations with testing tools, with deployment tools, with notification tools. And one option would be we are just building yet another proprietary tool to tool integration set and then we have to maintain it every time when these tools are changing their APIs. And so we wanted to solve this problem anyway to come up with a standard. And no better way than defining, declaring and pushing a standard through the open source community. That's the first thing. The second thing is I think it gave us a lot of great exposure as well, right? I mean everybody wants to be seen as an open source player or being contributing to open source and what better way of doing it to actually contribute a whole project to the cncf. And it's obviously not only Captain that we are contributing to, it's also open telemetry. We also, we just launched Open Feature as another open source project where we are trying to standardize feature flagging. So I think we as Dynatrace, some may perceive us as kind of the legacy enterprise observability or monitoring or APM tool, which we are not. Right? Because we are clearly, I think a modern, very interesting company that also actively participates in open source. The last thing is we see a lot of our users now and whether this is existing Dynatrace customers or also people that are just interested in Captain, they start with open source, but then they see as the technology works for them, they say, well, it's great. We now know that technology works, it solves the problem. But we don't want to run this in our organization because that's not our field of expertise. So this is where Dynatrace also provides the orchestration capabilities to our customers as part of the Dynatrace platform. So as a Dynatrace customer, you not only get observability, but you also get the whole orchestration that Captain provides because we are kind of providing it as part of our platform.

That's a great point. A lot of companies do start with open source and then realize working for enterprise, you probably need more of a vendor based solution. And it sounds like if you're using this type of open source platform because it's a standard, then you can, rather than have to create all kinds of crazy integration. It's already, it's easy. You can consume other vendor tools basically with ease because it's a standard.

Exactly. And the great thing about this, again, this is not just a benefit for us as Dynatrace, this is a benefit for any tool vendor in the industry. Because I mean, you've worked with many tool vendors. One of the biggest pain points is to find engineering resources that are building yet another integration and then maintaining it with one of your ecosystem partners. The biggest pain is we're all building the same integrations. And with standardizing the way these tools communicate by standardizing the event payload, it will be not only easier for us, but for the whole community. Because tool to tool integrations have always been tough, especially to maintain. Because once you get a new version of Tool X, the API changes and therefore you break all of your existing integrations and then you update them. So with our efforts to standardize all of this, and we're doing this by using cloud events, which is kind of the core layer of our eventing. But then there's a special interest group in the CDF in the continuous delivery foundations, it's called CD Events, we are really now standardizing all of these payloads and we are one of the contributors. There's many other organizations and projects that are working together to come to a consensus on how do we trigger a deployment tool, how do we trigger a testing tool, an observability tool, a notification tool, and a Remediation tool. And I think once we are getting there it will benefit everyone.

Nice. Now besides being a standard, the article that caught my attention last week was it says Captain's a reference implementation of Google's SRE's principles. So what does that mean?

Yeah, there was a really nice comment from Tarash. Tarash is a performance engineer at Facebook. He invited me last year to speak at his performance engineering conference and I was speaking about Captain and when he saw my presentation he said, I think he made this statement on LinkedIn and the Nice thing about it was because at the core of Captain, remember what I said earlier, orchestrating sequences of tasks by sending events to trigger different tools. But at the core of the decision making which task to trigger next we are looking at the observability tool and you can define so called SLIs and SLOs, service level indicators and service level objectives. And this is going back to SRE that one of the core pillars of site reliability engineering, the way Google defines it is that you define your indicators and your objectives and then you can make better decisions on whether your system is in a healthy state, yes or no. And if not then you need to make course correcting actions. So we at the core of Captain use the same principles. You as somebody that has ownership of an application of a service you can use Captain and Captain is always validating and enforcing that you are meeting all of your SLOs. And if you're not meeting your SLOs because you made a code change and now after the tests are executed, you can see that you're not meeting your performance criteria, your resiliency criteria, then Captain will not let this build go into production. Or if there's a problem in production and one of your SLOs is breached, Captain can then orchestrate remediation sequences to bring the system back to where your slos are green again.

Awesome.

So who would you say is adopting Captain? Is that there are different people that would find value in captain or is it just SREs?

No, I think we see a lot of different, I mean we have a lot of, we have actually now several public references. I think on the one side we see people really liking the SLO based evaluation. So the nice thing is with Captain, while Captain can orchestrate a complete end to end life cycle. And that's kind of like the goal where we are connecting all of your existing tools, but we're orchestrating it. We see a lot of people starting exactly with that SLO based evaluation capability. So traditionally we see a lot of performance engineers that say, hey we are running our performance tests right now and we want to automate the analysis after the performance test is done. And that's what they use Captain for. Because after they run the load runner test, their K6s test, their neoload, you know, whatever it is, they then call captain and say, do me a favor, evaluate my objectives, give me a score back. We call it an SLO score. And then I can report back to the developers whether the build is good or not. And then they expand from there. Then the LET CAP also orchestrate the deployment, the testing and the evaluation. And then we see organizations that are really taking this as they prove out that Captain really helps them. And then they are bringing it to a state where Captain is orchestrating the whole end to end delivery. Where for delivery we see more and more people now using Argo CD. So Argo CD is a great delivery tool. It provides a great GitOps based approach. But Argo CD is really just doing that, right? Argo CD is making sure that the things are deployed that you want to be deployed. And once ARGOCD is done, then Captain takes control again and executes your tests, validates your SLOs and then promotes any configuration change to the next phase. So this will be a classical use case for DevOps engineers or platform engineers. And the last user base that we see again could be SREs again around auto remediation. There's a problem in production. What do we do? Do we just create tickets and somebody needs to manually look at the data? Or do we let Captain orchestrate a sequence where if the disk is full, Captain can trigger a tool like Ansible to clean the disk and then validate if the problem has gone away. If it's gone away, great, no human interaction needed. If not, then create a ticket to escalate or use case that we see now more and more is people are adopting feature flagging. Somebody turns on a feature flag and all of a sudden it turns out that feature flag was bad. Captain retrieves that event from, let's say a Prometheus alert is turning off the feature flag by calling the feature flag to turn it off, validating. If turning off the feature flag really solved the problem, mitigated the problem. If so, great, harm minimized. If not, then you can either execute a different action or just escalate to a human being. But in most cases the system is brought back to a state where the SLOs are green again without much human interaction.

Great.

It sounds like with all these different types of orchestrations Captain can do if you're not Using Captain, it sounds like you'd have to do a lot of scripting. Does kept minimize that because it's using a standard or because does it have things that already built in that you just call, make a call and it already knows I know how to deal with Ansible. I don't need to write a separate script to do that.

That.

Yeah. So what we've seen, and this is also stuff that you see when you look at our website. We have people that say they are saving about 90% of scripting with Captain. Why? Because there is no hard coded need anymore to call tool A, wait for the result, call tool B. We are taking care of this, but the way we take care of it is we are separating process from tooling. So in Captain itself there's no scripting. Captain uses a declarative approach. So you have a declarative way of defining a sequence of tasks and Captain is then sending these standardized events that can then be subscribed by your tools. Now right now not many tools support our standard events that we're currently standardizing. This is why right now we have two options for you or three options. We have a couple of, we call them Captain services where we basically built a little proxy that can consume a certain event, let's say for deployment and then trigger the target tool. So Jenkins is a great example. You can have Captain orchestrate a sequence of deployment, test, evaluate, and for the deployment we have an integration where we can trigger a Jenkins job. And once the Jenkins job is done and it does its job so you can reuse your Jenkins pipelines. And once that's done, great, you go back to Captain and then Captain continues for test again. We have some integrations already built for certain tools. Like K6 is one tool and then we basically translate that event to actually calling K6. The future is that we want more and more tool vendors as we are standardizing these events through the CNCF and the cdf. The tool vendors just by default support cloud events, our CD events and then they can just subscribe to our event bus. Another interim solution that we also provide already is a so called webhook capability where you can say hey Captain, if you need a particular task to be done, please call that webhook. A very classic example again would be you can say Captain, every time you finish a deployment, then I want to send a notification to Slack. So just here's a Slack webhook or coming back to Jenkins again, same examples. Jenkins also provides an API so we can just subscribe a webhook and every time Captain wants to deploy something, we can just subscribe a webhook and then make a call to Jenkins or even any other CI cd, pipelining tool or testing tool that provides an API where you can trigger a tool. The point here is that Kaptain itself is not doing deployment. It's not doing testing, it's not doing notification. It is orchestrating all of these tools so you can keep the investment of your tools that are doing a pretty good job in what they're doing. But we don't want you to spend the time and effort to create long scripts that are pulling them all together, that are connecting them together. I always say friends don't let friends build and maintain their own orchestration. That's what Kipton is for.

Nice.

And it sounds like not only is it open source, but the way you developed it is you don't force people into a certain life cycle. So many different companies have different pipelines run different ways. It just seems like because you're just making a call, it doesn't force you to change how you're developing software or the way your company creates software. Is that correct?

That's correct. You can see it as a general.

Does that make sense?

Yeah, it makes perfect sense. It's a general purpose orchestration engine at the core, making data driven decisions on how to orchestrate these processes. But you are completely free to define the process and then you're also completely free which tools you want to support that process. And the beauty of this is you can define a process, we call them sequences of tasks. You can define certain sequences on how you want to deploy, let's say software. But different teams in your organization can then use different tools to actually do a certain task. Because maybe some of your teams, they like Jenkins and they already have a Jenkins pipelines for deploying. Some may want to use Argo, some may want to use Azure DevOps. But Captain orchestrates the same process. But because of the loose coupling and the separation of concerns between process and tooling, you can then decide which tool to use in which environment to do a certain task. And Captain is doing all of the orchestration and making sure that context is passed from one tool to the other.

So I know there's a lot of tools out there and a lot of times sometimes teams get overwhelmed. What should we use? What's the right tool for us? To me it sounds like Captain could be used by anybody, but is there use cases where maybe someone wouldn't benefit from Captain? Or if you're using an open Source technology, you probably it would be a perfect fit.

It's a good point. I think the major use cases we see are around delivery and operations, meaning auto remediation. The reason being is because at the core of Kaplan is this data driven decision that is based on SLOs. So really the cool thing about it is that the CAPTAIN orchestrates sequences of tasks. It can always reach out to observability platform and make a better decision based on that data. So you're really benefiting from Captain if you have an observability platform that you can hook up with Captain to then make better decisions or let Captain make better decisions. But in general, you're right. In general Captain could be used to orchestrate any types of tooling. But then there's also many tools out there that can just call five tools in a row. It's like the if this, then do this. But the reason why we are different I think is again the data driven aspect, the separation of concerns between process and tooling, the event driven way, how we are calling tools. And this is why I think especially DevOps engineers and site reliability engineers will have a close look into Captain because you can connect all of the different tools that you have without having to write all of the connectors and without having to worry about how to integrate your observability at the core. Which I believe is important because we as humans we make data different decisions and so should your orchestration layer do. And that's what Captain does at its heart.

Cool.

So someone's listening, says how hard is it to get up and running with Captain? What do you need in order to use Kaptain?

So Kaptain is a container based solution which requires Kubernetes. So you need some type of Kubernetes flavor. But we have some quick start guides where as you probably know, there's many very lightweight Kubernetes distributions available that you can run on your laptop, whether it's K3s, K3D kind, the micro Kubernetes. So we have a couple of quick start guides on Captain Sh where you can just, you know, run a script. And we are installing a lightweight Kubernetes cluster and on top of that the CAPTAIN components. So what it really takes is a Linux machine essentially or even a Windows machine, because on Windows I can also run Docker. Kind is kind of Kubernetes on Docker that also works. So you need a Kubernetes based environment. That's what it is. And then what you get is the core components. We also have a nice ui. We call it the Captain's Bridge. There's also a Captain API so that you can integrate Captain with your existing tools and you then want to connect it with your observability platform. But yeah, to get started, follow our Quick Start guides. I think they are out there on the website captain sh or go to GitHub.com captain you also find more information there.

Awesome. Okay Andy, before we go, is there one piece of actual advice you can give to someone to help them with either their performance or SRE efforts? I think you already mentioned some sites, but what's the best way to learn more about Captain as well?

Well, I think we also have a YouTube channel because the community we have weekly developer community meetings where the development teams like the development community comes together and they show the latest things they develop. They also discuss kind of the backlog items and then we also have monthly Captain User group meetings where users present what they are doing on Captain to users and all of that ends up on the Captain YouTube channel. So so if you go to YouTube and search for the Captain YouTube channel, you basically see what others are doing. I think that's a great way. And if you have questions, we also have a Captain Slack workspace. So if you go to Slack Captain Sh, this is the most active way to get or the easiest way to get in touch with the whole community if you have specific questions.

Thanks again for your Performance Testing Awesomeness Links to everything and value we covered in this episode. Head on over to testguild.com P90 and.

While you're there, make sure to click.

On the Try them both Today link under the exclusive Sponsor section to learn all about SmartBear's two awesome performance test tool solutions, Load Ninja and Load UI Pro. And if the show has helped you in any way, why not rate and review it in itunes? Reviews really do matter in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Performance and Site Reliability Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack Performance Testing Awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Performance and Site Reliability Podcast. Head on over to testguild.com for full show notes, amazing blog articles and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.