Automated K8s Load Testing with Matt LeRay

Foreign.

Get ready to discover some of the most actionable performance engineering, testing and site reliability device with some of the world's smartest engineers. Hey, I'm Joe Calantonio, host of the Test Guild and SRE podcast and my goal is to help you succeed in creating application performance awesomeness.

Hey, it's Joe. And welcome to another episode of the.

Test Guild Performance and Site Reliability Podcast. And today we'll be talking with Matthew Laray all about automating K8 load testing with someone who describes himself as actually hating manual testing. So really get excited to get his insights around that. If you don't know, Matt is the co founder and CTO of speed scale, has 20 years in the monitoring and observability space and Matt started Speed Scale around three years ago because as I mentioned, he hates writing tests. So I have to know why this is. But you don't want to miss this episode if you want to hear the answer why? And definitely get a lot of value from it. So let's get into it. One second.

This episode is brought to you by the awesome folks at SmartBear. Listen. We know load testing is tough but necessary. So investing in the right tools to automate tests, identify bottlenecks and resolve issues fast saves your organization both time and money. And that's why SmartBear created a SaaS load testing tool to help teams get full visibility into performance so you can release quality software faster than ever, make testing effortless and give it a shot. It's free and easy to try. Head on over to loadninja.com and learn more.

Hey Matt. Welcome to the guild.

Joe, glad to be here.

Awesome to have you. I guess. Before we get into it, is anything.

I missed in your bio that you.

Want the guild to know more about?

No. No, I like long walks on the beach and. I don't know. That was pretty much it though.

Awesome. Awesome. So why do you hate manual scripts or manual testing?

I hate manual scripts because. Well, one is I'm lazy. But beyond being lazy, the when you go to Kubernetes native architectures or like cloud native architectures, everything has to be automated or it falls behind. And so what I don't like about writing manual scripts is that pretty much the moment you write them in that world, they're going to be out of date as soon as someone changes something. So there's this ethos in Kubernetes. I don't remember who originally said it or I'd credit them, but it's a conversation about cattle versus pets. And so when you pre Kubernetes, you had to carefully curate your virtual machines and your, you know, you had integration test environments and everything had to be just so. So you treated them like love, you know, love pets. You would dress them, you would take them to the grocery store, you know, you'd introduce them to your friends. But Kubernetes says, let's just blow that whole mindset away and we'll treat them as herds of cattle. And when you have herds of cattle, you certainly care. Right? You want everybody to be healthy, but at the same time, you know, no individual cattle is as important as the entire herd. And you have this attitude that you can go and shift things around at will and change it. And I think that, you know, writing manual test cases falls in the first category and that's. And so it doesn't translate as well into the cloud native world.

Absolutely.

I actually have a pet cow, so.

I kind of not sure if I relate to the cattle, but I guess if you have a bunch of them, it's different.

So that is the first time anyone's ever said that in my entire life. So congratulations.

We just get animals here. I don't know where they show up. But anyway, so along those lines, I guess, you know, scripting or automation is difficult. So in Kubernetes, as people go to the cloud, it's, I guess for a lot of people it's somewhat new. So how difficult is it to automate these type of processes, especially in using kubernetes or against kubernetes?

Yeah, so I think that it's so hard that most people don't do it at all, to be very honest. So there's a couple key challenges that come up when you try to automate testing in the world of Kubernetes. The first thing is you've got to get your environment correct. Right. So in, you know, in other types of testing, with other kinds of systems, monoliths, et cetera, you go and you create, try to create as close to a replica of production as you can. Now, in an interesting way, Kubernetes makes that problem actually a little bit easier because with Kubernetes you have things like infrastructure as code or get ops as they as it's kind of morphing into now that allow you to basically define your integration test environment the same way you would define your production environment using essentially a script, if you will, that creates the infrastructure. So that problem gets a little bit easier, actually. But problems that get a lot worse. Right. Are the problems around data. Right. And understanding the User interactions. So piecing off the data, the data part, you know, in the pre kubernetes world you would use something called test data management. And there were leaders in that space. And what he basically amounted to was I'm going to make a big copy of my Oracle database. I've got this big Oracle database and make a huge copy of it and I'm going to carry that into pre production and then I'm going to go and try to anonymize the data. I'm doing data loss prevention to try to get rid of PII, etc. That's still possible. But the problem is, is that the production systems, the data involved, is so large generally and so diverse. The data sets are so diverse and so hard to anonymize. That copying process is not practical. This is something we hit very commonly, you know, amongst all database types, even the really new stuff like BigQuery, it's still very hard to take a 400 gigabyte BigQuery database and appropriately move it into pre production. So there's the data problem, right? The second and the third problem is this concept of real user interactions or real API interactions if you will. So you go and let's say you write some test case. You're a real smart developer, so you know, you go and you really good insight into how the code's going to be used. Even the absolute best test cases will not represent how the API will really be used in production. You know, we took a shot at this with things like Open API and swagger. We've tried right over the years, but it's just not possible to understand all the weird things that will happen in production. So those interactions are all the weird things that users will do. So those kind of the three things that make it really hard to migrate a sort of a normal testing process into Kubernetes.

All right, so I guess what some tips then for migrating to Kubernetes. Also, I assume having a hybrid approaches to the cloud make it even more difficult to know what the heck's going on, where things are going.

Yeah, I gave you the easy case with pure cloud update you get into hybrid and let me tell you, SAP does not like kubernetes. Or maybe it does now, I don't know. Didn't last time I tried. Okay, so some of the tips, the first tip I'd say is start from the right base. And what I mean by that is start with the concept of GitOps and infrastructure as code in mind. So when you do a big migration, a Lot of our customers, I work at a company called Speed Scale. A lot of our customers are doing cloud migrations. They're moving from either first generation cloud architectures or data centers into like real public cloud with managed Kubernetes like EKS or Google's GKE or one of those platforms. They're the Kubernetes engines for the public cloud systems. And so the first thing you do is don't try to migrate your current way of thinking into those systems. Give you a concrete example, one of my customers, they moved over their virtual machine infrastructure to Kubernetes and they kept static names like pets. They named all their containers, right? Or their virtual machines. They named all their virtual machines and they carried these names over into Kubernetes. And the Kubernetes controller was, this is such a strange thing to do to name all of your containers in this very specific one off way that they just were fighting Kubernetes constantly. Because Kubernetes is designed to treat them as what's called deployments or like replica sets, which is to say I have five of these things, I'm going to name them all independently and when one of them dies I'm going to bring another one back. You know, it's designed to have this sort of automatic load balancing and resiliency. And so they went and just tried to replicate virtual machines. And so my advice to them was either get with the Kubernetes program or don't do it at all. Because you're better off running virtual machines, you know, in a virtual machine way on virtual machines, you know, rather than trying to go to Kubernetes. So that's number one. Number two I'd say is, and of course I'm biased in this regard. I've seen it work. So I'm going to, I'm going to evangelize for it because I think it works is right as few manual tests as you can. Now obviously unit testing is a fabulous practice. Everyone should do it. I mean, I don't know if you, you know, Joe, you're more knowledgeable about this than I am. But I don't think anything has replaced unit tests as sort of the par excellence for doing small scale testing. So do lots of that. Right. However, think about integration testing differently. So we at SuiteScale have been popularizing a technology called traffic replay and we've been maturing that technology over the last few years into something called a production data simulation. So what we do is we go into your production system and we record all of the interactions coming into the particular container or application. Then we also record all the interactions going out of the container. So what that looks like is let's say we have like a payment service, something everybody can relate to. Probably credit card numbers, user information come in one side and the outside it goes to a database or a payment service, one of those things. So we go and we capture basically like snip out a certain segment of time. Like let's say like you have a Black Friday spike or something, right? We go and we capture that and then we take it, we massage the data and cleanse it and then we go and automatically create test cases from it, from what the real API, what really happened. And we automatically create a simulation of the Stripe payment API and then the DynamoDB database that you're using and all those things. And we basically we kind of spoof it. Essentially we make that container think like when you bring it pre production, we'll make a container think it's running in the real production environment with real responses. And so the reason that's key is one is, I mean yes, it's super cool, like it's just neat, right? It's kind of a neat thing to do. But more than that is you're not writing test cases anymore. Instead what you're doing is you're curating test cases. So it's like you're starting from this real production data and you're saying I'd like a little more of this, like I'd like it to do a little bit more. Like I want to load test, right? So I'm going to amp this thing up and make it 10x, you know, 10x as many of these kinds of transactions and I'm not really interested in, you know, heartbeats, you know, like monitoring heartbeat. Okay, I'm going to filter all that stuff out. And so it changes the job of the test or the SDET to or SDT to sort of curating what's being done in the pre production environment and putting it into the continuous integration pipeline rather than just crunching away at test cases and mocks that are going to be out of date as soon as you write them.

That's almost like a model based testing, but for real. So like a lot of people may think they know what their applications are doing, what the users are doing. And it's almost like this is better because besides the complexity of Kubernetes and all the containers, what's happening, it's automatically going to tell you exactly what the users are doing. And it sounds like it automatically spins up the simulations for you. It knows that, hey, this is a stripe thing. Does someone have to go in and manually, like kind of do some sort of correlation and try to fix things?

Yeah. So I come from a monitoring and observability background and I've been doing this a long time. I was actually there when the first transaction tracing was, came on the market, which is now open telemetry or open tracing and opentelemetry. Right. It's matured into this open source project. But all these things, because I kind of came from that background, me and my co founders, we said, let's start, let's treat this problem as a monitoring problem. Like let's observe a system, right. And then let's pull that back into pre production. And I think that that gives you a different view. Like that everybody calls it shifting left, right. So you're trying to shift, but in a way, if you start all the way. Right, right. In production, you look at the production system, it gives you kind of a different perspective on it where you say, well, I'm not starting as a developer and figuring out what I can develop to make production easier. I'm. I'm starting with what's going on in production and I'm trying to bring that back to the developer. And so we went and we basically wrote like what looks like an observability agent. Right? Well, that's not our business. We're not in the observability business. But it looked like an observability agent. It captures all the stuff that's going on and then it, and it pipes it straight back into production and even onto the developer desktop. So that allows us to go and auto mock things and you know, do all that work for you.

So just something just came to my mind. You know, site reliability engineering is a big thing. Now I'm getting a little off topic, but it sounds like, is it always recording? So say for some reason, like you gave Black Friday, something occurred that you normally don't anticipate and it broke. Something can, like an SRE take that case then, and then use it almost for experimenting to say, okay, here's a use case we know broke us last time before we release code of production, let's make sure we're able to handle this scenario again if it was to happen again.

So that's an excellent question because I actually did that this morning. So in my company, we have no outages, we write perfect code, there's no bugs ever, unless I touch the code base, in which case Bugs will sneak in. But our professional engineers, they write no buggy code. So we actually did this morning we had an issue with our reporting server, we'll call it. It's like a reporting server. And I was like, how did the user do this? Like they broke something in here. What was it? With a normal observability tool, you could try to piece that together by looking at the events, like the exhaust fumes from the application. So you'd say, well, like what do the logs say? What were the metrics? Was there any systems down? Like any kind of weird stuff? The problem is that's not a full fidelity picture of what happened. Right. So what I did instead is I went into our system, which is always streaming. So that answers your first question, Joe, is it is always streaming. We're always collecting. That isn't keep everything, but we're always collecting. And then I went in, I said, let's create a snapshot of during that time period and let me actually run it in my debugger on my local desktop. I'm going to blast my local debug application. I'm going to see what actually happened inside of this thing. Anyway, I was able to figure out it was actually something about the query parameters that they were sending. Our report server didn't like it and caused it to go in a loop, et cetera, et cetera, et cetera. So that's the kind of stuff you can do because it's just always there and you're not going to break production because you can. And if I say, okay, well I need to see the database that the reporting server is talking to, well, I can just bring that right back to. Right. I don't have to copy the database. All I have to do is know what the right answers were for the exact questions the report server asked at that time. And since we're recording that, we can, our little mock engine, our auto mocker, can go and give the right answers. Now if you ask it a question, it doesn't know, well, you better programmed it or whatever. But for the things that the app application is going to do during that time period, it knows the answers.

It's very, very cool. So, you know, when people think of performance testing, I think in the cloud they just think, oh well, auto scale. So therefore we'll have no performance issues. So when Kubernetes gets involved, why does performance testing matter? And I guess the next follow up question after that would be like, what may be different with Kubernetes as compared to another environment? When performance testing is involved.

Yes. So auto scaling has made production outages less frequent in some ways. So I think you're dead on actually is that many like small scale performance problems you can sort of throw money at. Right. And in the past you kind of couldn't because you'd say, well, we can't get 10 more servers in here or whatever, you know, our cloud, but now you kind of can. And that is how you get multi, multimillion dollar cloud bills. That's one of the ways you do that because you just keep doing that, you know, solving that problem and you know, it's great. The thing that actually still remains the problem that we run into quite a bit is someone that the application actually works reasonably well and then a developer makes a change and they cause, you know, an inefficient use of resources in some way and then cause themselves to hit their cluster or node limits in Kubernetes. So there are still limits even in a system like Kubernetes. And it might be a hundred nodes, it might be, you know, it might be whatever. And then that difference is so enormous that they end up still overrunning their, you know, they still overrun their limits, so to speak. Or they get a massive cloud bill that's door number two, which makes their boss very unhappy or more importantly the customer's affected. So some problems like latency more hardware doesn't fix. Right. Depends on the kind of problem it is. Right. But if you're going, let's give an example, like in the old days we called it death by a thousand paper cuts, right? Which is like, let's say like you have an authentication header that you're asking for over and over and over from some downstream system instead of caching it, you know, you don't have the right caching layer in your client library, right. So you're doing that. Well, the thing is, is that there's a fixed network latency. Even in high speed networks there's a fixed network latency to go get that authentication key. And it doesn't matter how much hardware you throw at that. Right. There's no way to speed it up if you're doing that 100 times when you should do it once. And so that kind of stuff creates out to production quite a bit and things like auto scaling don't solve it. That's obviously a really simple example, which there are much more complicated ones, but you get the idea.

Nice. So now what's the impact of Kubernetes then on performance testing? Is it more just configuration testing and then Running a load to see how Kubernetes handles it or is there anything specific to Kubernetes coming from like a monitoring and observability background? Are there any key metrics you have to look at rather than like used to be like CPU and memory? Are there anything like Kubernetes specific you need to think about when you're doing performance testing?

Yeah. So I'm not an expert on this but I, we, my team does work on this quite a bit. But I'll tell you some of the things that I see commonly. So one of the things people do is instead of testing the absolute limits, they test their autoscaler rules. So the auto scaling rules can pick different algorithms for how you auto scale in Kubernetes and different kinds of limits. And so they'll use a load generation tool like K6 or speed scale or whatever to go and figure out what those how well that scales. Right. In fact there are a bunch of talks coming up in Kubecon about that actually. So now the actual metrics that people look at. Are you familiar with the Google SRE handbook?

Yes.

Okay, so the Google SRE handbook talks about four golden signals which are really just kind of what you already talked about. It's like cpu. I'm sorry, it's actually utilization. It's more fine grained than that. It's things like latency saturation. Anyway, these four signals sort of look at the health of the application more than they do the specific Kubernetes cluster. And so that's. Usually a lot of people are into those kind of metrics now and that's the stuff they're looking for, which is pretty much what we've had all along. It's just a little bit different phrasing of what those signals are. One thing I will say about Kubernetes, some of the unique problems you'll have is that you'll get containers that will have some sort of startup condition. And so as it's trying to auto scale under a load test, some subtle like startup condition won't be met when you run 10 of the same thing. Right. Some backend dependency it won't be able to handle. You'll run out. Like I actually had a customer a couple of weeks ago that ran out of connections to Redis and so their auto scaling looked fine until they heard a certain problem, you know, like a sitting threshold. I think it was like 500 or something. But it was huge. But it was still there is a limit. Right. And so what'll happen is that the container won't start, and it'll be kind of hard to figure out in kubernetes that you have this container frozen. Right. So it has some unique challenges. But anyway, lots of papers being written about that right now.

Nice. So how about tooling? Obviously, speed scale would be one of the tools. Are there any tools that you use along with speed scale, or is speed scale like a equivalent of, like you said, a K6, like you mentioned earlier? Are they different tools for different needs?

Yes. So to answer your question, one of the great tools to use with speed scale, or just in general is Postman. Because. Yeah, because who doesn't use Postman, right? I mean, unless you're using Insomnia, which is also good. But. So Postman solves the bootstrapping problem for APIs. Like, it's really good. And when I say bootstrapping, what I mean is I'm writing a new API. I need to kind of flex it. And I just need some, like, quick and easy way to say, like, give me an end, you know, hit this endpoint. Okay. Now change the parameters a little bit. Okay, we'll call, you know, and you end up. What you end up with in a way in Postman, if you think about it, is you end up with a set of test cases, at least, like, you know, the embryonic beginning of test cases. Because it was as you were developing the software, you kind of have an idea of the different things that you were flexing right. As you built your Postman collection. And so one of the tools that's very common is to. Is we will. The first thing we'll do is we'll import a Postman collection into Speed Scale and we will use that as the starting point for a replay. So, you know, because if you think about it, it's like a bootstrapping problem. I've got a new service I want to run, Replay. Yeah. But I don't have my service in production yet, so. So what do I do? Right. I can't record anything. It's on record, Right? Okay, so then give me the Postman collection, and that will at least give us a starting point. And then we'll go and we'll massage the collection of stuff and we'll make it look like a huge load test, and we'll vary. We. We have an engine that will transform data, right. And it'll make it look like it's multiple customers when it's only one. It does all this stuff. And so. But you can kind of start with Postman so that. That's Number one, if you're working with Kubernetes, I highly recommend a tool called K9S. Have you ever heard of this one?

I have not, no. That's a new one. Rami.

Yeah, it's an open source tool and we sponsored actually in a tiny way because we just love it so much. But it is a command line interface for Kubernetes. So there's a great tweet that went around and again I don't, I've seen it so many times. I don't know who originally said it but it said one of my co workers asked me to explain, explain Kubernetes and now both of us are confused.

Kubernetes is complicated. Like there's lots of stuff going on but K9s gives you like a little interface into kubernetes. So I definitely check that one out. Some of the other tools that we see commonly, we see K6 a lot actually because folks will, they'll have a bunch of K6 scripts and they'll want us to interact with that and we work with that just fine. With speed scale. Another general purpose Kubernetes thing we see is there's two major sort of management systems for Kubernetes that are running around right now. There's something called Argo CD which is very, very popular. And then there's another one. This is. Well, there's another one people just using Customize which is the, that's kind of the built in version of a sort of a little deployment system. So we see that quite a bit. Argo seems to be really, really popular right now.

Nice. Is there any other tooling that people you think should have in place for kubernetes besides like performance tools or like monitoring? I guess speed scale would take care piece as well. But is there anything outside of that that you think when anyone's doing performance testing or anything to do with kubernetes they need something, some other infrastructure, other tooling to help them.

Yeah, that's a great question. So the other tools, if you want to run Kubernetes there's a, there's sort of a path that I think you follow. Path number one is you use the tools, the monitoring tools provided by your cloud provider. So in AWS that would be like cloudtrail or you know, Cloud Watch or whatever. Very quickly you're going to outgrow those tools very, very quickly. No offense to those tools, but they're, they're just kind of the beginner stuff right from there a lot of folks use a Combination of Prometheus, Grafana and Loki. Okay, so Prometheus is your metrics engine, right? It's become kind of a standard for your metrics engine. You definitely should start with Prometheus and then Grafana is the visualization layer, like it's the charts. It's a really, really intuitive user interface for that. And then Loki is an engine that understands Kubernetes and collects logs. One of the things about in the early days of Kubernetes, collecting logs was actually harder than it sounds because the containers are very ephemeral, meaning they come and go a lot. And so like you got to make sure you grab the logs before the container is gone. Right. So the newer systems have solved that, you know, and so things like Loki are sort of native for that. Now the issue you run into as you continue like kind of growing the monitoring, the observability practice is that you hit a problem of long term data storage and scale. So as great as Prometheus is, you know, there's. It doesn't really come with a long term data storage system. At least not like a real mature one. Right. And so you start to say what happened last month? And you go, well, I don't know. Right. And so then you get into things like the commercial tools generally, like, you know, your datadogs and your, you know, your new relics and your, you know, some of the New York startups, of course you get into that kind of world and you hit the same problem on the log side is that you say, I'd like a little bit of log retention. Okay, now you're getting into. You need to start investing the commercial solutions. But you can get a long way with Prometheus. Usually we deal with a lot of large enterprises and the large enterprises have a, what do you call it? A lot of the large enterprises have a. They'll go like pretty far into Prometheus and then they'll sort of make the jump when they get to, you know, when you start getting like five clusters or whatever, people kind of make a jump to something bigger.

Awesome. So, you know, we talked all this time, I don't think we ever even mentioned what is speed scale. So.

Oh yeah, yeah.

What is speed scale?

So we'll start with the marketing answer to so what is speed scale? We'll start with the marketing answer. Then we'll get into the technical answer that's probably more relevant. So the bigger picture answer is that speed scale is a production simulation platform. In a nutshell, it shifts left the production environment and brings it Back to the continuous integration system and to the developer desktop. Okay. So it allows you to simulate all kinds of things about production without actually breaking production. So the way it works kind of more technically is we record traffic from the production system, we scrub it using a data loss prevention engine, and then we turn it into test cases and mocks. Right now what that allows you to do is every time you open a pr, you essentially get a preview of the performance and the API compliance of a particular. A new build of a service. So, you know, I go and I build. I have payment service to use that, you know, we use that example for. But I have a payment service and I have a new build. Now I've done a bunch of refactoring. Let's just take the simplest case. I've done a bunch of refactoring. Will it still perform the way it's supposed to and hold up under realistic production scenarios? Well, one school of thought is let's test in prod, right? And so test. And prod has its uses, for sure. You know, there's definitely things, you know, things that work that way. However, a lot of the proponents of test and prod, like the Googles and the Facebooks of the world, they have a different business model than most other companies. So Google and Facebook, you know, their customers are actually their advertisers, right? And then their users are the product, right? And so if you say, I'm going to test in prod, you start talking about things like blast radiuses and you say, okay, as long as it only affects 2% of my customers, it's okay. Who cares? Because they'll just hit refresh and Gmail will come back. And that totally makes sense. It's a smart thing to do for Google and Facebook. However, if you're a bank, 2% of your transactions would be a major problem for you. Like, this would be a major issue if you were, for instance, a fast food restaurant. We have a, you know, one of our customers is a fast food resident. I did not realize this, but look, if you're trying to get a meal and this is not our customer, just to be clear, but if you want to get a meal at McDonald's, right, and the payment service does not go through, right, you will just go to Arby's, right? Now, I never realized people were quite that fickle, but they are. If it doesn't work, they'll just go over to Arby's. They don't care, right? So that kind of approach of like, let's just like accept that some things are going to break in production is not, doesn't work for everybody. And so that's where speed skill comes in. We come and let you say, okay, I've got a new build on my service. I've got, I've created a PR like you know, a peer review for or a merge request for this new build. We're going to build the container and then Speed Scale is going to run performance tests on it and then it's going to check the API contracts and make sure it still works the way it's supposed to under realistic production style scenarios. And only then are we going to let it move forward and go into production. So that's kind of use case number one, right? And then the second thing people do with this production data simulation is they take it and they use it for local development or for like migration testing to replicate their downstream data Systems like their APIs without actually having to replicate them. So let's say I'm doing a mic. Well, I already gave you the debugging. I'm debugging on my developer desktop. But another case is like I've got a, I'm running an EKS and now I want to move to gke. So I'm moving from AWS to Google, right. Like they're different kubernetes systems but I don't have all of my data data migrated over. Cause that's immensely expensive, right. To go and migrate it out of aws. So instead I'll have Speed Scale replicate the things that I actually use and then I can performance test in GKE and without actually having to replicate my entire integration test.

Nice. Very cool. So is this an open source version? Do you have or is there a free version of your product?

So we have some free tools that we have, but it is not, our model is not open source at the moment. We're considering it, let's just say. But the thing about open source is that is it's a wonderful way to get people aware of your tool and for adoption. But it also requires the size of engineering team where you can really maintain the open source properly and really do justice to the community. You know, people don't, you know, if you're going to say you're going to do community, you need to really do community like open source, you know, so we didn't want to kind of do it halfway because we're still a very small company, right. And so for the moment we're closed source.

Cool.

And then it is pretty reasonable. I actually, I think you list Your pricing out there, I mean, so seems really, really reasonable to me. But you know, I know.

Well, thank you. We're going to raise our prices based on what you just said.

Don't blame me, people. All right, cool. All right, Matthew, before we go, is there one piece of actual advice you can give to someone to help them with their Kubernetes performance testing efforts and what's the best way to find contact you or learn more about speed scale?

Yeah, so the one piece of advice that I would give for the Kubernetes someone make the migration over to Kubernetes and load testing or performance testing is to try it on your desktop first. Use something called minikube and learn about Kubernetes. Kubernetes has very rough edges. You can't see the rest of my body, but I'm covered in cuts from Kubernetes. Every time I interact with it, you know, it is it. You really. It is not as simple as just saying it's like a new vSphere or whatever, like a new, you know, VMware, whatever. You really. There's a thought process that goes into it, but the good news is bring it on your desktop. It's called minikube. There are demo applications like Google has one, I don't know, it's similar to Sock Shop, there's a Potato Head, there's a bunch of them. Go learn about it, you know, run the stuff and then I'll say that when you're ready there is a free trial of speed scale. You can go to our website speedscale.com and you know, check it out. Also you can join our our support system or community in Slack. It's a speed scale community and we answer questions lightning fast. And I will also point out that one of our contributions, the way we try to give back is we try to help people who are new to Kubernetes learn about it. And so if you jump on there and you ask us ridiculous Kubernetes questions, we will give you non ridiculous answers to the best of our ability. No question. We are all started in the same place so we just like to try to help out there. So come and join the speed scale Slack if you want to talk Kubernetes with us and performance engineering. So that's how you can find us. You can also find us on Twitter and LinkedIn and all the usual social media platforms at Speed Scale. Speed Scale Incorporated.

Thanks again for your performance testing awesomeness. Links to everything of value we covered in this episode. Head on over to testguild.com P99 and while you're there, make sure to click on the Try them both Today link under the Exclusive Sponsor section to learn all about SmartBear's two awesome performance test tool solutions, Load Ninja and Load UI Pro. And if the show has helped you in any way, why not rate and review it in itunes? Reviews really do matter in the rankings.

Of the show and I read each.

And every one of them. So that's it for this episode of the Test Guild Performance and Site Reliability Podcast. I'm Joe.

My mission is to help you succeed.

With creating end to end full stack performance testing awesomeness.

As always, test everything and keep the good.

Cheers. Hey, thanks again for listening.

If you're not already part of our.

Awesome community of 27,000 of the smartest testers, DevOps and automation professionals in the world, we'd love to have you join the fam@test guilds.com and if you're in the DevOps automation software testing space or you're a test tool provider and want to offer real world value that can improve the skills or solve a problem for the guild community, I'd love to hear from you. Head on over to test guild.info and let's make it happen.