Live Kubernetes Optimization in Production with Stefano Doni

Foreign.

Get ready to discover some of the most actionable performance engineering, testing and site reliability device for some of the world's smartest engineers. Hey, I'm Joe Calantonio, host of the Test Guild and SRE podcast, and my goal is to help you succeed in creating application performance awesomeness.

Hey, it's Joe and welcome to another episode of the Test Guild Performance and Site Reliability Podcast. Today we'll be talking with Stefano all about optimizing configurations of K8 and some really cool technology that they just announced with their company. Really excited about this. But before we get into it, if you don't know, Stefano is obsessed with performance optimization and is the CTO at Alchemist. With more than 15 years of experience in the performance industry, he's worked on multiple projects, both national and international enterprises. He's presented at a ton of different events like Srecon at the CMG Performance Summit, the Pack, and I think he's going to be speaking at Kubecon coming up in a few weeks, which we'll talk about. In 2015, he won the CMG Best Paper for contributing to capacity planning and performance optimization of Java applications. And in 2017 he shipped one of the first in the market, Kubernetes, Capacity optimization solutions. I'm really excited to have them on the show. If you have anything to do with sre, anything to do with Kubernetes, you don't want to miss it. Check it out.

This episode is brought to you by the awesome folks at SmartBear. Listen, we know load testing is tough but necessary. So investing in the right tools to automate tests, identify bottlenecks and resolve issues fast saves your organization both time and money. And that's why SmartBear created Load Ninja, a SaaS load testing tool to help teams get full visibility into so you can release quality software faster than ever. Make testing effortless and give it a shot. It's free and easy to try. Head on over to loadninja.com and learn more. Hey Stefano, welcome back to the Guild.

Hi Joe, Thanks a lot for having me today. I'm really excited.

Awesome. Great to have you. Like I said, it sounds like you're doing a lot of great work at akamedi, so I'm excited to find out more about that before we do. Is anything I missed in your bio that you want the Guild to know more about?

Well, I know it was. I guess it was great. So thanks a lot for summarizing that.

Sweet. So last time we spoke, I think it was Last year, episode 81. Like it's almost been a year, I think it was in November. So you know I've been seeing the shift from performance to SRE and I don't know if that's something you've been seeing as well. I think in the pre show you mentioned something like maybe Alchemist is maybe helping roll out more features to help sre. Is that a direction you've been seeing more and more of?

Yeah, exactly. So actually what we have been working on lately is the new capabilities on the product which is the 3.0 version of Akamas which brings some major innovations around. Exactly around those topics. So for those who are familiar with Akamas up to now, basically the way we work was to actually use AI to help mainly performance engineers to automate the tuning of the configuration of their application and infrastructure. Things like JVM configuration, database configs or kubernetes, resource settings, et cetera. Mainly in pre product environments. So leveraging load test and tuning, automating the tuning of the configuration there. So mainly focus on performance engineers. So before the application was released to production or when, I don't know, a new business event might arise, et cetera. So that's actually we have performance engineers do before the 3.0 and now with 3.0 we are actually very excited because we bring major innovation into the platform which is the ability to help optimize applications directly in productions. So that's kind of a new shift for the product and that is actually as you were mentioning, more geared toward SREs. So people that actually are interested in taking care of reliability and cost efficiency performance of application running in production.

So is this a shift or is it just added benefit? So you still service performance engineers but you have more and more features now to help SRE as well.

Yeah, great question Joe. Actually Akiva 3.0 becomes a platform so it's a unique product. It's a unique platform that helps you basically address both scenarios. So that's the performance optimization, performance engineering kind of development and pre pre production staging environment still makes a lot of sense if you think about the DevOps trend. So being able to for example make sure configuration are great to sustain higher level of load that we are expecting. So something that we still don't see in production today or for example addressing even cause engineering scenarios like for example how do I identify the best configuration for my application that maximize reliability under certain failure scenarios. So there are still a whole lot of interesting use cases for which you may want to use Akamas. In our pre product environments we are actually adding a kind of new part of the product that lets you do the next step. So once you have launched your application in production, typically you have new needs to actually make sure it's optimized based on the new traffic, based on the actual way that use the platform that might be different with respect to what we have designed in low test scenarios or even keeping track of changes that happen to the application to the infrastructure in production. So it's an addition and comes with a field, although becomes a holistic actual platform that can address both scenarios.

So this might be completely off the wall and incorrect, but is the shift because is pre production not as relevant as it once was? You know, with cloud native and you know, using all services in production, you really can never get the real performance or real optimization that you could in pre production. Is that one of the reasons for this? But I'm just off base here.

Yeah, that's a great topic. I see actually thinking about what we see on the market, we certainly see a shift towards actually moving directly in production. So doing lots of much more experimentation in production as part of actually the DevOps culture of kind of shipping small increments of code even to the production and being able to roll out quickly, being able to test what would be the best product even for the user. So it's a kind of mind shift, mindset shift that we are seeing and that facilitates of course doing those kind of optimization data in production. Also it's great for even for our use cases. But actually for other company we see that they are actually preventing their investments on performance testing. So I guess it depends a lot on, on the culture, on the business, actual business needs. We see kind of the two trends. Some people, some organizations are kind of not investing anymore on performance engineering because they see the complexity, they see the skills that are required, the investments and they're moving towards a more kind of agile way to test things in production. But the other other kind of company, like for example financial company, we see that they are actually investing more in anyway making sure reliability is there before shipping to production in order to avoid incidents once they have launched.

Great. So what's the benefits of then testing in production? Basically yeah.

Actually the thing is that when actually SREs look at an application in production, they typically have different kind of use cases, different kind of business needs. So one thing that we see a lot is that being able to optimize the cost. So that's for sure one of the biggest drivers. So we see that in the cloud native space much of the responsibility for optimizing the cost relies on application teams because at the end of the day the cost of a Kubernetes cluster is dependent mainly on the way the application is configured. So how big are those containers, those pods powering other applications? So they struggle to do that. So that's a matter of being able to help teams reduce costs while not impacting performance, which is the other kind of side of the coin where teams are struggling not having better performance or linear reliability. So that's kind of the other benefit because again, due to the kind of tricky way that Kubernetes manage resources, we find that still many, many teams are struggling with that. So they risk of actually having production incidents which actually can do a lot of scaly things like reputational damage, et cetera, due to the way that the complexity of the environments of the stack and that is creating lots of, lots of issues. So we see those are the benefits that drive. And the other thing is that of course there's a kind of skill shortage in the industry, so it's not easy to find people that are able to do this kind of work and it still takes a lot of time even for them. So the idea is that of course let's provide a smart system that can help those guys in making better decision in quite a drastically less time, basically.

So I think last time we spoke, maybe it was you, maybe it was someone else. A lot of the issues we saw were due to configurations of Kubernetes. So how does this work now in live? Is it all automated? And you mentioned there's a skill shortage. So are you trying to do all the heavy lifting? How does that work?

Yeah, that's great. So the way we work with especially in the live optimization, so when we optimize in production is basically we try to address the whole, the whole problem. So actually Akamas first of all observe how the application is working in production. Why we do that, we observe basically infrastructure usage, but we also observe the traffic, the patterns, the response time. So we observe also application metrics and then Akamas basically is able to recognize the patterns of workloads. So it's able to understand if there are peaks, if there are valleys, et cetera, in order to actually understand how the application is behaving. And then Akamas based on the AI models that we have built actually start providing recommendations. So if we make the, if you think about for example, how we work in pre product environments, so the whole optimization cycle was completely automated in pre product environments, so basically no people had to stare at the screen and see the changes because actually the whole process could be automated. When we move to production things requirements change a little bit, meaning that people have more head control. So once Akamas identified the next configuration to try, it doesn't simply automatically change the application as they work in production, but now Akamas recommends a change to the user so your user will be able to actually review the changes. For example, I want to change the size, the CPU limits and memory limits for, for this Kubernetes pod, I want to change the GVM parameters, like changing the garbage collection or the heap size. So people as a result want to have control, want to understand what will be pushed to production. So that's a kind of human in the loop approach that is actually accepted by the market because they want to have this level of control. Next step is actually we need to roll out the changes to production and that completes our optimization cycle and that is again done typically integrating with DevOps pipelines. So typically people again don't want any tool to change the application as they work in production kind of randomly without control. So what we do is typically we integrate with DevOps processes that customers have. This means typically, for example, pushing a configuration to a repository, so not changing the actual cluster, but pushing the new config on the repository where the DevOps team, for example has the chance to again review and merge the request into the the repository. And then typically a pipeline, Jenkins pipeline for example, will actually roll out the change to production. So that's the how it works.

Awesome. Yeah. I'm not sure if you're familiar with visual validation. A lot of times it does a baseline and then it sees a difference and then it asks the user, do you want to make this change based on this difference? Is this the same kind of approach? Does it just prompt you like, hey, we noticed, blah blah blah. Do you agree? If so, do you accept these changes?

Yeah, exactly. So we provide parameters. So our recommendation is, at the end of the day is a change to the configuration. So let the user see what is the current configuration and then what is the next. That is actually suggested by Akamas, so you can actually see what are the changes. So you can kind of familiarize with what we are going to change. By the way, users also have the ability to revise. So let's say that, I don't know, you want to skip some particular change or change the values of the parameters so they are allowed to do so. And also in our UI we also present, and I think that is important also in terms of explainability of what AI is Providing we also present analysis so the users can actually right in our product actually see how the application is performing. Am I slowing down? So perhaps a change is a kind of direct action because the application is slowing down. So we want to increase the CPU limits that we assign to the bot, for example, in response to a performance degradation, something like that. So that's kind of explainability capability that we're providing to the platform, which I think are key for people to accept those changes.

Nice. As the holiday season rolls in Black Friday, will this help you scale up if you're having issues and if so, does it say hey, we can scale up to blah blah blah, but there's going to be this cost associated with it. Does it give you that type of data?

Well, yes. So what we do typically is to right size the container. So for example, selecting the proper CPU and memory limits for your containers. We can also act on the, on the auto scaling parameters. So that's kind of another use case where typically people already have built in autoscalers from kubernetes like the horizontal pod autoscaler. So with Akimas we help actually optimizing the horizontal portal to scalar threshold, for example, to make sure that your goals are met. Like, let's find out the best configuration that keeps cost at a minimum, but still provides the required level of performance and reliability that I'm expecting.

Because you are focusing on live production, I think you mentioned sre, but you also mentioned, I think Chaos engineering. Are there any features that help you with Chaos Engineering? Can you do that? Like little segments to see what happens with an experiment and if it, if it works, give some insight. Hey, that worked. Maybe you should push that to your configuration going forward or anything like that.

Yeah, yeah, somewhat. So on the Chaos engineering we typically work, we find people today at least prefer to start. It's kind of a journey, you know. So typically people prefer to start in a pre prod environment. Of course we know that the Holy Gate will be doing things in production, but the way we see is that people are starting to actually adopt Chaos engineering in pre prod. So that's kind of one scenario that we see. So leveraging low test because at the end of the day you need to reproduce things in pre product environments while working on production. One practice that we see a lot is using canary releases. So for example, when, let's say the archonists recommend a change, the question is where do we apply that change? So sometimes we apply the change to just a small portion of the pots that are powering for example, I don't know, an online front end for an E commerce website. And then you can also observe how these subset of the, or your canary or just a subset of the pods will perform and then start actually comparing against the main control group in a way. So that also kind of another way to actually try out different settings directly. In production of course you need to be much more safe because actually in a pre product environment you are allowed to actually do anything you want, change any parameter etc. In a product environment you need to be more careful and that brings to a whole lot of safety capabilities that we have to include into our platform to make sure that actually the recommendations are good for production and they are not causing any kind of adverse.

Very nice. Because it is using AI and machine learning. If you're using the same application over time does your technology get better with better insights or does it get more, does it get smarter over time if it gets a large data set?

Yeah, that's a good question. So I guess that's super great because actually that's the core of AI. So we see by the way a lot of marketing around lots of solutions that use AI to optimize applications. So one distinctive fact of real AI is that it's a learning piece. So as you said, it's exactly that. So the fact that we have seen different workloads, we can see what configuration works best with respect to the different work tools and actually akamas actually learn. So it will identify that for this particular microservice, for example, I don't know, this specific garbage collection for the JVM works best. So they may be contrary to another microservice which prefer, I don't know, another setting of the JVM or the containers. So this learning is key because actually AI can adapt to the specific microservice and make sure that we can extract cost efficiency for each application by learning how to tune each single application. So that's, that's very different from, I don't know, other approaches that just use thresholds. Like I don't know if you are above 80%, just add more CPUs. That is a much simpler approach where new learning is actually there.

So how much of autopilot can you put this on over time? Say like you've been running it for a year, you know it's been working, you've been accepting things and you're like look, I get the insights but I don't have time for this. Sometimes just check this off, just I accept it, do it automatically. Is that an option?

Yeah, that's a great question. Actually we have two modes in the kind of production optimization, so the manual approval and the fully automated. So manual approval is basically the process that I have just described. So basically a human will get a recommendation to actually review and then apply to production according to the OS processes. Actually we envision as the customer gets trust into the recommendation, we already have the ability to put the kind of fully autonomous mode where actually hackers will be able to actually skip this manual validation process and just for example, either apply the changes directly to your Kubernetes cluster or actually again leveraging your processes, changing the configuration of the microservices again in line with the infrastructure as code or everything as code approach and then trigger a pipeline completely automatically. So we also envision kind of gradual, that's kind of the only future state where customer can reach those kind of maturity. We also envision some in a way gradual path towards that. So there are some parameters might be changed live even with our starts of the application. So they might be kind of automatically configured by Akamas without any prior approval. So you have those capabilities. So you have also the ability to define ranges and increments. So for example you can say okay, I want to change the parameter within those ranges because so that I'm safe, that AI won't want, I don't know, identify big values for various more values for my CPUs that might be risky. So you get those kind of safety policies that you can define within the platform. And I think that that's one of the main features, main differentiator that we have because people want to have those kind of control and in order to trust a solution like that.

Absolutely. So I know in performance since I've been involved all these years, it's always been like either throw more hardware at it or optimize your application to be more performant. And I guess as more people go cloud native they start using kubernetes. I assume this isn't a magic option. Like if your application is not optimized to be performant, it's not like is this going to fix all that? Or like. Or do you have any lessons learned on like people using Kubernetes and Java optimization challenges you've seen with your clients or customers?

Yeah, actually in fact we have been working on especially with many customers around Kubernetes lately and especially Java is still one of the most popular languages for modern microservices leveraging framework like Spring boot, et cetera. So we have learned quite a bit of interesting lesson learned there. And I like your pillars as we call them. So the fact that we typically have dealt with those kind of problems by throwing hardware or by fixing the code. Actually there's this middle kind of layer which is the configurations. So those configurations we are finding that especially not simply the configuration at the container level, but at the runtime level. So at the end of the day, what runs within a container or a port kinetics pod is actually the runtime like the JVM or golang and they are ultimately actually responsible for driving for example resource usage. So the amount of memory, your Kubernetes application, the amount of CPU they will be using, of course that's dependent on the actual application, the actual business logic and the code. But actually there's a non trigger part which is actually the responsibility of how the JVM for example manage memory, how the JVM manage the cpu, so how the gabbage collector is working. So what we have found out is that for example one of the biggest issues is that people are suffering reliability problems in in Kubernetes application. So the dreaded out of memory kills. So Kubernetes is very different when it comes to managing memory with respect to, I don't know, operating system like we have worked in the past. So there's no swapping, there's no slowdown. So Kubernetes suddenly kills your pod, your container as the moment that you reach the level of memory limits that you have configured. So a lot of teams are struggling with that. And the issues there is that again the way that the runtime works is key. So for example, what we have found out is that there are many organization that configure the JVM with the heap size very close to the memory limit. So the problem is that the JVM for example allocate extra memory beside the heap side and that is very easily triggering out of memory errors and all of a sudden bringing services down in production. So that's the problem is that actually it's a hard problem because there are no well defined configuration options for the jvm for example, to actually keep the whole footprint within the limit. So people do a lot of trial and error there. So the key lesson learning is that you need to configure properly also not just the CPU and the memory setting at the container level, but also review what is the configuration of your runtime, especially the jvm. But we are seeing similar and interesting in a way development also on the Golang runtime, which is kind of Following a similar path. So it's not just the JVM itself, but it's pretty much the same is happening to pretty much any gamma's collected language, which includes also net for example or golang or even js. So that's kind of the common thing that we are seeing.

As for those issues with live production, does it help you because actually adds costs I guess if you don't have things sized right. Does it help you with POD resizing and node right sizing also?

Yeah, we work at the basically what we call application aware. We means working at the different layers of the stack. So we work at the POD level. So basically being able to right size the containers, the pods, the containers that run within the pod. So CPU and memory requesting limits. But also we have actually identified the best settings for the so the runtime that run within so like the jvm, et cetera. So that's one part. The other part is that is the differentiator we have is that when doing those kind of optimization we find people are struggling because actually due to the way that again Kubernetes memory management or CPU management works, you can have lots of side effects if you don't properly size the configuration. So for example, another problem typically people run into is what is called CPU throttling. So you will get slowed down by the Kubernetes resource management mechanism if you are not properly putting your CPU limits in place. So the answer that we have is that we identified that the only way to actually be safe on one side regarding reliability and also ensuring performance while reducing the cost is being able to actually drive the optimization based on the actual application response time. So that is key because actually there are lots of solutions. Simply look at the infrastructure like I'm using 10% of my CPU usage, so let's bring down the CPUs because I'm using very little. But then you run the risk of introducing this kind of advanced throttling, et cetera, which are very counterintuitive. We find that people are having a hard time managing with that. So the way we tackle this problem is actually we actually see the workload. We gather metrics from observability tools like application response time and the throughput, the patterns, et cetera to being able really to recommend configuration AT can reduce cost but minimize the risk of introducing slowdown or reliability issues.

That's a good point. Observability has been a big trend the past few years. And it sounds like something you're highly integrated with as well.

Yeah, exactly. Because again Many solutions just simply and we try to solve the problem at the infrastructure layer. So you simply need monitoring or observability tools just looking at the infrastructure but we of course we need lots of metric also from. From the whole stack in order to be able to drive smarter to smarter recommendation. And again our approach is not having any agent but rely on the integration. So we have native integration for example with Prometheus which is I guess one of the most common stack and open source stack that we find for Kubernetes settings but also dynamics we find we are very much integrated with them because again it's one of the most common tools commercial tools that we found on the market.

Absolutely. So it sounds cool but I'm sure people would love to see this, see a demo, see it in action. You did mention I think in the pre show something about Kubecon. So what is Kubecon? If people want to see this in action, maybe give us a little more information. I believe you're going to be there.

Yeah, exactly. So that's great points Joe. We will be having a booth there. So Kubecon is basically the leading CNCF cloud native computing foundation conference around the Kubernetes ecosystem today. So really it's I think it's one of the biggest conference that we have today in it. So next edition in North America will be actually in two weeks time in Detroit. So I'm very excited to be there. So we have a booth with our team so just hang out, come to the conference if you have a. If you can do that then we will be having live demo so if you want to see actually the live product in actual. I guess that would be awesome to chat with the attendees.

Awesome. And I believe Scott Moore is going to be there. So I think everyone that lists this knows Scott Moore so he's always a who he might be in his office outfit. So you definitely should check that out.

Yeah, exactly. So Scott will be with us at the Kubecon so very excited also to actually hang out with him and we'll help us actually driving the demos and talk with the customer. So I guess that's that. We're going to have a lot of fun.

Awesome. Okay Stefano, before we go is there one piece of actual advice you can give to someone to help them with their performance sre Kubernetes testing efforts and what's the best way to find you, contact you or learn more about Akimus? Right.

So we have a website Akimus IO we have published several resources there several blog posts around exactly those those topics. So we have white papers that describe better in more detail how we work in production. So how do we optimize kubernetes microservices with application aware approach so people can download it and that is it. So feel free to I always love getting in touch with practitioners so people are having those kind of issues and that's you can also find me on Twitter and on LinkedIn.

Thanks again for your Performance Testing awesomeness. Links to everything of value we covered in this episode. Head on over to testguild.com P98 and.

While you're there, make sure to click.

On the Try them both Today link under the Exclusive Sponsor section to learn all about SmartBear's two awesome performance test tool solutions, Load Ninja and Load UI Pro. And if the show has helped you in any way, why not rate and review it in itunes? Reviews really do matter in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Performance and Site Reliability Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack performance testing awesomeness. As always, test everything and keep the good. Cheers. Hey, thanks again for listening. If you're not already part of our awesome community of 27,000 of the smartest testers, DevOps and automation professionals in the world, we'd love to have you join the fam@test guild.com and if you're in the DevOps automation software testing space or your test tool provider and want to offer real world value that can improve the skills or solve a problem for the Guild community, I'd love to hear from you. Head on over to test guild.info and let's make it happen.