Kubernetes using AI to optimize Performance with Stefano Doni

Welcome to the Test Guild Performance and Site Reliability Podcast where we all get together to learn more about performance testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Performance and Site Reliability Podcast. And today you're in for a special treat. We'll be speaking with Stefano all about Kubernetes AI Performance Optimization jvm. We'll see where it takes us. He has a lot of experience, so really excited excited to dive into especially the AI aspect of this. If you don't know, Stefano is obsessed with performance optimization and leads Akimus Vision for Autonomous Performance Optimization, which is powered by AI. He has more than 15 years of experience in the performance industry and he's worked on projects from major national and international enterprises. He's also presented at multiple conferences including SRECON, CMG Performance Summit, PAC, and in 2015 he won the CMG Best Paper award for his contributions to capacity planning and performance optimization for Java applications. And in 2017 he shipped his first Kubernetes capacity optimization solution to the market, which hopefully we'll touch on as well. You don't want to miss this episode. It's a really hot topic with AI and especially Kubernetes. So stick around, listen up and let's get into it. Check it out.

This episode is brought to you by SmartBear. Listen. Load testing is tough. Investing in the right tools to automate tests, identify bottlenecks and resolve issues quickly could save your organization time and money. SmartBear offers a suite of performance tools like Load Ninja, which is a SaaS UI load testing tool, and Load UI Pro, an API load testing tool to help teams get full visibility into UI and API performance so you can release and recover faster than ever. Give it a shot. It's free and easy to try. Head on over to smartbear.com solutions performancetesting to learn more. Hey Stefano, welcome to the Guild.

Hi Joe. Hi everyone. Thanks for having me today.

Awesome to have you. So Stefano, I guess before we get into it, it seems like AI and Kubernetes is really a hot topic. But just for folks that don't know, can you give me maybe a high level overview of what is Kubernetes?

Sure. So Kubernetes is a platform to orchestrate containerized applications. So when people are moving from VMS and Phys system to containers and they are doing so for lots of benefits such as the ability to use less resources for example, or the ability to portably run their application on multiple cloud provider and on Prem data Center, for example. So you have this kind of question of how can I run reliably my application, how can I make sure that it can scale across my fleet of servers both on prem and on the cloud? So really Kubernetes is designed by Google to solve this kind of problem. So it automatically can takes care of ensuring your application stay healthy, stay can provide service to their users. It automatically can also scale based on the traffic levels and make sure that it's reliably run into a large fleet of servers.

Cool. So anytime someone talks about containerization, a lot of times companies automatically think oh, we can just set it to scale automatically, not have to worry about it. So just curious to know what are some benefits of using Kubernetes for performance and maybe some misconceptions or things you need to look out for when you're trying to get your applications performing when you using a Kubernetes type environment?

Sure. So Kubernetes actually was designed to provide lots of benefits from performance and reliability point of view, in the sense that it takes care of actually automatically scaling the application. So it provides auto scalars both from the application point of view and also from the underlying cluster point of view, which is very interesting, very unique. And also takes care of automatically scheduling the application into the underlying fleet of servers in a way that is as efficient as possible. What is often referred to as bin packing. The other thing that it can also ensure is is what is called performance isolation, which is a typical problem that you run into when you are actually trying to pack many applications into the underlying machines in a way that it's so tight because you want to save costs, you want to make sure that many applications run in a smaller footprint. But when you are doing that, if you are not careful, what you can do is that you start impacting the application performance. So the different application can actually impact the performance of each other. So Kubernetes tries to also improve that situation by providing lots of capabilities to the application teams in order to make that these kind of bad results don't happen. But at the same time there are actually challenges because many organizations that are actually moving to the Kubernetes are actually migrating their application to this new containerized orchestrator are actually facing a couple of issues. They are facing issues on a cost perspective. And I guess we are going to touch more on this because actually depending on how you configure your application, the actual capacity of the cluster can be used or wasted. Better said. So you may run the risk of actually wasting lots of hardware infrastructure based on how you configure your actual application. On the other side, you may also face issues related to performance and reliability because actually Kubernetes provides some very in a way coming from also my experience from the previous kind of virtualized environments, the resource management techniques that are implemented into Kubernetes are pretty in a way new, pretty tricky. And what we have found also is that lots of team are actually not really understanding how that works. So the way that they configure their own application is not actually the best way for Kubernetes to run efficiently. So you may run into performance issues when your application run into Kubernetes or even availability issues, because again, due to those kind of particular resource management mechanisms, Kubernetes may even kill your application even if they do not respect the limits that you set for them. So you may not only risk slowdown or risk spending more, but you may also face availability issues. It's a great platform, so that must be said, but you need to be careful in configuring your application to make the best use of it.

Cool. So I used to work at a company where this is before Kubernetes was out, we used to use virtual machines and they always say no, we need to do performance against a bare metal server because virtual machines are unreliable. So curious to know, I haven't really used Kubernetes a lot. How reliable is performance on a container environment like Kubernetes compared to virtual machines? And what's the difference if there is any?

Yeah, that's a great question. So actually when an application got initially moved to Kubernetes and Docker, so there were lots of also studies in terms of what is the overhead of migrating application to container. Is there any since due to the fact that it seems like the container is some kind of light containerization, light virtualization. So actually from the main resource users perspective, there's actually little overhead. So from for example using performance using CPU and memory, it's actually pretty native because actually there are really no, there's one thin layer of resource management which is built into the Linux kernel, which is called control groups, which by the way runs in every Linux kernel kernel, even if not containerized, that can in a way shape the access to the resources. But it has been provided built into the Linux kernel by the Google engineers back in the time. And that's something that in a way has to do with how do you configure the application. So if the application are configured properly, I would say there's no real overhead as regard the main resources like CPU and memory. You may face some additional overhead in the storage and network area because it is here that there are additional layer into the architecture like virtualized and overlay networks, for example, or other kind of storage mechanism where actually there are more interaction layers in a way. So you may face slowdowns there. So for example sthesis and database windows. Not recommending to run your most critical databases on Kubernetes, but I would say I guess this is going to change because I see lots of focus also to of course improve the performance across the board of this platform.

So how many vendors do you think concerns are unfounded because maybe they don't understand resource management in Kubernetes? I know a lot of times people just blame the environment necessarily invest time and okay, maybe we don't really understand the resource management Kubernetes is using. So rather than look at that, we just say oh, we don't recommend it. So does that happen?

Yeah, I guess so. I guess so. It's. It's actually must be said, I guess that it's pretty tricky because actually it's. Yeah, it has been designed in a way that may impact also the. Not just the. In a way the low level infrastructure like the operating system, the CPU and the memory, but it also depends and that is an interesting part on the runtime of your application. So for example, the jvm, if you take the jvm, it has been changed by Oracle in order for the JVM to actually discover what are the settings of the control groups that I was referring to before. So one other thing that is actually pretty tricky is that. So for example, you can run your container and say to Kubernetes, okay, I want to run my container and assign it to CPUs. But let's say that your host has eight CPUs. What really happens in this case is that your application will see all the 8 CPUs. That is what the host really has available. But in a way it will be constrained, restricted to use just two CPUs. So this kind of resource management in a way and quota mechanism, it's actually pretty tricky because it's something that didn't exist in the VM world. So in the VM, if you had two CPUs you really had two CPUs. So the application actually sold two CPUs. So there was no issue on that regard. With containers, your application can actually see eight CPUs. The runtime of the languages was changed and was in a way improved. To being able to automatically adapt to this kind of new reasoning. But it can be tricky and it can be language dependent. For example Go JavaScript, VM and also the JVM may behave differently. And also this impacts also your application layer because there are actually application frameworks that need to discover how the underlying platform is being built in order to automatically configure themselves. For example, as regards thread pools or fork join pools for the JVMX for the Java, etc. So we don't really know, but there are lots and lots of settings into the stack that actually may be configured and then automatically configure based on what the underlying platform really expose. And that might be tricky because Kubernetes brings some kind of new and tricky reasoning around that. So you may run into some interesting surprises here.

I think at your latest presentation you did for the tricentis pack, you went over four key areas of Kubernetes resource management. So I thought maybe we hit each one so get people familiar with it that may not be familiar with it. The first one was a resource request. How does resource management handle resource requests within Kubernetes?

Sure. So resource request is actually the first important concept that Kubernetes brings. So by my request actually is the amount of resources that your application are guaranteed to get from kubernetes. So for example, you may say I want to run my container, my pod, which is actually Kubernetes concept that includes containers and I want to have for example two CPUs and two gig of memory. So when you do that and Kubernetes in a way runs your application, you will be guaranteed to get actually 2 CPUs available all for you and 2 gig of memory from the underlying hardware. So the impact of that is that of course Kubernetes make sure that those kind of resources are used just by you. And it really depends on the application developer puts in this kind of number into the YAML files that define the Kubernetes resources, the POD in this case or the deployment. The implication is that those CPU and memory resources are guaranteed. What you will end up doing is that you may actually make your cluster full of capacity if you are actually asking too much resources. And by the way, your infrastructure may be in a way no longer usable for other application teams, for other parts of your organization. Because actually Kubernetes are already committed in a way reallocated all those kind of resources for you. And at the same time those are the important concept to remember is that those are in a way requested Resources. So those are allocated resources. So it's completely unrelated to the actual resource usage. So one of the first challenges I personally ran into several years ago when we started working on Kubernetes is that all of a sudden my cluster was capacity full, so I could no longer schedule application. But if I look at the actual CPU util, it was pretty much idle. So again, this kind of lack of. We called it back. Also in the virtual machine era, the lack of over provisioning on the resource request is very important. And again, it's something that we didn't have before because if you think about for example VM, where we could actually over provision or allocate much more VMs into the physical infrastructure with respect to the available resources, that is something that cannot really be done on Kubernetes, at least as regard resource request. The thing is different with respect to resource limits, which we are going to cover later I guess. But that's actually another kind of surprise that when you run into when you actually try to configure Kubernetes application, I.

Believe that was actually the second thing you covered was resource limits might strongly impact performance and stability. So what's that all about?

Yeah, sure. So we request I can actually ask Kubernetes I want to have this kind of amount of resources and it guarantees that those are available to you. But interestingly actually your application can actually use more. So if you ask for two CPU requests, your application can go and use 3 and 4, et cetera, of course, depending on the available capacity of the underlying machine machines. How can we actually limit the maximum amount of resources that can be used? So that is the second concept, important concept that Kubernetes brings you, which is this resource limits. So again, we talked at the beginning that Kubernetes provides facilities for performance isolation that is done actually through limits. So you can say, for example, my container, I want to have two CPUs requested and I want to have three CPUs as my limit. So my container, CPU usage for example really won't go above 3 CPUs because by the way, that is enforced by the Linux kernel level. So there are really no ways that you can consume more. Depending on the key question becomes from a performance engineering perspective is what happens when we kind of try to use resources up to our limits. Kubernetes treats in a way resources differently here. CPU and memory. So CPU resources, they are called compressible resources in a Kubernetes documentation. What this means is that actually the container will simply be slowed down. So Kubernetes makes sure that you cannot use more than three CPUs in my example. And when you are approaching these kind of three CPU limits, your CPU access will be artificially constricted. So this means that basically you won't be able to use more CPUs and it's kind of like that your CPU will be start slowing down. And when you are approaching this kind of limit, so the net effect from application performance point of view is that you will be facing application performance slowdown because of course you won't be able to access more CPU cycles. So on the other hand, just want to mention also what happens on the memory side, because the memory is treated pretty differently because memory is referred to as a kind of incompressible resource. It's like water in a glass, so you cannot really compress it more than the application really needs to use. So when your application tries to go past your limit, actually Kubernetes will kill your container actually. So there is no really slowdown like we were used to with in the virtual machine world because typically Kubernetes host actually don't use swap as default. So there is no mechanism like paging and swapping. In the typical scenarios, default scenarios, your container simply be terminated. So all of a sudden you will face some availability issues on some part of your application. And that is another kind of interesting discovery that people do when first approaching Kubernetes.

So I guess that touches on the third thing is the CPU limits may disrupt server performance. I think you've touched on that a little bit for sure. Anything more to add to that?

Yeah, yeah, sure. That is actually an interesting topic to discuss because there's from this kind of high level discussion, the CPU limits makes totally sense. But again the dev is in the detail because actually due to the way that the actual CPU limits are implemented at the Linux kernel level, there may be interesting surprises on a service performance level. So what people figure out when moving application to Kubernetes is that you may actually have performance slowdown like peaks, especially in your 10 latency, like P95 percentile, et cetera. Even though you may be pretty lowly utilized, your container may have a CPU used that might be far low with respect to the limits. So in this situation with low CPU usage with respect to the limits, you may be, you may be thinking that okay, it's all right, so I have plenty of additional residual capacity that I can use to sustain my traffic load. But due to those in a way thread management, that mechanism, quota mechanism that applies at the CPU level you will get. There are chances that actually you will get a kind of slowdown from this kind of quota mechanism, even at low CPU usage. So at Akamas we have run, we do a lot of research also in that area because those are actually parameters that can be configured. And it's interesting to see, for example, how does the different level of throttling change with respect to the way that you configure CPU limits. And what we discovered was that CPU throttling, which is the mechanism that actually kubernetes put in place to actually constrain your CPU usage, may start as low as 30% CPU usage with respect to your limits. So again, it's not that kubernetes throttle your application when you hit 100%, which would be totally reasonable, but rather chances that the throttling can happen much earlier. And that is something that it's also hard to explain because for example, in a microservice world you may get latency peaks in some microservice in your application stack, which typically have high fi now. So your application may call hundreds of different microservices. So it's enough that one single microservice will slow down due to one of these in a way throttling unexpected situation that your entire application may slow down. And the interesting thing is that there are no upper and codes in of this kind of slowdown because actually something that happens very, very low at the kernel level.

Wow. So it just seems like you need to know a lot and how much time do you need to actually spend then sending the resource requests and limitations to ensure that the Kubernetes is stable? I mean, it's different for every application. Is it something that you need to play with? It seems almost impossible from what you've said so far.

So yes, it's actually pretty tricky. So we face, we see teams literally spending weeks or even months for tuning those kind of resources to ensure that run cost efficiently on one side and you run with the required level of stability. And the additional challenge that of course people have today is the whole kind of continuous release process because people are adopting of course DevOps practices. So it's not that you have actually two months of time to make sure your requests and limits are actually properly set in the best possible way, but perhaps the next day you will have, or the next week you will have a new application version, a new application released that may actually require different kind of resources you in a way settings. So it's something that it's becoming one very big issue for teams that are adopting this platform.

So I guess is that where AI machine learning comes in then? Because it sounds like you have to deliver software faster, quicker, and like you said, if you get a little change, almost your microservice change, it consumes more than you thought. It's going to throw everything off from your normal baseline. So how does AI and machine learning help then with all these issues we talked about so far with Kubernetes?

Yeah, that's a great question. So the way we see Mashilomi helping on this regard at Akamas is around four for kind of major areas. So on one side it's the complexity. So you have lots and lots of parameters to be tuned. So as regard Kubernetes, you have those kind of request parameters that are actually a subset of what you will probably need to configure. Because actually auto scaling is another kind of part where people spend time configuring because you have other settings there also, but also you have the runtimes level. So most of the time, for example, the way that you configure Kubernetes resources is also related to the way that you configure your JVM from example. So you have a kind of lots of parameters that are actually connected to each other. So the way that your application behaves, it's kind of independent of all those parameters. And similarly, even the most expert performance engineers are having a hard time actually figuring out how those relationships are among the parameters and how to best optimize application performance. On one side, AI can help in actually understanding and making sure that you can actually best configure your containers and your applications without spending that much of effort. The kind of trial and error process that people are using today. So that is one piece. The second piece is that it's kind of AI can reason about multiple level of layers of the stack. So it's not just again tuning the Kubernetes parameters. But the interesting thing is that AI can handle this kind of complexity and for example, add the parameters of the gvm or you can add the parameters, for example, of your application framework. So all those parameters in reality play a role in your end to end application performance reliability. So the fact that you can, as a performance engineer, all of them at once is really something that we, I guess we all wanted to do, but it's something that it was impossible simply because you don't have the time. Those kind of relationships are so complex. And the beauty of AI is that it can take that kind of challenge the first thing which is very, very powerful, I think it's one of the major areas where AI can help us is that the optimization is not simply in a way looking at infrastructure metrics like we might do as performance engine. So for example, we see an application peak in the response time when we dive deeper and we sort of trying to find the root cause. But the beauty of AI is that it can actually implement an optimization that is goal driven. What do we need by that is that we can actually ask AI to okay, let's minimize my response time of my whole application and the AI will automatically figure out which is the microservice to tune, which are the best parameters to be tuned in those in that particular microservice or the other microservices. So for example, at the pack I've shown an example example where we tuned the Google microservice application, which is called Hipsters Shop, which is composed by 10 microservice. Again it's already exposing some complexity for the performance engineers, but it's a way a simple application with respect to what we see in real life customer settings. But the key thing is that we actually optimize the end to end performance of the online boutique application, for example, reducing the cost of the application while maintaining the level of performance or reducing the performance peaks that you see in the application latency. And the AI was able to actually optimally configure all the 10 microservices with respect to their CPU requests and limits. And for example he find out interesting configuration where for example some microservices were under provision. So AI found out that it needed to add more resources to them. And for other microservices they were over provisioned. So AI found out that it could reduce cost by removing resources to them. So the nice thing is that the whole optimization becomes goal driven and our task for performance engineering become of course defined in this kind of goal. And then the problem process becomes actually completely automated so that we don't have to spend more time and we can also leverage the night and the weekends and where we are away from the office to let the AI work and make sure that our system are optimized with respect to our goals.

Yes, that was my next question. How much human involvement is involved? Does it bubble up and say hey, these microservices change and if you do this, this and this, it'll resolve it or you just sit on autopilot and it does its thing?

Yeah, there are actually several scenarios in which you can use AI in this kind of environment. So the way to I would say it's to use kind of optimization as a service, meaning that the application teams or the performance engineer need to in a way revisit the new settings of the container or any runtime, et cetera. Perhaps due to a new release of the application or new architectural changes, you are moving to the cloud, et cetera. So you kind of fire up a new initiative that is kind of ad hoc initiative. The next thing that you can do is actually integrate this kind of optimization into your pipelines. This is very interesting because this way you will be sure that when a new application application will be released to production, not only it will pass your traditional pipeline test, for example security, functional testing or performance, no regression, but you will be also sure that also your configuration settings will be optimized for your goals. That is something that it's very interesting because can also automate the whole optimization process in a way. The first step that we are working on is also to in a way optimize the application directly in production, which is actually the kind of next step for ML to in a way being almost totally unsupervised and doing things. Whenever it thinks it needs to optimize the application, it will kick off new optimization without any of us in a way interfering or having to do any kind of action.

How much data is needed or how much time is needed before this really helps? I guess because if you have just one run, one baseline, it's going to take time for it to know what's a real norm and what's not a real norm before it starts injecting maybe what could be not optimized changes.

That's a good question. Actually it's, we'll say one of the biggest area where we invested from research point of view in the work that we do. So we had to invest lots of time in researching the best algorithms and design new ones actually machine learning algorithms with the purpose of actually optimizing the application in a way that is as fast as possible. So it's an interesting challenge because you have lots of parameters, as we said, that may interact in complex ways. So it's hard to understand how those parameters actually impact the end to end goal, which might be your application performance. At the same time, we need to be fast because actually that's the way it works today. So we cannot say generally stop the customer environments for one year just for the sake of optimizing them. So it typically takes in the order of hours to a couple of days to reach a very good optimization result. Most often actually we see a great improving results in the initial runs because actually due to the fact that this problem is complex, actually nobody actually is doing that manually. So most of the time also the initial experiment executed by machine learning are already better with respect to the existing customer configuration. But in if you let more time and let it work, you will learn from the past experience and it will be able to extract additional performance for free.

Well, what I thought was interesting, we don't have time to dive into this is you had another session on Kubernetes with JVM and how AI helped with garbage collection, all that type of things. And a lot of the solutions that came up with AI were not intuitive in the sense that it kind of went against the grain of what performance engineers were taught about what to do with certain settings. So I assume that applies to everything. Maybe it comes up with something that you like that's not a best practice, but maybe in this situation, in our environment, it knew that this is actually the best practice for your situation. Does that make sense?

Yeah, that's very, very interesting area, especially for us as performance engineers. So it's not actually just the jvm. And I think the interesting thing is that as long as we run those AI driven optimization, we actually like to spend time as performance engineers to really understand because sometimes we see that the results are really unexpected and something that we wouldn't actually recommend based on the best practices. So I have several examples about that. One of the things that for examp we saw in optimizing that was not the JVM, it was a database, so it was a MongoDB database. So we asked Akamas to actually improve the number of queries per second that the database could achieve. And we optimized the database settings and also the Linux operating system settings. One of the best configuration that machine learning came out that was totally unexpected was actually a configuration that improved the way, in a way the storage. The new storage system actually issued I O request to the underlying hardware. And in particular it was actually having an impact in the number of requests and how they were merged before actually being issued to the storage. And if you look for example, in this case at the response time of the I O request at the Linux level, in a way the best configuration was actually having a higher response time. So it looked like from I O perspective, the response time of the IO operation were in a way higher than the baseline. But at the same time this kind of settings made the database which was sitting on top much faster. That was due to the fact that the I. O that were issued to the storage were larger and they included much more data because of course this way you saved the roundtree time. So machine learning found out that it was better to actually request bulk kind of request and requesting more data and requesting it less often so that the database queries for us needed to do full scan, etc. I have all those kind of data already available in the cache. So that is an example on the database. But I have lots of examples also documented in. You may read several blog posts about the GVM tuning in our website Akamas IO and that is another very interesting area because actually it is back also to the default values that we have today in our system. So for example, the GVM garbage collection is what the most common in a way topic of tuning for the gvm. So people performance engineer in a way hate it because you need to tune it, but you know that spending lots and lots of time, you will be able to extract additional pretty significant performance from your application without touching the code. So it's a pretty hot topic in the industry. We found actually several interesting stories there. One of the things we found is that for example, the default garbage collection that you find Today on the JVM, it's called the G1, it stands for garbage first. So it's a garbage collection that was designed in a way to collect all the garbage that the application was generating, so all the memory without the developer being able to being needed to be to reclaim the memory. So the JVM has this garbage collection concept that makes the memory management automatic for you. But of course when the JVM the garbage collection kicks in, your application will be stopped. So this particular garbage collection that is the default that you have in the GVM 11 today, it's also the default for the new one. In a way it's designed to reduce the amount of pauses that your application will get. And it has a lot of internal heuristics to in a way decide for example, how when to kicks in, when to in a way interrupt the application, how much garbage to collect, how many threads, so how many resources you CPU memory, the garbage collection itself needs to use to do the collection, etc. So there are lots and lots of in a way also parameters that are exposed by the JVM itself, by the JVM designer to let you in a way tweak the way the garbage collection operates. So one thing that we have found out is that for example, the default garbage collection was interfering with application performance because it was using too much threads, too much CPU resources, and it was stealing away those kind of resources from the application threads. So the interesting thing is that G1 was designed to reduce the application process, so make the application run faster. But in doing so, it needed to use much more CPU resources that ended up slowing the application even more. So we used AI to tune that. And the configuration that actually improved performance by significant margin, I GUESS it was 30%, was actually a configuration that in a way make the garbage collection in a way work, stop the application for a longer time, which is something a little bit counterintuitive because if I stop my application for a longer time, I will have higher pauses on my response time. But considering the whole in a way kind of interactions, also on the CPU level, this kind of configuration was actually making the application run faster because this way the garbage collection was using less CPU and the application could use more CPU when the application could run. So that's just an example that I guess illustrates all the complexity that we have under the COVID in our application and all the tunables, actually all the configuration spaces that I won't say that nobody is looking at it because actually there are lots of skilled performance engineers that are actually willing to embark in this kind of journey. Journey. But it's really, really complex. And the beauty of machine learning is that you can learn without us telling machine learning how the application is built. It can automatically learn what are the best settings for the application performance, for example.

Yeah, it is really revolutionary what you're doing there, what's going on. I think it's going to change the game. And I'll have a link to your blog post on all the things you just covered in there. If people really want to take a deep dive. Like I said, I think AI and functional automation is kind of overhyped. But I think machine learning with performance is really the sweet spot because I think it really can help with a lot of the issues that testers are facing. Okay, Stefano, though, before we go, is there one piece of actual advice you can give to someone to help them with their performance testing efforts? And what's the best way to find or contact or learn more about your solution?

Okay, great. Thanks a lot for the question. So I guess for performance engineers, I always like to in a way expose this kind of complexity because I really think it's also interesting for us to understand that we have this actually space of potential improvement in the performance. So a lot of time performance engineers are focused on, for example, improving applications, application code writing better, queries to our database, etc. That is of course fine and it's still in a way needed. But it's interesting to know that if you are looking under the COVID you are in a kind of lots of stuff are happening by the runtime, by the kubernetes, by the Linux operating system. So I would suggest people invest time in understanding those and understanding that you have those kind of configuration options because they really can impact your application performance. But also the costs from an organization perspective are really tied to the on the day, to how you configure your application. And also the reliability, which is a kind of hot topic today, is very much impacted by those kind of configurations. As regard the other topic you may find, I guess the best way to reach out to me is via LinkedIn. And if I may suggest, if you want to try machine learning based technology Akamas, you may do so because we have launched a free trial initiative this year and actually lets you in a way play with the tool and you can even download Akamas in your laptop and play with a sample application so that you can realize the potential of tuning and seeing how AI driven optimization can really work in practice. And we are finding that performance engineers are loving it because it's so much evolutionary, it changes so much the game that we are getting such a great feedback. So something that you can do and share with us the feedback. It's something very appreciated.

Thanks again for your Performance Testing Awesomeness Links to everything of value we covered in this episode. Head on over to testguild.com P81 and.

While you're there, make sure to click.

On the Try them both Today link under the exclusive Sponsor section to learn all about SmartBear's two awesome performance test tool solutions, Load Ninja and Load UiPro. And if the show has helped you in any way, why not rate and review it in itunes? Reviews really do matter in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Performance and Site Reliability Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack performance testing awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Performance and Site Reliability Podcast. Head on over to testguild.com for full show notes, amazing blog articles and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.