Zero to Hero DevOps Performance Testing with Giannis Papadakis

Get ready to discover some of the most actionable DevOps techniques and tooling, including performance and reliability with some of the world's smartest engineers. Hey, I'm Joe Colantonio, host of the DevOps Toolchain podcast, and my goal is to help you create DevOps toolchain awesomeness. Hey, it's Joe and welcome to another episode of the test guild, DevOps Toolchain. Today we'll be talking with Yannis all about monitoring Key6 through Grafana and also performance testing and a bunch of other things that he's involved with. Really excited to have Giannis with us. He's a seasoned tech professional with over 13 years of experience in QA engineering, currently steering the testing strategy as Director of QA@GWI. He's a lifelong learner. He has a deep rooted interest in agile methodologies and testing techniques. And outside the office, he's also a passionate software testing and DevOps advocate. You've probably seen him on the socials, but he fosters connections and knowledge sharing through his Athens based meetup group and he's recognized for his contributions. He's also a proud Grafana K6 champion, which we'll dive into as well. Really excited to have him on the show. You don't want to miss it. Check it out.

Hey, if your app is slow, it could be worse than an error. It could be frustrating, and in my experience, frustrated users don't last long. But since slow performance is a sudden, it's hard for standard error monitoring tools to catch. That's why bugsnag, one of the best production visibility solutions in the industry, has a way to automatically watch for these issues. Real user monitoring. It detects and reports real user performance data in real time so you can quickly identify lags, plus get the context of where the lags are and and how to fix them. Don't rely on frustrated user feedback. Find out for yourself. Go to Bugsnag.com and click on the free trial button. No credit card required. Support the show and check them out.

Hey, honest. Welcome to the Guild.

Welcome to it. Nice to be here. Oh my God. It's a dream for me. I mean, it's really. I'm really excited for this.

Well, I appreciate that. Thank you so much. I've been following you on the social media and I've been seeing you post a lot of different things. I guess the first thing that caught my eye is you're doing a lot in the community. With the meetup in Athens, I thought maybe we could talk a little bit. Why you did that what is it? Is it a Ministry of Testing thing or is this your own user group?

No, actually the testing community in Greece started with Ministry of Testing, which was the first, let's say, meetup group organized like three, four years ago. You know what I wanted to make. So it's a different meetup group. It's not Ministry of Testing, but we're working together with the guys of the Ministry of Testing very, very closely. I would say that we're going to join forces in the future. But what we're trying to do, why I created this group was like example in K6 or Lambda. This doesn't matter. But the good thing is I'm part of a broader community and I found problem, especially in the Greek community. Like we wanted more technical solutions, you know, more workshops, more knowledge sharing in the low level. I would say more technical, low level and say, why should we not create a group for this? And you know, we can exchange this knowledge. We have done workshops live. We are about to have more speakers globally that are coming in Greece. So for example, in September we have Manoj Kumar coming Greece for the meetup. So there are. We get attention from other people. We want to work with other people in the, with other professionals. But we want to focus on the need of having more technical, let's say, topics covered within the community. And this is what was, you know, my first, let's say, envision of creating this group. Right.

Love it. So any tips on how to start a group? Did you survey the group and you found, oh wow, people really want technical stuff. Let's make this technical heavy? Or is that just what you felt or that's the direction you'd like to go?

The beginning I said, you know, I didn't get any feedback like officially from the community. I started like, you know, there was the Ministry of Technical, it's covering a lot in the community. And they said, let's start by creating some online workshops or some episodes. Like we are doing podcasts every two weeks with different people from different companies. So I wanted to have live talks or office hours podcasts within the group so we can share more lightweight even with people from other companies, like professionals from other companies. And I said, let's go for it. Let's see how we can grow with our members and see how it goes. And I'm extremely happy because after we are operating seven, eight months, I think, and we have around 300 members right now and we are growing, so I'm pretty happy where we are going with group. Yeah.

What's the name of the meetup. I love the name of the college you went to. Aristotle Technology. Assam. What's the name of this meetup group?

No, the meetup is Athens Zet from Software Developer Engineering Test.

Okay.

Athens Deck Meetup.

Cool. All right. So, you know, you said little. You mentioned something about being a champion. So you mentioned K6, and I think you mentioned Lambda Test. I've been seeing an uptick in people being champions or, you know, ambassadors. So maybe is there a benefit to being a champion? Could you tell a little bit more about what that means to be a champion or an ambassador?

Yeah, first, I think I wouldn't say the recognition, but some people, you know, they like the recognition, having their name, you know, recognized within the community. I wouldn't try for that. Mostly I'm. It's the knowledge sharing. You have a direct, like communication and the networking channel with the engineers or the dev relatives organization, which they are, you know, the best names in the market, some of them. And you can actually work together with them, you know, plan ideas, how you can contribute for the different organizations and the different communities. And it's super amazing because sometimes, you know, you are working on your tasks in your daily routine, but you need to think big sometimes how you can expand your knowledge, how you can cooperate with other people that are professionals, but they are champions actually in the market, so in the community. And they are. This is for me, my motivation that I applied for this ambassador program. And for now I'm really passionate for sharing the knowledge through the meetups, podcasts, even the communities that these companies have created for the people, for the professional. So, yeah, that was the first motivation.

Very cool. Another thing, it seems like as you are focusing on performance testing, a lot of testers sometimes miss out on performance testing. This is the DevOps toolchain. Do you see performance testing as a piece that does belong in DevOps? And if so, maybe what is performance that we're talking about here?

Right. I would say performance testing. Traditionally there it was a topic that DevOps can cover or system engineers in the past, before DevOps actually being. It was a concept back then. But I think performance is something that belongs to engineering teams nowadays. So it should be part of the culture and the mentality of the delivery flow of a software house or a software company. So I wouldn't say it's something that only DevOps do, but the whole team should be responsible for. Like, you need to start small, like from a proof of concept within your team, and then, you know, expand, of course, in the way that you are expanding your product, developing your program. At the same time you need to expand your performance testing. So I have seen that a lot of tools in my experience and the most common problem is you have something that you need to operate everybody. So you need like a technology stack which can be easily used by front end engineer, maybe from backend engineers, from QA professionals or even DevOps. Right. And in the way that you know these tools are integrating the with DevOps, let's say technology stack, it's you can do everything, really everything. It's amazing. And I would emphasize on K6 because I used K syncs my, you know, 3, 4 years experience. Now I use JMeter before K6 as a performance tool but we migrated the K6. It was really lightweight. The way that you can scale your test in, you know, an environment, the cost, for example, how many resources you need to scale your tests will significantly better. We can do a comparison between the tools but you know, I will give the question the answer based on your question. I don't want to extend too much if I start. I'm just speaking too much.

Yeah, probably a different episode. K6 versus JMeter would be a great episode for sure.

Right? Maybe it's another one. Yeah, yeah, yeah.

Cool. So, so interesting you chose K6 though. You mentioned, you know, a lot of times you want because it's a part of the DevOps pipeline, you want your front end developers to get involved. It seems like to me, once again, I'm not saying one's better than the other, but K6 to me seems like it would lend itself better to a front end type of developer's workflow than like JMeter. Am I correct in that?

Yeah, yeah, yeah, yeah, it's correct. It needs to operate on top Of Go and JavaScript. You know, someone with a front end engineering background, it's more easy for him to operate. Of course the QA people, you know, QA people right now, they have mostly everybody have development background at some point, more or less. I wouldn't say we have a traditional manual QA process. So K people, front end engineers and definitely DevOps are people that are involved a lot with K6 in DevOps because it's a part of the monitoring. They need to set up everything on Grafana, you know, the Time series, the base and everything. So everything is there but the dev should be working on alerting the teams creating SLOS metrics and everything is part of the DevOps pipeline. Front end engineers and QA people are responsible for owning the coverage. Let's say of the whole stuff, the scripting and everything.

Absolutely. So I think another kind of myth with performance is that it's only for load testing or stress testing. But within DevOps it seems like you could use for all other types of performance related testing. So is there any other types of testing you think performance wise people are missing out on because they maybe just relate it to like a stress test?

Definitely there are a lot of types. So load testing has different categories. For example, one category is the smoke testing. A lot of people doesn't know they know, but they don't use it a lot when it comes to load testing. So smoke testing means that you can safely run continuously small parts of your test without any, you know, options like stress stressing the environment for hours or whatever. So smoke testing, it's more related minimal load. Just validate that your APIs are working as expected in smoke testing is really good if you have continuous integration, continuous deployment environment and you know, you need to have a CI pipeline that can run per commit or whatever. You know, when we have changes during the development, you can actually run this type of test very quickly and validate if you are okay with your system. We have load testing as you said, we have stress testing, we have the endurance or stroke tests and you have the spike tests which the spike is derived from stress testing which is another category. With K6 it's really easy to differentiate the configurations in terms how you know, what options do you need in order to have, let's say a low test or a stress test or a stroke or shock test. So there have this meaning of stages in the options that says you can define the duration of your test in the target, meaning how many concurrent users you need based on the duration. So with these two attributes you can simulate every type of this test that I told you about. For example, you need a soak test. What does mean, what does it mean to have a soak test? I need like 4, 5 minutes to warm up the system with for example 100 users and then I need to operate in that load for a bigger period of time. Like I need to load it with 100 users for eight hours and then suddenly stop the system. So this is an actual example of a shock test. Of course, this type of test that take a lot of duration, it cannot be part of your shift left testing, let's say strategy because you cannot like actually wait eight hours to validate a commit or something that you need to deliver in production very quickly. So but you can run it in different scheduling based on how the team is operating. So I would say yeah, this is the mostly more or less all the categories. It's pretty easy with K6 to simulate all these categories based on the two attributes that I told you about. And it's extremely useful for, for any performance engineer to be able to run all of them at some point for the system. Right, nice.

So you said a performance engineer to run all of them. I'm just curious to know as we go in the DevOps pipeline though back in the day always was a separate team as part of a performance team. We did performance right before it went into production or to do a proof of concept. But in your real world experience, do you actually build in certain type of performance tests when people check in code you run a performance smoke tests or did you resilience tests or is it always a separate activity outside of the CICD pipeline?

I hope not. But it basically, I mean I'm a really advocate of stiff left testing because it's. You can prevent things happening in an early stage of development. So for example, this is based on the architecture as well. For example, we are working on GWI micro front end architecture. So our front end is developed on micro frontage, meaning each micro front ends a separate repository with its separate API, for example. And you can deploy each micro front end separately on an environment. So for example, I want to test the shopping cart micro frontend, for example, which is one part of the whole product. I can deploy it, I can deploy the API and actually run my stress tests only for this specific micro front end. And it's super easy because the developer that actually responsible for the micro fronting can actually also create this type of test as part of the, you know, the specific architecture. Now if you have a monolith like the product's monolith, so meaning that you have the whole product everywhere. What I have seen from my experience in monoliths, the problem is that you do the development, you deploy on staging on UAT or whatever environment you have and then you are doing your testing, as you said, in a separate stage, which is outside of the development phase, which is for me, both have value. Of course it has value to run your tests after you deploy something, but to run it as part of your team that it's developing a specific API of the system. It's also really, really valuable because the defects that you are going to find from your performance tuning is going to come before even doing any deployment. Like, you know, going to a more stable environment, integrated environment.

Absolutely, totally agree with you. So besides That I have a secret slide deck. I I believe you're giving this a talk on performance in Athens and I'll have a link to this in the show notes people should definitely check out. But in one of the slides you have SLOs and you talk about who the different users are that would benefit for certain SLOs. So could you talk a little bit more about what is an SLO and how for performance, like what kind of SLOs would help?

Right, so service level objectives, it's a term, like it's an agreement that you make with your customer as a term. I mean generally speaking. So even with performance, it makes sure you give an agreement to your customer that the uptime of your application is always in a good shape. This is one metric, for example, or the latency of your requests for specific services that the customer is really keen on using, it's going to be very, very low. So these all agreements that you make with your customer is type of SLOs. Now the SLOs can be used in terms of performance, as I talked to you about. And the QA people are actually have a way with K6 to codify the SLOs, meaning that they can actually develop the needed conditions in order to make sure that we have an agreement in place and it's not violated as an agreement. So you need this, maybe the agreement could be not on your customer. Maybe an agreement that the QA that is developing the script is an agreement internally with your product. Maybe the product needs to know, you know, what's the uptime of the application or if we have a good throughput on your system, you have zero or very, very little request latency on your services. So codifying SLOs, it's about thresholds. The threshold is a K6, let's say feature. You can actually use the thresholds to make sure that you can codify through your performance tests what you need to in order to make sure that you have a condition not violated when you are doing your performance testing. It's really amazing because I'm going to have a talk in September about SLOs and SLIDES and SLA. So these are meanings that comes with your customer. And when it comes to performance, you need to make sure you have a reliable system. So the reliability of the system is in a good shape for your engine, for your customers and how you are going to achieve this. You need to actually make sure you have metrics in place on your Grafana dashboard. You have the appropriate alerting when a threshold is violated. Meaning when an SLO is violated, you need to make sure you alert your things and you have on call engineers. Most probably you have the DevOps which can be on call engineers and they can actually pick up when the system is having an issue during your performance, let's say execution.

Nice. So I know also back in the day when I used to do performance testing it was mostly back end testing and we just have like a one runner script just testing the front end. But it wasn't very accurate. Nowadays is there a difference now? Is it more performance testing focused on the front end and if so, does K6 handle that? Is it all still backend type of activities?

Yeah, I agree. I remember myself 10 years ago that the with JMeter that would, you know we have this nice tool with a graphical ui. We knew the endpoints that are, you know, the backend APIs that we need to invoke and we were stressing this based on the resources on the back end. Now the, the thing that how the application have evolved, how the web development is evolved. Now the backend is one part, let's say of your performance testing strategy. You need to figure out the whole user experience because you have both front end and backend meaning that you have the performance of the backend. But this is the server side of the things we have the client side. You have a protocol communicating frontend with the backend. So there are a lot of layers like front end APIs that are actually communicating with the backend can be websockets, can be HTTP whatever protocol you are using in your teams. But you need to figure out how in all this, let's say journey of a request like your customers authenticated to the system. What does it mean? It's you know, make the user credentials and submit and the submit is actually triggering the request to the backend. Right. Back in the days we said let's load the backend and see how it operates. If the database can occupy all the transactions, if we have any memory leaks, let's say on the backend. But now it's all about the user experience as well. The customer needs to know how fast something is loading. What does it mean it's loading. We have the Google Web Vital which is metrics that developed from Google. A lot of action tools are trying to do front end performance monitoring. Also K6 of course we have Marie Cruz that it's the devrel and it's actually we're working together on this one. She's working on the front end part. It's amazing because all the tools are trying to figure out what it's missing in terms of performance testing. It's missing the whole journey of making a request from your front end. What does it mean? How frequently the backend response, the response times of your backend, how the front end is actually operates. For example, you have a slow network, you have a 3G fast network or a slow network, if you have different geolocation, you have your users operating from another country. How fast it's actually your site is rendering on that. So the part of the user experience or the mobility is actually shifting and changing the way you are monitoring through performance tests. It's amazing. I mean the whole idea I want to measure everything and how K6 is actually have this hybrid approach. You can actually load your backend and actually load your front end at the same time. If you have good resources, of course. And then you can actually see both if you are loading your backend and your front end, how things are going, how the metrics are going, how the Google Web vitals are operating. And you know, we have Lighthouse, I can say another tool. So I worked with other tools in the past. You have Lighthouse from Google, you have site speed, IO. There are open source, very nice open source. We have pingdom. There are a lot of possibilities there. But think about K6 is something common. You can do back end in frontend at the same time. So. And it's one technology suck. You don't want to bring too many technology suck within the group. Right. So it's really cool. I think that, you know, it gives this possibility.

All right, so someone listening to finance might be like, oh my gosh, where do I start then? Like how do I create a realistic scenario? You know, I don't have no idea.

Right. So do you have any tips or like how do you create scenarios that actually emulate them? What's happening? Because you do have APIs and microservices and authentications and then like front end putting back end. How does that all work?

Yeah, I think the best is as I told before, it's to start small. Start with a base case that can cover for example your backend based. First you need to know about your product and especially as you said, the authentication. Do we need to have authentication? How we're actually pushing this authentication to our performance scripts. You know, by the way, K6 has this ability to correlate. So you have set up teardown. You can do your setup as in unit test or in, you know, end to end functional test, meaning that you can authenticate a user through the system and then you can actually Move your token, your any access token for example, that is generated, you can move it to the actual body of the execution. What I would say before going to technical is know how the system operates first you need to know how it operates. Do we have an application? You know, we have microservices. What does it mean? Do we have to say we have a different API that can be invoked? Can we actually start with a small case on each of these APIs? And the good thing with the K6, when you have the knowledge of each part of the system specifically you need to know how to create a user flow.k6 and each performance tool, try to imitate, or to imitate actually a use an actual user flow. What a user can do through your system, it can authenticate and then it can actually purchase an item if it is for example, initial. So you want to imitate the whole user flow because this is what you're trying to do. You want to try to mirror what your users are doing in the system. So this is extremely. Even if you do from front end or backend, both should be doing an actual user flow. So I would say start small, start with your backend. This is what advise each thing. It's the traditional way. We need to have performance tests on our API, so we need to start on the backend side of things. And then when you have a, you know, a good sense or a good governance in your backend, you can actually, you know, go with the front end as well and have a hybrid approach within your development. I know it's a little bit frightened, you know, how should I start? But I think the best way is to start small. Don't try to solve everything like from day one, right?

So Yanis, you know, one thing I know people struggle with, when I started with performance testing, I had an enterprise end to end solution that did the load testing, I had the monitoring, had the reports, had everything baked in and then we went open source. And you're like, okay, I don't have any of this insight, but it seems like K6, I think it was acquired by Grafana like a year and a half ago, two years now. Does Grafana, does K6 provide now any data with Grafana that gives you like monitoring capabilities that you may not have with some of these more open source based performance tools, right?

K6, the cloud version of K6, I would say they use their own dashboards in their cloud version and it's really cool because you don't need to create your own way how to query data from time series Database, I mean, or configure your database. I mean everything is running over their data center, the tests are running on their data center and it's actually very cool because you can actually monitor from their dashboard everything. And now with the moment to Grafana as well. They provide the cool and nice dashboard predefined from the Grafana cloud. So you don't need to actually develop anything. So if you have a small team you can invest on going to the cloud version, it's really super great. I can share a lot of information to anybody. For me that I worked on the open source solutions, I wouldn't say the community is really big. It's not that difficult, let's say to use the community edition of K6 and the Grafana Community Edition and create your own, let's say configuration based on your needs. It will need some, I mean you need some people from the DevOps team to work on that task and maybe from the engineering team. But it's really nice with the cloud you don't need any of these. So with the Community edition you need to choose your time series database. You can choose Influx B or Prometheus. You need to know where you deploy that if you have an internal, let's say a data center or whatever or you can operate on Google Cloud. That doesn't matter. It's only matter for the cost that you are going to have. I don't think these data are really very costly for the companies. I mean to store time series data from your performance that it's really lightweight in terms of resources and then you need to find out you, you have your Grafana, you need to set up your dashboard. The tricky point with Influx, I create an article about this. The community is not supporting the influx data version 2. They have dashboards predefined only for version 1. But I have created an article how you can actually do backwards compatibility on influx data version 2, which is tricky there you need to do some tweaking because version two supports buckets for storing data. Version one has a different way how you can store your performance data. So there is meaning that there are a few things that you need to invest when it goes to the open source part or the community based part, when you go to the cloud where all these nice cool things provided from Grafana and K6 out of the box.

A few of the resources you sent me mentioned Grafana Loki.

Oh right.

Is that open source? Is that. What is that?

Look, it's an integration, it's an a log aggregation system which is integrated On Grafana, we use it in ZWBI mostly to we have open telemetry for our backend. So we want to keep traces somewhere centrally. The log, it's actually storing all the logs. It's similar to new Relic, another system or datadog. But we use Loki, which is very nicely integrated with Grafana. It's very lightweight, you don't need too many resources. Again, it can keep your logs for longer period of time. You can have of course retention, data retention policy. You're lucky. And you can query like for example, I want to see all the logs from a performance run. I can go to Loki through Grafana. It has a very nice graphical UI that I can query my logs to find like my traces if I support open telemetry, for example, for my requests. So log is really nice. Think about it like a log aggregation, like elasticsearch in case Stacker, you know, it's still very similar.

Great advice, Arkansas. Before we go, is there one piece of actual advice you can give to someone to help them with their DevOps before performance efforts? And what's the best way to find or contact you?

As I tell you, I'm. I'm K6 champion. I'm on the Slack channel from K6. I think it's open. It's open for the whole community. You can join the slack channel from K6. I'm there. You want power Luckies. You can just jump and we can discuss anything. We have also a group, a community channel in K6. A lot of people are joining, discussing about any issues they have. I can even share a lot of information. I'm talking with Deveresh, Leandro and Marie. We have, you know, a plan. Now the champions are trying to create, to spread the knowledge with different things, podcasts, you know, live works or whatever. And we're bringing together like a plan how we can schedule stuff in order to, you know, communicate to people so they can freely join us on these efforts. Really it's. It's all about the people and professionals, what we're trying to do and of course LinkedIn. You can find me there, of course. Okay.

And for links of everything of value we covered in this DevOps toolchain show, head on over to testguild.com P122. Remember, latency is the silent killer of your app. Don't rely on frustrated user feedback. You can know exactly what's happening and how to fix it. With Bugsnag from SmartBeer, see it for yourself. Go to Bugsnag.com and click the free trial button. No credit card required. So that's it for this episode of the DevOps Toolchain Show. I'm Joe. My mission is to help you succeed creating end to end full stack DevOps toolchain awesomeness. As always, test everything and keep the good. Cheers. Hey, thanks again for listening. If you're not already part of our awesome community of 27,000 of the smartest testers, DevOps and automation professionals in the world, we'd love to have you join the fam@testguild.com and if you're in the DevOps automation software testing space or you're a test tool provider and want to offer real world value that can improve the skills or solve a problem for the guild community, I'd love to hear from you. Head ON over to testguild.info and let's make it happen.