Is it Observable with Henrik Rexed

Welcome to the Test Guild Performance and Site Reliability Podcast where we all get together to learn more about performance testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Performance and Site Reliability Podcast. And today you're in for a special treat. We'll be catching up with Henriks all about observability, what he's been up to. He's recently made a career change. I'm just curious to dive into all the different things he's seen and performance and observability, set reliability, all that stuff. If you don't know, Henriks is a cloud native advocate at Dynatrace, which is known as the leading observability platform. And prior to Dynatrace, he actually has worked as a partner solution evangelist at Niotis, delivering webinars building prototypes to enhance the capabilities of neoload. He's also has worked in the performance world for more than 15 years, delivering projects in all different types of context, including extremely large cloud testing and a lot of different things on the most demanding business areas such as trading applications, applications, videos on demand, sport websites. He's done it all. So he's here. I'm really excited to have him. He also is one of the organizers of the conference named Performance Advisory Council. So you probably know him, but if you don't, you definitely don't want to miss this episode. Check it out.

This episode is brought to you by SmartBear. Listen. Load testing is tough. Investing in the right tools to automate tests, identify bottlenecks and resolve issues quickly could save your organization time and money. SmartBear offers a suite of performance tools like Load Ninja, which is a SaaS UI load testing tool and Load UI Pro, an API load testing tool to help teams get full visibility into UI and API performance so you can release and recover faster than ever. Give it a shot. It's free and easy to try. Head on over to smartbear.com solutions forward/performance testing to learn more.

Hey Henrik, welcome back to the Guild.

It's great to be here. It's been. It's been a while. I think it's been almost nine months that I didn't come to your podcast, Joe.

Absolutely. And what a lot has happened in nine months. It's something I've been seeing a lot of, so I'm curious to dive in there. So I guess before we get into it, say anything in your bio that I missed.

No, everything is completely true. Can also add maybe has the negative reflex to drink beer when he has ended his work.

But oh you know what I did Mrs. You have actually have a new YouTube channel we're going to talk about later that people definitely should register not register for subscribe for which is all about observability, which is one of the topics we'll be covering today. So Henrik, I'm just curious to know. You used to work for a performance testing company. I guess is how I would see it as almost like a performance role. And a trend I've been seeing is a lot of people have been moving over to say reliability engineering and so so I know you recently moved over to Dynatrace, so just curious to know what that transition is, what your new role is. Give people a little bit ideas what you've been up to in the past few months.

So yes, I clearly joined Dynatrace on the 1st of April and since then I took the role of taking the topic of observability related to cloud, native technology and also anything related to SRES as well. So for me spending I would say 90% of my career doing performance engineering engagements or working for a vendor for me was the right timing to take a step back and have a broader vision on project because in performance you do a lot of things but still you're still stick to a specific delivery related to performance testing or performance engineering. And I think moving to more to the observability I think it's great because you see so much different technology. So since I started, yeah I knew few of those technologies because due to my work but I had to clearly spend time digging on those technology trying them reaching out to customers understand the pain because working in the performance since many years I knew from my experience the biggest frustration the pain that you can have when you deliver performance testing or performance engineering. But now with the observability something that I had touched but not so much so I had to reach out to customers and understand what was their main issues, their main pain and try to figure out how we could yeah.

Resolve them. Cool. So I don't know if this is true. It seems almost like site reliability is merging into performance testing or it's a cooler term for what used to be known as a performance engineer. Am I wrong? Is performance still a component of sre? Are they still completely different? How do you see it?

I think the early performance engineering is not part typically with the SRES is going to be more from the continuous development or continuous integration. So more on the DevOps related topics, but the tests that you achieve closer to your production environment Then clearly yes it's going to be SREs. I mean, I mean we used to do 15 years ago test where we were doing failover. We, we will naming those tests failover test where you were running some tests and say okay so now after 30 minutes of test we're going to shut down the web server one and then we're going to restart it and see how we recover or we were doing other stuff with load balancers or application servers and that was 15 years ago. So that was sort of almost what we are naming today. Chaos Engineering. But Chaos engineering is everything about automation. It's, it's smarter, you can do more, more more interesting disruptions on your environments. And clearly if you do chaos engineering you need load testing and the best way of measuring how reliable your environment is before you move to production is obviously combine load testing and Chaos Engineering. So I think yeah, clearly that part fits clearly in the scope of SREs.

Nice. And another reason why I ask, I see a lot of large companies that are mostly known as observability companies like Grafana as an example acquired a K6 and just curious to know what is the overlap there then that you've been seeing Observability companies want then to acquire performance tooling. What's the gist?

It's an interesting trend or topic I would say because as an observability platform you try to be agnostics when it comes to the toolings that you're going to use to generate some load on your environments and be able to ingest any type of traces logs or so utilizing a specific in house tooling for that it could make sense. But mainly K6 is an open source project which is great. A big shout out round of applause to the Grafana labs acquiring them and to all the team that works at K6 and Grafana. It's a great tool and I think it makes sense because it's more people are more utilizing K6 as an open source product more than just like a commercial one. So for them it's a way of connecting all those great products that they have came up on the last couple of years. So grafana of course k6 now low key for the logs. So they have so many small solution and once you connect them the story is resonate pretty well. So I think it's for them it's, it's a really cool opportunity to, to get more followers and more fan of their stack.

Very nice. So I guess before we dive in any further here, I don't know if people have different Definitions of observability. But what is your definition of what is observability?

Observability is for me is obviously you need. There are distinct pillars that comes in observability. But for me is being able to get all the relevant information related to your system and your environment or even your car. And that will help you to understand so understanding a situation. And then you can. From the data that you have received, you can do the right. Take the right actions either automatically or manually. And I think the. On the Observatory stack, especially in the IT world to be able to understand there are of course the traditional, I would say monitoring approach where we're going to collect some metrics, either application or framework based metrics or customized metrics that I have. Also I'm exposing metrics from my application and I want to ingest it because it will help me to understand more on the business level. Also the logs because we all writing logs when we code. So in logs there are so many great information that we can take advantage of. So if there's a problem, you had a stack trace, you can share it with your developers and then you can speed up the resolution of your incident if there's a problem. And of course last is traces. Traces been there, I mean most of the observability platform of the market. I mean Dynatrace including they all have been doing traces since many years. But they were doing in a property way with their own format. Some solution were able to automatically instrument and get all the details somewhere or less powerful. But I think opentelemetry when it arrives is just the ability to put a standard. And OpenTelemetry tell us the ability to ingest metrics, traces, logs, all the things that we expect from to fully observe an environment. OpenTrim is doing it. So I think it's great to have an open standard. So then if I want to use today I don't know elasticsearch and tomorrow I want to move to Splunk or Dynatrace the effort will be very minimal in case or even there won't be any effort, any effort at all because I just use the standards. And that. That is great. I think for the most how many.

People are following the Standard? I know OpenTelemetry I've been hearing more and more about. But is that like something you've seen a lot of vendors actually embracing?

Most of the vendor in the Observatory world are all yeah support opentelemetry in a. In a way. So I mean if you look at the contributors of the specifications I mean Datadog, Splunk, Dynatrace, new relic. I mean all, all of them were there. So for them it's great to come to like a standard format that anyone, everyone can use. And then for them it's a way also to utilize the community technology to extend or improve the way they were ingesting some traces from their end.

So this might just be my ignorance. I haven't been a hand on, hands on performance engineering years. I usually think of APM solutions when I thought of Dynatrace for some reason. But now it's called an observability solution. Maybe it always was. So what is APM and how does it fit into observability? Is APM gone away now? Is that like an old way of doing things of monitoring? And observability is now the, the name you use for all types of monitoring you do across your pipeline when you're doing software development.

I think for me it's just almost like a marketing term. So I think the terminology, we should confirm that with Alexander Podelco. But he, he made a presentation about the history of performance and I think it was Compuware who came up to that terminology saying APM. So that was in 1970. I don't remember the date. We will have to ask Alexander to give the exact date for that. But I think it was interesting because they came up to that word and if you look at the history of the APM product, they were mainly doing metrics quite well. Some of them was doing traces. Logging was not so much covered. And then suddenly over the few the last years logs has been more and more used. There was an interest for that and I think because of collecting not only metrics and traces but also other stuff it turned out to say. Now yeah, let's, let's be clear. We don't do only application monitoring, but we do than that with the observability because application means I just want to look at my app and that's it. And here if I'm deploying my workload on Kubernetes, of course I've got my app, but my app is running an environment which is shared and distributed within our various teams. So I have other problems or concerns. That's why I need to probably extend a bit more. The way I'm collecting things will be interesting to get the events, Kubernetes events as well to see how those deployments and how the workload is behaving in my cluster. So I think that's why we don't stick to the application anymore. We are more broadly monitoring yeah the entire IT in fact, more than just apps.

So can you give an example of observability like the life cycle, like an example of maybe how it helps someone so people get a feel for why observability is such a hot topic nowadays?

I had a story, it's a story that came up. I was this summer. So it's a story with happened at the Dynatrace. And the funny story is that basically Most of our SaaS platforms relies on cloud providers. So if a cloud provider has a major outage, then there's a big chance that we would be suffering of that outage because we rely on them. And at Dynatrace we use our product in everything we do. We use our product because it's a way of first of all making sure that the product is doing what we expect and it's useful. And for a lot of reasons we do that. So the team internally used it and they basically detected the problem through collecting all the various monitoring events from the cloud providers from other environment. And then because of all the nature of the Dynatrace product with auto remediation and so on, there was already rules set up in case of. And the auto remediation just happened, did what it had to do and we didn't even even be in touch by or impacted by the Amazon. I mean there was Amazon, but the Amazon outage. So that was a great story because at the end we just put some automation. We just put the right level of instrumentation to get, like I said previously, log metrics, events and traces. And then once you have ingested everything on your observability tool, then you can create the right alerting, the right rules, the right auto remediation task if you have a problem detected. And this is what our engineers did that day and no impact. So I think that is just a great story. So just discovered the next day that there was a problem on Amazon and then say oh that's funny because we rely on Amazon and we didn't have anything. So so that's great I think.

So was the remediation someone manually doing something, switching something so there wasn't an outage where they get alerted automatically or is actually rules that say if this happens then do this automation go to this other backup server or something.

So yeah, the remediation is you basically defined in case of this type of situation, you do this. So if you define several rules to manage. Yeah, an outage, because outage could be from different sources, then he will basically trigger the right actions for that. So. So I think that's to define the right action that you have to set in place for your remediations. Yeah, I think like we said sre and load testing, Chaos Engineering is the best exercise to come up to the right set of rules and the right set of actions to avoid impact on your customers.

So that's interesting. Then before to actually set up these alerts, then you can use Chaos Engineering, say what would happen if the service we rely on went down and then set up a remediation type rule within your platform. So if it actually did happen in production, then it's handled. Sounds like correct. Cool. So what tool, what, what technologies do people need to know to, to get this in place or how, how easy is it? It sounds like oh, you just put it in this system, you've set up some rules and you're off and running. Is it that easy or.

I mean it really depends on. Because when you're working in the cloud native space, if you look around you there is so many great framework and great open source technology that are out there. So there is a natural reaction to say hey, I'm going to do, I'm going to do it by myself. I don't need, I don't want to utilize Datadog, I don't want to use standard Trace, I don't want to use new Relic, I can do it myself. I'm going to set up Prometheus, I'm going to set up Jaeger, I'm going to set up a collector like Loki which is open source as well. And I can do some fancy nice dashboards and alerting if I want. So that's the hard part. But the other problem is if you do goes on the direction is you need obviously time, you need engineers, you need to have also the process that manage the entire observerly platform that you came up on. So it's work. So I think when you are moving in the observatory space, it's really important first of all to understand the solution that you were planning to use. So either a commercial one or open source. Understand all the various framework that are there. Because ingest logs, there's so many ways to do it. And just ingesting logs you can say hey it's easy, you take a file and you send it over, you index it and that's it. No, there is more things about it. So probably you don't want to saturate your storage so you want to optimize a bit the way you're going to ingest the log. So you're probably going to filter, add more information to the top of the log. So then you will be indexed with the context of the app. So even just a small task of ingesting logs could be a project by itself because you need to know the technology, you need to know the logs that you are going to collect, the format of those where you want to send it over. So there is so many things to consider. So I think for someone that starts and start working in that space, cloud and observability, it could be quite scary because you have to look at everything that comes up on the market and if you just look at Twitter, you could be just amazed by the number of solutions coming out every day. So then you say, ah, how can I keep track on everything? It's quite scary.

So you did mention that you've seen an uptick in companies using cloud native technologies. Is that why there's also a correlation between people getting into observability? Because now there's so many things outside your control that you need to be able to monitor it if a service or third party thing goes down. Is that why there's a people going to cloud native technology? You definitely need to have observability in place or you're not going to be successful.

Yeah, I mean everything started with the, of course the digital transformation where people, they had this big monolith and they need to distribute their content or services to more channels, web, mobile and other type of stuff so they have to rethink their architecture. And I think microservices was a good practice. And when you transition to microservices then the natural choice say hey, I heard about Kubernetes and it's going to be easy and everything's going to be by just defining YAML files to deploy. It's wonderful. So let's do it. And then you start maybe manage your own Kubernetes environment and say, hey, it seems complicated to configure. So it seems that Google has a great service or Amazon or Azure has great services. So why don't you use a managed Kubernetes service that they provide and that makes your life easier for sure. But then comes the other part of the iceberg where you say okay, so I manage everything on the cloud, I rely on cloud providers and also I need to auto scale because I want to basically reduce potential issues if I have some huge spikes. But again, there is another problem that comes in the picture is the cost. So a lot of people managing their communities Clustered, they try to sort of a build specific dashboard to figure out if they are overusing or in the opposite, they are not using in an efficient way their nodes that they are paying on daily basis. So maybe they could rethink the structure. So yeah, moving your workload to the cloud and moving to microservices it bring more complexity because your application is more distributed, more complex. There is tons of dependencies everywhere. So you want to understand if your services is slow due to your own perimeter or maybe it's a new service that we just opened recently and that is just over consuming my service. So there are a lot of questions that comes up in the top of our head from the moment we transitioning microservices and then of course to the cloud.

Nice. I know Dynatrace also has a tool called the Captain. Does that have anything to do with the cloud native and observability or is that something totally different?

It's an open source frame of solution that can do tons of things. You can either do the entire CI cd. So it's an event driven framework in the opposite of traditional CICD tools where you define a huge pipeline YAML file or in Jenkins you're going to have a groovy scripts or whatever. And why we decided to work on this project is we looked around us and say hey, it's funny, people are building pipelines so they're building code to validate their code. And then they say okay, so project A say I want to use Selenium, I want to use BitBucket, I want to use, I don't know, a set of toolings. So they are building this integration within their pipeline. And then you, you step back and you look at team B and team B is doing another type of pipeline with other tools. So they are also spending a lot of time just to integrate those tools there. And then suddenly you have, you look at a step back and you look at team C and you realize that Team C copy paste part of your pipeline. So they copy paste your code and they reused it, slightly changed it and then you just leave it there. And then you look around one year later or maybe six months later and then you realize that this small code that has been copied, you realize that it's almost in 2000 plus pipelines of the organizations and say okay, so who is going to maintain it Once the creator of that code believes the company, it's going to be a nightmare. So yeah, there is a snowflake effect of those CICD process. And Captain wanted to do a Very simple way of describing your pipeline. You describe stages and then you notify your captain which uniform he's going to wear. So today my captain is wearing Selenium Jenkins or GitLab is wearing JMETER for load testing and he's wearing Prometheus to collect metrics. So you just precise which toolings you want to use and then because it's event driven he will reach out to the right toolings to do the right actions on your pipeline. So it makes your life easier to be honest. So we don't do only CI and CD processes. We are pretty much reflecting the the approach of the launching a spaceship with NASA, nsa. It's you have a team that's in charge of putting the spaceship for launching and then once it's on the ground and the engine starts, it's finished. It's similar to the it, there's a team building the app and then once the the rocket has been launched, it's almost in has quit the ground the earth. Then the other team mission control takes over. So you captain is the same. So you have, you can either do everything which is before, you either do everything which is after production. So auto remediations, problems detections, stuff like this or you want to do everything. And what I really love about captain over the last years when I was still working in the performance engineering world was the quality gate. When you do pipeline and you want to do automated performance validations at the end what is lacking in our automation sometimes is just performance. You realize that project are waiting 62 or 72 hours to have someone coming in, looking at the results, giving his feedback and say okay, red or green and then share the results to the project owner or leader and he will say okay, we can go to production and this is not possible. It's not automation anymore. It's just we just posed the automation to ask the experts to give his insights on what was happening. And the quality gate resolved that pain with reusing this notion introduced by the SREs. So sli and slo. So you define which indicators make sense for your project, what are the objectives related to those indicators and then the quality gate will, will evaluate all of those objectives and will give points. And what I love is that the scoring mechanism that comes out of the quality gate because then you can say okay. For me a good release is a score of 90 and otherwise if you're under 90 you don't deploy. So I think it's a way of trusting the green the positive that comes out of performance that usually in the past I remember we're using mainly SLAs and SLAs that you cannot do. You can have rules that are more important than the others. The notion of priority doesn't exist so much. So if a rule fails, then it fails your test. And sometimes you. There are some basic checks that you want but you don't want if they are just failing. It's not a big deal. I mean, if you take the example of an E commerce application, add to cart is a critical transaction or search is a critical transaction. But go to contact page. I don't want to block my deployment just because the contact page is not responding in two seconds. To be honest, I don't care. So that notion of what is more important compared to the others is being introduced with the SLI slo and that is just wonderful, I think.

Awesome. Awesome. So you seem to be on fire with observability. I think you even started a new YouTube channel called Isit Observable? So just a quick plug, what is this channel all about? What type of topics do you cover?

So the main objective of Is it Observable? Is basically to bring educational content on observability. So the idea is that if you don't have time and you have to work with new frameworks or new techniques related to observability and then check it out on my channel, I just want to give you free videos where you can understand the technology, the architecture, how to use it. And for each episode I deliver a tutorial, step by step tutorial, where you install the demo app, you install the Prometheus, if you can take the example of Prometheus. And then you do all the things we do some dashboarding. So basically you know how to get started, start. And that's the most important thing, especially when you're dealing with new technology. Sometimes you look at the documentation, sometimes say hey, is that a documentation? And. Or sometimes say oh, there's no documentation yet. Okay, so I'm going to try it out. So that's the idea of easy observable. And for those who are not Dynatrace customers or users, it's not a big deal. I try to do my all the episodes with open source technology as much as possible. Sometimes I do some exercises or tutorials where I say, okay, today I'm going to use Fluent bit and I just want to connect it to Dynatrace. I mean, I know that Dynatrace is going to ingest the logs, but it's just the case of showing that taking a pipeline where I have some log stream pipeline where I want to transfer my logs to the right format of data trace and send over. So that's just an example, but it shows you how to utilize Flynn bit, for example. And that's. That's the idea.

Awesome. And I'll have a link to this channel in the show notes for everyone that's listening. So all you need to do is go to testguild.com p like performance76 and you'll see the link. You definitely want to subscribe to this channel. As I said, this is a huge trend that you want to get on top of before you miss out. All right, Henrik, before we Is there one piece of actual advice you can give to someone to help them with their observability efforts and what's the best way to find or contact you?

First of all, you go to YouTube, you search for is it observable? You can just subscribe to the channel. So you will be notified on the next episode coming in. But just to let you know, there is one episode every two weeks and they are released on Thursday. So otherwise you can reach out to me on LinkedIn or Twitter. So LinkedIn you just search for Henrik Rexhead and you'll find my LinkedIn page. So don't be afraid of if you see a LEGO avatar avatar. So that's me. It's not a fake account. And same thing for Twitter. I also have this LEGO avatar as well.

Why you do that?

Why? Because I'm a big LEGO fan. So the avatar that you have on LinkedIn is this one. But I have another outfit so I collecting minifigures. I have Legos everywhere. And why I picked Lego because when I started to do advocacy I thought that Legos was great tools or concept to explain stuff. It's building brains, bricks and because now we do a lot of building small microservices, small things like bricks. I loved Legos by nature, but I thought that yeah, it's great. It's perfect to teach stuff in it in fact.

So I interrupted you. Did you give your piece of advice to give to someone right away that they could implement to help them with their observability efforts?

I think for the first question, I mean observability is the main important step when you start going to DevOps or SREs. You need something that will observe properly environment. So first select the toolings that would make sense for your organizations, I would say and then depending on the tooling few angles that I described. So event logs or traces are supported or not. If they are not supported. Don't worry. Like I said there is tons of open source frameworks out there so maybe you can start looking at them trying some few exercises. There's plenty of GitHub repos where you have demo or tutorials where you can go step by step try to implement that technology in how house and then from the moment you have did that POC successfully, you can maybe increase and go further by increment adding more and more application to your platform.

Thanks again for your Performance Testing awesomeness. Links to everything of value we covered in this episode. Head on over to test guild.com P76 and while you're there, make sure to click on the Try them both today link under the Exclusive Sponsor section to learn all about SmartBear's two awesome performance test tool solutions, Load Ninja and Load UiPro. And if the show has helped you in any way, why out rate and review it in itunes? Reviews really do matter in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Performance and Site Reliability Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack Performance Testing Awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Performance and Site Reliability Podcast. Head on over to test guild.com for full show notes, amazing blog articles and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.