Mastering OpenTelemetry and Observability with Steve Flanders

Get ready to discover some of the most actionable DevOps techniques and tooling, including performance and reliability for some of the world's smartest engineers. Hey, I'm Joe Colantonio, host of the DevOps Toolchain podcast, and my goal is to help you create DevOps toolchain awesomeness. Hey, it's Joe. And welcome to another episode of the Test Skill DevOps toolchain. Today we start with Steve Flanders, all about his upcoming new book. It may be new at the time you hear this, or it could be old. It all depends. So it's mastering opentelemetry and observability, enhancing application and infrastructure performance, and avoiding outages. A really important topic, if you don't know. Steve is a Senior Director of Engineering at Splunk, where he is responsible for the Splunk Observability cloud platform team and Splunk's OpenTelemetry contributions. He is part of the founding team of the Open Census and Open Telemetry projects and has over a decade of experience in the monitoring and observability space. So he really knows his stuff. You don't want to miss this episode. Check it out. Hey, Steve, welcome to the Guild.

Hi. Thanks for having me.

Awesome to have you. I guess. Before we get into it, I always like to ask authors of books, why. Why are you writing a book? It takes so much time. Obviously you're very busy, you're at a high level position at Splunk. So what's the deal?

Yeah, I've kind of always wanted to do it, right. It's been on my bucket list for a long time. And I feel like, at least for me personally, like, you have to really know a topic and you have to really be willing to put in the time to actually write the book. And I really felt like I had a fair amount of experience in both observability and with the OpenTelemetry project, and I felt like I had something to kind of give back. And personally, I learned a lot from others, kind of coming up in the industry, reading blogs from others, going to conference talks, reading books. And so I'm just trying to hopefully give something back that I hope others will also get value at.

Nice. So I guess the first thing is your audience. I know opentelemetry has been a big thing recently. I've had a lot of people on my guest talking about it. So, you know, what's your angle on it? Is this book for everyone? How does it flow? Is it a beginner to expert or expert only? How does that work?

Yeah. So I tried to take an approach where I hope that it's approachable to the widest audience possible. So it doesn't matter whether you have a lot of familiarity with either observability or open telemetry, or if you've been using it for years already and you already know it quite well. I'm hoping that there's something for everyone kind of in there. I approached it by trying to compare it to things that are maybe relatable to the real world. So, for example, like telemetry data metrics, traces and logs. I compare to like symptoms that you might have, like heart rate, blood pressure, height and weight that a doctor might look at, for example. So trying to relate it to concepts that you might better understand if you don't understand the concepts. But at the same time I'll drill down into like specifics of like, how do you configure the OpenTelemetry collector? What does the YAML file look like? How can you use this for advanced use cases like tail based sampling or filtering or aggregation rules? So there's really a mix of content in there, trying to cover the topic as broadly as possible.

Nice. How did you become one of the founding part of the founding team of OpenTelemetry?

Yeah, so I was part of a stealth startup called Omniscient that was acquired by Splunk. And at Omniscient we were building a distributed tracing backend, which is now the Splunk APM product. And at that time we really believed that there was kind of a shift going on in the industry where you were going to see basically open source instrumentation, data collection kind of components come to the forefront. Because we heard after like tons of customer interviews that there was a real pain point around proprietary instrumentation and data collection and the inability to really move between vendors or kind of get the value or support that was necessary and even the increasing costs from a vendor perspective. So very early on we actually reached out to Google and found out that Google and Microsoft were actually partnering to build something called Open Census. And initially they were focused just on the instrumentation aspects, basically what you put in your application to generate the traces, metrics and logs. And while that was interesting to us, at our startup we were actually more interested in the data collection aspects. So how can you deploy something like a host based agent and actually receive this data, either push or pull based, process that data and then export it to one or more different backends? And so we actually partnered with them. They had that as kind of their phase two initiative. We Said, hey, what if we help build the collector aspect of this? That was known as the Open Census Service. And then as people probably know, Open Census and open Tracing merge to form OpenTelemetry. And the OpenTelemetry collector is actually based on the Open Census service. So my initial kind of introduction to it was on the data collection aspects. Now I have a team that's kind of contributing to all aspects of otel, whether it be the specification, the instrumentation, or the data collection aspects.

Are you surprised by the adoption of this standard? Because it seems like a lot of companies are embracing it, which I'm not surprised by. Maybe I am a little, maybe I'm cynical, but are you surprised by the adoption of it?

I don't think that I am. And that's because it's an actual real pain point for both sides of the coin here. So end users have a pain point, whether that's a customer or an individual person, in that they only want to really instrument once, but they want to be able to continuously get value out of it. They can't afford to reinstrument to change like vendors that they're using or change like open source projects they're trying to leverage. They really need a single standard. So there's one pain point. But on the flip side, vendors, they actually kind of need the same thing. So prior to a standard like OpenTelemetry, most vendors had to actually hire a team of engineers that would build instrumentation data collection just for their proprietary backend. And so that's actually really expensive. It takes a lot of work because there's hundreds or thousands of different environments or integrations that you need to collect data from. And so standardizing this and having a single way to actually collect it and unifying those efforts mean that now any vendor, whether it's open source or commercial, can really take advantage of this data and then enrich it as they see fit. So given that like both sides kind of benefit, at the end of the day, I'm actually not that surprised that it's kind of taken off and I hope that it continues to kind of grow in adoption kind of long term here.

Nice. All right, so back to the book. Like I said, I sometimes go off the tracks here. The book ethics starts off with the fundamentals of observability. So are there any core principles of observability that you think are really important for people to know about to help with system reliability?

Yeah, I mean, observability is a pretty broad topic and it really means different things to different people. Right. It's like some people think of observability and they only think of application performance monitoring or apm. And while that is a part of observability, it may or may not be the only part, right? Like, APM uses traces. Just traces are very powerful because they have context and correlation. But in addition to traces, you have at least metrics and logs, the three pillars of observability, as they're often referred to. But even the three pillars of observability are not a complete picture of observability. What we're seeing is that you really care about, like, what is the user experience? So like, how is the performance from a browser perspective or a mobile device? How do you collect that data through like real user monitoring or synthetics data as an example? Profiling of the application also matters. So observability is just a broad topic overall. And I try to cover the foundations of like, what does OTEL think of observability? What does the cncf, which OTEL is a project within the cncf, think of observability? What does the concept actually mean? What is its origins? Where did it come from? What is kind of my definition of observability? So trying to add like a personal aspect to it. So yeah, I try to cover kind of broadly what the topic is and what you hope to achieve from it. And then I used it as a launching point for the other chapters in the book to really provide specifics on how hopefully you can achieve it in your environments.

Nice. So that's a good launching point. So you go from the opening chapter, then I think you launch into intro to OpenTelemetry. So folks that'll listen, I assume they know it. But just in case, what is OpenTelemetry and why? Why is it kind of like a. I hate the word game changer, but why is it really up the observability area, I guess.

Yeah, yeah. So an easy way to think about opentelemetry is it is an open source standard for telemetry data. And telemetry data means things like traces, metrics and logs, but it extends to profiling, real user monitoring, synthetics and the like, or at least it will hopefully in the future here. But in addition to providing basically an open standard, it actually has a reference implementation. So it provides language based instrumentation, things that you install within your application to generate what OTEL calls signals, these traces, metrics and logs that is language specific. So you're going to see all popular languages supported Java, Net, Go, Python, php, Ruby, you name it, right? Like there's a lot of broad coverage there. And then it offers a data collection component known as the OTEL collector. And this is used to receive, process and kind of export that telemetry data. And the collector is kind of powerful because it can actually do things to the data, it can enrich it, it can redact information, it can do additional processing like aggregation or filtering of it, depending on what your use cases are. But it kind of gives you control of that data so you can do with it what you want. Whether you want to send it to multiple different backends, whether you want to reduce the amount of data that's being like sent to a backend, all that's like fully within your control at the end of the day. But OTEL is a big project, right? Like covering 10 plus programming languages, offering collector components, having a specification for all of this and having a specification that's actually signals aware. Like it treats, has a full definition for traces, a full for metrics, a full for logs, and then it has kind of cross cutting components as well. Like the notion of resources which you can kind of think of as where does this application live in your environment, what host is it running in, what cloud provider? Or if you're running on premises, like what container or orchestration engine are you using? And it actually enriches all of this data, it provides context and correlation across all the different signals. That's really the power. It goes beyond just looking at an individual signal or an individual like product line, like apm. It looks broadly across observability as a.

All right, so this may be a really newbie question. So when I think of this, I think of production data. But are you able to utilize this while you're developing a software to kind of profile it as like shift left almost to get this data before it goes to live users?

Yeah, yeah, this is actually a great question, right? Like observability is common for production environments because you're trying to solve availability and performance problems in really complex environments. Today you have a lot of microservices and distributed systems. You have a lot of dependencies on third parties and calls to external APIs or databases or what have you. So production I think is kind of a no brainer. But then like if you think beyond that, hey, you have actual software development, lifecycle type things. You want to make sure that you're not like introducing regressions or performance issues. You want to understand the health and behavior of your system before it even makes it to production. And yes, open telemetry or Observability in general can help with this. Ideally, you should be looking at it through the entire lifecycle, not just when you get it to the production front door. And you can actually enrich this with additional information. So for example, there's this notion of events. You can think of them as like specific types of log records. So an event might be like I trigger a CICD pipeline. Well, I want to be able to denote that with the data that I'm collecting so I can figure out, for example, if something changed between build one and build two. And so that's an important like piece of metadata that you're going to want to tag on there. But yes, you should be doing this through every cycle, whether it's development qa, if you have like a staging environment or performance testing environment, all of these steps are great places to introduce the telemetry data and to start kind of observing your systems and understanding how they behave.

Nice. So obviously a hot topic in our industry, regardless of what people are doing, is AI. So now with open telemetry, I love it because you have all this data, but now it's kind of hard because you have all the data to, you have to analyze. So do you see AI being actually unlike other use cases, this is a good use case for it because it was kind of made for kind of going through all this data and kind of parsing it and bubbling up insights and things like that.

Yeah, and I touched on this on actually the last chapter. So kind of the future of observability. Because AI is still kind of an emerging thing and emerging technology and trends. I see a couple short term benefits and then many long term. In the short term, I'm sure if you've heard of generative AI where you can basically ask a system a question and it will come back with an answer. This is actually really powerful because query languages for backends, whether it's open source like Prometheus or some commercially proprietary backend that you want to get data out of, from an observability perspective, it's really hard to like learn those query languages and be able to construct the query in the right way. You either need to have a lot of expertise, read a lot of documentation, a lot of trial and error, but generative AI is making this way easier. Instead of me needing to understand like a PromQL query, I can just ask it a question like, hey, why is my checkout service slow? And it will go construct the queries for me, go generate that data, come back and give me an answer. Saying, hey, I noticed that you have higher latency here. You just deployed a new build, maybe it's related to that. So at least from a generative AI perspective there's a lot of benefits. But the future in my mind is like let's take the OTEL collector as an example. It will eventually have all the data in your environment if it becomes like the single aggregator. And then you can do some very powerful things from an AI. Like you can do like adaptive filtering or sampling. You could do like redaction of information because it identifies like pii, personally identifying information and actually cleans that up for you. You can do things like send some of this to a data lake, like maybe in your own premises, and then send like the rest of your data to a data warehouse or observability platform and you can dynamically change like where the data is going. So I think the real power of AI from an observability perspective will be what I call edge computing, which is something like an OTEL collector running in your environment between like the edge of your environment and where you're trying to send that data. I think there's a lot of opportunity there and you're actually starting to see a lot of vendors and startups kind of come up in this space. So I'm very interested to see where that goes in the next few years.

Sounds wild. So a little bit more about the OTEL collector then once again it could be off base. I know Google is in a lot of these ad platforms are rolling back what you can know about people coming to your website. Do the hotel collector have like parameters in place that you have to worry about that is this completely different. You're not necessarily collecting real user personal identification information.

Yeah. So if you think about application observability, so like backend type systems, that is typically traces, metrics and logs, it may have some PII in there, but it's less likely. Usually like a developer would have to explicitly add something in there that would contain it. One example might be you have a server that calls a backend database. Well, if you add the query parameters that you're issuing to that backend or backend database or you get the results of that query that could contain some pii. But like for backend systems, you're not going to get a lot of that from a telemetry perspective. But you mentioned things like real user monitoring, which is more like browser or mobile devices, that is much more likely to contain pii, because I'll have geolocation data, I may know like what cooking parameters you have or like what headers you're passing which could have your email address or name or other pieces of information that you entered into a form. So there is definitely some concern, at least from a real user monitoring perspective today, if you're generating real user monitoring, you're probably sending it directly to an observability backend. You are probably not sending it through the hotel collector. You could. Right. There are, I know of a few use cases where this actually happens today. But in general, because customers or end users are all over the world, you really want to send it centrally to the observability platform. And there they may have features where you can like toggle what PII is kept versus not what's redacted and what have you. If you have a air gapped environment or like an intranet, then you may actually use the hotel collector for something like from data. And there are options, right? They're called processors in the collector and you can process that data however you see fit, where you could actually redact, remove some of the information, you can hash that data. So if you want to have like a unique identifier that doesn't actually have the raw string that might be leaking sensitive information, you can do that. You can aggregate this information and maybe generate metrics off of the PII and only send the aggregated data. So you're not leaking pii. So there are definitely options depending on your use case. But I would say in the case of rom, more of these are going to be observability platform specific and not necessarily OTEL specific.

And there's a whole chapter on open telemetry collector. And I guess that brings me to opentelemetry instrumentation, which is a chapter. Curious to know, is there anything you see common challenges with people that try to implement open telemetry with the instrumentation piece?

Yeah, I mean instrumentation in general I think is actually very complex. If we take the tracing signal, that's probably the most complex signal today. Because in addition to generating there's called spans inside of a trace. In addition to generating a span, you have to pass what's called context. It's actually called context propagation. So the idea here is like if I talk to you and you talk to someone else, I need to know that you talk to someone else. We do that by passing an identifier. So I would tell you, hi, I'm ID1, and you would tell the next person you're talking to, hi, ID one, talk to me. And now I'm talking to you. And we can actually stitch that information together. In software, that's typically done through headers. So you will pass a header like a trace ID with a unique value, and that'll get passed through every single request or transaction in your environment. Well, doing that in instrumentation is actually very difficult. Now OTEL is trying to make this easier. So for example, it has built in context propagation for some of the libraries that are automatically instrumented. And it offers something called automatic instrumentation, which runtime, it actually injects itself into your code and allows you to do things like pass context easily. Now automatic instrumentation is nice in that I don't have to go manually update the code in my application in order to get the telemetry data, but it only works for things that the automatic instrumentation is aware of. So this is like known libraries or frameworks, typically open source or popular ones, and definitely not everything available today. So if you have your own custom libraries or frameworks, automatic instrumentation may or may not work for you. And then you need to learn how to actually add instrumentation properly within your environment to make sure things like context propagation work. So yeah, this is definitely challenging. But the good news is OTEL provides a lot of kind of choice along the ways. You can start with automatic instrumentation, you can add manual instrumentation later, and even using automatic and manual at the same time is fully allowed. So there's a lot of flexibility in the model of the OTEL framework.

Nice. So I think you also cover the power of context and correlation. My background, at least 20 years ago, I started off as a performance engineer. We run a performance test. We have all these monitors, we see a spike in response time. We're like, oh gosh, I don't know what relates to what, what's causing what all these different metrics coming on. So how's it changed nowadays? Like with context and correlation, is that solved where you know, oh, it's the SQL statement is running X amount of time out of this whole step that I ran.

Yeah. So again, context and correlation I think is a difficult challenge. Tracing or distributed tracing tries to make that easier by passing context. But metrics and logs, by default, they don't have that context. I actually refer to metrics and logs more like symptoms. In your environment, they tell you that something happens, but not necessarily. Why? Because they're individual records or like time series of things that occurred. But OTEL actually does allow passing of context with metrics and logs. That's one of the cool benefits that it provides. So across all signal types, you have some there. The other cool thing that it does Is it treats, they're called resources. I mentioned it earlier. It treats like the infrastructure or where you are type aspects as natively important to otel. So everything can contain resource information that's valuable because it helps with at least problem isolation, if not root cause. So by problem isolation I mean let's say I have a thousand Java servers all over the world that are, that run this checkout service and all of a sudden I'm having a problem with checkout services. I don't know whether all services are impacted if just a single region is maybe just one POD or one host is having problems. But with resource information, which is basically just metadata tagged onto these signals, traces, metrics and logs, I can start to answer those questions. I can be like, oh, hey, only this particular subset of my Java instances, maybe in one region is having problems. And I can further break that down and say, oh, in this one region I just deployed a new release and so I'm now seeing differences between my old release and my new release. So that's some of the context and correlation that's available there. A lot of this, if you use like automatic instrumentation is provided out of the box, which is kind of nice, but you can of course enrich it with your own conventions or your own metadata as well. So there's a lot of cool ways to kind of get context and correlation through otel. Some of it automatic, some of it manual, but it is trying to make it easier and trying to extend it beyond just a single signal and the, and the observability space.

Yeah, this is crazy. Back in the day to work for insurance company, we have to install an AGN machine, ship it to the insurance company, have to plug it in and then ping a site and then if it's down, like this is incredible what you could do nowadays. So that's awesome. So speaking of that, like sometimes people may be hearing this, they want to get their hands dirty and they may not have an application. They don't want to mess with their application. I think in the preshow you mentioned something about like a demo app. Is there like a demo app someone can follow along with these topics to really learn the concepts as they're reading.

Yeah, so. So OTEL provides one. It's called the Open Telemetry Demo, also known as the astronomy shop. OpenTelemetry's logo is a telescope. Right. So the demo application is like a shop where you can buy things for like telescopes or binoculars or things to that extent. It's actually, you can think of it like an E commerce application. So it has a bunch of services in the back end where you can like authenticate, view the front end you can check out, there's payment currency, there's a cart service, and all of them are unique microservices written in different languages. So it's a polyglot architecture and they've all been instrumented with open telemetry. They all send data to an open telemetry collector and that open telemetry collector actually forwards that data to open source observability platforms like Prometheus and Jaeger. So it's a basically batteries included version that you can just get up and get started with to actually experience the power of OpenTelemetry today. That demo environment only supports stable components in OpenTelemetry. So Open Symmetry is an evolving project and its GA or generally available equivalent is called stable and Open Symmetry. But things that are not stable is called in development used to be called experimental, now it's called in development in hotel. The in development ones are actually not part of the demo environment today. So they do wait for things to become stable before actually introducing it to the demo environment. But you can very easily deploy this either locally on your system. It supports Docker, it supports Kubernetes, and you can play with it. You can turn things on and off, you can kind of customize it as you see fits and they have a few extensions and things built into it too. It comes with a load generator tool so it'll actually start generating telemetry data for you. It comes with Grafana, it has built in dashboards. You can actually see some of the telemetry data and what you can actually do with it. So it really is kind of easy to inject even failures into it. It has feature flags so you can actually say hey, I want to cause 10% errors on the cart service. And you'll start seeing that in like Prometheus or Jaeger and be able to kind of troubleshoot your environment. So really, really cool overall.

Nice. So like any technology, I could see people pushing the boundaries and kind of abusing it. So I think you have a chapter on the anti patterns and pitfalls. Are any examples of maybe things people should avoid and maybe not do, even though it may be tempting to do?

Oh, there's actually lots of things. So with great power comes great responsibility, right? Like let's talk about instrumentation for a second. You could literally instrument everything in your application, every function, method, whatever it is throughout your entire code base. You could add tons of metadata for like everything single like Unique thing about it. Like what is the customer id, what's the user id, what's the transaction id, what host is it running on? Pod? All of that, that could be called over instrumentation. So eventually you generate so much telemetry data that it's just overwhelming. A, you have to process all of that data, so there's a cost associated with that. B, you have to like store it and be able to analyze it in like dashboards and alerts. That gets very expensive over a while. And C, like not all that data is valuable. Right. Like your end state is probably trying to be able to quickly solve availability and performance problems. You don't need to collect every piece of telemetry data in the entire world to do that. So one of the anti patterns is kind of talked about is over instrumenting. Of course the opposite is also true, which is under instrumenting. If you don't have enough of that telemetry data, then you can't actually achieve observability because you can't ask any question of that data. You don't have the data to actually ask those questions and live. Adding more instrumentation, like during an active incident, that's not easy. So something like automatic instrumentation would require a restart. Manually instrumenting typically requires a restart. You may not be able to reproduce the issue after you restart. So you have to really strike the balance between under instrumenting and over instrumenting. And opentelemetry can help here. Like automatic instrumentation is great, especially for known frameworks and libraries. It actually provides a good balance. And manual instrumentation you can kind of control and you can learn over time whether you need more or less. But that's a really classic example and I list dozens of different scenarios beyond instrumentation as well in that chapter.

Absolutely. So people definitely need to buy the book. We'll have a link for it in the comments down below once it's available. All right, so we're getting near the very end. So you talk about scalability and how to scale observability. I guess once you have these handy patterns, hopefully you're following them because I'm sure you'll find out if you're doing things wrong once you get to the scaling part. So any tips for scaling or is it. It's like any project, I assume is that constant refining that needs to go on throughout what you're learning from production and how you change throughout what you're learning from what you've done?

Oh yeah, it's definitely a constant process because your environment will change. Maybe you have spikes in traffic which generate additional Telemetry data, how do you handle that? And these are typical like distributed systems type problems. Right. If I have a stateless service, for example, the OTEL collector is stateless by default. So unless you configure something that's stateful, then you could do something like an auto scaler where you're like, hey, I'm going to, I'm going to measure the CPU or memory over my OTO cluster if it's running gateway mode and I will have it dynamically add more or reduce the number of instances. That is possible. But if you configure the OTEL collector in a state full way, for example, tail based sampling requires state management, then you have to be very careful because when you add or remove instances, you can actually cause disconnected traces because the ability to route the data to the right instance is at least temporarily broken while the cluster kind of resyncs to its new size. So yeah, you have to be careful here. You have to understand like what is available in terms of tools, what you need to be monitoring, and then what impact said changes have in your environment. Like anything. Like a lot of the recommendations I have is make sure you test this before you go to production because you're going to have a really bad time otherwise. Right. It's already too late at that point. But I talk about it from an application perspective, I talk about it from an OTEL perspective and I even talk about it from the observability platform's perspective because all of them have different dimensions of scale and things that you care about. And it's not just how do you scale up or scale out, it's also like the cost associated with that and whether you're getting value out of performing those operations. Sometimes the answer to scalability problems is don't send as much data, right? Going back to over instrumenting, if I reduce the amount of data, if I filter it earlier or if I aggregate it earlier, I don't have to scale the backend platform as much to support those use cases. And so those types of aspects are also discussed in the chapter.

Nice. All right, so we already touched a little bit on the future of observability, which how you end the book. Curious to know though. I get shown demos all the time. I've been seeing people, automation engineers and testers creating solutions that build like an SDK on top of OpenTelemetry and then they're able to get from production to create test because they know what users are doing. They're doing crazy things with APIs and things like that. So are you surprised by maybe how OpenTelemetry is being utilized? You see a future for that, people working on top of it to do other things that you may not have originally anticipated.

So one of the things I love about OpenTelemetry is just how extensible it is. So one of its guiding principles is to be open source and vendor agnostic. That second part is actually very important. Being vendor agnostic means it needs to be able to support a variety of different open standards and it needs to be extensible to the future as new things are being added. Be that testing, be that profiling, be that things in CICD pipelines, whatever the use case is from an observability perspective, it needs to be able to grow and evolve into it. And that was thought about from day one in the OTELLA specification. So that's really, really powerful. And I love seeing different use cases for otel. There's actually a few cool projects out right now that are doing LLM, which is going back to AI for a second. LLM observability, I love that. And it uses OTEL behind the scenes to actually like collect this data. OTEL actually has semantic conventions, which is just a fancy way to say standardized names for metadata. They have it for LLM just like they have it for HTTP or databases. Like these additional use cases are just so cool to me and I love just that, the community in general, it's very friendly and inviting and so you'll see people come in and come out depending on what their use cases are. And we can kind of all collaborate together. So I'm not surprised to see this long story short and I hope to see more of it going forward because that really is the power of open Source in my mind and it shows a really healthy community overall.

Absolutely. It must be a cool feeling to see how your work has grown and people are utilizing it for sure. All right, Steve, so you know, I got an early kind of outline of the book, but I know you've been working on it. Could you give me like an official date? I know it's kind of when you're writing a book, you never know, but when do you plan on launching the book? How could people get their hands on it? I know you probably travel around the world at conferences. Can people actually meet you? Get the book signed once it's live, all the stuff about the book, once it goes live, when is it going live and how can people get their hands on it?

Yeah, so it's actually available for pre order right now. It's not released yet, but you can pre order it on your typical sites like Barnes and Noble or Amazon or something to that extent. I'm in the home stretch of kind of incorporating feedback and going through the last versions of it, so that's very exciting. Looking forward to being being done with that process. And then right now the book is scheduled to be released sometime in early October to early November. The idea is that it will be published before kubecon North America, which is in Salt Lake City this year. I actually just found out that I had a session accepted there, so I will definitely be there and I will have copies of the book, so I would love to kind of meet people that are also going to be at the conference. I believe I'll be doing probably a book signing as well, so that's probably the best way to get your hands on a potential free copy of it. And then of course it'll be available on all your typical sites, be it physical copy or like online virtual copies, probably starting in the October to November time frame.

Okay Steve, before we go, is there one piece of actual advice you can give to someone to help them with their DevOps OpenTelemetry efforts and what's the best way to find or contact you?

Yeah, so my general guidance is if you're not looking at OpenTelemetry today, you need to be right. Like it doesn't matter what observability products you're using, where you are in your observability journey, I can almost guarantee you will get value out of OpenTelemetry, even if it's just that it provides data portability and the ability to be vendor agnostic. So I really encourage you to kind of take a look at that. In addition, the observability space is rapidly evolving. OpenTelemetry is part of the CNCF. The CNCF has a lot of great resources on observability, including different platforms that are available, trends that they're seeing, surveys that they've done from an end user perspective. So definitely took a look at anything observability related in the cncf and then regards in terms of like contacting me. I'm available on a variety of different sources, so I have my blog@s.flanders.net LinkedIn is always a great place to kind of find me. My blog has information about the book as well. And then you'll find me in the OTEL community too. There's a CNCF Slack instance for otel. I'm active there or just in the in the Otel GitHub repositories themselves and.

For links of everything of value we covered in this DevOps Toolchain show, head on over to test guild.com p161 so that's it for this episode of the DevOps Toolchain Show. I'm Joe. My mission is to help you succeed creating end to end full stack DevOps toolchain awesomeness. As always, test everything and keep the good. Cheers. Hey, thank you for tuning in. It's incredible to connect with close to 400,000 followers across all our platforms, each and over 40,000 email subscribers who are at the forefront of automation, testing and DevOps. If you haven't yet, join our vibrant community@testguild.com where you become part of our elite circle driving innovation in software testing and automation. And if you're a tool provider or have a service looking to empower our guild with solutions that elevate skills and tackle real world challenges, we're excited to collaborate. Visit testguild.info to explore how we can create transformative experiences together. Let's push the boundaries of what we can achieve.

With lutes and lyres, the bards began their song A tune of knowledge, a melody of code through the air it spread like wildfire through the land Guiding tester showing us secrets to behold.