Demystifying Open-Telemetry with Michael Haberman

Get ready to discover some of the most actionable DevOps techniques and tooling, including performance and reliability for some of the world's smartest engineers. Hey, I'm Joe Colantonio, host of the DevOps Toolchain podcast, and my goal is to help you create DevOps toolchain awesomeness. Hey, want to know more about open telemetry? Well, you're in for a treat because we'll be talking all about it today with Michael Haberman. If you don't know, Michael is a senior architect at SmartBeer. He previously co founded and was the CTO of Aspecto, which is a distributed tracing platform acquired by SmartBear. I forgot how long ago. We'll find out though. And he has over a decade of experience as a software architect. He's helped successful tech companies transition from monolithic applications to microservices. So he knows his stuff. His expertise in managing microservices complexities and has led him to develop solutions that simplify adoptions and management. All things I think you know about, especially in this day and age in DevOps. Before we get into it, make sure Also to download SmartBear's free what is OpenTelemetry? A straightforward guide using the link down below. Hey Michael, welcome to the Guild.

Hey Joe. Thank you. Thank you for having me.

Awesome. Before we get into it, is there anything I missed in your bio that you want the Guild to know more about?

No, I think that's covered.

Sweet. I think before we get into it, just in case, if anyone's listening, they don't even know what opentelemetry is. How would you explain what is opentelemetry?

So opentelemetry is an open source project that defines how to collect telemetry data. At the core of it, it's a specification how the data should be collected, and on top of that you have the implementation on how to actually collect it in different programming languages. So that's the core of opentelemetry.

Nice. I think once heard this before. It's a kind of works alongside the three pillars of observability, traces, metrics, and logs. So can you explain how OpenTelemetry supports each of those pillars? Maybe.

Yeah, of course. So As I said, OpenTelemetry is all about collecting the data. The telemetry data and OpenTelemetry initially set out to have three main telemetry types that would be traces, distributed traces, logs and metrics. Arguably those are the ones that you must have in order to monitor and observe a cloud native application. So logs, they've Been out there for the longest, I guess everybody knows. What's that? Basically, you write kind of the story within your code that then you can read and you understand what the application did. And then opentelemetry doesn't generate a lot of logs. That's usually up to the developer to write those. And OpenTelemetry would send it to destination of your choice. Metrics. It's trying to get a bit more interesting because OpenTelemetry can generate its own metrics. The OpenTelemetry is actually looking at the libraries, the frameworks that you use. So if you're using some REST framework and then it attaches to that framework and it would automatically create metrics such as how much HTTP calls did you get, their latency breakdown per HTTP status code, those kind of things, which you may have to do manually. With OpenTelity, you get that out of the box and you can also have your own custom metrics. Looking at traces, distributed traces. This is where OpenTelemetry started from. This is where most mature and traces in general are created automatically. Again, by attaching to the libraries that you use, it's able to create the traces.

Nice. Now, I know a lot of companies, a lot of people have really started to embrace opentelemetry. I think it's gained a lot of traction since inception. So do you know any recent updates or milestones for the Open Telemetry project that developers in general should be aware of?

Yeah, I think there are a few interesting things that are happening. First one is OpenTelemetry is expanding from the three core pillars. So there is a new telemetry type that is approved and starting to be developed, and that's called profiling. So you'll be able to profile code in production and opentelemetry will go and add more and more telemetry types as the project matures. I think that's a very interesting thing that's happening. Logs reached GA a few weeks ago or a month ago, which was very important because people made logs and it wasn't GA yet. So I think those are the two interesting things that happened.

So what's the big deal about profiling, do you think?

What's the big deal? I'm not sure if it's a big deal or people had the OpenTelemetry really pushed the notion of the three pillars of observability. And what people concluded from that is that OpenTelemetry is going to be only about GO3. And now once we introduce the fourth telemetry type, the fourth pillar, if you want I think people are starting to understand that OpenTelemetry is looking to answer all the observability questions.

Nice. So I think with standards like OpenTelemetry, sometimes you crush your fingers and hope that it starts to get adopted as a standard. Any idea how that's going? How any. Like how many vendors do you think are embracing? Any thoughts on that?

So I think it's. It kind of became industry standard. So you see all the monitoring vendors embrace the open telemetry. I think all of them can ingest open telemetry data. So if you have some monitoring solution, most probably you can send the telemetry data to it. Another very important sign is that the cloud vendors support it. So you can send your telemetry data to your cloud vendor and it would know how to ingest that. And one very interesting thing that we start to see in the adoption is that code frameworks and code libraries are starting to produce telemetry data by themselves. So if opentelemetry had to go ahead and patch each library that you use now, it's enough standard for the library themselves to implement OpenTelemetry within the library, which is a very important step.

Nice. Now, I mentioned in the very beginning how Specto, your company was acquired or moved over to SmartBear. Can you talk a little bit about Xpecto and why do you think SmartBear made that move? Does it just show the importance of forward telemetry nowadays and what you all were trying to do at specto?

So in aspecto, we were very early adopters of opentelemetry in very its early days. And we made a lot of contribution to the project. What we wanted to do is to help people get started with OpenTelemetry and also address some of the pain points that OpenTelemetry has. And what marble saw in Aspecto was kind of that they wanted to have their own offering of OpenTelemetry and allowing Bugsnug not only to be an AI monitoring solution, but rather a performance solution. And they chose opentelemetry to do that.

Nice. So speaking of pain points, then I know how hard is it then to start implementing OpenTelemetry for your application? I think there's multiple ways, but any gutches or things people need to watch out for or be aware of when they're trying to use OpenTelemetry?

Yeah, I think there are two things. So the first thing you have to do is to implement the OpenTelemetry SDK in the programming language that you use. For the most part, this is a pretty Straightforward things to do. Usually you just import the library. It's running the initial function and should be working. One thing to remember, usually you don't install it in one application. Usually your application is distributed. You will need to install it in multiple places the first time that you install. It's easy, but then if you have 50 services and you need to do one little change, you need to do it 50 times. So think on, you know, depending your configuration, how you avoiding that? I don't know if you have a shared library or something of that sort. So this is one thing to keep in mind. And a very interesting kind of side project of OpenTelemetry is the OpenTelemetry Operator, which is a kubernetes operator that you can run in your kubernetes environment and it would actually inject opentelemetry to your containers without the need to install it manually in each application. Which is a very easy way to implement OpenTelemetry. It doesn't support all the languages, but most of them.

Nice. So is there anything people need to be aware of when they're looking at the specification? Do they really need to adhere to it as close as possible? And if not, will it create compatibility or different issues with different visualization tools or anything like that?

So I think that the end user of OpenTelemetry shouldn't be too concerned with the specification. Unless you are contributing code to opentelemetry, then it's very important. And if you're a vendor, it's really important to follow the specification. So yeah, if you're just installing OpenTelemetry, you probably don't need to be aware of the specification. And I think most vendors are aligning on the specification quite well.

Nice. So for companies that have a lot of applications that exist already and they want to implement OpenTelemetry, is it all big bang or nothing? Is there an incremental way you can adopt OpenTelemetry? Is there like baby steps maybe or certain levels you can get at without just throwing it all out there?

Yeah, so I wouldn't throw it like the Big Bang deployment and hope for the best. I would pick some services that they feel comfortable on experimenting with better if they're communicating with one another. So we will be able to see distributed trace and then experiment with those, gain confidence, understand the different configuration that you may be required to do and then move forward to everything.

So I guess, are there any services or components that should be prioritized to instrument first? Once they see it, they're like, oh wow, okay, I get it let's move forward.

Yeah. So you have to start with the OpenTelemetry SDK. That's the component that's running within your code and then that component need to export the data somewhere. This somewhere could be some vendor or open source Tools or an OpenTelemetry collector. An OpenTelemetry collector, it's the backend component that can ingest, process and distribute telemetry data whether you are going to run or not run the collector. It's an important decision. It's decision that you can change, but it's important decision to think through. As a rule of thumb, I would say if you're a small company, you probably don't need that. Medium size may probably need that. And if you're a large company, you definitely need that.

Gotcha. So I speak with a lot of people I know, a lot of them are moving towards a microservice architecture. How does OpenTelemetry help or enable observability in those kind of really complex distributed systems? I guess it's a must have, would you say or like how does it help developers?

So OpenTelemetry again is the component that collects the data. So then we need to ask which data contributes to adopting microservices and for the most part it's going to be distributed tracing. What distributed tracing allows you to do is to see how an interaction that started with one service then propagated throughout different services. You can see the entire interaction in one view. So if service A called service B and then C, if you're looking at a trace, you will be able to see all of the interaction that occurred due to that usually API call. And this is something that you can't do with, with metrics and with logs. You can't do out of the box. People may try to have like some correlation ID between different logs in order to understand the whole story, but you get that out of the box with traces. And logs usually doesn't help with performance issue. And with traces this is inherent in the tracing data itself. So usually what the company that would implement microservices they would usually would already have some monitoring solution. And if it doesn't have distributed tracing and they'll have enough services, they would start to ask okay, which service is causing the problem? I know where I'm getting the effect, but what is the cause? And then they will start to look for distributed tracing. And in OpenTelemetry you get that out of the box.

Nice. So what's the future you see with OpenTelemetry? I know AI has been a big deal. Once someone has OpenTelemetry is it a leap to think that eventually AI will know or I had an issue in production, I know exactly what the issue is because OpenTelemetry let me go ahead and fix it, or is that not even in the near future is that a even normal scenario you could foresee?

So opentelemetry has very strict limits on what it's going to do. It's going to collect the data and it's going to be responsible on shipping the data to wherever you decide. It's not going to visualize the data, analyze the data, nothing of that sort. So OpenDelemity is saying I'm not going to interfere with what the vendor are doing, whether it's a vendor, an open source tool, send the data wherever you want. Once the data is sent, it's out of our hands. So where do I see opentelemetry going in that aspect? I think mostly we're going to see more telemetry types supported such as evolves. We're going to see more support for client side observability which is kind of lacking these days. I think those are the areas. Yeah.

Great. So I said you were one of the leaders at Aspecto, now you're at SmartBear. Have you seen, you must have. I know you've seen a lot of different organizations. Any success stories, you've seen how they implemented OpenTelemetry or seen how ways it's significantly improved, you know, their observability or application performance?

Yeah, yeah, 100%. So usually the success stories are on performance. You get this, we had a customer that usually the transaction, a specific transaction would take less than a second and every now and then it would take like 40 seconds and usually for the larger customers and the customer would complain and they added logs, they tried over and over and over to find what happened. And then using aspecto they were able to capture the right trace, look at the trace and then figure out what happened. There is a common pattern that happening with microservices called N&1 Query. You Query 1 thing, it returns a list. Then you do something for each item in the list. You expect the list to be short, but then the list is in hundred or thousands and then you get performance issues, which is exactly what they had.

Nice. So with the spectre on your roadmap, are you just waiting for OpenTelemetry to incorporate things or like what is. Is there anything on your roadmap that you're building on top of opentelemetry that you see making it even better. Yeah.

So what we focus mostly was around sampling. So when talking about trace data, so you throw this SDK in your code and it's just starting to send out traces. You don't control how much traces it's going to produce. And unlike logs, where you both have the control and also you have the verbosity of the logs, whether it's an info or warning or error. So you get a stream of data and you need to pay for that. And that could be too expensive to some customers, to some companies. So what we did is we implemented a mechanism where you can define the rules of what data interests you the most. So for instance, you can say if this trace had an evo, I want to sample 100% of the data. If this trace has high latency, I want 100% of the data. These traces belong to this customer, which is very important for us. We want 100% of the data, but for all of the rest, we want a baseline of 20%. So by that you can collect the data that's most interesting for you and then sample data for all the rest, and then you get the most and you don't need to pay for everything.

Awesome. So with open telemetry, who normally benefits from it? I know everyone benefits it from, but the developers just help them debug quicker in production. Like, why should they all be embracing telemetry and especially opentelemetry?

So I would say that open telemetry is a new approach to something that exist for a long time. Eventually we're talking about the monitoring space, everybody monitor the application. We start to call it observability because things change a bit. But it's an evolution of monitoring. So cloud, native and microservices created a different need to observe your application. But eventually it's kind of the same thing. What OpenTelemetry created, it created an open source, an open standard to collect the data, and with that allowing all of the monitoring vendor on the same thing, if you want to move today from your current vendor to a new vendor, you need very good incentive to do that, because it's very complex. Right. The SDK, the component that you have deployed, are specific to one vendor. Moving to another vendor could be very, very complicated. Talking about OpenTelemetry, you have OpenTelemetry installed now choose where to send the data. If you want to move from vendor A to vendor B. It's the same data, you just need to point to a different address. And now you are using a new vendor. So I think the motivation to use open telemetry at the core of it is in an open standard, open source version of collecting monitoring data.

Nice. This may be a dumb question. Are there any cons or downsides? Downsides to opentelemetry or things that could be, I guess, not optimal? We talked a lot of pros, but any cons? I guess.

Yeah, yeah, 100%. Everything comes with price. So open Telemetry is kind of a new project, right? It's on the road I think for four years now, something like that. If you look on the mature vendors in the monitoring space, there are, I don't know, 15 years. So opentelemetry does have maturity issues in certain places. As I said, I think it was like two months ago. Logs reach GA only now, right? Logs is very important piece of your application. So I think the maturity both in the feature set that you get from OpenTelemetry but also the documentation, in some instances it's lacking, making it harder to get started with.

Awesome. So any resources or tutorials or learning materials you recommend to help maybe a developer get started with OpenTelemetry?

Yeah. So first of all, OpenTelemetry has its own documentation which is significantly better every few weeks. It's mostly for the get starting things. It's very, very good. There are online free courses or lectures that you can find. We at aspecto created the OpenTelemetry Bootcamp. It's a YouTube series of getting started from the simplest to more advanced stuff. Six hours in total, which is a very good place to start with.

Nice. Okay, Michael, before we go, is there one piece of actual advice you can give to someone to help them with their open telemetry DevOps efforts? What's the best way to find or contact you?

Best advice would be, I think, to experiment with it. I think it's a very good piece of software that you get a lot from and I think developers are usually hands on. Play with it, gain confidence and you have a very good monitoring solution at your hand. Best way to find me is in my email address. Michael abermanmallbear.com yeah.

And for links of everything of value we covered in this DevOps toolchain show, head on over to test guild.com P14E. And while you're there, make sure to click on the SmartBear link and learn all about SmartBear's awesome solutions to give you the visibility you need to deliver great software. That's smartbear.com so that's it for this episode of the DevOps Toolchain Show. I'm Joe. My mission is to help you succeed creating end to end full stack DevOps toolchain awesomeness. As always, test everything and keep the good. Cheers. Hey, thanks again for listening. If you're not already part of our awesome community of 27,000 of the smartest testers, DevOps and automation professionals in the the world, we'd love to have you join the fam@testguild.com and if you're in the DevOps automation software testing space, or you're a test tool provider and want to offer real world value that can improve the skills or solve a problem for the guild community, I'd love to hear from you. Head on over to test guild.info and let's make it happen.