Observability Testing Using OpenTelemetry with Swapnil Kotwal

In a land of testers far and.

Widely journey seeking answers, seeking skills, seeking.

A better way o the test Illinovation Testing Podcast Guiding testers with our nation awesomeness. From ancient realms to modern days, they lead the way. Oh, the testicle of nature. Testing Podcast.

Have you ever encountered unexplained system failures or performance issues in production and asked yourself how do I diagnose and test problems quickly and accurately to avoid downtime? That's what this episode is all about. Today I want to share with you an Automation Guild session from our last online conference about the power of open telemetry. This session shows how OpenTelemetry unifies fragmented data silos into into a seamless observatory Stat future proving your infrastructure for unparalleled clarity and flexibility. This is an area I think more testers need to be involved in and aware of, so definitely check it out all the way to the end. And if you could do me a favor, just leave a comment or review and let me know what you think about these types of podcast episodes that are a straightforward interview. I'd like to know your thoughts and really appreciate it. Attention Test Automation teams. Are you tired of flaky tests, endless debugging and fragmented tools? Introducing browserstacks, test observability your all in one solution for smarter testing. It's cool because you can track all your tests in one place, detect flaky tests instantly and debug failures faster with their advanced AI powered insights and improve your automation stability with real time analytics and customer alerts. No more guesswork, just results. With over 25 years in test automation I've seen it all and that's why I trust BrowserStack to streamline your workflow and help boost your productivity. Ready to transform your testing? Support the show and visit TestGuild me BrowserStack and see the difference for yourself.

Hey Gil, I'm Swapnil Kotwa, Test lead engineer at SailPoint. For 12 years I have navigated ever evolving test landscape building robust test framework for startups and security critical infrastructure at CellPoint. From open source projects like Testng, APM Gatling to performance engineering at the helm, my journey has been one of the constant learning and innovation. Today I'm laser focused on architecting vendor agnotic observatory stack ensuring the performance insight fuel data driven decision making. The observability landscape is undergoing the systematic shift and OpenDelemeter is the driving force. Imagine the world where fragmented data silos disappears replaced by unified stream of metrics, logs and traces seamlessly revolving the inner working of Your system. And that's the power of open data. This is vendor neutral framework that shatters the chain of proprietary tools granting you the freedom to choose the best fit model. It's just not about the data collection, it's about future proofing. Your observability strategy. OpenTelemetry empowers both developers and operations team with simplified instrumentation and streamlined analysis fostering the collaboration and cluster clear insights. Don't get left behind with the observability revolution. Embrace the OpenTelemetry and unlock the future of understanding of your system unparalleled clarity and flexibility. So today's topic is Distributed system observability using OpenTelemetry. So in today's world, the SaaS world, basically systems are distributed. Hundreds of microservices are delivering your product. Okay, so and you need that clarity, you need that correlation between your traces logs and matrices. That's where the open telemetry came into the picture. I talk about myself a bit, so that's what I usually look like. And you can follow me on the different platforms as well. Let's talk about the open telemetry or in telemetry in general. So if you look at any spacecraft that is traveling to the space, there are some telemetry, right? It is sitting and looking at your spacecraft. And in similar fashion you need to have certain things deployed on your spacecraft so that you can monitor or observe the state of your spacecraft. In similar fashion, the cloud native applications needs the telemetry and based on that telemetry you would able to monitor that. Okay. And that's how it look like. So what is the observability? Is it any new thing? No, it's not a very new thing. We have been monitoring our systems using the many, I would say vendors like Datadog, Prometheus, Grafana and so many. Right. But there were never a unified way of instrumenting your source code and exporting those or emitting those traces logs or metrics. Okay. And that's the problem that bundle metric going to solve. It's just not solving the how you should instrument your source code, but how you emit that data, how you'd collect that data and send it to the your representation too. That is how the whole bunch of it is come under the OpenTelemet. So as were the continuous CDCI or continuous testing is not a very new thing. Okay. But this is a new, I would say space that is required that we need to constantly monitor our system. We need to see how our system is doing it. Okay. Before our customer can feel the heat. We should know the status of the system so that we can mitigate before anything goes wrong. That's where the continuous observability came into the picture. And OpenTelemeter will help in that case. Okay, so this is the big picture of it, okay. So when you make your system observable, right? Then you would able to monitor it. Okay. So you need to put some payload on your SaaS system so that it should emit the states of your service or your application so that you able to emit those traces. So this is the bigger umbrella of your monitoring. Okay. These are the basic pillars of observability. In terms of open telemetry, we call it as a signals. But these are not only three signals that it has. These three signals are the one which we usually see as a representation of our data. But there are two mores, which is a baggage and context. Okay. Which basically helps to propagate the context from one stack to another stack. Or to propagate the context the baggage is used. Okay, but let's not go to the technical part of it. So this is the foundation pillar that OpenTelemetry has. So you need to know the logs, okay? What are the problems that you can list it out Traces, the request scopes, traces. So which can show you the whole journey of your request that starting from the first service to the databases and coming back to the again that is being served. Right. And throughout this journey, if there is anything that is going wrong while that is happening, you should also need some matrices to see if there is some CPU utilization or heap utilization is growing aggressively or anything that is not something that we can control. Right. So metric would help in that case as well. So this is how they are the pillar when your system becomes the observability. I'll talk about the basic differences of it. And I also have one of the interesting findings over here. Right. So you observability means you need to make your system observable, okay. And then it becomes monitorable, right? So anything that is, we can monitor it. Okay. In observability you will see what is my system is doing in monitoring is my system is working or not. Most beautiful part of it is when there is some event or something happens in case of open telemetry, it emits the traces or logs or metrics rather than the other platforms where they constantly keep on polling on a certain intervals that whether I have something to get it at the representation tool. Okay. So in monitoring mostly traces are not there. The logs and events metrics over There but there are again some of the challenges like it won't scale that much that we wanted to scale. So when your system are growing, you need that proper observability. So that is how it differs, right? So that in case of monitoring is there is something wrong. That is you got to know observability. Use that data to tell what is wrong, how it happened, why it happened. Right? You can debug the things without going or touching the platform, right? You can have. You will have a data which will tell you the whole story that what exactly wrong with your system. Okay, so let me see how this open Telemetry came into the picture. The brief history is like Uber was doing his. It's in research for many years and it come up with a open source project called Open Tracing, right? It was mostly in specifications how the your client side platform should do the instrumentation for your source code. And on other side, Google and other companies were also working on open Census project. They were bit ahead than Open Tracing where they were trying to define the specification interfaces as well as the implementation as well. And data collection exporters also they come up with it. So around 2019 they came together, they joined the forces and they formed one common open source project called OpenTelemetry. OpenTelemetry is the second largest CNCF project at this moment after Kubernetes. And that is how the active community had. Okay, so let's see why we need the observability, right? So meantime, to reproduce the issue or to act on that issue, whatever testing burden that you have, you want to reduce that the whole control of it that you can achieve using the OpenElement. Okay, so these are the some topics, I just listed it out. You can go collect that data. You can have one system that unifies your all stacks, right? Say for example, some of your microservices are running on Azure, some of them are running on gcp, some of them are on aws. And maybe they are written in different programming language, right? If you are not having one common solution, if they are not compatible with the vendors and platforms and the programming languages, you won't able to streamline the fold system. You won't able to see that big picture. Okay, and that's where open Telemetry came into the picture that they almost all programming languages supported the standards and protocols are the same vendor neutral, platform agnostic and even programming language libraries are also available which is very systematic across the platforms. So there are one of the representation tool that OpenTelemetry has Jagger. Okay, so which Provides you how the traces look like, okay. And how you can look at it to solve it. I'll talk about it in a actual example. Say for example, this is how E commerce in your website look like, right? When you have one of the front end, that is a web browser which basically you are trying to purchase something, right? Which goes to checkout service, then the shipping service, then it goes to the payment and it adds to the cart, right? This is how it sends the confirmation email. And you need to see how this journey of the that particular request happened, right? Is there anyone? Oh, you need to find, right? So that is how if that service you want to see the traces came into the picture. If you look at this waterfall tree, it actually have some time, okay? And the data type, what kind of request it was, which kind of microservice it has and things like that, okay? And how much time was spent by that particular request on that particular method or that microservice as a whole. So this is the representation of it, okay? So this is whole thing we called as a trace. And the time taken by the trace or the request to complete, okay? And the bifurcation of it is underneath like whatever the microservice it was, it has called some functions and things like that. So you would have to take care of it doing the programming or instrumenting it. So that we'll see in similar cations. Say for example, you also want to see the metricsis a bit. Okay? So in case of you see there are some lag happening at checkout service. You need a matrices to see, okay? If there is CPU reaching to the 100%, okay. And then only you would got to know, okay, I need to look into it. I need to increase the CPU cores, maybe the heap size like this. In similar fashion, if there is any new code changes happen and there are lot many 500 requests being served from the front end. Those things will tell you the matrices. In case of logs, say for example, if there is anything happened, okay, what is exactly happened? The logs tell the story, okay? There are something wrong with that checkout service because of deep size is running out of memory and it is not able to serve the incoming request, right? So you would see those in a logs. But you need to have this all three pillars unified. You need to know their correlation. You don't have to do it manually. And that's the basically we are going to solve over here. It's not a microscope, it's the clarity under that microscope, right? What exactly happening over there? Okay, so that is the what? The data being emitted from the microservices, how data collection happens. Okay. There are standard agents and cloud integration that OpenTelemetry have it also have two type of instrumentation. You can do manual instrumentation instrumentation where you can actually do modification and addresses log metrics or you can actually use their libraries to do auto instrumentation as well. Okay, so there are ways you can use a low code or no code to do your instrumentation. And emitting those faces there is no cardinality limits. Okay. That's the beauty of it. It will reduce the burden of any company which is looking for this observability. This is what the few past where there were different libraries, implementation, data structure and format were there for each of these pillar tracing metrics and logs. But there was nothing one unified framework or platform was there. So which is being replaced by OpenTelemetry. And that's the beauty of it. And that is what we call cloud native telemetry. Okay, so I already mentioned this is the second most active project in CNCF today. So don't need to worry about it. It's going very well in development and things are going well. So what all things you would able to solve using it? Okay, so as you know the bifurcation of your request journey from the very upper layer to all the microservices and again back. Right? So you would able to identify where the time has the most of the time spent by this particular request. Right. And you would able to have the logs and traces, they are correlated with each other. You would able to see okay, what is the problem, how to solve it and how to make decision on it side. So that's the beauty that data will help you out at this moment since there were platform dependent, vendor vendor dependent solutions are available which makes it cumbersome to solve or integrate multiple platforms, programming language and services. Right. So those are the gaps being solved by opentelemetry. So as I mentioned, we would able to solve those problem how we would go for it. Right? You need to decide what are the goals, how you would able to solve it. Right? So for that you need to decide the goals and you need to work as a whole team to solve the problems. That is there open your project how that metrics and other project could serve. Okay, so let's talk about the architecture of it. As I mentioned, there are specification the API SDK and some of the helpful data is available. The collectors are the one which actually collects the data and send it to the your representational tools. There are client libraries which is vendor agnostic for your instrumentation and things like that. Right. So those all languages and platform are supported at this moment. You would able to seamlessly emit all these pillars, traces, metrics and logs to the using any platform. This is the basically your application. Okay. We basically delivered one of the agent. I would say a jar in case of you are using the Java. Right. And which basically emit the traces which collected by one open collector. Okay, Hotel collector. The hotel collector is kind of your sidecar or maybe the agent that is running along with your application. Which which basically sends that data to the upper layer. And if you have multiple such a kind of programming language are used, multiple microservices are here. And those microservices basically being used to deliver this E commerce website. Right? And this is the website which is having some of the telemetries or binoscopes selling, which I can go and add into the cart. I can look into the cart, review it and I can place the order. Right. In similar way. So this is the website which is being served by multiple e services microservices. Right. And so this is the Yaga where we can see what is the traces that is happening. Let me expand it for maybe three hours and see what the stresses. If I get any. Okay, I'll see if there is any traces. Okay, so this is how trace look like. Okay. The time taken the request originated at this 0 micro seconds to it actually took this much time. Right? So how much time it has taken, okay. For this request being served. That is the representation of it. It represent how much time it actually took and the bifurcation of it. So if I go over here, I will see the how much time it taken. Okay, it took 1.16 milliseconds. The next one took 100 microseconds. Likewise. So if at all you would able to see where the time has gone, you would able to debug it. Okay. There are ways basically show the architecture of it at this moment. It is not really cool, but if you go to the here. So it should show me something like this. The correlation of your microservice how it look alike, right? So that is what it it should show. It also show the physical memory how much time it has spent. Okay, so these are the some of the Grafana dashboard that we build using the open telemetry data that is being ingested to it. Okay, so these are the microservices that is being used. And what are the mean rate or the range, right. For that particular how many requests per second being served by particular service? So that we can know which service is extremely busy in our application. Which needs to be auto scale or needs to be further enhanced. That you got to know the clarity over here, the latency that is also represented. If at all there are any errors are happening. The error rates and a lot many matrices we can run. Here are some matrices which actually showcase what exactly wrong with your system when the peak was happened, how the data was handled in that case. Right? These are the matrices and the representation of it. So the first thing that we saw over here is the traces, right? The second thing we saw the matrices. So if you look at the graphana that grappana data being served by the Prometheus or here Jager or OpenSearch. Right? So here it is Prometheus that is basically showing the uptime of your system, how constantly it was running or is there any hiccups in it? Here it basically showing the. If I select the data type as a Jagger the bifurcation of your trace, how the journey of your request was. And here it basically showcase your logs with the whatever the problem statements you have. Right? So this is how you can use this all seamless integration which have the span id trace ID which using which you can unify. You can search between the logs, you can look into the traces and matrices and hit time and you would able to make sense. Okay. What exactly happened with your system. I'll quickly demo you how simple it is. Okay, so if you look at this example, for example log 4J. Here we are using OpenTelemetry and log 4J as a provider. Okay. And what we are doing is we just return one method which basically represent what happens with this particular request error message with the span or not. And that basically written at the bottom that if. If you want to showcase the span, it will show span as well with the error message or input debug level messages. Right? This is the where we instantiate our OpenTelemetry instance. If you look at there are three APIs like trace provider where you can build your own trace with some strategy and you can include it. There is logger provider which basically you can set up what kind of log you want to do, how you want to process them, whether in a sequence or in batch. And you can set it there. In similar fashion there is set metrics provider. Also you can set to the same instance of your OpenTelemetry SDK. And this platform will start emitting your all three pillars of your system so that you can see it in horizontal manner. Here is the one of the Interesting tool. I say for example, if you want to troubleshoot some issue, you can simply go and check it. Likewise, right? The drop down is shown here and in that fashion you would able to identify how much time it has taken. The 49 seconds were spent, but where the time has gone. Right. And how my request was served. You need to be very careful while instrumenting that how many traces are meaningful traces you are making. Say for example here I'm emitting a lot many traces, right? Which doesn't make any sense. Okay. So in that case you can see I have one. So if you look at this trace, right, it originate here and there are certain methods. It shows this method is started somewhere here, but it's not a part of sub method that is here or here. The error has originated here. That what it means that it the username and password wasn't correct. Right. But this is how you would able to pinpoint the problem. Okay. And that's the beauty of the traceability. That's how you would be able to solve the issue or troubleshoot the issue on production. So with that I'm concluding and hopefully this session will help you out to do the testing, troubleshooting and monitoring of your system health on production. So let's do it. These are the some of the references that I would say the one should go and start with. Thank you so much. If you have any queries, anything, any problem or any feedback, please reach out to me on my email address that given here on my social media handle. Thank you so much.

Thank you Swapno for your open telemetry awesomeness. For links of everything of value we covered in this episode, head on over to testguild.com a507 and make sure to check out this episode's awesome sponsor browserstack and how they can help you with testing observability. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe. My mission is to help you succeed in creating end to end full stack automation awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Automation Podcast. Head on over to testguild.com for full show notes, amazing blog articles and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.