Testing in Production with Eran Kinsbruner

Foreign.

Get ready to discover the most actionable end to end automation advice for some of the smartest testers on the planet. Hey, I'm Joe Calantonio, host of the Test Guild Automation Podcast and my goal is to help you succeed with creating automation awesomeness.

Hey, it's Joe and welcome to another episode of the Test Guild Automation Podcast. If you've been in the testing automation space for a bit, you might be wondering what the heck happened to Iran Kinsbruner. Well, in this episode we catch up with Iran to talk about his new gig and some new cool software you can use to help you test your applications in production. I originally had him on my performance and site reliability podcast, but I felt like you should hear it here as well. For some reason I don't have as many listeners on the performance podcast, so you probably missed this. So if you want to become a better tester by being able to understand the behavior of your live applications, you don't want to miss this episode. Discover why the traditional approach of adding more logs and redeploying the app is costly, time consuming and can impact business and SLAs and your software development and testing efforts. Listen up. Check it out. The Test Guild Automation Podcast is sponsored by the fantastic folks at Source Labs, the cloud based test platform that helps ensure you can develop with confidence at every step from code to deployment. For Every framework, browser, OS, mobile device and API. Get a free trial. Visit testguild.comSourceLabs and click on the exclusive Sponsor section to try it for free today. Check it out.

Hey Ron, welcome back to the Guild.

Hey Joe, happy to be here with you again and thank you for the nice intro.

Absolutely great to have you. So people listening, like I said, might be like what? Because you've worked for another company for a long amount of time, you've been in the automation testing industry for years and you've written a lot of books in this space as well. So just curious to know what happened. Did you have a new position? What's this about? Just to get people back up to speed where you are right now.

Definitely. So yeah, I've been over a decade with Perfecto and blazemeter Underperforms and did a lot about software quality and you know software quality is a challenge and we were always talking about shift left. So I like a lot about the shift left, the things that developers are more accountable and owning everything about software quality, software performance, software observability. Because when you look at DevOps, you know, even though everyone is talking about DevOps, dev and ops are still in different silos, so. So developers would deliver their code, would it be in a serverless deployment, kubernetes, you know, containerized application deployments and they would forget about it. Maybe not forget about it, but they would definitely be challenged to get access and insights into their code after it was kind of pushed into the production environment. And when you are getting tickets flying issues coming from production from your customers. So yeah, it's involving quality, but how you troubleshoot and meet all these SLAs and MTTRs that you are trying to reduce, how do you prevent these things from recurring in the future? So these kind of things were really exciting to me. I didn't really touch them too much in my mind while I was doing pure software quality software test automation. And when I heard and was approached by Lightron, I was amazed because you know, they are solving something. You know, whenever I'm talking with many gurus in the industry, they say, okay, if we're only able to do this live troubleshooting in production a decade ago, few years ago, you know, we would have been saving not just money but so much time and pain for our customers. So I think that, you know, this was kind of a nice transition for me moving from software quality to ensuring quality, but from the hands of the developers, from the native environments which are the IDEs. So yeah, I was really excited to join Lightron and I hope, you know, to see how they really penetrate into this entire CI CD, entire DevOps cycle and kind of change the rule of games. So troubleshooting isn't so painful as it used to be.

Awesome. So you did mention shift left. So is observability in this type of technology shift right or is it shift right? You get that information which helps you then shift left earlier. Does that make sense?

That's a great question. Because when you say shift left or shift right, you know, these are two things which are sometimes separated. Right, shift right. Yeah, people think about APM tools, logs, IO Grafana, all these dashboards which are very useful, very strong, but they give you insights from production. And who is the owner of production insights? The ops guys, the SREs and these kind of things, these kind of Personas. But when they get these logs and dashboards and stuff, okay, they don't always have the tools, you know, they're not the developers who wrote this service that is running in this remote pod, in this remote cluster, etc. So there is kind of a gap between this shift right and shift left because you know, these guys are able, the Guys from the shift right world are able to get up until a certain level of observability. But when you're trying to troubleshoot in a really fast MTTR cycle, you're not always getting the right tools and process in place to do this smooth transition. Even more, when you are dealing with remote clusters and pods and this kind of production running environments, you as a developer, you are unable to even access these remote environments. And these remote environments are very limited by tools. Sometimes they don't even have a shell. Okay. So to even open, you know, command prompts and run, you know, this kubectl and other tools that you want as part of your troubleshooting activities. So it's a nice way, nice requirement to have this shift right and shift left. Connecting, getting insights from production, shifting them left, Shift observability left to the developers to the ide. But in reality it's very hard. Limitations of tools security. You need to stop the running application in production because you want to put telemetry and breakpoints. You want to learn more because sometimes the logs that you have aren't good enough. So it is already a content for a different book if you like. But that's. It's a very challenging deployment in which we are living right now with the microservices and the cloud native applications. So that's where things need to change.

Absolutely. So you talked about Personas, so a lot of the copy I see on Lightrun's website and some things you ran out seem to be developer centric. So does a tester fit in this at all? Would a tester find any value from this type of information or are they more of just being aware to know? Hey developer, you have to check this out. Like where does a tester fit in this type of where you find yourself this type of role?

Now that's another great question, Joe. So you know troubleshooting and resolving issues is done by developers, right? But when developer is able to pinpoint the issue, get the full cycle observability into his IDE and be able to fix the issues as fast as possible. It does need to trigger a CI CD loop where it's also running some test cases. Right? In this automation cycle, what lightrun can actually do is actually also able to troubleshoot CI CD pipeline failures in test cases. Understanding the false positive and negatives that you have within your CI pipeline, you because at the end of the day you can add conditions, you can add telemetry into your test code as well and understand if the build upon the troubleshooting cycle is good to go or not. So definitely testers maybe later in the cycle can benefit from the CI CD workflow. And by the way Joe, we know together that within a DevOps family yes there are Persona titles but we are dealing with a squad or with a feature team we are which owns both development and quality. So yes the QA or the automation guys will be writing the test cases but they will tag along with the developers and will need to be in sync as developers are putting the fixes once they finish all this troubleshooting cycle which hopefully is faster than with light on Nice.

So kind of want to build some context of maybe how people can start visualizing this. So I know that for me what I saw you recently posted something on troubleshooting Java applications using dynamic instrumentation. I assume it's one of the first things you've published in this new role and I think you set the stage by breaking down why there's a new need here like modern application you touched on microservices, kubernetes, all these things have created new challenges. So where do we stand with standing like testing then from where people may be thinking how they used to do troubleshooting, maybe even you know, troubleshooting Java applications compared to now. But maybe some things that need to change or some mindsets, I mean need to be shift to address the modern needs. We need this type of tech.

So it's I would say the world is evolving into much more because it wasn't that complex prior. Now it's even more so. You mentioned earlier serverless side lambda actions, these kind of environments even to set them up for troubleshooting and testing and everything, it's quite challenging. Okay, think about these serverless applications are event driven programming solutions which means when a service is calling something an instance is being set up for him and sometimes these actions takes a matter of microseconds or 5 seconds up until few minutes. So in such a very quick back and forth service communication between Lambda Actions and other kind of Google actions as well Azure, all these cloud native applications which are either cloud native standard applications or serverless, you know that both the developers and the testers and the SLAs and the DevOps engineers are now challenged in gaining visibility into this environment because setting it up locally is I would say it's complex than ever before. Okay. And this is what I'm hearing from customers of Lightrun. You know they used to do what they call remote debugging up until now and that's a traditional workflow that you're doing from your IDE intellij pycharm VS code which is fine, but to do so you a need some kind of a remote debugging. To where? To an environment. But this environment is very far away. You have hundreds of pods that are running these replicas of your service. How do you find the right pod? And you need port forwarding and you need to. Once you're putting a breakpoint from the IDE to this remote environment, you, you need to stop the running application so it changes the state. So it might not be the production app, it will be a pre production but you still are stopping the running application. And by the way, when you're doing so you're trying to add more telemetry to do more add more telemetry. It means you need to add more code to add more code. It's a full redeployment. Okay, so this new way of DevOps with cloud native application software development lifecycle is becoming way more complex than before, which requires a mix of different processes and tools to allow you to expedite, as we say, the MTTR and also the MTBF when you talk about SREs and everything. So they say, okay, we got some fire drill from production MTTR, right? We have SLAs, we need to fix it fast. But how do you prevent it from coming back again? And that's the meantime between failures. How do you extend the time so this thing doesn't come back? Going back to your question about testers, right? So once you get this full observability, the full visibility into issues into the root cause by not stopping applications that are still running, getting exactly the user flow visibility into your ide, you learn a lot about what's happening in your real world and you can feed this information back to your testing teams and this reflects on the entire continuous improvement of everything around your software delivery life cycle.

Nice. So maybe I'm off base here, but back in the day you used to have a server in house. You knew what was going on, you knew the server database and probably that was like as complicated as it got. Now you mentioned cloud native, you have these services, third party things being consumed. Is that what makes this also difficult is you may not even know where to even debug or put logs because it's dealing with things that are. It's so dynamic that you don't even know nowadays almost.

That's exactly the thing. And when you're dealing with serverless, you definitely don't know. So yes, developers in this kind of new world are detached from their applications after they deployed or did a pull request, they put the code in. They don't know where and how it's going to run in production. And if you go go to my LinkedIn profile, you would see connecting developers to their live applications. Okay, we are trying to bridge this gap. So they do have this easier bridge to their applications to troubleshoot issues without the dependencies of a long full recycle or redeployment that involves the DevOps engineers and the QA and everyone else. So developers are getting the full power that they need to prevent or to troubleshoot and mitigate issues. And issues can be performance issues, it can be functionality, it can be security as well as. Because Joe, I think that you are familiar with the log 4J crisis that came and quite recently there was another Python issue or incident and these issues, everyone was kind of impacted. But how do you know that in your code workflow when you're running it in the runtime environment actually passes and uses this security vulnerability only by examining and getting observability and you might understand, yes, I'm using log 4J for my logging and everything, but my code in runtime never even goes to this code flow. So I would never be impacted by this security. So this security prioritization because today with Sast and Dust and all these security things, you're getting a lot of false positives, you're getting tons of noise and developers how they prioritize what to fix and what is really impacting our end or their end users. So this is another pain that we are seeing today involving observability.

Absolutely. So I think you mentioned so far the old way of doing things. There's main reasons that causes like developer productivity is one thing that's costly cost of logging obviously. And you mentioned mttr. So what's the alternative approach then? What is dynamic instrumentation and logging?

So dynamic instrumentation dynamic logging actually allows the developers to log only when and where they need. So instead of just testing, throwing logs at almost any line for a just in case situation, you know, maybe something will fail. So I have a log for that and this piles up into a very huge cloud bill. And today the FinOps foundation, that's a movement that is growing, it's an open source, it's a foundation that drives cost awareness and optimization and it's trying to shift left accountability to developers because developers not always care about the builds that they are going to get from their cloud providers. So they put as many logs as they think they need, but not always Maybe sometimes, never. They are using this kind of logs what they actually are doing when something is happening. They're finding out that these logs aren't good enough and they are just trying to add more logs. So with dynamic instrumentation and dynamic logging, these developers can observe add logs in runtime, as I demonstrated in my recent post. Right. They can add specific logs metrics, virtual breakpoints. Virtual breakpoints are like a breakpoint that you would do traditionally from your ide. But the big difference, the main one, is that they're not stopping the application. That's why they are called snapshots. You are taking a snapshot of a running application and you are observing through the entire variable path what's happening in this specific method. You are taking the code block of a specific application, whether it's Java Node, js, Python, whatever, and you are measuring how much time it took to exercise this method. But this method is still running in production. You're not stopping, you're not redeploying to add more logs. So the main difference between dynamic logging and traditional logging, for traditional logging you need more telemetry. That's fine. You add more logs, you open a ticket to Datadog and the other logging vendors and you redeploy the code to all these clusters so you can observe. So that's a very tedious, costly and inefficient way of troubleshooting, you know, post to the dynamic thing again, on demand debugging.

So you're just injecting things on at runtime. So you didn't have to like recompile code or anything. You're just going in, you don't have to flip any crazy switches in your code. It's just you're injecting almost.

And the beauty of that, you inject them directly from your IDE into live remote productions without changing the state and stopping the application. We have a full IP and technology around that. But to your point, that's exactly that.

Very, very cool. So I guess, like, how do people get started with this? If people. Like we mentioned Lightrun multiple times. This is not promo for Light Run, but you happen to work for Light Run. Just. Chris, talk a little bit about how Lightrun helps with this. Then you mentioned some. You have some unique technology that helps do this.

So Lightron comes as a free. So we have a free tier. You can start for free and get. So Lightrun basically is a platform which includes three components. The first one is SDK plugin. You download the SDK plugin whether you're using IntelliJ vscode, vscode.dev, the online web based IDE, PyCharm, Rider for.net and others. So basically the entire IntelliJ JetBrains IDE family is supported. So what language you're using, you would use this ide. You will download the plugin, it's free. You get the plugin to your ide, you authenticate from the plugin. Your user can be your Gmail or your business account, it's your choice. Once you authenticate. The SDK is now communicating with the management SaaS Cloud Based Portal of Lightron and that is the bridge, your live applications. So basically once you're doing that, you just need the agent. The agent is the one that is doing the dynamic instrumentation of your target applications. Okay, so once you're running the application with our agent, call it an SDK. Okay, we call it agent, it's an SDK. Once you're running the application node JS Kubernetes, deployed applications, containerized application, whatever, you're running it along with the application. And it sits and polls the management server. Management servers listens to developers request coming from the ide. Okay, so let's say you are using Java, you're using Intellij, you have the exact snapshot of your application. The Java application that is in production. Okay, you want to put a breakpoint, a virtual breakpoint. You want to put some logs into the running application in production. You will use the Lightroom plugin from the IDE. Say I want a snapshot of this line 4 times, I want 4 snapshots. I want these logs from this line 100 in the running application, the management server that listens to the SDK, sorry to the IDE plugin will ask the agent. Agent, please get me this information. And it's going one way only. Okay, so fully secured pii, everything included goes back to this secure channel to the developer's ide. So there is a checksum in the idea version of your code and what's running in production. So you're not really touching and changing states in production. The application is running in production, you're getting insight directly to your IDE and you can start free.

Very nice. So does this do any sort of remediation or is it just log files? And then it's up to the developer then to say I have all these logs now. I mean what does this mean? Or how do I know this is bad or good? What does that do?

So the developer knows his code, so the only thing that it allows him to do is suspect that, you know, the exception or whatever is happening in this area, in this method, you know, that's the thing that he needs to observe. It doesn't have enough logs from the logs that were deployed originally to production. Okay, so it's going to pull the application and get insights. And the insights are like any debugger. You get a full variable path, you get the full insights, the full conditional logging. You can put conditions, you only want logs from this production user ID or from this code path back to your id. So the developer can really pinpoint, you know, what he's looking for based on the output that he's getting into the log console that comes in the SDK and snapshots and performance metrics as well. We get performance metrics, he can then make some sense out of what he's getting and remediate fix, maybe do another round of troubleshooting. But this round of troubleshooting is very, very fast. Okay? These logs do not live forever. They're like kind of immediate. You put the log, you use it, it goes away automatically. If you don't touch it, it goes away in an hour. You can delete it yourself, or if you need to save it for audit trails or whatever, you can keep it for more time. But this is kind of a new agile way of troubleshooting production environments directly from the IDE and without this full redeployment TDS cycle.

And I assume this technology would allow you to see things that you couldn't see, because this is, you're looking at production where you may be using services and containers that you're not even aware of, but you couldn't even reproduce locally. So it seems like you may not have anticipated an issue. Now it's in the wild and you can easily, rather than trying to figure out, keep rebuilding the application and redeploying it. It's just dynamically just inserting it and getting the information that you need.

So I mentioned earlier the prioritization of security issues. You would only go based on runtime. I can give you another example. A lot of our customers are using today what we call progressive delivery. No feature flagging. That's another thing you would know on a segment of users, remotely distributed whatever, that they are using a version of your code, a version of your application, you would know exactly how to troubleshoot a specific issue that was only deployed to them. Okay, so that's another use case that you can only do and troubleshoot locally when you have this kind of observability. Otherwise it's going to Be probably a nightmare for you to troubleshoot.

You also mentioned this is also a performance podcast. You mentioned something about performance, but you didn't go into detail. Can you talk a little bit more maybe about performance metrics analysis that you get using this approach?

Of course. So what we have, we have logs, we have snapshot. These are good for troubleshooting. But sometimes, you know, and we're dealing mostly with backend issues. Okay, less of the front end performance coverage. But you would see on a web service, on a web application, you know, Node JS or whatever that actually we have seen, we had a pet online store, you know, that one of the services took way too long. Okay, so yes, you have different ways of troubleshooting performance. But we looked at how files and stuff like that and were able to find the area in the code that kind of is responsible for this service. So with Light Run we just gather this method or code block and ran some metrics. We can gather metrics of how much time it took to exercise the method, how much time it took to do a specific service call around this method and and back again to the ide. So if you are using Prometheus, you are using Grafana and other dashboards to visualize performance. So you are actually going to get a synchronized output of metrics, performance indicators from lightrun on a specific method, coming on top of what you already have and consume. But this is going to come directly from production. And think about one customer website, other customer website production, pre production benchmarking. So this can really give you wide, wide range of performance visibility for gathering different metrics. Okay. We call it TikTok by the way. You know, we actually use a clock and we measure the performance there.

All right, so I used to work for a healthcare company that had certain hospitals that had certain infrastructure and you find issues, you're like, we don't know, we've never seen this, that they're using something totally different. So could you run this on a client instance and get this information as well? So like all these weird environment issues that are different between hospital to hospital could be bubbled up using this.

Also as long as you have Internet connection, that's what you need. You can use Lightron anywhere. That's the communication from the production or from the other service back to the management server that I mentioned earlier back to the ide. So any environment complex as it will be, you know, you can use lightround to troubleshoot and analyze these kind of issues. And again, we are talking a lot about full observability cycle and we are pitching or we are working with companies like Dynatrace and Grafana and Instana and all these logs IO, all these vendors are working with us on a better together kind of story. Because you are not ditching your traditional dashboards and APM tools. You're just getting another X ray, another set of eyes for complex issues, for complex investigation, especially in production, in environments that you can simply never as a single developer replicate locally in your ide.

Wow, it sounds incredible. Now you said that you actually have to see this to believe it. So you do have a free version. You know, this is normally an audio only podcast, but I don't know if you had like a quick demo you can show and I could do it in the YouTube version or we could hold that off till maybe just different, different show or something. It's up to you.

I can quickly show something.

All right.

You know, and if anyone was skeptical about everything that I've just said, you know, let's try and see something live. Cool. And I'll just move and share my screen. Okay, so what you see right now, you see in my browser that I am authenticated. That's kind of the authentication sign coming from the IDE from Intellij. Once I'm authenticated, you would see that also in my management server. You see it here that I have no running agents. Okay. Which means I'm currently as a developer, I'm disconnected from my production environments. But let's look at these two windows. Okay? We have here two remote VMs. One of them is running in Amazon in AWS. The other one is running on Oracle. I'm going to connect to this remote distributed environments and I'm going to run a simple Java project. And what you see right now, that's the dynamic instrumentation. I'm running the projects with an agent path. That's the SDK that will pull for requests coming from the ide. At the same time, I'm going to run the same application, so think about replicas and different environments. So the same application which is just looking for article titles coming from Wikipedia. Okay, you see it here? It's, it takes every 30 seconds or so, it takes an article based on a pattern from Wikipedia, but again it's the same code running on a remote AWS environment and on a remote Oracle. So now what I'm going to do, I'm going to go to my Intellij, okay, And you see here that I'm connected. I'm going to add a log that I want to pull this Book titles. And I'm going to pipe this, this information both to the STDOUT and to the plugin of Light run. And I'm going to direct my log request right now to AWS prod. Okay? And you will see that I'm doing it right now and I'm going to do the exact same thing to the Oracle VM. So I tagged both the remote VMs. I'm going to get logs from both these environments. You can see here that both environments are running. And you see right now. Okay, I'm just. So you believe me, the application is still running. Okay, you see it here? Okay, I'm piping the logs to the production. You see here the book title, okay, from Wikipedia is being thrown to the STD out, but it's also being thrown to my Lightroom console in the ide. So the developer doesn't need to know or care where the application is running. It can be in 20,000 pods or thousands of pods, but is actually looking here into the terminal. Okay. If I want to add some metrics, we talked about performance, so I can actually look at all these lines and this is a very important method in my Java application. So I can actually take some metrics and I'm going to measure, you know, on the Oracle instance, okay, Call it performance metrics. Okay. And I'm only going to pipe the outputs to my plugin. So right now you will see kind of a blue clock that ticks and it measures. You will see it here, you know, once I'm. Yeah, you see, tick tock. Okay. So right now I'm actually calculating the time it takes to execute this specific method that simply goes and pause or fetches from Wikipedia the book titles. Okay, so that's amazing. The application is running in a remote environment. I'm able to take logs, I'm able to measure performance time for a specific method. And by the way, if I want to take also a snapshot so I can actually debug, I can actually take a snapshot here and here. I'm going to, I don't know, not going to use especially just say Joe Colantonio. No, I cannot write this here. Next time, Joe, I'll give you the shout out. Yeah, but I'm going to take a snapshot and you will soon see an image. You see, snapshot was taken. Okay, Once I click on that, I go to the lightround snapshot and you see that an active snapshot from the Oracle remote machine was taken exactly at the same time. And I can go here to the title and to the entire path and observe and see what book or what article was being taken at the same time you get the point. Dynamic observability shifting left back to the developers from the most complex environments giving you troubleshooting capabilities from performance standpoint logging and all you know, without really stopping or redeploying your code to the production sensitive environments. And one last thing Joe. And with that I will finish if you go back to this terminal and you reload the management portal, you'll actually see that right now the agents are live. There is one Java agent running on Oracle and the other one is on aws. And if I want to see what actions I took, you can see here that I was using Random the Wikipedia Java application from the plugin to do both logging captures TikTok metrics and just dynamic logging from the running application. So I think that kind of summarize this short demo that I wanted to show you.

Very awesome. I see why you you were excited when you heard when you learned about this company. That's very cool. So Aran, I like to keep this about 30 minutes. So I guess before we go, is there one piece of actual advice you can give to someone to help them with their observability efforts and what's the best way to find contact or learn more about this new approach to logging?

I guess so. I would say it's a complex advice. You know, as we started talking today, the marketplace is evolving towards much more complex applications. Microservices, cloud, native, serverless aren't new, but the adoption of these types of applications is growing. And with this growth comes the challenges of troubleshooting MTTR keeping your customers happy, but also keeping your developers happy. Because if developers spend too much time troubleshooting their applications and they don't have the right tools, they aren't happy, they are not productive and we know how it goes. So what I recommend or give as an advice, and I think that's where and I see other vendors looking into the same approach as we speak is to think about redefining or reviving the entire process of the SDLC within DevOps and considering new approaches of troubleshooting when it's needed. Okay, I'm not saying APM logs should be ditched. They are very useful, they are very important. Performance tools are very important. But there are use cases where only this kind of approach will help you be productive. So c how you fit it into your entire existing processes and get your developers more I would say accountable for this quality, but also accountable for the cost of cloud computing, which is also rising. We are in a very difficult year I would say from an economy perspective. So cutting logging costs, cutting cloud computing, looking at what finops are pitching is very important for the future of this cloud native application development. And to your other question, learning more about light on light on.com of course we have a blog, we have social media, we have monthly webinars where we are inviting many industry experts to talk about how they adopting dynamic instrumentation, dynamic logging, FinOps, shifting their FinOps awareness back to the developers and these kind of things. So either follow me or follow Light on and you will be in the know.

Thank you Aran for your automation awesomeness. Felix Everything in value we covered in this episode and on over to testguild.com a439 and while you're there, make sure to click on the Try it for Free Today link under the Exclusive Sponsor section to learn all about Sauce Labs awesome products and services. And they just did a rebranding, so if you haven't been to the site in a while, check it out. It's going to blow your mind. That's it for this episode of the Test Guild Automation Podcast. I'm Joe. My mission is to help you succeed in creating end to end full stack automation awesomeness. As always, test everything and keep the good. Cheers. Hey, thanks again for listening. If you're not already part of our awesome community of 27,000 of the smartest testers, DevOps and automation professionals in the world, we'd love to have you join the fam@test guild.com and if you're in the DevOps automation software testing space, or you're a test tool provider and and want to offer real world value that can improve the skills or solve a problem for the Guild community, I'd love to hear from you. Head ON over to testguild.info and let's make it happen.