Achieving Observability Goals with BugSnag with Eran Grabiner and Justin Collier

Get ready to discover some of the most actionable DevOps techniques and tooling, including performance and reliability with some of the world's smartest engineers. Hey, I'm Joe Colantonio, host of the DevOps Toolchain podcast and my goal is to help you create DevOps toolchain awesomeness. Hey, it's Joe and welcome to another episode of the Test Skill DevOps Toolchain podcast. Today we'll be talking with Aran and Justin all about the best ways to achieving DevOps your observability goals. If you don't know, Aran is the Director of product management at SmartBear, where he works on observability initiatives, so he really knows his stuff. His journey with Smart Bear began after being acquired by SmartBear for his observability startup Aspecto, where he served as CEO, so he has a background that includes diverse roles in cloud focused startups. We also have Justin joining us. I'm not sure if he remembers, but he was on my automation podcast way back in episode the 67 in 2015. But he is now the Senior Director of Product Management where he leads product teams to deliver significant value at Smart Beer's customers for Smart Beer's customers, while also focusing on the design and execution of business strategy. And before becoming involved with SmartBear's developer observability business, he led portions of SmartBear's API lifecycle business, focusing on API functionality, security, performance testing, API virtualization, API exploration. All the good things. So really excited to have Justin back. If you don't know, Justin's also featured in my book Automation awesomeness on page 36, so definitely grab it if you don't have it yet. And I'm really excited to dive into observability with two of the best in the area. You don't want to miss this episode. Check it out.

Hey, if your app is slow, it could be worse than an error. It could be frustrating and in my experience, frustrated users don't last long. But since slow performance is a sudden, it's hard for standard error monitoring tools to catch. That's why bugsnag, one of the best production visibility solutions in the industry, has a way to automatically watch for these issues. Real User Monitoring it detects and reports real user performance data in real time so you can quickly identify lags, plus get the context of where the lags are and and how to fix them. Don't rely on frustrated user feedback. Find out for yourself. Go to Bugsnag.com and click on the free trial button. No Credit card required. Support the show and check them out. Hey guys, welcome to the Guild.

Thank you for having us.

Thanks Joe.

Awesome. Really excited about this episode. I guess before we get into I'm always curious to know around why someone creates a tool for a certain solution. Yours being observability. So maybe a little background how you got into observability, why you got into observability.

Yeah, good question. So like in many cases in life you ask someone something and like in retrospective he builds like a strategy of why he did things and so on. But actually it was a little bit by chance. So for me it was a little bit by chance as well. Back then I by chance went into like actually the cybersecurity and DevOps space. Like a lot of folks in Israel, there is a very, very active cybersecurity scene. I'm originally from Israel and then at some point wanted to found something myself and to build something. And I got to know my co founder Michael and he was very much into observability. So I would say my partner took me in and this is how I started my journey there.

Nice. And Justin, as I mentioned, you've been all over the life cycle, end to end for software development, life cycle, all kinds of products and services. So what drew you to observability or what makes you excited about observability now being your main focus? It sounds like.

Yeah, I think it's really for me just an opportunity to really have an impact on what an end user is experiencing. And I think that that's kind of the fascinating piece about the observability space. It really is about what's going on with the end user. Are they having issues, are they having problems, is it slow for them, et cetera, and then being able to help companies improve that and ultimately just deliver a better experience for their end users. So I'd say for me that's really the exciting part about being in this space.

Absolutely. Very cool. So I've been hearing more and more about observability over the past few years. I'm just curious to know maybe if we looked at the observability landscape, how it's evolved over the past few years, maybe how things have been changing or where things are going. So Aram, let's start with you here. Any thoughts on where you may have started when you first conceived your solution and where we are now?

Yeah, so I think zooming out, obviously back in the days we had legacy applications and we had some legacy companies founding this domain of performance monitoring and the idea of selling you a product that helps you better understand your software. And they've been around for a few years until I think the big boom that happened, I don't know, maybe five, ten years ago when folks started to move very, very quickly into the cloud and you started to see tools that are focused on cloud native applications and on cloud applications, companies like Datadog and New Relic and so on and so forth. This new wave of new tools and new monitoring. And then I think in the last couple of years, in particular the last, I don't know, two to five years, there are few vectors I think that really influence this industry. The first one, and it's not happening just in monitoring. Developers have much more power and much more responsibility than what they had in the past. We see this shift. They have more tooling, they are more responsible about what's going on in production. Twenty years ago it was a wall between production and pre production and now you see engineers doing stuff while in production. And the fact that you have this new Persona coming in and, and responsible for performance and how their application is behaving in production is just different. It's a different Persona. They have different skillset, they have different needs and it is changing the industry. And maybe the other thing is, and it is connected is the push towards standards and open source again. Back in the days every vendor had like proprietary technology. The fight, so to speak, between vendors was like, you know, my agent is better, you know, I have a better technology. And nowadays we are moving into a world where everyone are essentially using the same protocol, the same open source foundations. It makes a lot of sense for users. We can talk about that later. But I think that's like another trend that we see out there very, very strongly.

Great.

And Justin, I have a feeling that you have like pulse of your smart beer customers. What they're asking for, what they're struggling with is observability. That seems to be a growing concern for them or something you've been getting asked by for certain features of functionality that they might need because they're struggling. Yeah.

Are they struggling? I would say yeah. How they're solving that problem I think differs, you know, on company size and maybe the applications that they're producing and the problems that they're trying to solve. But yeah, I mean there's no doubt that the importance of the end user experience for our customers is really, really high. Right. They're trying to deliver the best products that they can and ensure that they're their customers keep coming back. And if you look at the data about poor experiences for people having whether they're online or on their mobile applications if things are slower, you get customer drop off, they don't move through your funnel, they leave you and move to another company. So yeah, absolutely the entire experience for a user is very critical.

So another term I've been hearing a lot about is digital experience. Back in the day I used to call it apm. I don't know, are they the same thing? Is digital experience the same as APM and observability? And I don't even know if APM and observability are the same anymore. So any thoughts on the difference between these types of terms or what people are trying to achieve? Yvonne, I guess we'll start with you or Justin.

Yeah, I think there are slight differences obviously there are again like in every domain in life some things are just evolving and you have like new names that are. That sounds better or you know, those kind of things. But there are, I think few slight differences. For me when we speak about APM was very focused specifically on the application performance and very focused on I would say system health. Is it working? Is it not working? Getting the right metrics in place and speaking about observability, at least it's supposed to be something a bit deeper. If you look at the term observability coming to this domain, coming from actually control theory and so on, it's the idea of understanding how your system is behaving by observing it from the outside meaning to really understand how things are behaving and why. So I would say the difference is like few layers deeper in your understanding. But that's a natural evolution I think with digital experience. I think that in the last couple of years with the Internet boom and so on, we saw the rise of companies that are user intensive and a lot of Internet based B2C capabilities, mobile applications, E commerces, gaming and so on and so forth. And as Justin said, those businesses are really affected by those users and those users are interacting with your software in a much, much faster pace and a different way than things happened in the past. And also they are doing it over platforms that you don't necessarily control like they have their mobile, they have their web browser and so on. And I think digital experience for me it's a subset of observability that is very much focused on their particular experience like how a user, our customer is experiencing your software, interacting with it, what kind of experiencing he has. So you are less interested in how the very deep back end service is working. You are more interested in the real life experience that the user from a certain geography is now experiencing.

Well, I was just going to say I think it's like a technical impact versus a business impact. And I see in our space for what we're doing is we really are focused on what is the business impact.

So speaking of business impact, a lot of times we talk about user experience, but obviously it comes to how can we cut costs, how can we save money? Can you use observability to help not only optimize performance or user experience, but also maybe cut costs or point out things that people can optimize for to help them with that?

Yeah, I mean, I would say when you think about being able to pinpoint specific errors or events that are occurring, being able to find performance issues, being able to do that very quickly and easily is absolutely optimizing for your cost. Right. If you don't have your engineers, which are generally speaking, you know, expensive team members that you're able to, you know, have them focus on getting to resolution very quickly and being able to move on and ultimately drive value back into your business through building strategic roadmap initiative projects or innovating or whatever it is that they might be needing to do versus focusing on potential issues that are coming up in the product. I don't know. Ron, other thoughts?

I think smarter people than me said lately, slow is the new down. And it goes back to what I just said about those new businesses where you have so many businesses where you if something happens in two seconds versus three seconds, it actually affects your bottom line. You lose users that way, you gain less per user that way, and so on. So in digital experience, improving the performance of these flows in your system can actually affect your bottom line.

I love that. I'm going to use that for the title Slows the new down. I think that'd be cool.

It reminds me of the Navy SEAL thing. Right. Slow is smooth and smooth is fast.

Yep, I love it. Very cool. So you do all work for a vendor. I love working with vendors, but I know a lot of the trends here are with open source. There's opentelemetry and things like that. I think there's a lot of benefits with a platform that people can get from, say, a vendor solution. Are there any benefits or differences between open source versus, say, like a paid solution?

Yeah, I think obviously there are few differences, but at the end of the day, I think that businesses should focus on their business. We've seen in the last couple of years where companies were raising a lot A lot of money. You've seen companies like Uber and Netflix and Lyft and so on, like building enormous technology capabilities in house. And once there was a downturn and so on, you saw these companies like starting to go away from these initiatives. It cost a lot of money for companies to maintain those infrastructures and it's not your main business for me as an entrepreneur and so on, you should focus on your business. And once you're getting into building those capabilities internally, it has some advantages because you obviously have more control and it's more tailor made and so on. But also it's defocusing your R and D which I think is more important than even how much money you are paying for the vendor and so on.

That's a great point. I've seen companies that invest in, in house, proprietary, their own solution and then someone leaves and the product, no one knows how to pick it up and go forward with it and they've wasted tons and tons of money on something that you said isn't their core competency. So I highly agree with that. Justin, any thoughts with this also?

No. I would 100% agree with what Aran just said. Right. And it's so attempting to go off and build your own thing. Like you said, at some point somebody leaves and then you go how are we supposed to maintain this thing and that? Yeah, it's a huge cost and then you basically are starting over, trying to rip and replace. And there's a cost associated to doing that.

Absolutely. So I guess who's responsible for observability nowadays? I think you said around it's you know, a lot of people, everything's on the devs nowadays with this definition of done throw observability on top of it. We try to get rid of silos. So who's responsible for observability?

Everyone and no one. Yeah, you know it's like back in the days we had the QA department.

Right.

Like someone is responsible for quality and then we understand like it's foundation, everyone is doing it and so on and so forth. So also with observability, I think that as you look at like more and more modern companies, you see how it becoming horizontal thing where a lot of people are touching it and different organizations are approaching it differently. But I think one interesting trend, at least I've seen in the last year or two is where you see more and more observability teams that are horizontal and responsible for that for the whole organization. It goes hand to hand with all the platform engineering trend and so on. But I would say that when you go into the traditional company nowadays, you would still see who's owning the APM vendor and so on, usually the DevOps. But now you see this trend of how it's slowly moving into R and D and dedicated team that are responsible for that. What do you see out there, Justin?

Yeah, I think that's a very valid and true perspective. It is a situation where I think traditionally we're seeing DevOps own that it continues to spread out though and more and more people are having to touch it and get involved and be responsible for it in some form or fashion. And so yeah, it's definitely, I think it's a team effort. A little less about the individual.

Yeah, I can see it being difficult though. You know, just like automation testing a lot of times I see there's an automation architect and then he sits on top of all the sprint teams, but he helps, you know, empower them, get them up to speed with things. He doesn't necessarily do all the automation, but he disseminates information, education, tooling frameworks. Do you see some place for like an observability architect that would act as like someone that isn't a bottleneck but helps facilitate making sure all the best practices have been happening with all the different teams with observability, I could be totally out there too. So I don't know if that's something you've heard or seen.

Yeah, I think that like when you are starting to touch something that influence like a lot of people in your organization and touches the code and it's not anymore. I think like another thing that we're seeing out there is that observability. It's not anymore like a black box product that you just buy and get. It's becoming like another layer of your application. You're building stuff around it, you're interacting with it. It affects how your application is behaving. Like if something happens you are doing something differently. So it affects a lot of things. And once you're getting into this level, you obviously want to have someone smart looking at it and making horizontal decisions and so on. So yeah, I also saw in many cases where this observability team or the tech lead or architect that is responsible to make these decisions is someone that is seeing stuff horizontally, very deep, understand the depth of the system, understand how it's going to affect the tooling that we're choosing, how it's going to affect the other components that we have out there, like the platform we're using. The databases we're using and so on and so forth. So, so you definitely see like the person that is, I don't want to say the smarter, but probably have a lot of experience and knowledge about the system going into those decision making processes.

Yeah, I would agree with what Aran just said. We have, you know, especially internally, we have individuals that are responsible for that. And you start to look at just how skilled and capable they are and how well versed they are in a number of different disciplines and then being able to pull all of that stuff together and make sure that the right people are doing the right things, that we're making the right types of changes that don't have adverse impacts down the line. I'm humbled, actually, whenever I work with folks like that because they just the amount of stuff that they know and ensure that we're always making the right decisions is critical.

So Justin, with your background in testing traditional testing methods, do you see how does observability complement or does it complement other traditional testing methodologies and methods that you've been involved with?

Yeah, it's an interesting question. How does it complement the testing piece of it? I would suppose that if I had to think about the testing piece at least, I mean, maybe internally I think about observability and what we're doing and maybe I can apply it to that. We use our tools to help us ensure that as we're making releases or shipping new product, that we're using the observability tools inside of the applications that we're producing and tools that we're producing and then we're immediately feeding that data back in so that we're always making the appropriate changes and that we're getting real time, live information back. And I would say that our customers are ultimately trying to do the same thing. I think one of the challenges ultimately that people end up facing is, okay, well, we have this, we've pushed production, we now have these, you know, these events that are taking place. But how do we feed that data back into the test that we're actually creating and that we're building? And so I think that closing off that loop is really important for a lot of organizations and not all of them have necessarily been able to solve that. And I think that that's where observability is able to feed back into the testing process and provide additional clarity and make it easier for people to then create new tests or go down that path.

So Imran, I guess, are there any best practices then for integrating observability into your software development life cycle that you've seen work for others?

Yeah, I think you should start with what you're trying to accomplish.

What are the things that are important for you as an R department, as an organization? I see in many cases Justin talked about it the difference between a business impact and a technological impact. In many R and D teams it's very hard for them to really understand what affects the business. Like they are deep into their technology and trying to figure out exactly what are the things that are important for you as an organization to monitor. Right. And to do the, you know, back thinking from, you know, this is the metric that is important for us as a business. How these things boils down into our tech stack. And then my personal recommendation would be like don't build spaceships, don't get into, you know, huge initiatives, don't try to invent the wheel. There are so many companies out there that went through these processes. Knowledge is so much more available nowadays than it was, you know, a few years ago. Get into stuff that like already work protocols and communities that are large, that a lot of people encounter the same challenges that you are encountering and you can learn from them and try to make sure that you are not becoming, so to speak, like a slave of that, that it starts to control you and it's like instead of observability helping you to move faster and to better understand your software, you are becoming like someone that is chasing the observability so, you know, nowhere to stop. And lastly, I would say there are many great vendors out there, including smartware, that gained a lot, a lot of experience in these domains and can really help companies just make this leap forward faster and kind of like forget about it and as I said earlier, just focus on your main business.

So Justin, any best practices? You seem helpful or you think people.

Should know about best practices? Yeah, I think I would echo a lot of what Aran said, which is focus on the things that you're good at. I think getting the data back into the developers hands as quickly as possible is incredibly vital. Again, ultimately cutting down costs and allowing them to be productive. And ultimately I think they'll enjoy their jobs more because they're able to focus on the things that excite them. And I think that's really an important part of the observability space is being able to provide that level of information and detail back and allowing people to ultimately build the things that they're passionate about building.

I know the answer is yes, but can a modern software development company or team survive without using observability minus survive.

Yeah, it's going to be hard. So many moving pieces nowadays. And also observability, it's not a binary thing. It's not like either you have it or not, you have something. As a developer, you have your very basic logs that you're writing somewhere. So it's like saying no observability whatsoever. It's like going through the street with your eyes folder. You do have something. It's more a question of like, you know, what is the level of observability that you have and how it impacts your workflows and processes.

So Justin, you mentioned in house you have a lot of people working on observability. You work for a vendor, a lot of tools, I assume. Do you use your own tooling within your company to do all your testing?

We do, yeah. I think what's interesting about SmartBear is that, you know, obviously a lot of the stuff that we're building is is web and so we do some mobile related stuff, but primarily just from a testing perspective and providing that capability through tools that allow you to do testing in the cloud, on mobile devices and web and et cetera. But for the most part what we're building is SaaS, platforms and tools. Right. And so yeah, our teams are absolutely using bugsnag and other Smart Bear products to ensure that we're driving quality throughout.

Awesome. And I failed to mention this. The observability tool that you tend to use, is it bugsnag, is that the main is that was acquired and then relabeled as bugsnag and added onto.

Yeah. So bugsnag is interesting, as you mentioned, is an acquisition as part of SmartBear and then yeah, through the years it's continued to grow and then most recently that's kind of been added to the observability business, if you will. Or tooling of smartbear is aspecto nice.

So Ryan, are there any features added on to bugsnag that makes observability that much more awesome? Here's a feature that you normally would have to build yourself, it would take you a long time, or here's something out of the box so you flip on and bang, you got it.

But yeah, I think what's like for me, what's unique about bugsnug, obviously there are a bunch of features out there that are cool, but what's unique in what we're trying to build, I think is actually the full stack approach of it from front end to backend. There are a Lot of vendors out there, obviously, but the fact that bugsnag became like very, very professional in helping the digital experience journey and specifically to front end engineers and front end platforms. And now as we are expanding further, like deeper into the stack, I think it gives you like an added value that is very hard to get otherwise, to really see a complete view or observability, if you like, from what your user is experiencing all the way down to your application stack. I think this is something that is very hard to achieve end to end and we are also struggling with building it. Right. But I think that already the current capabilities that BugsNav is offering around this are very unique.

I think at the end of the day it's just about providing a holistic perspective on what's going on from an event or error standpoint and then also what's happening with performance and being able to provide information to folks around that. So I wouldn't say it's necessarily features like that's the core of what the product is ultimately trying to do. I would say that one of the things that we often hear from our customer base is that they just really appreciate the usability aspects of it. So less about feature, but really about, you know, you've got folks that are in here day to day trying to perform a job and we want to make that as easy for them as we can. And so it's less about a feature but just more about, hey, how do we make the experience good for people? And it's the same problem that they're trying to solve, right? How do they make the experience good for their folks? So yeah, I would say it's less about the. Less about the specific feature and more about the experience.

So I would think just like any type of testing, especially now you said devs need to work on it and they're overloaded. How do they know what is a priority? Does this help them know, like prioritize issues in observability like you would, Is it risk based? And help them know, okay, yeah, you have 10 issues, but these are the two most important issues that are risky that from observability point of view should be fixed first. Is anything like that baked in?

Yeah, I would say, I mean that's the nice thing about observability is, you know, kind of coming back to what do you want your devs focused on? One of the unique things about being able to have observability tools is the ability to say, hey, what are the priorities? And as issues arise, how do we flag those and Set a priority for them and then make decisions. Is this something that's affecting one user? Is this affecting a thousand users or a million users? And so when companies are able to make those decisions and do it effectively, then they're able to prioritize and then again, ultimately get back to the things that they want to be building and are excited about being building. So I would say, yeah, absolutely, I guess, is the answer to your question.

All right. So, you know, I can't let you go without asking about AI and machine learning. It's everywhere and every tool. I would think with the observability, though, because it's analyzing a lot of data, you can actually leverage machine learning to really help you bubble up insights. Is, is there any sort of machine learning or AI built into bugsnag or anything on the roadmap? You see it helping with observability down the road or even now?

Yeah, I mean, I would say that those are today. No, there's not anything specific inside that is tied to AI. Is it stuff that we're thinking about, looking at and assessing? Absolutely. And I know Iran's been actually running a project around that and, you know, so, yeah, it is something that we're focused on. I don't know if there's things you would want to specifically share about it.

But yeah, yes, still in the oven. But we're definitely looking at it like, you know, from. From different angles. I would just say like, you know, high level. There are a few layers here. There is just a layer of the user. Productivity engineers, DevOps engineers, they are spending a lot of time in observability tools. And our goal as an observability tool is to find a way that makes it faster for you from the moment you get in to the moment you get your answer. And just about the productivity piece of it. How easy is that for you to understand what you are seeing right now and to take actions? I think that what we've seen in the LLM space and so on, this is an area where it can be very, very helpful. And yeah, obviously the other piece is you are collecting a lot, a lot of data and machine learning algorithms can make a lot out of it. I think one of the challenges over there is the fact that every company is slightly different and you're trying to figure out what are the things that are common to everyone. Right. Because each company has its own loads and loads of metrics and logs and so on and so forth. And it's not like, you know, it's not like OpenAI when you just go out to the Internet, scanning like all the things that are out there and have something that help everyone and this is like a real technical challenge, how to build something and how to gain insights from data like that that would be again, horizontal and relevant for everyone and not for a particular company that owned this data and so on.

Okay guys, before we go, is there one piece of actual advice you can give to someone to help them with their observability efforts? And then what's the best way to find contact you and learn more about bugsnag?

So Justin, what would I say? So first I'll just say, yeah, you could just go to smartbear.com and absolutely go learn about our.

That was my recommendation.

Go learn about observability. I would say one piece of advice, I think we've been really talking about it the entire time, which is what are you trying to do from a business perspective? What value are you trying to drive for your end users? And where are the priorities lie? And then what tooling exists out there to help you solve that problem in the most efficient way possible so that you can focus on the things that really matter? And how do you drive value back into your customer base? And so I think that that's probably the tip I would say is help let other people solve those challenges for you.

Yeah, definitely. As I said earlier, so many communities out there, so many channels where you can reach out to folks and learn from their experience. So many written pieces about those things. Me personally, I'm specto and we are coming out of the open telemetry domain. So this is what I'm mostly familiar with and I'm seeing it every day. Just go into the OpenTelemetry Slack community space. You have a channel for everything and experts for anything that can answer you in minutes. This is amazing. It's something that people didn't have a few years back. So I would say get engaged. Getting to those places where knowledge is out there would be a very good first step. And personally, you know you can reach out in any way, Twitter, LinkedIn, whatever, we are out there. Happy to help. My partner and co founder, also working for Smart Bear Michael, he just released like an open telemetry course in Udemy. You can just go ahead out there and learn things so many ways. Don't be shy, don't afraid to say hi and reach out and we will be happy to help.

And for links of everything of value we covered in this DevOps toolchain show, head on over to Test Guild.

Remember, latency is the silent killer of your app. Don't rely on frustrated user feedback. You can know exactly what's happening and how to fix it with Bugsnag from SmartBeer. See it for yourself. Go to Bugsnag.com and click the free trial button. No credit card required. So that's it for this episode of the DevOps Toolchain Show. I'm Joe. My mission is to help you succeed creating end to end full stack DevOps toolchain awesomeness. As always, test everything and keep the good. Cheers. Hey, thanks again for listening. If you're not already part of our awesome community of 27,000 of the smartest testers, DevOps and automation professionals in the world, we'd love to have you join the fam@testguild.com and if you're in the DevOps on automation software testing space or you're a test tool provider and want to offer real world value that can improve the skills or solve a problem for the Guild community, I'd love to hear from you. Head on over to TestGuild Info and let's make it happen.