Stop Monitoring. Start Observing (Observe) with Jeremy Burton

Get ready to discover some of the most actionable DevOps techniques and tooling, including performance and reliability with some of the world's smartest engineers. Hey, I'm Joe Colantonio, host of the DevOps Toolchain podcast, and my goal is to help you create DevOps toolchain awesomeness. Hey, we're all entering an era where simply monitoring a system isn't enough. And it's all about gaining deep insight and foresight. And this is the world of observability. And I guess today Jeremy Burton from Observe Inc. Is here to turn everything that you thought you knew about data on its head. So prepare to have your perspective changed. Hey, I'm Joe and welcome to another episode of the test guild, DevOps Toolchain. And today we'll be talking with Jeremy all about what I'm calling stop monitoring and start observing. So after a long career in large enterprise software companies such as, I think Oracle, Veritas, emc, Jeremy is now the CEO of a fast growing startup. I think they just got a lot of funding recently called Observed Inc. Which is a SaaS observability offering built on top of Snowflake. So Observed promises to lower the meantime to resolve incidents by 2/3 and at 1/3 of the cost is what I'm being told. And Jeremy also sits on the board of Snowflake and McLaren F1. So really excited to have him on the show. You don't want to miss it if you really want to learn more about observability. Hey Jeremy, welcome to the Guild.

Thank you. Big intro. Hopefully I can live up to that. Big intro.

I'm really excited. I know observability's been getting hot, hot, hot lately. I'm just curious to know why. I know you're an entrepreneur and you must have looked at all these areas of, okay, what is maybe lacking and where can I create something that's going to help people? So why observability?

Yeah, I think it really starts with us building as an industry, building applications differently. And you can probably track this back to maybe, I don't know, seven, eight, nine years ago when kubernetes became commonplace as a sort of fabric for building applications on top of container based development. And really the benefit there was that developers could go fast, they could write code at their own pace and then deploy it into production every day. And the problem that that creates for the poor folks that have to troubleshoot when things go wrong is that you've got a lot of change going into production every day. You don't have time. There was a time, believe it or not, when we used to test a lot of things before we pushed them into production. And the line now is you test in production. Why is that? Well, because you can't reproduce a hundred thousand or a million users in test. And so we essentially test in production. And so you see problems that you've never seen before. And the way that you troubleshoot those problems is you have to be able to investigate. And that's really what observability is all about. It's investigating these unknown problems, things that you've never seen before.

No, I love that. I think a lot of people, when they're older, like me, they think back in the day when they had a raised floor and they actually had everything under their control. It was all deterministic. So you could test before you went to production and you pretty much, you were good. But nowadays you really can't, I guess because you have multi clouds, you have different services that could be audio control that are up and running. So is that why we see this even more now? Because going cloud native and all these moves that we've been doing lately, developers.

Sort of rule the day, right? They wanted to go fast. They want a instant gratification. I write my code, I want to push it into production. And obviously, you know, the world of apps on the phone, people want regular updates. They expect that I'm going to get new capabilities every day. And so the pace of delivery of new features has increased dramatically. And so that forces you then to push more often to production. And so when you see these problems, obviously it's a race to figure out, okay, what went wrong and how do I fix it? Because if you don't figure that out, you're probably going to have a negative impact on the customer experience. And you know, from big banks to small mom and pop shop, they do business online. And you know, that ability to give people a good experience is a differentiator, you know, to the business or to the brand.

How do you know what's going on then if you really can't? If it's so undetermined almost of what's happening, anything could happen. I know. That's why a lot of people do chaos engineering before, because they're trying to test for this unknown activity and you have all this data, all these points like, how do you know what to even focus in on?

Yeah, I mean, look, and this is sort of the mantra of the company, we believe, look, everything is a matter of data. If we collect all the exhaust fumes that these digital applications emit the infrastructure and the application and we collect them together into one big database and then we try and make sense of it, or we put another way, we help SRE teams and DevOps teams to make sense of those sort of digital exhaust fumes and track down what the problem was. So that was sort of the genesis of the company was, hey, couldn't we collect all of this data, not just the logs, not just the spans and the tracing data, or not just the time series metrics, but can we collect it all and could we make sense of it and could we help folks troubleshoot these unknown problems?

So, so obviously you have a platform called Observe. What makes it unique in this space? I've seen all the tools that do something similar. But what's your secret source that? Does it make it easier to troubleshoot all these different points and give you insights or like, what is it that makes it so?

Yeah, there's a couple of things, but number one, we started later than most, so we're a relatively new company and obviously as a new company you can take advantage of new technology and a new architecture. Number one though, we put all the data in one database and that sounds like the most obvious thing in the world to do. But believe it or not, historically that is not been the way because you know, you take a company that's done very, very well over the years, like a splunk, they really built a database to help people analyze logs and they, they made it sort of bespoke for that purpose. And then a company like Datadog came along and they built, they have a time series database. Why? Because they help people analyze metrics, CPU utilization and all that good stuff. And so we were like, okay, enough of this, like different database for different type of data. Why don't we just put the data all in one database and we could take advantage of new technology. Snowflake because it can handle time series data, it can handle unstructured logs, it can handle semi structured JSON blobs. So putting the data in one place is sort of the foundational thing. And then the real magic is that we don't just allow folks to do like a breadcrumb search, you know, I've got a container id, let me go search, you know, thousands of terabytes of logs. We actually curate that data and we structure it. And that's what makes it easier to navigate and therefore troubleshoot. And so the magic in the product is taking this sort of messy unstructured data and adding structure to it and adding relationships to it, which allows people to follow the path of the problem and identify root cause.

It seems like it takes, you know, logs, it takes metrics, but it's all in one location. So it's able to make correlations better that because you have everything in one place. That's able.

That's right. The analogy that I always use to help people sort of grok it is if, if I put a blindfold on you and I dropped you in an unknown location and took the blindfold off, like, what would you do? Right? You know, if you're in San Francisco, you'd be like, oh, you know, Golden Gate Bridge, Transamerica building, you know, Golden Gate Park. And based on your position relative to those places, you'd be like, okay, I think therefore I'm in the Marina district. And so troubleshooting an unknown problem is exactly the same. You pick a data point, maybe not the Golden Gate Bridge, but it might be a metric. Okay, show me the logs for that metric. Show me the traces or the spans for that metric. Now if I correlate around those points, I should be able to identify where in my application the problem is. And so we take that exact principle to heart in observe in most companies today, by the way, that correlation is done by like a subject matter expert. There's always one or a hand, there's one or two people in the company who know everything. And typically today, I kid you not, what happens is, folks, there's an incident, they get everyone on the line and they say, okay, take a screenshot of what you're seeing, Bob, and put it in the Slack channel. Okay, Mary, you next. What are you seeing? And this goes on for the first hour of the troubleshooting call. And then one of these subject matter experts comes along and eyeballs the screenshots in the Slack channel and does the correlation in their brain. And our viewpoint is like, that should be done in software. And the only reason it's not done in software is because the data is not in one place.

Love it. Kind of similar, I started my career as a performance engineer in order to do performance engine. You had a database guy, a mainframe person. You know, you had all these people looking at their little piece and then you come together and go, okay, let's correlate. It sounds like this is all in one location, one dashboard. It's almost like a team flight deck. Almost where you could see everything at a high level as well.

Yeah, I mean, you won. What nirvana is, people want to be able to see the top level view. So you know, one of our biggest customers is a big bank, right? And they use Observe to help troubleshoot their problems with their mobile application. And they want to see the customer journey from login, right the way through everything that they've done. And then when something goes wrong, they want to be able to drill down. Hey, show me that service. Okay, now show me the metrics for that service. Show me the logs for it, show me the latency for it. Okay, yeah, it's not there. Let's back it up, let's go here today. That's like a hundred people on a call, each given a discrete view of what their area is seeing. And so really what we're trying to do is eliminate that chaos at the beginning of the troubleshooting call where you don't have all the data and you're not able to correlate it. If we can get rid of a lot of that, then actually finding the problem and then fixing it once you've identified the problem tends to be trivial in most instances. So if we can eliminate a lot of that messing around at the beginning of the call, then I think we can help people make progress, I guess a finding.

An issue though, all this data, so much data, how do you. And you have to do it in real time really fast. Does it bubble up insights or do you use something like generative AI to query it, to say, hey, tell me in plain English whatever you're looking for and it'll bubble up insights for you. How does that work?

Yeah, so there's a couple of things there. I mean, the data, first of all, the freshness of data, it takes about 15 seconds from when the data is created to it being inobserve. And then you can actually, you know, query it, number one. Number two, then on your question around generative AI, I think we're probably not going to get to the Star Trek like, hey, computer, what's the problem? Anytime soon? But I'm pretty bullish on the impact of generative AI because think about it this way, like our founder, Jacob, anything that goes wrong with our system, if you ask Jacob, he knows what to do. And so the thought with generative AI, you know, yes, we can do the sort of GPT help and the copilot, and we've got all of those things and the product. But what we believe we can get to is if we could train an LLM on your prior incident history. LLMs are statistical engines, right? They don't inherently have an ability to reason, but they can statistically tell you what the Next right thing to do is based on prior history. And so what we think we can do is for a particular class of incident, we could dynamically create a runbook for that incident. So runbooks, for years we have been written and maintained by humans. It's sort of best practices for troubleshooting. But why can't you automatically generate those from prior incident history number one? And then number two is if you can do that, why wait for the human to take the steps? Why not go and execute some of these steps? And so maybe when a human enters the troubleshooting flow, all of the data that they would need for the investigation is in front of them. And in most companies that's probably half the time it takes to resolve the incident is getting the data together. So that I think the promise is that you could half the time to troubleshoot by just eliminating a lot of the putzing around before you get your hands dirty and start investigating. Particularly with an unknown problem, like if you've never seen the problem before, there's probably not a precise answer to get to root cause. But there will be a general set of steps that you would follow in order to triage that and sort of move the ball further down the field. And I feel like we can start with the ball on the 50 yard line instead of the 10 or 20 yard line.

Yeah, I love it. I mean there's a lot of promise in AI, but it seems over hyped. But I think with machine learning, with all this history and logs, with a large language model specific to your incidents, I think is it was made for.

Machine learning almost, it's massively overhyped. But I think like, I mean, you know, in tech, I mean I've been in it a long time, we love a good, love a bit of fashion, we love a good trend and a bubble. And I think this one is absolutely a bubble. Absolutely 90% of the investment dollars that have been spent right now on companies, those companies are going to go out of business. But there will be some big winners. And I think, you know, for those of us that have worked with LLMs today and know the reality of training an LLM and the hallucinations, it isn't perfect, it's kind of a little bit ugly. But I think the promise in terms of like getting people ramped and productivity, I think it's going to change the nature of the way we interact with machines, which I think we've talked about in tech for probably decades.

Right. I think that the feasibility though of triaging at least giving you Like a quadrant to look at rather than everywhere is a big win for sure.

Yeah, that's the win because as you mentioned earlier, right, the data volumes are vast. How do you eliminate most of the data and then dive in? And I think the LLM is going to help a lot with that.

So what's the big benefit of this? Is it able to detect issues before your customers start complaining? Or is it just like an incident happened and then, you know, it's almost preventative? Or is it just trying to fix something that's just plain old broken?

Yeah, no, it's a great question. I think certainly things like anomaly detection have been around for a while. So when you see something an anomalous, you can alert and that may well be a sign of, you know, something more profound happening. I think with observability it's an opportunity to sort of rethink how we're alerting and how we're finding problems. You know, there are literally thousands of alerts set on every conceivable piece of infrastructure. And our viewpoint in the world of observability is much more like, ok, but what's important, did it affect a customer? If it didn't affect a customer, maybe we don't care so much. And so can you actually set an alert or determine even whether a problem is affecting a customer? And when I mentioned earlier about, you know, taking this machine data and deriving relationships, you know, one of the pieces of magic in observe is that, you know, you can see your customers and you can see your metrics by customer and if there's an alert you can see which customers are affected. And so maybe I don't necessarily need to pay attention to a thousand noisy alerts in my infrastructure, but I do if it is impacting a customer. And I think that is, I think observability. The opportunity, if you like, is to rethink how we're looking at our applications and systems and very much take a customer perspective on the problem. There'll always be ways that you can, you know, show outliers or show anomalies and that might give you a heads up of a problem. But I think at the same time we can probably lose a lot of the noise and the alerts by actually alerting on the things we care about versus the underlying infrastructure that we might not, you know.

No, I love that. I know every time I speak to an SRE they're always dealing with noise and even with security, you know what's really an issue and be able to prioritize it is huge for a team. Right. Because, you know, you have all this information happening really quickly. You need to be able to triage quickly and pay attention to the things that really matter. So I love that approach.

Yeah, the sort of discrete alerts without context. And this is again, back on observability. What's it about? It's about having more context. Why do you need context? Well, if it's a problem you've never seen before, you know, back to my analogy, if it's a location you've never been to before, you need context to establish where you are. And so I think the great observability tools are going to give SRE and DevOps teams that context that they never had. Should I care about this alert? Did it affect anyone? Okay, it did. Let me investigate. Okay, didn't. Maybe I can ignore it.

Absolutely. So isn't this a tool everyone should have? So are there challenges to implementing an observability tool or is it just. Is it new? Is it education? Is it a hard sell?

Yeah, it's relatively new. You've. Although it's becoming more widespread and certainly we bump into directors and VPs of observability quite often now. And so it's a little bit of a mindset shift in the organization because, you know, I do believe in the idea of engineering for observability. It's been the case for many years that engineers write code and other people clear up the mess. And we are moving much more to a world where engineers have to take some responsibility for investigating the problems that are found in their code. And the best way they can do that is they can better instrument that code. And so, you know, I do buy into this idea that if you're going to go down the path of observability, it's not just renaming the monitoring team or the DevOps team. The observability team, you know, IT ops became DevOps. Did they do anything different? No, they just changed the name. What does DevOps mean? It means development and operations working together. But many DevOps teams still feel like they're always on the hook to fix engineering's problems. It's like, no, no. The dev in DevOps should mean that they're also in the boat, rowing in the same direction. So I think, you know, there is definitely some work to do on building a culture of observability within a company. Then once you've got the team bought into that, you should have more instrumentation in your code. So with more instrumentation, it should be easier to find out where the problem lies. And so that to me is, you know, we always talk about technology, but people are involved here as well. And there is a cultural aspect to this to get people to understand the world from maybe not their historical perspective. And certainly engineers doing their share of work on call helps with that.

Right.

I mean, if you're on the front lines of dealing with problems and you realize that your code hasn't got the level of instrumentation to answer the questions you need to answer, that's the best way to have an engineer run back to the desk and do it.

So would then getting a system like this then actually highlight what they need to focus in on for better instrumentation? Almost, yeah. So we.

There's a couple of approaches to observability. There's an approach which is, hey, if you've done logging or if you've done custom metrics, you're doing it all wrong. Reinstrument your code with, I don't know, open telemetry or something like this, and then everything will be perfect. I think for newer companies, that's a viable approach, you know, but I think for most larger companies, they've got years of code that is not instrumented with open telemetry. And so we took the approach early on with observe that we don't care whether it's structured open telemetry events or whether it's unstructured logs, give us it all. And we believe that we can make. Even if you've just got like, like your code instrumented with logs, we can make more use of those logs because we add structure to the logs after the fact. And then look, if there's blind spots, there's no magic here. If there are blind spots in the code, we can be precise about where the blind spots are. Don't reinstrument everything, but add this to this set of logs and then you'll be able to answer that question. So we tend to pitch observability. We also tend to go after logs first because it tends to be a quick win. We can show the value of the product. And also I'd say, particularly at this point in time, these systems are getting quite expensive. We can show folks how they can not only troubleshoot faster, but they can save a buck or two.

I love that approach. A lot of times someone hears something and it's cutting edge and they may be on a very small team startup and they can implement all these. I'm used to working for health care and insurance on legacy code, so being able to actually consume all these different formats, I guess would Be a big win for sure.

Yeah. No, I mean my view, by the way, I think in the, I mean the environments that you're familiar with that is going to be logs with a bit of metrics for the foreseeable for the rest of this decade. That is going to be the predominant amount of telemetry that exists. I think open telemetry is a good thing, but it's a new thing and new things tend to be adopted by new companies much, much more quickly than you know, all the more traditional companies. But it is not the case that you can only have observability if you reinstrument your code with like OpenTelemetry or something new. That should not be the case.

Absolutely. So you did mention you are a newer company. So I guess a little background with the company, how long you've been around and like you have any customer success stories like I mentioned, I think you had a claim about able to cut incident times for like 1/3 or something.

Yeah, so we've been around just over five years. Took us about four years to build the product and then the last year to really get it to where we wanted it to be. So some things don't change in software. I've been doing this a long time and writing code still takes time and getting it right still takes time. I still have never seen a release that shipped early, so I don't know, some things don't change. But yeah, We've got about 60 customers right now. Most of them are sort of 200 to 2,000 employees in size. So you'd say sort of mid sized companies. We're working with a couple of larger enterprises now. Our biggest customer is a big bank. They do about 120 terabytes of data a day and we're working with them on how they troubleshoot their mobile banking application and we're quite bullish on what we can do to improve their troubleshooting time. And that is probably the like the biggest use case for observe observability, troubleshooting, reducing time to resolve incidents. We also work with some customers because in smaller companies the DevOps team tends to be the security team. You know, if you've got a team of two or three folks, you know, they get a hold of observer and they realize that you can ingest logs. Well you know you can ingest access logs, you can ingest security logs and we have a language in the product called Opal and you can go manipulate that data directly. So I'd say probably 20% of our users use the product not just for observability, but also sort of security and even compliance type use cases, because they're essentially running queries against data coming from the identity and access management systems and from the firewalls and things like that.

I actually have that written down, security. So there is overlap. So it's almost like you can kill two birds and one stone if you're. Obviously you need security, better security. So implementing observability and especially observe, sounds like it would also help with security also.

Yeah, I mean, we've got a. Maybe this is counter to a lot of folks, we've got a very broad definition of observability. We don't believe it just pertains to distributed applications. We believe it, it should also pertain to the infrastructure. I mean, that's what the applications run on, after all. We also think that, look, you can take telemetry information from a monolith as well and be able to troubleshoot that more quickly. You can take data from firewalls and make sense of those. And the term observability actually was coined back in the 60s by a guy who was studying control systems theory, a guy called Rudolph Kalman. And it was really just about, you know, understanding or asserting what is going on inside a system from the external outputs. It's quite a simple concept. It's like, hey, can I figure out what's going on inside from the outputs? And so, you know, security. If I'm. If I've got access to outputs, which will be the logs, can I determine what's going on inside the network from that? Sure you can. So why wouldn't observability apply to security as well? I mean, I don't see why it wouldn't.

How are you able to handle the data volume that we're seeing nowadays? Is it because of Snowflake? Is Snowflake a database? It's the first time I've heard of Snowflake. Like I said, I'm a newbie here.

Yeah, no, that's a great question. And I think one of the fundamental differences that Snowflake has, and even though it was the biggest IPO in history. Right. Most people still don't really understand why it was such great technology. And I mean, I mentioned earlier that it can handle different data types. That's interesting. But that's not the main event. The thing about Snowflake is that to run queries, it doesn't build an index. And I was at oracle for almost 10 years and that sort of bends your mind that like hurts your head Right.

Y.

Right. Because, you know, hey, if I want to find something in a book, what do I do? Well, look it up in the index. So how could it possibly be faster to not have an index? And the trick is that what Snowflake does is two things. It organizes the data it ingests into what's called partitions or micro partitions. And then it organizes the data very efficiently in storage. And so it doesn't use an index, it scans data. But because the data is organized efficiently, it's able to eliminate a lot of the data when it's executed in the query, does what's called pruning. And so the punchline is, is that it's faster in many cases than an index based database, but it doesn't build an index, it scans the benefit. Then the reason why that is so cool is we just dump all the data we want Snowflake to query into S3, right into Amazon. S3, so cheap as chips to ingest it, we don't even charge for ingestion. And then Snowflake queries it from there. Every other system it's that they, when you ingest it, it starts building an index which gets expensive. And the more data you have, the bigger the index is, the more they have to charge. And so this really changes the economics of observability because it's fundamentally a different query engine and a different database technology.

Okay, Jeremy, before we go, is there one piece of actual advice you can give to someone to help them with their DevOps observability efforts and what's the best way to find contact you or learn more about Observe?

Yeah, I think the number one topic we get inbound right now is cost. I think folks budgets are under pressure and rather than just looking at can I get a discount or you know, can my vendor do this more cost effectively, I'd sort of pull the thread and say ask why? You know, why is this more expensive? And usually what that comes down to is that the data volumes that people are seeing today is maybe an order of magnitude or two more than their current tooling was designed for. And so, yeah, I'd say asking the question why? And if someone can give me a discount, does that solve the problem or am I back here next year? Pull the thread on that. Because what we're seeing is, you know, the architecture of incumbent tooling was designed for a different era and a different volume of data and the economics are very, very different. So yeah, I'd say, number one, if that's your pain point, pull the thread and ask why? Technically why? Because the price list usually reflects the architecture of product awesome and the best.

Way to learn about Observe.

Oh, best way to learn observing.com if you go up there, obviously free trial for those that are interested and blogs and resources should give you everything. The very beginnings of Observe. The founding story right the way through to current day.

And for links of everything of value we covered in this DevOps Toolchain show, head on over to testguild.com p134 so that's it for this episode of the DevOps Toolchain Show. I'm Joe. My mission is to help you succeed creating end to end full stack DevOps toolchain awesomeness. As always, test everything and keep the good. Cheers. Hey, thanks again for listening. If you're not already part of our awesome community of 27,000 of the smartest testers, DevOps and automation professionals in the world, we'd love to have you join the fam@testguild.com and if you're in the DevOps automation software testing space or you're a test tool provider and want to offer real world value that can improve the skills or solve a problem for the Guild community, I'd love to hear from you. Head ON over to testguild.info and let's make it happen.