Observability at Scale with AI with Jacob Leverich

Get ready to discover some of the most actionable DevOps techniques and tooling, including performance and reliability with some of the world's smartest engineers. Hey, I'm Joe Colantonio, host of the DevOps Toolchain podcast, and my goal is to help you create DevOps toolchain awesomeness. Hey, what is AIOPS? Well, you're in for a treat because today we're talking with Jacob all about AI and observability. If you don't know, Jacob is the co founder and Chief Product officer at Observe. He's he previously directed engineering at Splunk, it was co founder at Kuru Labs. I may have botched that, but I'll find out. Listen in to discover how AI and data lakes are reshaping observability, the role of knowing knowledge graphs, the cost of tool silos, and what engineering leaders often get wrong when trying to scale visibility, and much more. So whether you're a tester, a developer, SRE or team lead, this episode is for you. You don't want to miss it. Check it out. Hey, before we get into this episode, I want to quickly talk about the silent killer of most DevOps efforts, that is poor user experience. If your app is slow, it's worse than your typical bug. It's frustrating and in my experience and many others I talk to on this podcast, frustrated users don't last long. But since slow performance is as sudden, it's hard for standard error monitoring tools to catch. And that's why I really dig SmartBear's Insight Hub. It's an all in one observability solution that offers front end performance monitoring and distributed tracing. Your developers can easily detect, fix and prevent performance bottlenecks before it affects your users. Sounds cool, right? Don't rely anymore on frustrated user feedback. But I always say try it for yourself. Go to smartbear.com or use our special link down below and try it for free. No credit card required. Hey Jacob, welcome to the guild.

Hey, thanks for the warm intro, man. Appreciate it.

Yeah, it's great to have you. You've had quite a journey. So I'm just curious to know how you got into this, how you got into DevOps and the whole story.

Oh, right on, right on. Yeah, it's actually, it's kind of the beginning of my career and it's kind of like now where I am today and there's in the middle, there's a lot of different stuff. And let me tell you about a little bit. So yes, on the cpo it observed now we Observe ability. Startup. And the product role is a newer one for me. I kind of came into this, you know, co founder, you know, head of engineering. So very much on. On kind of the engineering side of it. But prior to that I actually worked at Splunk. I was an engineer in the core search engine, just like banging out C code to scale it out to bigger and better use cases. That was a really good experience. Learned a lot there. Prior to that, spent a lot of time buying big companies, failed startups. I've spent time at Google, spent time at HPLabs, spent time at IBM, did the whole grad school thing. None of it matters. So when I started my career, I was a Linux system administrator, you know, like late 90s, early 2000s, you know, that's kind of how I cut my teeth. That's how I paid for college. That's how I kind of like, really got into like professional life and technology. And it's just like banging out mountains and pearl scripts and monitor and maintain small fleets of servers, manage servers and colo facilities. You know, wore a pager, a Motorola pager.

Fun times. Yep, yep.

You know, waking up at 2 in the morning to kind of, you know, troubleshoot fires. And so. So I kind of started my career that way and kind of like really loved kind of everything about that, you know, kind of the good and the bad. And then I went through all this other stuff in my career, but then ended up at Splunk. And I kind of realized like, holy crap, if I'd had this software when I was a practitioner, I would have been way better at my job. So it's like, no longer like, kind of just like making homegrown tools to like, kind of build. I just kind of monitor, maintain these things and solve like, the challenges of bringing all this data together. It's like, actually if you have a tool that like, helps you, you know, solves the hard technical challenges of bringing all the data together, that's cool. I don't have to do that myself anymore. But then also, like, kind of once you have it all in one place, like, having something that facilitates the business process of like, everyone, you know, being able to look at the same data and triage the same alert and kind of work together to solve a problem actually is like, very meaningful for businesses that depend upon their digital infrastructure. And so I get. Long story short, I kind of figured out like, oh, wow, I'm actually really passionate about this stuff. Like, like, I used to do it and it was my life. And then, you know, kind of I went through this whole technical journey and learned a lot about like building large scale systems and you know, big engineering teams, all sort of stu. But then realized like, oh man, I actually, I want to get back into this and I actually want to bring like new technology to bear to solve the problems that people have, you know, kind of doing this job. And so. So long story short, that's kind of how I ended up, you know, kind of, you know, starting this company and I think, you know, kind of getting to the product side of things. You know, it's like kind of the company is sort of operating enough and we have other people that are way better at engineering management than I am. And so, so I kind of focus on our customers and kind of being more, a little bit more outward facing and making sure we're still going after problems that people care about. So anyway, that's kind of, that's awesome. Yeah, the journey there.

It's a pretty big leap though. You've named some big companies you've worked for. Why become a co founder though? Why through all seems like a lot of headaches, but it seems like you were passionate about a problem obviously that did something draw to you to observe that like you want to solve this so bad that you had to be a co founder.

I think that actually had already kind of always been a little bit of my predilection kind of the startup thing. I mean, so like I take it like early 2000s, late 90s, you know, we're kind of going through the Crucible, the.com era. Right. And so I was in, I lived in Austin, Texas and sort of, you know, I was just kind of managed to get into, you know, small startup circles and work for small companies and kind of always loved that part where it's like you're, you know, kind of, you know, you're always working with the owner of a business and the owner cares about, you know, the business and they care about the customer and they care about the product and there's kind of, you actually have the autonomy to do like you know, just a little bit of everything and actually have influence and impact over it. And you can see like your ideas take flight pretty quickly, you know, whether good or bad. And definitely went through some failed startup experiences around that time. Right after I got out of grad school, you know, I kind of did a startup and that didn't perhaps go anywhere. So I spent some time at bigger companies and that was good. Like learned a lot about like, well, what do big companies care about and what makes Them tick and sort of how do operations work at bigger companies? And so I'd say it was good to have that exposure at larger companies. But I'm not sure that, you know, I'm the kind of person that would like, you know, really be satisfied, you know, working at a mega corp for 10 years. You know, I kind of, you know, I always want to learn, I always want to have impact and I always want to, you know, kind of connect the dots between what I'm doing and what the customer gets out of that. And so just that, that kind of input that the insatiable need, you know, to like connect to like kind of the person who gets value out of the thing I do, but then also to explore just like everything I can about the technology and just the cur. Curiosity about it. It's hard to find that outside of a startup's, you know, kind of environment. And so I think it was always kind of my destiny to escape the bigger companies and to go back and just like bash my head against the wall at a startup, you know, over and over again. Like it's just sort of, that's where I get joy.

Nice. Nice. So it looks like observe I could be wrong. Really focuses on observability obviously by the name and scaling though. So why scaling is this? It seems to always be a hard, complex problem. Why attack what seems to be something that's really difficult, I guess to do?

Yeah, no, great question. I think it's always kind of this observability is a pretty established space. There are tons of tools in space like why on earth could we possibly need another observability tool? And I think the search is always for hey, who doing this thing is actually unsatisfied with their current solution and have unmet needs and are actually having a hard time because of the, you know, what kind of existing solutions like can't perhaps offer them and I think, you know, the lived experience for a lot of people so site goes get down at 2:00am and you know, it's actually, it matters like it's some part of the business that actually is important and like customers use it or it generates business or you know, revenue for the business or whatever. So like it matters and so people want to fix it urgently. But you know, the types of systems that we operate these days are pretty goddamn complicated. It's like, it's usually not just like one thing that went wrong. It's like, you know, it's like it could be any of a number of things that went wrong. Perhaps all at the same time. And it's usually not just like one signal that's like the thing that like was the root cause. Like you might need to like sift through a lot of data to figure out like, well, where actually did the problem really originate and what am I going to do about that? And I think the lived experience for a lot of people were a couple of different things. One is, well, you know, they'd have a bunch of different sort of signals and different tools for those different signals. And it was always a struggle to figure out, well, okay, you know, which one's going to show me where the problem is, by the way, who actually knows how to use that one. I might need to get like 10 people on a call, you know, to kind of start to figure out where the problem is. And in fact, you know, we have a lot of clients where it's like not just 10 people, it's hundreds of people, like on a war call, like trying to figure out like, who can see where the fire is coming from. You know, so that's kind of, that's one aspect of it is all the preponderance of different signals that are out there. But then also particularly at larger scale, just the quantity of data becomes inconvenient to manage. And I think sometimes people don't realize how much data is being generated by the infrastructure. Kind of in larger businesses it can be not just tens, but hundreds of terabytes of log data per day or a petabyte per day. And just moving it around is hard, but then certainly getting any value out of it is very hard. And often when you're trying to firefight these issues, like you kind of don't know what you need and where to look. And so, so there's, and so what people end up doing is they end up throwing out most of that data and they're kind of flying blind and just guessing around like, oh, let me just like turn this knob and see if it fixes it. It's kind of hard to actually deal with these situations, you know, kind of at large scale if you don't have the right data. And so our perspective on this has always been like, well, this is just fundamentally a data problem. Like this is a data management problem. And it's sort of a lot of the, I guess, you know, it's not like, like necessarily a new problem. Like, like kind of large scale data management has been something that people have like, you know, dealt with in other industries or in other, you know, contexts before. And you know, the solutions you Know, tend to be around the lines of like, well we need to get it all like in the same place or close together so that the right people have access to the data. We need to I guess like correlate the data or contextualize the data so that someone knows how to go from one breadcrumb to another. They don't have to like use their brain to figure out like where to look next. And so just organizing the data a little bit and then also just confronting the scale challenge. You know those like, hey, there's going to be a lot of data and the data volume's growing and if we aren't using kind of the latest and greatest technology to like cope with that, then we're going to have a hard time and we have to throw away all the data and we're kind of going to be stuck at square one. And so just seeing like all those different pieces, you know, we kind of. And also for what's worth, like recognizing there's new technology available, there's like kind of disruptive ways to go about doing this. You know, we started to just sort of see if we could make the match. You know, kind of what people are doing, let's say more on like the data engineering or bi side of the world. You know, they're talking about data lakes and data warehouses and data lake houses and they sort of solving their way problems they have in business analytics. Like hey, perhaps this technology and these techniques are something that could be of utility in the tehop space and sort of, you know, there's all sorts of ideas over there around like you know, hey, raw data. But then making silver and gold, you know, sort of versions of the data using cloud object store as the data store of records. You have like infinite scalability but it's also very economical. Like there's like these things that can actually be applied to this problem and just sort of we can think about, you know, sort of the data problem and the DevOps and observability in SRE space in similar ways. And you know, if we could do that, what, you know, what better outcomes could we achieve?

Absolutely.

And it really just always comes back to like hey, you know, I'm at 2 in the morning, you know, the like the world's on fire, I have 200 people on a call, like how can I make that whole experience better?

Totally. So I started my career as a performance engineer. It's not exactly this problem, but it was similar. So when I ran a performance test I'd have all this data metrics Collected from the mainframe, from the network, from the database. And you see response times go up. And now like, how do I correlate all this? So you had to get all these people on call the database person, look at it while you run the test. And so it's sometimes people suffer for not having enough information. But it sounds like in here you have too much information. So how do you do that? Correlation then is observed like a platform. Then that actually brings the teams together to find the gaps maybe in visibility that they may miss.

Yes. So a couple of different things. I mean, so at the end of the day, some of this is actually pretty easy. We kind of treat it as like almost just a relational data modeling exercise. We have all these different data. Within that data there tend to be correlation keys of some sort. And if we're talking about like structured logging and distributed tracing or something like that, then great, I have a trace ID and I can use that to correlate all the data. That's good. But even if you don't, you still have things like, well, which host did this run on or which pod did this signal come from? So that's a correlation id. So that's one way you can correlate the data. But then I think even more than that, the thing where we find a little bit more joy is like, well, often there's like a user ID in there or back in the day with mail logs. It's like you have a message id, right? And you can stitch the flow of data through all these different systems by using some correlation ID like that. And so our trick is to take all this data in to find those IDs and to basically just do a basic relational data model. And it's like, hey, this is a key that it can use as a foreign key to other data sets or it can use as a correlation key with other data sets. And so like that's sort of like the gist of how you like connect all this data and navigate around. And I actually. So that's kind of just the way that we think about that. But then I just want to go back. So you mentioned that like kind of behave performance engineering. You're doing all these tests, the benchmarking, you have all these different data, you want to correlate it all. Man, that like I identify with that so hard for a couple of ways. So it's kind of. When I was in grad school, what I was doing was computer architecture. Actually I was doing like hardcore like low level performance, you know, engineering optimization, working on Like HPC workloads and stuff like that. And you know, my day to day job was basically doing cycle accurate simulations of like chip multiprocessors and memory systems and stuff like that. And I was generating profoundly large amounts of like, just like all sorts of telemetry data from these systems and also correlating that with like, you know, benchmark runs on real world systems to make sure that the simulator does what we think it does, all that sort of stuff. And it was the same kind of challenge. It's like, man, I have like hundreds of terabytes, like all of this job like trace data and I actually need to, I need to correlate all these signals and I'm trying to do linear regression with all of it. And so it. But it also just basically boils down to like very basic things. I need to like correlate things in time and I need to correlate them by you know, whatever identifier, you know, is shared between these different signals. And so that was the trick for always doing that. And then, and for what it's worth, you know, I was using, you know, tools like Hadoop just doing all sorts of stuff to like do that large scale data analysis. When I ended up at Google, I was working, I was doing performance engineering on the MapReduce team. And it was just basically just like, hey, like I need to figure out like, you know, how to make these jobs faster or use fewer resources, be more efficient. And the way in which I went about doing my job, you know, in addition to like using profilers and all that sort of stuff. But it's like, hey, I actually just need to do analysis on the job logs. And so I'd spend all day long like looking at the logs of data coming out of these large, you know, workloads and trying to find, you know, hey, so how can I stitch together all the different pieces from all this big distributed system? Like how can I get all the relevant data together in one place and then basically just ask, you know, analytic questions about it and figure out, well, which properties of this job make it know faster than these other jobs kind of finding those signals. But it ended up just being like, like log analysis at the end of the day, large scale event analysis. And fortunately we had a lot of tooling inside of Google that was like incredible for this namely Dremel was an internal tool that like kind of everyone used for like large scale, like you know, kind of log analysis. And for us that that backend eventually became BigQuery. So they commercialized it basically as a big analytics database. And so, you know, kind of when we were, when I was like later on like thinking about what does this industry need? You know, it's like, well I'd love to have like everything that I had at Google, but like it doesn't really exist commercially yet until like the mid, you know, teens. And so you know, things like, you know, Snowflake and databricks and open source tools, you know, kind of like also coming out as well, they sort of started to resemble the things that you know, you could use adequately for this type of data volume and for this type of workload, you know, and sort of based on data that's sitting in an object storage. But they're also like these columnar analytics databases that can be very efficient for processing large amounts of this data. Very, you know, this. It's also have a relational data model so you can use things like foreign keys to relate data together with like correlation IDs. But then also we're becoming very good at handling semi structured data, you know, because like not everything comes in a nice table. It comes in as like slop, you know, just like raw strings and you gotta like figure out how to get the IDs out of those. Or it's like just like, you know, profound amounts of like random JSON data and everything's like different shapes and different, you know, attribute names and like it's just all slap. But these engines are becoming very good at processing that kind of data and doing it efficiently. And so that was actually a little bit of the idea here is that like, wow, there's actually, there is like some new stuff on the block that's like commercially available off the shelf that is very good at doing this type of analysis and correlating all those different like, you know, disparate data of all these random shapes and sizes. And so, so it's like, wow, we actually finally have the things like needed to solve like the really hard problems at scale that a lot of, you know, sort of like medium to large business, you know, toil with. And yeah, they devote engineering teams to build their own internal data pipelines to deal this kind of stuff. It's like, no, actually there's an opportunity to actually build a solution that we can just, you know, just hand to people on a platter. And so sorry, it's kind of a long winded answer. But you can kind of see like how you know, kind of there, there actually was this interesting confluence between this very hard problem at medium to large scale, particularly around the diversity of data and the volume of data, but then also like, oh my God, we finally have the technology that might help this. So now we need to kind of figure out how to make these two play nice with each other nice. And so that's kind of the idea.

So let's talk about those technologies you mentioned, data lakes. So is AI the secret sauce now that's helping to accelerate all these things? I know everyone slaps AI and everything. I think though, machine learning probably would apply best to data than probably a lot of other things uses.

So yeah, so I think that the answer, the truthfully answer right now is no, it hasn't quite like been like the differentiator yet. And for what it's worth, you know, the last couple of years it's been pretty frothy with like all the like the LLM stuff. But like, actually we're starting to see like, really cool things like happen with, you know, with LLMs and kind of what people are doing recently with AI. And so, however, there's like, maybe a way that I think about this that it's sort of like a little bit different and how do we actually bridge the gap between like where we are now and like where we could like optimally make use of these new AI technologies for this use case. And so I think the first observation is that, look, if you just like take like all your log data and like jam it into an LLM and ask it what's going wrong, like, you're going to fail. You're going to fail so hard because, like, it can't fit into the context. Like, it just doesn't make any sense. And so you don't really solve like the, just generally like the troubleshooting or incident response problem by just like throwing all your data into an LLM. You actually, you know, you kind of need to, I guess, for lack of a better, like curate it, like kind of figure out like, well, like, hey, you know, so of all the data, like what is the useful data by the way of the useful data? Like who are the like, interesting, like verbs and nouns, like kind of within that data. And so just kind of taking like all the data that we have and like kind of figuring out a way to express it and sort of turn in simple terms that are easier for a human to understand actually gets us a lot of a way towards, you know, what an LLM would actually be capable of taking advantage of. And I, we always thought about this, you know, kind of in the early days as like, you know, we're actually trying to make humans better at troubleshooting. And so the, you know, kind of one of the ideas that's really something that we've. We're kind of stealing from the bi space and trying to bring to the, you know, this domain is, you know, at the end of the day, I don't necessarily care about my raw logs or my metrics or my traces or any of this. It's just like telemetry. It's just the raw data. It's great. It's sort of like the raw factual material that I might fall back to. But ultimately I care about my user, like Frank, who like, just had a bad experience and like either had a page that didn't load or like a checkout that failed or whatever. Like, I care about Frank and so I want to ask questions about Frank. Right. And so like organizing and curating the data and elevating, like, kind of like the thing of interest is actually kind of one of the ways in which, you know, even as a human, it can help me organize my thought process. It can help me communicate with other stakeholders in the business. It's like, yes, this major customer, they had a problem, but here's everything we know about it. Here's the metric that sort of corresponds to that customer. And by the way, it's good now, so that customer is green, you know, and like, kind of just having like a way to talk about the business and what's fundamentally important about the business is like, how you facilitate better troubleshooting and better communication with stakeholders than dealing with like the raw data. So likewise.

Cool.

This is actually very, I would say, apropos for LLMs. It's if we can humanize all of this, like technical telemetry. And by the way, that also kind of makes us get it a lot smaller to something we could possibly jam into the context, one of these things, then, yeah, we might actually be able to take advantage of it. And so that's kind of where, like, I think just being thoughtful about the data management strategy and, well, what's the purpose of like bringing all this data and sort of like, how do we need to distill it in ways that are useful for this thing that I feel like no one in the industry has kind of cracked the AI nut yet. But we're starting to figure out, well, here are the building blocks and this and the intermediate problems we need to solve before we can kind of get the full realization of the power of these new AIs. And then I'll just like, I had like one other. Sorry Just stream of cautiousness. But like, like I think just compounding on this idea of like bringing more data together. And I think, you know, kind of like the thing that, that was the kind of original sort of like, you know, opportunity for us was like, well, we need to bring together like the infrastructure data and the log data and all these like vent data and we're going to bring it as business data. We want to bring it all together into a data lake so we have like some place where we can like have everyone together asking questions about it. But when you think about like what I want to have at my fingertips when I am in the middle of a firefighter, I want more than just like that telemetry data. Like I want postmortem reports from previous incidents. I want to know like, kind of generally like what is the slack discussion about this issue or this customer, this like module like Ben, I'd love to have runbooks at my fingertips. And so I think where all this like ends up going is actually it's, you know, solving all the data management challenges. That's great. But actually ultimately, you know, kind of where the destination is actually we can bring together all of this other business context and when I'm asking questions of my, you know, like, observability solution, but ultimately I'm trying to figure out like, you know, is it healthy or not, or is this user having a good experience or not. You know, if we can bring all of this contextual information in as well, then we can supercharge the ability, you know, of these tools to actually provide useful insights to me, to suggest to me, like, oh, I see that you're running this in eks. You know, I recommend you go look at this setting. You know, like, it can actually, if it knows that context, it can provide much more interesting, you know, sort of suggestions.

Yeah, yeah, I love that line. Making humans better at troubleshooting, not replacing them with AI, but actually enhancing them to do the job better. You know, in the pre show, we talked a little bit about knowledge graphs and how do they contribute maybe to maybe breaking down the silos you might see, and provide maybe a better unified view of, you know, observability data that we've talked about.

Yeah, well, I think it's in a nutshell, it's what I've been kind of talking about, you know, so far is that like, hey, I don't just care about my logs. I care about the things in the environment or the users in my environment, whether there's infrastructure or people or just entities in general, and I think about it very much, you know, people are familiar with Google and like it used to be the case decades ago, you type in a keyword, it gives you a bunch of search results. But then it became where you could ask a question about, say, the San Francisco 49ers. And it's not just going to be a bunch of search results. It's actually going to tell you something about 49ers, and it's going to tell you, like, here's the upcoming game, you know, or something like that. And so I think, you know, they used to call that the knowledge graph. And it was just the idea that there's actually, like, richer information and there's kind of, like more contextualized information that. That, like, actually may be relevant as an answer to this question. And so I think, you know, this concept, like, very much applies to, you know, kind of like where the industry is going, where this technology is going. It's like, it's more than just the raw data. It's actually, there are things about the business, there's things about the infrastructure that are ultimately what I want to know about, what I want to ask questions about. The knowledge graph is more than just like, kind of the inventory of those things, but it's also like how they connect to one another, you know, and so I know that, oh, the 49ers, well, that's part of the NFL, and so I can go to the NFL and find the other teams. And that's a way to browse the data. So it makes like a search space when you're troubleshooting that's like, sort of navigable. And then when you think about, well, what could an AI do with that? Well, first off, it has a context in which it can ask questions, but it can also an agency flow. It can sort of start to follow these links and figure out, okay, so if this system's down, well, what about the systems that are connected to it? Maybe I had to go look at those. And sort of like the human troubleshooting process there is very straightforward. And so the troubleshooting process for an agentic workflow ought to be similarly straightforward, so long as you can provide it with that context. And so the knowledge graph, I mean, it's. It's. I wouldn't say it's not like a single technological thing. It's just this concept that actually, you know, we're talking about more than just the raw telemetry. We're talking about things in the world. They are connected to each other. You know, in in all of our solutions, we just represent this as relational data model with like foreign keys between tables. It's no more complicated than that technologically. But the point is like, how are you using it and how do you like, you know, both as either as a human, but then also, you know, kind of what does that set up for the future for these, with these AI tools? So that's kind of why this concept matters. And it matters even before the AI stuff, you know.

Yeah, for sure. So besides that you mentioned it's before the AI stuff. That sounds like a mistake many companies might make. So what are some other common mistakes companies make? Maybe when trying to implement observability, especially at scale. I always hear about people struggling with scale. Why is that? What are some common really quick mistakes? You think they get this in place, then they're going to have a better time scaling?

Yeah, I mean, so I guess maybe one of the mistakes, and it's sort of near and dear to my heart, is like 10 or 15 years ago it was a case that logs were so expensive and just dealing with large scale event management was so expensive, it's absurdly expensive that naturally people are like, this isn't going to work, I can't do that. And so it went like hardcore onto metrics. Makes perfect sense. It just makes so much sense to particularly for a lot of just kind of real time monitoring use cases. Yeah, I mean, I don't need the raw strings anymore. I can have these aggregated sort of signals to work off of when I'm doing my learning. It makes perfect sense. But then I think, you know, people kind of like maybe swung the pendulum like too hard and now it's like, oh, like, well now like I can only see things at an aggregate level, but I can no longer figure out what's going on at the individual user level or at the individual host level or individual, you know, just anything like that. And so, so people like talk about the cardinality problem and so. But I think when people are talking about the cardinality problem, it's because they swung the pendulum so hard, you know, kind of towards metrics. Because at their scale, like there was no event management system that could possibly serve their needs. And so I think, you know, the truth is that right answer is somewhere in the middle. You know, you kind of, you want to use metrics for some things, but then, you know, for like the deep analysis and deep troubleshooting and you actually want to like look at things at a more granular level. You need something that can handle the Events as well. I think that's just something I've seen a bunch of times is folks that just like swung the pendulum over so hard and just sort of abandoned sort of the more granular data because they assumed that it was too expensive. But like that I think their dynamics of that have changed significantly in the past decade. And so that's a big one. And then the other, I guess the other thing is that you know, people tend to like when a grouped in teams grow and when lots of different job functions get involved, you know, everyone kind of brings in a different tool to kind of do their thing. Makes perfect sense. But it like kind of at some point it sort of precipitates like kind of some of the problems I was talking about earlier. It's that like, hey, the site's down but there's like 10 different like tools. We have to like look at this thing. I unfortunately don't know how to use nine of them. So like what do I do? I got to page my friend and tell him, get him to wake up and help me out. And so, and it's not just like on the technical operations side of things. I mean this is like support and usage analytics and it could be finops, it could be security analytics. Like it's all these different like parties that actually have a vested interest in the operations of the system. They all have like their own, you know, needs and they tend to, you know, need to like actually work with, with you know, kind of monitoring the systems or monitoring those data. And so you end up like duplicating the data. They end up like sort of making just like, you know, just hard trade offs and sort of compromises. Like one team sort of is at the mercy of the other. And so people run into a lot of, you know, trouble with that kind of stuff. And so I think that's, that's just a fact, right? That happens. Yeah. I'd like to think that what we're building is sort of, you know, sort of the start of the answer to a problem like that. You know, if we can load all this data into a data lake, load it once and we can give a lot of different people access to that data. And for what it's worth, like hey, if it's in the data lake, like hey, bring your tool to the data. Like, you know, like this should actually like ideally be an ecosystem around it. And I guess what I see is, you know, kind of long term this leads to somewhat of a disaggregation of this market. And so, you know, We've already seen this with the data collection and instrumentation. You know, like for the longest time I used fluentd for, you know, years before I even heard hearing this blunk. And you know, thank God OpenTelemetry has like started to gain traction and now like everyone can sort of see like, oh, this is my way out of you know, kind of proprietary vendor specific instrumentation and data collection. And sort of like we're seeing a commoditization of like the data instrumentation data collection layer. And that's great because that kind of means like okay, so cool. On the data collection side I can actually send the data to two places. And so if I have two different people that want to consume this data, that's fine. And I think that the next shoe to drop there is on the data management side as well. You know, it still is very expensive to load this data into all these different systems. And so like that kind of sucks. I'd rather economize that and load it into one system. Load it once by the way, like I'd rather not load it into a silo. I'd prefer to load it someplace where I still actually own the data. I can still actually do whatever I want with it. I can sort of apply my own governance policies on top of it. You know, sort of like that, that seems to be, you know, kind of what most people would want. And so the last kind of piece of the puzzle to fall into place is like, okay, so now we have open data, like sort like open data collection, we have it stored in like data lakes, ideally in open data formats. And then so now it's like, okay, now it's a place that other all the point solutions for the engineering operations or for customer support or for security, they could all come to it and sort of, you know, kind of put their own personalized view on top of it. But it's not like only one person has access to that data. So that's kind of, I guess the way I see that unfolding over time.

Nice.

Yeah.

So can I explain it and see if I understand it correctly? So it sounds like observe is it consumes all this data that all these other tools could feed in. But it's the source of truth almost that anyone can grab any of the data from and find information just from that one source of truth. Did I understand that correctly?

That's the idea. Yeah.

All right, nice. Cool.

Yeah. And I think I just going back to like, you know, like just on my personal journey, you know, I used to went as a early budding, you Know, sort of assistant administrator. I was building all my own personal, special, you know, my own like my own tools to like do a lot of this kind of stuff and my own tools that kind of help people kind of just try, troubleshoot and kind of analyze this particular part of the system that I was responsible for. But when I got this blunk, it was like, oh, wow, like this makes me better at my job because it's not only does it solve all the technical challenges, it actually helps me share a hyperlink with my coworker and we can all work for the same data. And so, and that's great. But that was logs. You know, I think just like thinking like more broadly about technical operations, particularly at scale. Like it's not just logs. I have lots of different data, there's lots of different business contexts I'm going to bring together. And so yeah, so it's. Where is the common ground that someplace someone can go, sort of the trusted repository of this data. I know if I go looking in this one place, I'm going to find what I'm looking for because everything is there, you know, and that sort of helps people figure out, well, you know, I'm trying to solve a problem, where do I go? You know, like actually where could I possibly find this answer? And if you have a repository that sort of people have, you know, I guess consolidated onto, then you have a prayer at answering that question.

Love it. Love it. All right, Jacob, before we go, is there one piece of actual advice you can give to someone to help them with their AI scaling DevOps efforts and maybe what's the best way to learn more about observe and contact you as well?

Oh, sure, yeah. So I guess the one piece of advice is obviously this. We're building on technology and stuff like that, but never ever forget that this is a people thing. Like, you know, sleep with the SRE book by my side. Like there are just like good principles and practices that have to come to bear to actually like run a system that people, you know, are would be happy to support. And so don't forget, you know, your blameless postmortems. Don't forget having like a regular reliability, you know, sort of review cadence. Don't forget to prioritize the work needed to make a system robust. You know, all of those follow up items that you find in your postmortems that like, oh, we ought to do this or that or it'd be better if we could monitor this or that. Like it's one thing to say, yeah, we're going to spend 10 to 20% of engineering time on that stuff, but it's another thing to actually live it. And there's no piece of software that's going to solve that problem for you. Like, you still have to like, actually, you know, do the responsible kind of like management of a practice like this. And so that that's like, maybe my primary piece of advice is like actually, you know, it is within your power to like do these things and there's just a little bit of leadership, you know, kind of required to kind of bring a team along and then for us, yeah, to find us, you know, LinkedIn's great. We're on LinkedIn. You can email me@jacobserverinc.com and yeah, our website's observe inc.com I have a link.

To all these awesomeness links down below. All right, before we wrap it up, remember frustrated users quit apps. Don't rely on bad app store reviews. Use SmartBear's Insight Hub to catch, fix and prevent performance bottlenecks and crashes from affecting your users. Go to smartbear.com or use the link down below and try for free for 14 days. No credit card required. And for links of everything of value we covered in this DevOps Toolchain show, head on over to test guild.com p187 so that's it for this episode of the DevOps Toolchain Show. I'm Joe. My mission is to help you succeed, creating end to end full stack DevOps toolchain awesomeness. As always, test everything and keep the good. Cheers. Hey, thank you for tuning in. It's incredible to connect with close to 400,000 followers across all our platforms and over 40,000 email subscribers who are at the forefront of automation testing and DevOps. If you haven't yet, join our vibrant community@testguild.com where you become part of our elite circle driving innovation in software testing and automation. And if you're a tool provider or have a service looking to empower our guild with solutions that elevate skills and tackle real world challenges. We're excited to to collaborate. Visit testguild.info to explore how we can create transformative experiences together. Let's push the boundaries of what we can achieve.

With lutes and lyres the bards began their song A tune of knowledge, a melody of code through the air it spread like wildfire through the land Guiding tester showing us secrets to behold.