Streamlining DevOps Focus with Ivan Merrill

Get ready to discover some of the most actionable DevOps techniques and tooling, including performance and reliability with some of the world's smartest engineers. Hey, I'm Joe Colantonio, host of the DevOps Toolchain podcast, and my goal is to help you create DevOps toolchain awesomeness. Hey, it's Joe. And welcome to another episode of the test skill, DevOps toolchain. And today we'll be talking with Ivan Merrill, all about streamlining the DevOps experience. If you don't know, Ivan is the head of solutions engineering at Fiber Plane, based in Amsterdam. And before joining Fiber Plane, Ivan actually spent about 15 years at large financial enterprises. Helped them or helping their teams really understand the power of monitoring and observability. So he really knows his stuff. Also, while leading large scale development of monitoring tools on rather important banking systems, he has a bunch of experience I think you're really going to get a lot of value from. So you really want to stick around. If you want to learn how to streamline your DevOps focus with SLOs and all these type of cool capabilities that Fiber Plane has, you don't want to miss this episode. Check it out.

Hey, if your app is slow, it could be worse than an error. It could be frustrating and in my experience, frustrated users don't last long. But since slow performance is a sudden, it's hard for standard error monitoring tools to catch. That's why bugsnag, one of the best production visibility solutions in the industry, has a way to automatically watch for these real user monitoring. It detects and reports real user performance data in real time so you can quickly identify lags. Plus get the context of where the lags are and how to fix them. Don't rely on frustrated user feedback. Find out for yourself. Go to Bugsnag.com and click on the free trial button. No credit card required. Support the show and check them out. Hey Ivan. Welcome to the Guild.

Thank you for having me.

Great to have you. So I always ask everyone, is there anything in your bio that I missed that you want the Guild to know more about?

I don't think so. I think that kind of pretty much sums it up. Really. Kind of, yeah. Far too passionate about monitoring observability. That's kind of my jam. Something I kind of fell into. And yeah, been doing it for a long time now. So yeah, love it.

So what drew you to a Fiber Plane then? What made you want to get Fiber Plane and really build something?

So, honestly, the first thing is that it's a really cool product. I really like the idea. So Fiber Plane's kind of core product is collaborative notebooks for SREs. And having been in, you know, large incidents and seeing that kind of every team has their own kind of data source and every team is trying to go away and check their own things kind of in private. And then kind of actually that ability to collaborate and everything was quite hard. And you invariably ended up kind of getting lots of links to data and screenshots and things posted in your chat tool of choice. And that was quite hard to kind of follow things through. Right. And understand what was going on. And I think just the idea of having the whole incident kind of investigated in the open is a great way to build trust amongst teams, which is so important.

Absolutely. So you have over 15 years of experience. I think SRE is newer than that. So what has the evolution been like? I know, like when we talk about observability. I used to be involved with performance testing. It was more like application performance monitoring. But that seems to have morphed. So any thoughts on SRE and this new kind of wave of monitoring?

Yeah, so I think one of the things that always kind of interests me with a lot of this stuff is that a lot of the things that we're talking about in the SRE world, a lot of principles have actually been practiced by a lot of really large scale organizations, you know, particularly in the banking world where reliability clearly, you know, it comes, you know, a failure comes with potentially a heavy fine and front page news. So things like chaos engineering that we used to kind of, that was part of our kind of go live process where someone would try and break your thing and see how it failed and you needed to kind of work out, you know, resiliency. We used to do regular site switches between kind of one data center, all of these things that kind of reliability practices that we've been doing for many years. But I think SRE really kind of codifies a lot of this stuff and makes it more accessible to people that aren't working in these kind of huge organizations that don't have the resources to spend on this thing. But in terms of the evolution. Yeah, I mean, in the monitoring and observability world, when I first started it was all infrastructure monitoring, Right. It was kind of CPU patrol and you know, it was what we used back in the day, BMC tool and was very much checking an amount of things, you know, like run a check. Is this CPU above a threshold? No, everything's good. And you do that, you know, kind of over and over again forever. Right. And it was very, very hard to, to equate anything kind of user experience wise. And then obviously we kind of got into APM world and that was great. And I used to go around and say to everyone, you know, your users don't care what your CPU metric is like. Right? They care on their experience. You know, interesting. Listen to that real user monitoring thing at the beginning of this because you know, that was one of the things that we were, we were interested in. We were, we were trying to roll out and that's been really great. And I think, you know, that focus is absolutely right. And now, you know, just the evolution from me into observability is really big because it's no longer become this check based thing. You know, application performance management is great and you do generate a lot of data and you know, you've got, you know, kind of different aspects to it. But the idea from observability, you know, the understanding of the state of your system and being able to be more interrogative is that word interrogative, you know, of your system is really powerful. Right. And I think it just also moves on the way that people think about these things. Not, you know, these tools and you know, these practices is not just something to do in an incident, but actually, you know, driving a lot of other really important processes, engineering processes and stuff.

Right? Yeah, I love it. I mean it almost seems like a mindset shift. Like you said, we used to look at CPU usage and disk usage and memory, but now it's more user. Do you think that's why? One of the reasons for that is we're moving towards like cloud native type applications where before we had systems under our control, where we know, okay, we know this can handle this, but now who knows like where we're scaling up, what containers are scaling up, what services we're using that are outside our control. And we're like, I can't even test this if I wanted to.

Yeah, absolutely. I mean, I think we've mentioned the word DevOps, which I think means by law we have to talk about the kind of cattle versus pets analogy. Right. I mean, such a great way, but it's exactly that. Right. I mean, I think there is this, you know, the whole kind of the architecture of what we're running. Right. You know, CPU memory monitoring and stuff like that. It didn't matter so much back in the day when there was ASIS admin that, you know, understood that pet. Right. And you know, when I do talks, I literally have a picture of My cat, Steven. Right. And it's like, you know, I know what time he wakes up, what time he likes to go to bed, what time he likes his meals. You know, like, if I give him a little tickle under the chin, you know, I know his behavior. Right. And that is, you know, very well understood. And the mental model that we can have of these systems when they were, you know, much smaller is more complete. Right. And I've been fascinated looking at some of the things around kind of reliable systems and, you know, Dr. Richard Cook, you know, kind of how complex, you know, complex systems fail. And all of this stuff is really, really interesting. And we are moving into this space of having complex systems where no one has a complete mental model. And that requires, you know, a fundamental shift in how we approach things and what we do to actually be able to detect, you know, if they're not working as expected, where and why and everything else like that.

Yeah, absolutely. So is that another reason why we see a rise in like service level agreements or objectives? You know, we've had them before, but it seems like they really are transforming the way we may have done alerting traditionally. So maybe. I know you've written a few blog posts on this as well. Could you dive into a little bit about like, SLOs, how we can understand them better, how they maybe be the preferred choice for modern businesses or something like that?

Yeah, it's. So Service Level Objectives, SLOs, as seen in the SRE book originally and everything else like that. I like to think of them as a bar of quality that a team chooses to hold itself up to. Right. I'm building a system. What is the bar of quality that I hold myself up to? That's always kind of how I personally have chosen to explain it. And I think that's an important thing to think about. Because service level agreements, they're things that you put into contracts, you know, you sign them with your customers, they are often legal. You know, when I was running a monitoring team, we had vendors that we use. They gave me an SLA. But SLOs are what we choose to hold ourselves up to. And that is a, you know, a very important distinction. And I think they are, you know, well, if you're done right, they are based on what is important to your customer. Right. And so what you're trying to say is, give me some SLI service level indicators of, you know, the really important interactions that my customer has with my service, my application, and what is, what is the bar of quality that we hold ourselves up to for those interactions? And there are a number of different things that you can use here. You know, kind of the number of errors that you serve to your customers, your number of the latency, in fact, are two of the most obvious ones. And it's about not being perfect because we all talk about, you know, kind of there is no service that's 100% available, right? It might be for a short period of time, but, you know, it's impossible to keep that up. So you can't have a hundred percent perfectness, but you can have. We often talk about kind of four nines or five nines or however many nines you want. So we want maybe 99.99% of our customers get a successful request. And that is the bar of quality that we wish to hold ourselves up to. And what it's doing is that it's kind of giving us a way to say every single interaction that our customer has was this good or bad. Did they get a 200 okay, or did they get a kind of 500 or above, or, you know, whatever is the way that you choose to think about it. And that's, again, really important. And one of the things that I found most powerful about kind of going around to engineering teams and implementing them is it's forcing people to have this level of thought that perhaps hasn't happened before. You know, they might have thought, we've got nice, nice HTTP error codes, that's great, but which ones do we care about? So, you know, a common one, like if we're getting customer typing in their password wrong, that, you know, I've seen people have that pop up as an error, but that isn't an error. You know, that's. So that HTTP code that you've provided there, I mean, it's appropriate for that. But we don't want to judge that as an actual error to us. So we need to. You kind of exclude that from our calculations. And what are the things that we can definitively say are good? And, you know, what does that leave that are essentially definitively bad? And that is a really powerful concept that then you can apply through and start measuring and do all kinds of crazy things once you do it. So, yeah, it's a hugely powerful concept and literally books have been written on it. But in a simple level, it's a bar of quality that you choose to hold yourselves up to.

I love that term bar of quality. So how do we know where we have our SLOs aligned with the customer really needs? Sometimes we have thoughts of this is what our customer wants and need. And sometimes we get so focused on that that we don't even change based on what we're observing in production. So how do you make sure that they're aligned with your customers?

Yeah, with difficulty. I mean, it's not easy. I think the first thing to recognize in this space is that SLOs aren't permanent. They evolve. Your service evolves. Right. I mentioned briefly Dr. Richard Cook, who was talked about humans being the adaptive part of the system. We are constantly introducing change to our system, right. Where new features releases up, changing our architecture maybe. So with that, it's right that our service level objectives change. If we add something that makes it more reliable and we consistently see that we are more reliable, then we can maybe increase our service level objective if we think we can regularly meet that. But ultimately it's about being not perfect, but good enough. And that will depend on, on service to service and that will be some interactions with customers. I mean, ultimately what we're trying to do is to make our service reliable enough. There is this great idea that I really kind of firmly believe in, that reliability is any product's number one feature. Right. And it's this idea of if it's not reliable, then users won't trust it. And if they don't trust it, then they're gonna look for alternatives. And, you know, and if your system has got no users, then what's the point in it existing? So we need to make sure that it's reliable enough. Now what is reliable enough? That is a very difficult question. And I don't think really I can provide any single answer because there will be, it will be different for different systems. I mean, I've actually known in the banking world that some services provided a response too quickly. You know, if you're kind of thinking about a decision around are we going to accept this person as a customer? Yeah. And someone receives no, then they would. I don't think I believe the computer here, right. It said no, like so I don't believe it. So in actual fact, in this particular case, right. Having a really quick SLO that's really, you know, got really low latency is entirely unimportant because your customer doesn't, you know, you, in actual fact, you want to almost add in a delay in this time. Right. So this is entirely dependent around the customer interaction. And I think part of the thing from the engineering world, and if it is engineering people that are pushing out the SLOs and things, is talk to other people in your business, you know, talk to your product owners, talk to your marketing people, talk to your customer support people. This should be a thing that it's a bar of quality that you are providing. But it's, and it's what your team can provide, but it needs to be done in conjunction with other people because it's no point me saying that, you know, my bar of quality is four nines, but actually our marketing team are expecting five nines and are talking about we're being the most reliable, whatever. So yeah, there's not a right answer, but it needs to be in conjunction with everyone else. And yeah, normally I think, you know, it's pretty quick to find out if your users aren't happy.

No, it's a great point. Especially like you said that example, you gave the customer what they're expecting and even though it is technically slow, it may not be slow because, you know, like you said, that's a great, great. I love that example. So how do we get this information though, fed into our SRE teams and our DevOps teams? Is it all hinged on observability and how it's baked into our infrastructure and applications and how we're able to get this data to be able to come up with these better and better SLOs?

You're asking an observability person whether it's all based on observability? I'm going to say yes, it absolutely is. Always. Observability is the answer to everything. Yeah, I mean, let's face it, right? I mean, I'm not sure it's entirely true, but wasn't that, was it Robert McNamara made that quote, you know, what you don't measure, you can't improve. You know, there is that quite famous quote, isn't there? And whether that's true or not is, you know, up for debate. But it's certainly very hard to measure or to improve something that you don't measure. Right. So I think there has to be a grounding in it. Now that's not to say that you need a full on absolutely perfect monitoring system. To me, fundamentally for an slo, for the actual kind of the maths bit of the slo, you need a way of deciding for everything that you're measuring. So normally that's user requests is something good or bad because ultimately what you're trying to do and one of the most powerful aspects of service level objectives is providing a common language to the entire business of, you know, reliability and whether we were reliable enough. And in the DevOps world that means, you know, actually aligning devs and ops on Was this good or bad? So kind of a simplistic level. If you have the ability to take whatever is your unit of measurement and say this one was good and this one was bad, then you have enough. But hopefully, you know, that will then evolve into maybe more kind of granular slos or kind of better quality slos or anything else like that. I mean it's, you know, certainly something you can start off with quite simply and then evolve as you get more experience.

Great. You know, last time I spoke with the folks at Fireplane, I think it was like November. And I, I noticed recently you had a new observability. It looked like feature called Autometrics. So I know we talked about SLOs, why observability is so important. So is autometrics one of the solutions that someone can use? I believe it's open source as well to help them with observability.

Absolutely. And that's kind of part of the reason that we created it, invented it and put it out there for the world to use. So when speaking to lots of different companies about, you know, kind of the situation for them and their organization around, you know, monitoring observability, how well their applications were instrumented, we kept seeing similar kind of patterns of issues. Right. Actually companies were struggling to get good instrumentation in the first place because maybe developers or you know, were unsure about how to do it or didn't know what to instrument. Even like things like naming your metrics is difficult and then particularly, you know, once you've got those metrics, how do you go about querying? There's, you know, like there's so many different tools, each of them has their own domain specific language, you know, like everything else like that. So what Autometrics is a function based metrics for a few different languages. So what you do is you decorate your functions within your code base, within your ide. And for that you automatically get the number of invocations of that function, the number of errors from those invocations, and the latency of the invocations. And actually it's a very. Because it's quite a kind of curated way of doing this because the metrics that are created are given a very specific name. You can actually then follow it through. If you know the metrics and you know the number of errors and the number of invocations, then you can start to work out a percentage on the success rate. And if you know the latency and you're given a latency target, then again you can start to work out are we meeting that target, which then means that SLOs can be very quickly generated based on those metrics and you know, alerts based off those SLOs. So it's a very kind of, you know, great way to get started with observability. A really great way to kind of get, you know, some pretty good observability. Very simply. And I think one of the. Well, there's a couple of really, really big benefits to me. One of the things that I really like is that it takes away observability being this kind of this problem to overcome this thing that you need to do, this thing that requires a lot of effort because actually, you know, it's really simple to go and decorate your functions, you know, with the autometrics decorator and then, you know, to not have to think about anything else. But from that get good quality metrics that particularly metrics that make sense. You know, I briefly touched on mental models at the beginning of this. But you know, for someone that is, you know, writing the code that is building the system, their mental model is based upon how they interact with it. And that's their code, right? That is the functions that they write. So providing metrics that are in that kind of paradigm, you know, that makes sense to them that align with their mental model means there isn't really translation to be done between someone writing code and you know, the metrics that are output. And that's incredibly powerful in helping people kind of quickly easily consume those metrics and actually do useful things with them.

Now this just popped in my head. Is this also a way for sres to maybe open up communication better with the developers because developers are the one decorating their function. So it almost opens up a, a dialogue between SREs and developers. Almost like a backhanded way to get them to talk to one another. Like I said, I just, just popped in my head.

Absolutely. I think it's certainly in my experience it can be quite hard to go out to some of the engineering teams and say you need to instrument your code. You know, it's not actually a very fun task. Right. No one really kind of wants to go through and say, you know, today we're going to be kind of adding metrics to our code base. Woohoo. That's not something that, you know, people really. So being able to kind of as you as and when you touch your code base, maybe you're refacting some code, maybe writing a new feature, but just being able to add in this decorator, nice and simply get the metrics means that it's really easy, but it enables, as I said. Well, firstly, you're getting some really great metrics that make sense to both the SREs and the developers. I think that it helps the SREs understand the code base better because the metrics that they're getting, you know, consuming are based on the functions of the code. Again, it's this idea of helping people speak the same language. Right. Which is really powerful.

Absolutely. So how does this all tie in then to. I think one of the main objectives of Fiber Plane is to help people kind of streamline their focus, especially prioritize service level objectives and SLO capabilities like that. So I guess, you know, obviously with autometrics and all this stuff baked in, it's going to help you prioritize or get more information to be able to prioritize. But is there anything else that folks will be able to do with Fiber Plane to help them with the service level objectives or anything around this area?

Yeah, it's a good question. So, I mean, SLOs are the basis of such so many kind of important things. If done rightly, they're not just a way to say we're reliable enough or we're not reliable enough. And we briefly touched on the fact that you can base your alerting on the back of that. You know, you've got. Alert fatigue is a very real problem in the SRE world and everything else. But in terms of streamlining, you know, kind of the, the overall process and developers and DevOps things and everything else like that, it's also about. Well, firstly there is a prioritization, right. You're able to kind of make a better understanding of do I need to prioritize reliability or do I not? And with the Fiber Plane notebooks, you've got a continual idea that. So there's a notebook, you've got an incident you're going through, you can understand, hopefully you've got your application instrumented with autometrics and everything. So you can see things like your error rate and therefore your burn rate, which is another concept, maybe for another day, kind of going into the complexities of service level objectives, but you're starting to see kind of which incidents are costing you a lot of time, a lot of money. And you've got a, I think of the fireplane notebooks as, as kind of incident artifacts. Right. Kind of looking at, as you're investigating an incident, you are recording that investigation as you go through, which I think is really cool. And as I said, one of the reasons I kind of wanted to get involved with this company. So if you're doing all that stuff, then you can start to really make an, a more data driven decision around do we need to invest in maybe our release process? Because the last five incidents that we've had that have taken so much time have been, you know, around releases. And so is it right that we put in maybe more automated testing earlier on as we go, you know, through our continuous integration cycles and all that kind of stuff? So SLOs start off as this thing about kind of measuring reliability, but once you start to kind of bring them in and use them in a more thoroughly throughout the organization, as you get more mature with them, as you start using, you know, them as a basis for do I have an incident, you know, investigating those incidents, understanding, you know, what is the cause of this incident, what is causing us to reach our SLO target, then, you know, they've become such a wider, more powerful thing that can really bring a huge amount of benefit. So yeah, many things come. I mean, I could talk about this for a, a long, long, long time, but definitely there are lots of opportunities there.

Absolutely. So it sounds like a lot of data, just like performance testing. You have a lot of this data and sometimes it's hard to tell, okay, what does this mean or which one should I prioritize? So any tips on how to turn maybe some of these complex SLOs and metrics into more clear, pretty much actionable insights? Is there any sort of visualization baked in or anything like that to help the person figure these out really quickly or triage quickly?

Yeah, so certainly if we come back to things like Autometrics, there is, you know, an open source, but part of that open source project is what's called Explorer, which is a really nice, simple UI to say these are your service level objectives, these are the ones that on track, these are the ones that are off target. And again, don't forget that each one of your service level objectives is, you know, should be, if done correctly, based on customer experience. Right. So and it's, it comes down to a percentage, right? It comes down to a very kind of simple percentage. So you kind of, if you've got two SLOs and one of them is at 95% and one of them is at 97%, it is a pretty kind of clear thing. This is part of the common language they provide, go, one is better than the other. So yeah, there is things like that and actually there were even some Grafana dashboards provided there as well. So once you kind of get to the point of, you know, you're collecting the data. Yeah. There's a number of different ways that you can really visualize it and just become. I think everything is easier with a picture. Right. So, yeah, getting that clear view is really important. So, yeah, the Explorer find dashboards. There you go.

Awesome. Okay, Ivan, before we go, is there one piece of actionable advice you, you can give to someone to help them with their DevOps SRE efforts? And what's the best way to find contact you or learn more about fiberplane?

Okay, quite so. The first one is probably the easier one. So a couple of things we've talked a lot about. Autometrics. That's autometrics.dev get involved. As I said, open source. We'd really like to see people on there. Come join our discord and then you can go to Fiberplane.com to sign up for the notebooks. In terms of one bit of advice for SREs and things, I think one that I've given before is this is quite a new practice, right. And I think that we all have to understand that we are continually learning this, right? And I think don't be afraid of failure and understand that that's part of the learning process. I think that service level objectives really do kind of epitomize that. Where you might start off with a service level, one service level of objective, but it will continually evolve and you should continually evolve. And you know, that's kind of something that we must all accept, right? I mean, I always like to think that reliability in our tech world is pretty new compared to kind of aviation or, you know, kind of some of the other, you know, health and everything else like that. And a lot of the people that we're taking inspiration from are from that world and they have an awful lot more experience with it. You know, a lot to learn from it. We are constantly evolving it. So don't be scared of it. Give it a go and understand it. It probably won't be perfect first time, but that's okay.

And for links of everything of value we covered in this DevOps toolchain show, head on over to testguild.com P132 and while you're there, make sure to click on the Smart Bear link and learn all about Smart Bear's awesome solutions to give you the visibility you need to deliver great software. That's smartbear.com so that's it for this episode of the DevOps Toolchain Show. I'm Joe. My mission is to help you succeed creating end to end full stack DevOps toolchain awesomeness. As always, test everything and keep the good. Cheers. Hey, thanks again for listening. If you're not already part of our awesome community of 27,000 of the smartest testers, DevOps and automation professionals in the world, we'd love to have you join the fam@testguild.com and if you're in the DevOps automation software testing space, or you're a test tool provider and want to offer real world value that can improve the skills or solve a problem for the guild community, I'd love to hear from you. Head ON over to testguild.info and let's make it happen.