Collaborative Notebooks for Debugging Your Infrastructure with Micha Hernandez Van Leuffen

Foreign.

Get ready to discover some of the most actionable performance engineering testing insight reliability device with some of the world's smartest engineers. Hey, I'm Joe Calantonio, host of the Test Field and SRE podcast, and my goal is to help you succeed in creating application performance awesomeness.

Hey, it's Joe. And welcome to another episode of the Test Guild Performance and Site Reliability Podcast. And today we'll be talking with Mika all about collaborative notebooks for debugging your infrastructure. If you don't know, Mika is the founder of Fiber Plane. He previously exited a company focusing on CICD to Oracle. Mika also has dedicated his career to improving the workflows of developers. And now with Fiber Plane, he's building a collaborative notebook platform for DevOps and SREs. So we're going to dive deep into this and you don't want to miss this episode. Check it out.

This episode is brought to you by the awesome folks at SmartBear. Listen, we know load testing is tough, but necessary. So investing in the right tools to automate tests, identify bottlenecks and resolve issues fast saves your organization both time and money. And that's why SmartBear created Load Ninja, a SaaS load testing tool to help teams get full visibility into performance so you can release quality softW software faster than ever, make testing effortless and give it a shot. It's free and easy to try. Head on over to loadninja.com and learn more.

Hey Mika, welcome to the Guild.

Thank you for having me, Joe.

Awesome. Before we get into it, is there anything I missed in your bio that you want the Guild to know more about?

Yeah, I also run a fund called NP Hard Ventures which is focused on investing in pre seed and seed dev tools infrastructures, building block for tomorrow type companies. So same wheelhouse, but then on the investing side.

So that brings to my first question. It's not even related to what I was going to ask you. I'm always curious to know how founders they sell something Oracle. Sounds like you invest in other companies. Why start up another company?

That's a good question. Yeah, so I guess sort of the origin of Fiber Plane came from, I would say sort of our frustration that we experienced in the previous company, which was Wercker and but also inside of Oracle and that's specifically around sort of debugging our infrastructures and we can dive into that more. But I'd say sort of, yeah, this sort of this thing crunching away in my head and some of our struggles that we had was like just, you know, I wanted to pick it up and start a new company around it.

Great. So the company was, I guess, original that you sold to Oracle was CI CD based. This is sounds like more of an SRE solution. So I guess the first thing, it seems like a lot of companies are going cloud native. How has that changed? Maybe the landscape of DevOps engineers that maybe were solely focused on CI CD and now maybe SRE with things that they struggle with now that you think maybe they weren't struggling with before.

Yeah, so I'd say, you know, Kubernetes and containers and, you know, even serverless, I think they've become more prevalent. You know, when we started Worker, the previous company, we were super early in that sort of container journey. The original version of Wercker was actually running on lxc, not even Kubernetes. There was no Orchestrator. You know, once Docker came on the scene and. And Kubernetes, we completely moved to that. And that was around, I think, like 2014 when we made that move, you know, initially on CoreOS fleet, so we're using Containers, but there was no Kubernetes yet. And then later on, I think 2015, 2016, going full on Kubernetes. So I think, you know, I think the worker story is sort of indicative of what other companies are experiencing now. It's like, well, great, we now got. We are full on containers, we're full on Kubernetes. Now we've got a bunch of new problems. Right. And specifically around the observability and SRE space.

So does it also make it more complex for debugging applications and services as we go to cloud native? Because maybe they're not all under your control.

Yeah, exactly. So that was actually the reason. Well, one of the reasons for starting Fiber Plane was, you know, how hard it is now to debug your infrastructure. Right. So, you know, one of the challenges that we had at Worker and inside of Oracle is like, great, you know, we're running the CICD system. It's composed out of a bunch of microservices running in a multitude of containers, you know, abstracted on top of aws. All these layers of abstraction, multiple moving parts. Like, how do you reason about your infrastructure and all these services running inside of it? Right. And that for us, you know, was quite a challenge. Cause we, you know, so we were doing CI cd, so we were running users arbitrary code inside of this distributed system. And that makes up for some great downtime occasionally or, you know, incidents or stuff that we need to debug. So that was part of our frustration, like we need better tools to sort of to reason about this stuff and that was sort of one of the reasons to start a new company.

Nice. So also, I think maybe before Cloud Native maybe you didn't find issues as often as you would in production and therefore people maybe didn't debug in or test in production as often. Is that another reason why you see a need for a solution for people to be able to like do production based, not development but production based debugging and things like that?

I think to some extent, yeah. I think also in CI CD is sort of the move towards agile, whatever that means, has sort of accelerated the need for that as well. Right. We're building these things very fast. Instead of going from this sort of 1.0 to 2.0 version, we're doing these incremental updates. Developer velocity has increased. You kind of like you are testing in production, right. Sort of. Obviously you know, you shouldn't do that, but we're moving quite fast and that also has resulted in sort of this need for better tools.

So, you know, I don't know where I saw this. Something to do with dashboarding. A lot of companies focus on a dashboard like one place where everything can be done. I thought you had a blog post maybe on something on the single pane or the myth of everything in one place. Can you maybe talk a little bit more about that?

Yeah, I think that's sort of one of our, part of our, you know, our thesis and sort of DNA of the company. And this is not to bash on the dashboarding companies out there, but dashboards are great for when you know in advance what you got to measure. Right. And which effectively means what can go wrong. Right. It's sort of this cockpit view of your infrastructure. You've set it up in advance, you've reasoned, you thought every scenario through, you've reasoned about everything and you put this thing, put this into a dashboard and then you know, nowadays maybe less so, but you know, originally people would put it on TV inside the office that nobody looks at. Right. And I think nowadays, and that's again part of our thesis that, well, that's great for the known knowns but not so great for the unknowns. Right. And the unknowns are more prevalent now because of all of the reasons that I just mentioned. Right. Sort of these move towards microservices. We're running things inside of containers. It's this sort of orchestrator system like Kubernetes, that's you know, scheduling all of these Things we've got users acting on our services, right? Yeah, you don't know everything in advance. So what we're thinking is that, well, you need a more explorative form factor to reason about your infrastructure and your services. Right. And sort of obviously inspired by Jupyter notebooks and you know, the data science space, let's use this notebook where we sort of can gather all these different your metrics, your logs, your traces, possibly also other types of data, your deploys, your builds, and then put that in this explorative form factor and then reason about it.

And so I assume that helps with collaboration. Is that one of the big benefits this helps with SRE collaboration?

Yeah. So that kind of brings us to, I guess the other origin point of the company is that, okay, great, there's better form factor for looking and debugging your infrastructure, that's one thing. But the other thing that I thought was striking is how we've benefited from all this collaborative software out there. Right. Notion for productivity, Google Docs, of course we got Figma in the design space, you know, Framer for building websites in a collaborative manner. But it's like weird how actually developer tools and then specifically DevOps tools have not really benefited from this move towards collaborative software. I think honestly like the things that we got right, we got the pull request, which is quite old by now, I think since GitHub came around and we have pair programming, collaborative programming in Visual Studio, Code Live share, some of these collaborative ides. But other than that we haven't really benefited as a sort of industry, right. DevOps or developer industry from this whole move towards collaborative software. So that's the other piece that we bring to the table is that explorative form factor, but then also in a collaborative way, right, where you can mention people as you would in Slack, you can sort of start discussions, you can assign people and you have this sort of system of record and shared visibility into your infrastructure and then how people are working on that infrastructure or reasoning on that infrastructure.

And I assume the reason this is also critical is like you said with the dashboard, it's like no knowns SREs, a lot of unknowns what's going on in production. So maybe you need this real time type of collaboration. Is that why this is so important?

Yeah, and I think the other piece of it is, you know, the status quo of you and I perhaps looking at an issue inside of our infrastructure is, you know, you start a zoom and you share maybe your datadog screen, which is another dashboard and then you and I talk over it, right? That's the status quo right now. But I think also Covid accelerated that thinking quite a bit. Like, okay, we need collaborative software, right, front and center to do this in a more collaborative and shared visibility and transparent way.

So I guess to really be able to dive into this more, I guess we need to talk a little bit more about Fiberplane. What is Fiber Plane? And then maybe we can say how that helps with collaboration or tools like Fiber Plane helps with this process.

Yeah. So imagine very much a Google Docs like environment, right? You've got this writing, collaborative editing experience. That's sort of one thing, right? And I can like Slack, as I mentioned, or notion, you can add, mention people, attack people. You've got your basic writing capabilities. Then where it gets interesting is we have the notion of what we call providers, which are effectively plugins that allow you to fetch data from your. Right now, your observability stack, right. So we support for instance elasticsearch where you can fetch your logs. We support Prometheus where you can fetch your metrics. We've got a Sentry provider as well and a few others up our sleeves. So this allows you to sort of suck in all these metrics, logs, traces inside the notebook form factor. You know, present this type of data, be it in a, in a chart that is like real time. It's not a screenshot, right? Like, or an iframe, that is actual fiber plane rendered chart of your metrics. Or we allow you to get some logs in there, display them, filter them, also sort of highlight them and sort of say, hey, these log lines seem interesting. It could be the culprit of the thing that I'm investigating. You're able to sort of save that and build up that system of record. So that's sort of in a nutshell what fiber plane is. Then on top of that we have templates. So very much like a notion or any other type of, sort of productivity type piece of software where you can sort of codify that knowledge as a template or as a runbook, right. When you're running a sort of investigation, go through these and these and these steps. But also before you dive into anything, at least look at these charts, right? Or these logs by default and you're sort of able to codify that inside the template and you know, render these charts or logs beforehand. So great example here would be, you know, say a pagerduty alert goes off. That's something that we integrate with automatically. A notebook gets created based off a template where You've maybe codified, you know, the service name that is down, which could be like the billing API. Then we're able sort of to render some charts related to that billing API, give you some logs related to that billing API and then getting the right people inside the notebook and then start debugging that thing.

So it sounds like it saves a lot of time for triage because it's only bubbling up the charts that specific to that type of incident. And then does it automatically alert people that maybe are tagged for that type of incident so you know who to even get on board to help collaborate with?

Yeah, so right now, indeed, within the product, within Fiber Plane itself, people get notified that they should, you know, do something or. Yeah, you can sort of codify the mentions as part of the template that, you know, always Joe should be in this thing and help out.

Very nice. So does it do any sort of automatic triaging for you as well? Because it has all that data. Does it bubble up insights and say it sounds like it'll throw in the correct charts or places, but does it do anything that says, hey, maybe you need to look at this. This has been the way it was for the past month, but now we see a spike here, so that's where you should start looking at or collaborating on.

Yeah, so I wouldn't say we're doing anything in the ML vein yet where we sort of make suggestions, you know, based on certain data or spikes that we see. What we do support is another good example of like the provider model, the integrations is where you're capable of actually pushing events. We have an eventing API push events to Fiber Plane. Right. And a good example here would be deploys. So say you are using GitHub Actions for your CI CD and you're doing deploys. Each time you do a deploy for your service, you're able to sort of actually call an action, a fiber plane action that we've built. And it sort of pumps in that deploy data maybe with some labels attached to it as well, which is another sort of powerful concept within Fiber Plane sort of label that event. It's the billing API, this version. And what you're now capable of doing is sort of overlaying the events with the metrics. Right. So now maybe you see some kind of spike of 5002 and then before the spike you see a big fat red line which is your deploy. Now, you know, hey, maybe these things are correlated. Right. Like I just did a deploy of this service. The 500 error rate has gone up. Maybe these are Related So at least give you, you know, being able to correlate these different data types. Right. That's an important concept inside Private Plane.

Cool. So this whole concept of templates, I assume it's good for incident responses. Are there any other uses for templates? So you see been popular.

Well, I think the incident response is one thing I would say sort of runbooks like how we sort of need to do certain. Perform certain actions to our infrastructure which might not always be an incident. Right. Maybe some kind of upgrade or rollout. So you're also able to codify best practices processes within your organization as a template. Right. And that's not just useful for this incident response use case, but also maybe for onboarding. Right. New people join the company or new people join the team. This is sort of the process that we go through when we deploy a new service. So you're also sort of capable of codifying that knowledge and building up that system of record.

Very nice. I think we also talked about how maybe a lot of tools around observability and monitoring kind of in siloed environments. So it sounds like this also helps unlock that for the whole team to get these insights.

Yeah, exactly. I mean that's the other piece, right. Where for maybe for your logs you go to Kibana and for your metrics you go to Grafana or Prometheus. This sort of brings in all these sort of different data sources into one form factor and one space to work and again make it actionable with, you know, through collaboration.

And this is different than a dashboard because it's real time, specific to what's happening in your environment at that time.

Yeah, I think dashboards are still great for information gathering. Right. I think if you look at sort of the categories of observability, you've got, you know, the alerting piece, the notification piece. Right. With Pagerduty as a. As an example, you've got Datadog in the dashboard space or Grafana where you know, it's great for information gathering, looking at that again that cockpit view of the entire system. But when it comes to action and you need to do something having actual intelligence related to that specific service across different types of data. And again it could be metrics, logs, traces, but also as I pointed out, right. We support sort of this eventing mechanism allows you to sort of have all these sort of disparate types of information into one place and make an actual.

Very cool. How long has Fiber Plane been around for?

Yeah, it's a good question. So I think we started a company around two years ago, a little under two years with not a single line of code written. So we, yeah, we've been building for quite some time. In all honesty, there's also a lot to build because we're building both beyond like this sort of rich text editing environment. Right. As you would come to expect from a notion or a Google Docs, but at the same time doing this infrastructure piece. So we're kind of building two startups at the same time, I would say. Turns out building a rich text editing experience is not trivial. So we've invested, you know, a lot of time and effort, you know, building our own editor, as I said, sort of provided a plugin model, being able to gather the different data sources of these different systems. Yeah, there's a lot to build.

So there's a lot to build. So is the reason why you didn't have a line of code is because you want to learn as you go along rather than have a preconceived notion of what people want. Like, did you have a preconceived notion of what you were going to build? And then after two years, like, oh, wait, to watching the customers use this, we realize it's actually here is where they really need help with.

Yeah, exactly. So I think we have this sort of thesis around this form factor and you know, when we spoke about fiber planning, you can almost like visualize what that means. Even if, you know, say like collaborative notebooks for DevOps and SREs, you can kind of fathom what that could look like. Right. But then indeed we did a lot of customer discovery and talked to a lot of companies and how they're dealing with their infrastructure, how they've set it up, how they're dealing with sort of incident response and debugging their services, what types of tools they're using, all of that. And then, yeah, we sort of adjusted the roadmap accordingly and sort of this is where we are now.

Cool. So we're almost in the end of October. So I always ask people this regardless of what the interview is. 2023 is anything you see on the horizon that maybe site reliability engineers need to be aware of or maybe get on board before it takes off?

Yeah, I mean, obviously Fiber Plane will be coming out in a couple of weeks, so I'm very excited about that. So we'll be doing sort of our, our launch so people can sign up, take it for a spin, and then particularly we're super interested in getting feedback around what providers. Right. What plugins are people interested to see. So again, Right now we've got elasticsearch, we've got Prometheus, we got Loki, we got Sentry, perhaps Cloudwatch has been a much requested in talking to sort of our first users. So very interested to hear from the folks out there what they want to see in terms of new integrations to build.

So what's the model going to be? Are you going to have like an open source version? Is it all a paid solution? Is it both?

Yeah. So initially it will be freemium, you can just take it for a spin. And then I expect us to sort of introduce the business model in the new year. We've got some ideas on what that should look like, but obviously also very keen to get feedback from the users and sort of talk about pricing and what seems reasonable and all that. And then on the open source front, we will be open sourcing our provider stack. So the plugin model, sort of the SDK provider development kit to build new providers open sourcing our template templating stack as well alongside our cli, which we haven't really talked about yet. So we also have a companion command line interface which is quite interesting. Like, apart from the usual stuff where it allows you to interact with the Fiber Plane managed service, you know, you can create notebooks and initiate power published templates and initiate triggers and create notebooks from the triggers, all of that, but it also has two interesting features called FP Run is one thing. So, and this is for a scenario where usually like when people are debugging their infrastructure, you know, they might be in, you know, AWS and dashboards and all of that, but usually they also operate from the terminal. Right. Like they're behind their notebooks and typing things, using all these different command line tools to interact with the infrastructure. Like Kubectl is a good example. Right. But the downside of that is like from a team perspective, I have no idea what Joe just typed to sort of help get the system back to normal, right? So we've got this thing called FP Run which allows you to pipe the commands to a notebook, right? So you're able to sort of pipe kubectl, get pods or get logs and able to pipe that into the notebook. But on top of that we kind of can parse the data. So instead of just like rendering plain text, we can understand that, hey, this is actually a log and we're able to sort of visualize it as a log, we're able to sort of filter it and as you would sort of in a logging product. So that's Interesting. And then the other thing that we have is shelter cells. So imagine the same scenario, but it's more for long running sessions. Right. You're doing a bunch of stuff from within the terminal. You're able to sort of record that entire session, all the commands that you're typing for prosperity. So you know, your teammates can later see, hey, this is sort of the path that Joe went through.

Oh, that's cool. It's almost like a knowledge base. So if incident happens again, you're like, wow, all right, this is what fixed it before. Can you replay it and it'll automatically type the commands again for you and all that.

Yeah, exactly right. And sort of codify that process and indeed built up that system of record. And then of course maybe you can, you know, save that as a template. And now indeed next time something happens, you know what to do.

Love it. So is there anything else we missed for functionality? I mean the command line sounds like right up an alley for developers. They're always using the command line. This seems like just normal extension.

Yeah, exactly. No, I think we covered most of it. So yeah, the providers again, you know, we're starting off with obviously observability tools so support for Elastic Prometheus and as I said, you know, CloudWatch is coming up but it's going to be interesting, you know, doing looking into non observability providers. Right. So GitHub has a great example. Obviously looking into Slack as well. We're able to sort of interact with Fiber Plane from within Slack. Nice.

So as I mentioned in the pre show, I am not a site reliability engineer. So if a site reliability engineer is listening to this, so what is like the grabbing thing you find make people gravitate towards Fiber Plane or would want to try it out when it's officially released, is there like one like game changing functionality? You think a site reliability engineer, when they hear, they're like oh, that makes sense, I need to get my hands on this.

It's just easy to get started with it. Right. Like it's a very friendly, if you look at the product like it's a very friendly environment which actually you know, geared toward perhaps stressful situations. Right. You know, if something is down or you need a sort of tagline that we use a lot. So when everything is on fire, it's actually a very pleasant experience to use because it looks very much like a sort of consumer enterprise product, has a very great usability, good developer experience.

So it seems like if a Fire lab goes off rather than like you know, looking all over. Okay, where do I look? Where do I start? This is like this will help get them in line and what to focus on.

Yeah, and you know, as a team allows you to remain, you know, calm and collected.

Nice. Love it. Before we go, is there one piece of actual advice you can give to someone to help them with their SRE testing efforts? And what's the best way to find or contact you?

Good question. Or you can contact me via Twitter. I'm I M I E s on Twitter and Fiber Plane is well at Fiber Plane on Twitter. So feel free to reach out, shoot us a message and in terms of, you know, advice or just sign up for the product, I would say we're happy sort of to do an onboarding call, get the feedback. Curious to sort of to see you go through it and looking forward to help us inform our roadmap.

Thanks again for your performance testing awesomeness. For links to everything of value we covered in this episode, head on over to testguild.com P1.02. And while you're there, make sure to click on the Try Them Bolt Today link under the exclusive Sponsor section to learn all about SmartBear's two awesome performance test tool solutions, Load Ninja and Load UiPro. And if the show has helped you in any way, why not rate and review it in itunes? Reviews really do matter in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Performance and Site Reliability Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack performance testing awesomeness. As always, test everything and keep the good. Cheers. Hey, thanks again for listening. If you're not already part of our awesome community of 27,000 of the smartest testers, DevOps and automation professionals in the world, we'd love to have you join the fam@test guild.com and if you're in the DevOps automation software testing space or you're a test tool provider and want to offer real world value that can improve the skills or solve a problem for the Guild community, I'd love to hear from you. Head on over to test guild.info and let's make it happen.