A Day In The Life With Dev Op with Michael Martinez

Get ready to discover some of the most actionable DevOps techniques and tooling, including performance and reliability for some of the world's smartest engineers. Hey, I'm Joe Colantonio, host of the DevOps Toolchain podcast, and my goal is to help you create DevOps toolchain awesomeness. Hey, it's Joe. And welcome to another episode of the test guild, DevOps Toolchain. Today we're talking Michael Martinez. All about a day in the Life that with DevOps. If you don't know, Michael is a DevOps manager and lead at DevOps team, responsible for Stoplight and Smart Bears platform products. So he leans on previous experience as an SDA and a Full Stack developer. So he knows the full pipeline and he brings focus on automation to all aspects of the development and release cycle. You don't want to miss this episode. Check it out. Hey, before we get into it, do you know that you can elevate your entire DevOps testing process with tools designed for you to do more, better and faster, manage, automate and execute all your tests all in one place? A simpler way to guarantee app quality. Check out SmartBear's test hub and say goodbye to your DevOps blind spots by using it for better test coverage. Scale your automation, boost release, confidence and more. Head over to TestSkill, me TestHub or use the link down below. Hey Michael, welcome to the Guild.

Hey, thanks for having me on.

Awesome to have you. So you have quite a background. So I'm just curious just to kick it off first, how you even got into DevOps.

Yeah, so I started my career as a Full Stack engineer at a really big company where, you know, pretty much the deployment process was, you know, packing everything into a jar file and sending it off to India and, you know, crossing your fingers. I guess so, yeah. Like kind of moved into like more of the startup space after that. Moved into like as a role as a tested where I, you know, it was at a 10 person company and kind of wore a lot of hats there. Really saw like an expansive. Really saw my role as a more of a expansive role. Not just, you know, writing, you know, just writing tests, but really like understanding where testing fit in the pipeline, how that fits in the deployment process. Through a series of, I guess, right sizings and things like that. I kind of ended up being the last developer left. So I became responsible for the whole thing. Wow, like the whole. Yeah, the whole deployment process from start to finish, like building on all the ephemeral environments and you know, that was like, I was really responsible for all the automation there. Kind of similar thing happened at the next company and kind of here at Stoplight, like I had like the opportunity to kind of grow and have that more formally recognized and been leading the DevOps team here at Stocklight for the last two and a half years and you know, we're still alive and kicking, so pretty great.

Nice. So what's it like to lead after first being thrown into it, probably with no experience and now you're leading one. What was the transition like to there?

Yeah, it was pretty interesting. I mean, I feel like I've, you know, kind of always taken on a lot of like responsibility, but it was like more, it was interesting to kind of like have that role more formalized and have like, be more. Yeah, like have like more formal meetings with like, you know, stakeholders across the company. Right. Whereas before it was like everything was a fire, everyone needed everything and it's like, okay, now we have more formalized processes for triaging things, for collaborating and form and you know, really being able to build out those long term plans and projects with stakeholders across the organization.

Nice. Because once again, it sounds like you were kind of thrust into DevOps to own it. Is there one thing you wish you knew before that happened other than you knew this was going to happen?

Yeah, I mean, I think it's like, you know, the main thing is stay calm. Right?

Yeah, right, right, right.

It's like everyone thinks everything's a fire and sometimes things are a fire. Right. But I feel like, yeah, it's like, you know, really like, you know, like I feel like I've always had an. One thing that I think I've always been good at is having empathy with all our stakeholders. So understanding where they're coming from, but also not letting that kind of trigger your immediate reflexes is like, hey. It's like, hey, all right, let's take a step back, let's take a breath and let's see what we need from each other here.

Love it. So I want to give this almost like a starter kit for people to get up to speed with DevOps. So I want to go over some main principles and get your views on it because like a lot of people that listen to this have your background as an SDAT and then got into DevOps or maybe they think about switching careers or switching roles. So what are some of the main principles of DevOps? I guess a collaboration would be one of them. Automation, continuous improvement, customer centric action. Maybe we could talk about each one of these.

Sure, yeah. I mean, I think one thing that's really important, or one of the really, like, foundational principles for sure, is collaboration. Right. As I kind of touched on a little earlier in DevOps, like, you're working with stakeholders across the organization, both internal and external as well. Right. We have responsibilities to our customers who are like, the developers, of course, but we have SLAs, like, especially for uptime. Right. With our, you know, our customers, outside infrastructure issues, you know, tickets come through. So, yeah, so we have to like, really collaborate with the, you know, and we need to help developers meet their goals as well. Right. So, you know, having really good tight connections, like, and really good even, especially personal relationships with folks across the organization to make it, you know, easy to communicate and, you know, and bring up our concerns, like, is really important to the success of any DevOps organization. I feel like we don't operate in a silo. You can't just throw things over the fence. We're, we're, we're, you know, that's like one of the main, like, yeah, we need to work together to accomplish these goals as a whole.

Nice. So I guess you had another wrench thrown your way. You got into DevOps and then you were acquired by SmartBear.

We were acquired by SmartBear. Yep, yep.

How did that affect collaboration then? Did you have your own processes and now you're like, oh, my gosh, no, I need to collaborate against this. Large organizations with all these silos. Now, how does that work?

Yeah, it's a lot different. Right. Going back to the small, like, small company where, like, yeah, these relationships we built, it's like you have to relearn or you have to learn, like, the new organization structure, the new, like, you know, we, you know, when you're a startup and like, you're the only DevOps team, you're very empowered. Right. Like, we have the keys to the kingdom. Right. And now here we're like one piece of the bigger whole. So really, like, you know, negotiating that and like figuring out like, ownership across there. But I think, like, one thing that we really have done well in that acquisition is bringing the energy that we've had like at as a startup right at stoplight to, you know, inject that into the larger organization as well. Kind of like leading by example and like, really digging into their processes as well and like, you know, helping bring, you know, an automation mindset to their stuff too and really breaking down those barriers or. Yeah, yeah, working more to break down those, like, silos that sometimes exist, especially in like, you know, larger Organizations. It's like, do I talk to about this? Like, well, we'll see.

Right, Absolutely. So I love how you said automation mindset. What does automation play in the principles of DevOps? Most people think of just functional automation. They miss out on other automation. What is an automation mindset?

Yeah, I think it's like, you know, like, especially with DevOps and especially traditional ops. Right. There's a lot of like manual operations and requests that come through. Right. I think a lot of it is like thinking, you know, through processes and requests that come through and like being critical about what comes and like thinking about how we can better improve the process and how we can better improve like from a technical standpoint and. But also from, you know, like a higher organizational standpoint too. Right. Like, for example, like, you know, we have sometimes there are manual requests, manual database requests that come through. Right. And if it begins a pattern, you know, after the second or third one, you know, we kind of bring it up, you know, from. To the developers and like the product folks see if we can productize that. But, you know, that takes its own timeline. Right. So then internally we, you know, we work on creating tools to make that more seamless. Right. Like, you know, create like automate JIRA tickets that come through that can, like we can use and have our, you know, automation on the other side where we can, you know, process those requests like in a consistent and repeatable manner because, you know, those are dangerous and we don't want, you know, we don't want those. Yeah, we don't want. We want to make them as safe as possible. Right.

So how do you get buy in? Like a developer probably has a large backlog definitions of done they need to get done. Now you're like, oh, can you automate this database script or something? Do you get a lot of pushback or is it more just like you said, collaboration, Trying to educate.

Data helps.

Yeah, Right, right, okay, right.

Data helps. Yep. So for. Yeah, exactly. Like, yeah, you know, usually it's like collecting the number of tickets and like the time that we've spent like on this manual task, also showing that we've, you know, done work on our end to improve it, you know, that we're not just kind of like throwing things over the fence ourselves as well. Right, right. But also, yeah, putting together the data and you know, showing like what it's actually costing our team, but also the organization at large. Right. I think that's like. And yeah, I think like, yeah, coming from a place of empathy. Right. Understanding that they have their Own backlog, as you mentioned, and we have our own backlog and you know, like working together to have like, you know, things in the interim, like, you know, short term fixes that we can have medium term and like the long term goal that we're working towards together to kind of bring that all together.

Absolutely. You know, and just in my experience, I find a lot of times when people have a process in place, they hold tight to it. Maybe because I come from a regulated environment, but I know one of the key principles of DevOps is continuous improvement. So I assume you had a certain system when you were a startup and then you're part of this larger team now. So how do you work in continuous improvement to learn like what works and what doesn't work specific to your specific organization, what's going on there?

Yeah, it's like, you know, I think the first part was like understanding where, you know, like they're coming from and where we're coming from. Right. No process is perfect and everyone has a reason for, you know, whatever process they have. Right. So like, you know, being flexible is a really big important part. Right. Like, as you said, people hold things like near and dear to their heart, but really nothing is sacred. Right. Like, you know, like really like and taking a critical eye towards everything. Right. Our own processes included as well. Right. So especially like being part of like, you know, this larger organization at start, at Smart Bear, there were more or there have been more, I guess, metrics like from like the product management perspective, I guess, that they manage and like that they are, you know, like from the Smart Bear side, like there's a whole organization that monitors like project metrics, I guess. Whereas like in a startup it's like, hey, did we get it done?

Yes or no. Right. And we had our own pointing system and things like that. So kind of like, you know, working together with like the stakeholders there. Right. To understand what they needed from us. Right. And like what they needed from us and like what boxes needed to be ticked and like, you know, how, like, you know, so that we didn't, we weren't outliers in the whole organization, but also making it a process that we felt worked for ourselves as well and that we can kind of bring to other folks too. Especially for example.

We use story points on our team to take risk and other things, risk and effort into consideration. Whereas previously the organization was tracking hours and just time to action. So kind of like bringing that kind of mindset and explaining to other stakeholders why we use this and how we feel that better Most accurately shows the actual work that was done or the projected work that can be done. Coming up.

Awesome. So I guess last principle is customer centric action. Does this also have to do with internal? Because it seems like you're very internal as well with customers, your developers, your product owners and things like that.

Yeah, exactly. I feel like, you know, especially for ourselves, like our team has a very unique or I think DevOps in general, like has a very unique place within the organization where we have, you know, two kinds of customers. Right. The internal customers and the external customers as well. Right. So like, whereas like other parts of the. Org are really external focused. Right. We are as well. Right. Our team is responsible for uptime and incidents response and monitoring and all the infrastructure that goes along with that. Right. So we have to keep those SLAs and like those objectives. But also we're responsible for the pipeline. Right. We're responsible for infrastructure internally as well. Right. So it's like making sure everything is running for our developers, making sure that the developer experience is seamless, making sure that code can go out. You know, we're deploying, I think like at the peak of our development, we were deploying like almost ten times a day. Right. Making sure that the pipeline is clean, like having everything going out, you know, like, and changes like smoothly. So, you know, we need to work internally with the devs as well as like making sure that like our. Our commitments are kept to our. Our customers as well outside the custom. The company. Yeah.

So how do you get to that velocity of having not only releasing quickly, but having confidence in releasing it that quickly?

Yeah, I think like it's having. Making sure that the process is repeatable and that every step of the process is or like especially every environment is as close to production as possible. Right. When I originally came in here as an sd, for example, like our testing or so we deploy. Yeah, using Kubernetes and helm. Right. But we were using CircleCI's like kind of like zombie Docker container kind of situation to create a weird ephemeral environment. And we were having a lot of issues like with tests not or you know, regressions making it to production because like those things didn't match. Right. So we revamped that. We use the same helm charts, like just different values files and like, you know, in like very similar environments, we made sure that, you know, bring everything into infrastructure as code. Right. So like that, you know, we can see like, you know, just, even just looking at the code that everything is like as similar as to production as Possible bring in, you know, as much automated testing as we can and making rollbacks like as simple and as, as possible. Right. So if we find a regression that we can, you know, we're confident that we can roll that back easy. And then also knowing that, you know, and socializing that so like that the developers like are comfortable and confident in deploying to production. Right. Which all it does is take a merge. Right. You know, like letting them know like you open a PR is like, you know, if that, you know, all those tests are green, you're good to merge and then it's good out in the wild. And that's okay. Yeah.

Awesome. So how about uptime? Like when does DevOps end and SREs begin? Do you use SREs or are you responsible for the pipeline? Uptime sounds like you're definitely responsible for provisioning and maintaining infrastructure. Like how do you handle uptime?

Yeah, so we're responsible for. Yeah, we're like also sre, you know, like for the SRE role as well. So like we're responsible for uptime. We're responsible. We have an on call rotation that we've actually gotten developer buy in into. So like, you know, it's kind of, it is our job to keep things up. But like we have like gotten buy in from the devs as well to like, you know, have like that kind of shared responsibility. Right. So like we're on the escalation rotation. But also like triaging like is kind of a shared responsibility across like a lot of the organization. And I think that's something that's, you know, really great about the culture that we've built here at Stoplight that like, you know, it's not again going back to collaboration. Right. It's understanding and like building that kind of like shared ownership where you know, we own the infrastructure but like we're also, you know, here to help each other.

So do you have any advice for triaging to make life easier or any steps or procedures you follow?

So for triaging, I mean having a runbook, obviously, like documenting like. Obviously, yeah. And like making sure we have different levels of like alerts, right. We have alerts that could be kind of like more like yellow lights. Right? Like our lower priority alerts, but we also have like a separate set of like definitely like everything's on fire.

Like alerts. So being. Yeah, and like socializing and documenting that I think and having that in an easy space has really helped us be able to like have like especially devs who are not as confident with the infrastructure kind of like, you know, take on that triaging role and knowing that we're going to be here to escalate as well. Right. Like that we're on our own rotation and there's a safety net. Right, right, right. So that's good.

Awesome. So do you have any favorite DevOps tools that you think everyone should be using or that you find really helpful with any of the DevOps pipeline issues?

I mean, I've like. I mean, one of our projects was moving into datadog. I feel like datadog has been really great for monitoring and. Yeah, for monitoring and you know, obviously it's one of those tools where they make it really easy to spend a lot of money. Like, I feel like that's part of like the management job that, you know, you don't really hear. Like, you know, like that that wasn't like, I guess cleared too at the be not clear. But it's like, you know, before it's like, oh, yeah, you know, like we just provision a lot of things and it's like now like the bill comes and you're like, why'd you spend this money? It's like, oh, snap. It's been great to have like a centralized location for like all our monitoring and logging and being able to like cross reference that. I feel like that's been really great for us. Yeah, I mean we've. Yeah. Using some of the more modern CI CD tools. Like, you know, my first couple jobs, we use Jenkins like here, you know, we use Circle, CI and GitHub actions. I really like, you know, those are more flexible. You know, you can have more directed like asynclic graph kind of workflows, which is nice. And it's. Yeah. I mean, and there's a lot of integrations with like the other tools that we use already like that are native and they have their own runners that can, you know, run in your own. In your infrastructure so you don't have to worry about firewall stuff as much. So that's all been. Those have been some pretty good, you know, tools that we've used here. Cloudflare. We use cloudflare as our waf, which has been pretty nice.

Nice.

Yeah.

All right, so this probably isn't controversial anymore, but you know, there was a movement to start calling DevOps DevSecOps. How much responsibility do you have leading a team with security? Do you have like a separate security team or is that something you really think you need to encourage people to have DevSecOps as part of the whole.

Yeah, there's like all the Dev, like dev fit ops dev, like all sorts, right?

Yep.

Like all the ops.

Right.

And I think like for us it really depended on the size of the organization. Right. Like at Stoplight when we were just a startup, you know, I was, my team was responsible. Like we took a lot of like the responsibility for you know, getting us up to ready for a SOC 2 audit. Right. And getting our SOC 2 certification. We did a lot of that. We stood up the infrastructure. Like we were answering a lot of like those security questions that came through. Right. But the second we got acquired by Smart Bear, there's a security team. So it's like, you know, like there's a whole organization dedicated to that. And you know, on one hand like we're responsible for the implementation of like whatever security things come up. Right. But like again like anything else, the responsibility is always just shared, right. In a different way. Right. So you know, if they have their, you know, the kind of like scan like the agents that run, you know that they have that they run typically on their infrastructure, you know, they open a ticket with us. You know, we created and you know, instead of us it coming from us like that, hey, we need to run this certain thing which we had our own kind of like security agents. You know, agents that were running, you know, they have their own flavor and that comes to us and we do the implementation side but it's still a conversation back and forth. Right. So you know, we still do a little bit, but yeah, I think it really on that, you know, it kind of depends on the way your organization is structured and you know, like how you want to share those responsibilities. Yeah.

So do you have security built in to DevOps pipeline? When someone checks in code, maybe it kicks off or anything like that? High level kind of sanity checks.

Yep, we do some like. Yeah. And also like, you know, we, we have like automation around like making sure our dependencies are okay. We have code coverage, security, you know, like code analysis and that front as well. You know we also have I guess like now that it's been handed off, I'm not sure but we had like contracts with some vendors who would do like, you know, regular pen testing and things like that. So. Yeah, and I'm sure the security team has their own stuff right there as well, so. Yeah, for sure.

Nice. So every time I look at DevOps online, I see images. I always see the DevOps life cycle. Discover, plan, build, test, deploy. Now the holding, operate, observe continuous feedback. Like as a beginner I've never done DevOps. I've been an automation engineer all my life. How much do you spend on each of these steps? Is it really this discreet or is it just like, like in the day, in the life, how much do you spend on each of those little segments in that little, you know, figure eight?

Yeah, I mean, it really depends on the day, right? But like, for example, like, you know, we work like we work in an agile framework, right? Like develop it and cycle. So like, for example, every two weeks we have our sprint rituals where we do like a lot of the planning, right? Where we do, you know, grooming, planning things of that sort. We do a retro, right? So like kind of like, you know, that's kind of like the feedback part of it, right? So like there's dedicated time for that. Ideally, like deploying and testing is kind of like automated, but we're always like building, you know, we're also always building improvements to that as well, right? Like, yeah, building improvements to our automation, like, and that test integration as well, you know, and operate, like. Yeah. And observability, like, you know, like building again with going back to the automation mindset, like, you know, making sure that our observability tools are up to scratch and like that we have processes in place there and like for operations that, you know, that there's flexibility, like high availability, you know, like that our infrastructure is already designed, right, to like be flexible with those things. So, you know, it really depends on the day and the week and, you know, what's going on. But like, ideally we like, we do spend a lot of time on the building so that the other parts can kind of like, you know, kind of like flow naturally and like, you know, flow back to that and we can iterate on that again.

So this is probably a dumb question. So back in the day, my company's like, oh, we're going to do DevOps. But they didn't focus on deployment, so we had to do manual steps before tests kicked off and on the environments. So it wasn't built to be deployable, so it was kind of stupid. Is that still an issue? Do people automatically build it to be deployable? Do you ever have to say, you know, you have to build an API or a hook or something to be able to get this to run seamlessly.

Yeah, I see. Like, I've actually kind of been surprised, like sometimes, like what? Like by how common it is that people don't just like automatically deploy like that. Not as much automation. There's still like a lot of manual release processes out in the world. And I think like, for Us, like, you know, when we see that, we try to address that and make it, you know, empower the devs, right? Empower, like, create a process like, you know, to empower developers to kind of deploy their code and also educate. Because I think one of the main reasons, you know, when we came to SmartBear, one of the things that we saw were some teams weren't deploying or they didn't automate their pipelines, their releases, because they did not or they didn't think that that was compliant with SOC 2 regulation, like the SOC 2 certification. But we, you know, like, we had done that in a SOC 2 compliant way. And so, like, you know, kind of like educating them and like, you know, bringing that as well and educating, like, our smart bear counterparts, right? And like, bringing them up to scratch, like, with those, like, you know, letting them know that, you know, it can be a fully automated process has been good. So, you know, I think that's the goal and, like, that's what we strive. And like, I personally, like, you know, here and wherever I go, you know, in the future, that's kind of like the first thing I, you know, you see that needs, you know, sometimes needs to be fixed for sure.

Absolutely. All right, so speaking of future, let's put on your little psychic hat here. I'm asking everyone this. Obviously, AI has been a big deal. I guess it's almost impossible to know, but just because you're part of a larger company now that probably has a lot of different customers, a lot of verticals, probably see things a lot of people don't. Where do you see, like, DevOps heading in the next five years? And how much of AI is in play or how much is it going to change things, do you think?

I don't know. I like, personally see AI as more of a tool. Right. Like, I know some people get afraid and I was like, oh, it's a replacement. But, you know, like, we work for, you know, even now we're using kind of like we are using AI in our normal job, right? We have licenses for, you know, GitHub copilot, right? And like, that helps, like, make a lot of, like, the simple things, you know, like you have like a terraform template or something. And it's like, okay, here, these are the basic parts instead of, like, having to go look at the documentation. So I see like, AI, like, as, like, you know, helping us, like, build better tools to, like, empower, you know, our developers going forward. I don't think it's necessarily the end all be all I think it's like anything else. It's like, you know, also especially it's like a lot of trust but verify, right?

Yep. Right, right.

But yeah, like, with that, I know like Amazon has tools to like, you know, kind of like help like with cost and like efficiency and things like that. So like, I think like, you know, we'll see a lot of like AI empowered, you know, kind of development going forward. But I think, yeah, I think you'll kind of speed us up but not might eat my words in five years but not replace us.

Yeah, absolutely. All right, Michael, before we go, is there one piece of actual advice you can give to someone to help them with their DevOps efforts and what's the best way to find or contact you?

Yeah, I think, I mean, to help with the DevOps effort, I think it's like just collaboration, I think, is one of like the things. And empathy, like, have empathy with your stakeholders. Like, don't be afraid, you know, to reach out to people and like, you know, and work together. I guess, like, it's not a silo, it's a group effort. And I think, like, that's a good guiding principle that I've used that I think has like, really helped a lot. Yeah, but yeah, I can, I mean, yeah, I have a LinkedIn, I think that I sent that, you know, that, you know, I can be reached there. And yeah, I think like, that would probably be the best way or. Yeah, to reach me.

And for links of everything of value we covered in this DevOps Toolchain show, head on over to test guild.com forward slash P1P. So that's it for this episode of the DevOps Toolchain Show. I'm Joe. My mission is to help you succeed with creating end to end full stack DevOps toolchain awesomeness. As always, test everything and keep the good. Cheers. Hey, thanks again for listening. If you're not already part of our awesome community of 27,000 of the smartest testers, DevOps and automation professionals in the world, we'd love to have you join. Join the fam@testguild.com and if you're in the DevOps automation software testing space or you're a test tool provider and want to offer real world value that can improve the skills or solve a problem for the Guild community. I'd love to hear from you. Head ON over to testguild.info and let's make it happen.