Preparing Dev Leaders for the AI World with Will Blackburn

Get ready to discover some of the most actionable DevOps techniques and tooling, including performance and reliability for some of the world's smartest engineers. Hey, I'm Joe Colantonio, host of the DevOps Toolchain podcast, and my goal is to help you create DevOps toolchain awesomeness. Hey, is AI freaking you out and you're a developer or maybe you're a dev leader? Well, you're in for a treat because today we'll be talking with Will Blackburn all about preparing dev leaders for the AI world. And a bunch more if you don't know. Will is the co founder of Dev Clarity, an AI powered one on one tool for software engineering managers. And Will previously went for a first employee at a CTO at a financial technology startup that successfully sold in 2022. Really knows his stuff. Really excited to get his input around AI and how it's changing things. You want to miss this episode? Check it out. Hey, before we get into this episode, I want to quickly talk about the silent killer of most DevOps efforts, that is poor user experience. If your app is slow, it's worse than your typical bug. It's frustrating. And in my experience and many others I talk to on this podcast, frustrated users don't last long. But since slow performance is as sudden, it's hard for standard error monitoring tools to catch. And that's why I really dig SmartBear's Insight Hub. It's an all in one observability solution that offers front end performance monitoring and distributed tracing. Your developers can easily detect, fix and prevent performance bottlenecks before it affects your users. Sounds cool, right? Don't rely anymore on frustrated user feedback. But I always say, try it for yourself. Go to smartbear.com or use our special link down below and try it for free. No credit card required. Hey Will, welcome to the Guild.

Hey Joe, thanks for having me. Excited to be on today.

Awesome. Awesome. So, hey, what got you into AI to begin with?

Yeah, I think just as a technologist, you know, it's something that you're always trying to do is stay on the latest trends and, you know, be looking at the new technologies that are coming out. And so no doubt when we had this large explosion of large language models over the past few years, you start asking yourself, wow, what is this technology? And I've always been a user of technology. You know, no matter what wave is coming out, whether it's crypto or this latest AI trend, I'm a big believer. And you know, the only way to understand it is to get your hands on it and try it out. And so as I began to use large language models, you know, really that chatgpt moment that I think was a lot of people's first experience with generative AI, you're like, wow, this is interesting. And so since then, I've continued to stay engaged with it and have been blown away by it, while also have seen some of the limitations that generative AI has.

All right, I want to jump into some of those limitations and where you think it works. But first, so you were a CTO in 2022. ChatGPT, I think, came out in November 2022. So as a CTO, how much did you have a pulse on, like, when it came out? It was probably just when you were about to leave the startup.

Yep.

Did you see as like, oh my gosh, this could have helped us, or as a cto, this really can help me with this at this company.

Yeah, it's a great question. So we at that company, we were already going through an acquisition process and we could see that the company was going to be acquired. I'd gone from the first employee, a founding employee, to chief technology officer. We had a team of 35 on the development and product team. So. And yeah, I was feeling that, wow, this is, you know, coming out. Copilot was just getting released in some way where people could get it into their code editors. And I think I felt what a lot of development leaders have felt, which is, on one hand, wow, this could be this really amazing tool that increases our productivity and lets us write good code. And on the other hand, wow, this is also scary. And do we really want to our code base, our data going into these models? What does that mean? And also.

Are developers going to use this tool in the right way? Are they going to read the code that AI is outputting, or are we going to really reduce the quality of our code because of AI? I felt a lot of that at the time.

Gotcha. So for current CTOs, what would you recommend then? What's the approach you would give them to? Because they probably get pitched all the time with AI promises. Like, you're like, I don't know now at this point what's real or what's not?

Yeah, I think the world has changed a lot in two years. And I think now the models and the way that these tools are available, you absolutely should be exploring these tools and using them inside the development team. And that doesn't mean that, you know, you just give everybody a subscription and you turn away from it, it means that you work with your developers, you work with your other development leaders, you work with your DevOps teams who are going to have to be supporting all of this code that is getting pushed out by the software development team and you establish best practices in your organization around the right way to use these tools, what tools you're going to use, and then you're constantly learning from them. So one of the main suggestions I make when we're talking to development leaders is, you know, set up a slack channel that is specific for AI coding and tell your development team to put in your wins. But also the challenges that you had with AI, like where did you see AI go off down some rabbit hole that was just a complete waste of time and was going to reduce the quality of your code or add technical debt. Share that and have other developers add to that so you can all learn together.

Love it. Awesome. So if I'm a developer, then I'm listening to this. Obviously they've probably been using Copilot for a while now and all the other things. How do you see I changing or affecting software developers as software development?

Yeah, it's a great question and I think that the change that's happening in software development has already begun and we're going down a path that we're not going to come back from. I like to tell this story that there's this developer and she got told by her boss that tomorrow we've got this new tool that we're going to be able to use and it's going to change the way that software development is done forever and you're going to be able to be more productive and write better code. And so what's that developer thinking? You know, she is worried about her job, she's worried about what's going to come tomorrow and how that's going to impact what she does day to day. Maybe she liked the way things were done before. And so this developer, she goes home that night, she doesn't get much sleep, she comes in to work the next day and, and there in front of her is magnetic tape. And this is actually 1955 and her company has just moved from punch cards to magnetic tape. And you know, I tell that story because while we've never had this transition from, you know, not being able to use generative AI to generative AI being such a core part of software development, software developers have gone through transitions before, whether it's from punch cards to magnetic tape or high level programming languages which, you know, let so much of the underlying machine language code be handled by compilers and not by us. We've gone through transitions like this before and so while this is a change and it can be scary, I would say we should lean into this and it is going to change software development forever.

Yeah, for sure. I definitely agree with that. So. So do you think it's underhyped, overhyped, or properly hyped? I think it's under hyped because I think it's going to be a paradigm shift and I think a lot of people are burying the head in the sand kind of. Maybe I'm wrong, but that's a pulse I get right.

I think, like, I think when you look at it from a headline standpoint or a news standpoint, like that's maybe where it's overhyped. But to your point, like you get to Talk with Development DevOps leaders all the time. That's, you know, we serve development leaders as well at dev clarity and when we talk with them, I am surprised at how slow people are moving in some of these things, even just to try it out. I'm not saying you should be using AI agents right now to write all your code, but it does feel under hyped in a lot of ways. And I think to your point, this is a big paradigm shift and whether we like it or not, this is changing the way we're going to write code. And so I completely agree with you. When I talk to developers, they don't want to accept that in a lot of circumstances.

So who you think's in a better position for this though? You would think it would help junior developers, but it almost sounds like I saw news stories. I don't know where it was where top college grads with computer degrees used to always be in demand and always get a job. And now for some reason there's a lot of resistance. I don't know if it's the economy or if it is AI. So I guess what I'm asking is where do you see like how it's going to. Do you see an impacting new. Newer developers or maybe is it going to really be a good benefit for people that already know development and make them more like higher level developer leaders almost?

Yeah, that's a great question. And when I've studied this topic and again talks with people who are much smarter than me, I've kind of developed like two ways of thinking about this. I think one of the interesting things when we just think about generative AI and general, but it also applies to code output is that when They've studied who this helps the most. It really helps bring up that average player to that a plus level. Like that is where you see most of the gains. And of course there's diminishing returns at the top. And so if you already have an excellent developer, you know, it's harder to get more of a return out of them. So I think that that is just an interesting thing to be thinking about, but then we do have to think about, like, how is this going to actually be used? And one thing I have heard is that development leaders, CTOs like myself in my former role, they are more reluctant to give AI to a junior developer and let them, you know, go loose with it and start writing a lot of code. And so I think that right now is what is hurting junior developers. Now I think that that will change over the next six months as companies figure out the right way to roll this out to their organizations, to institutionalize best practices with it, to make it part of the dev review process, the DevOps process. If you're tracking the DoRa like metric, like, how can you understand what AI is doing to those kind of key output metrics that you're tracking? Once that happens, I think junior developers are going to be seen as that, hey, maybe they're not the skill level that a senior is, but if we augment their work with AI, we can actually get a lot more out of them. And then the last point I would make is if I was a junior developer, I would be looking at this as like the greatest learning opportunity of all time. Because if you're using a tool like Cursor, which is an AI enabled editor, and you figure out how to use it well, where it can understand your code base and have the context there and give it official documentation and instead of saying like, hey, I just want it to write code for me to say like, hey, I want it to be a personal teacher to me as I learned this new pattern. As I think about anti patterns to avoid. It's just an amazing opportunity instead of having to go to stack overflow and copying an answer from two years ago. Now you can get something that's generated for you for your code base and then have official documentation cited by your editor. That's incredible.

So I love that. I was actually speaking to a good friend of mine, he's been a software developer, he's a CTO now from over 30 years and he talked about Cursa, how he used it, and he was blown away by how quickly and easily he was able to get up and running with an application. So here's someone that has 30 years of experience developing and he says this, he admits it. So how do you make sure a junior developer doesn't get discouraged? I guess, like you said, it's a great opportunity, but how do you not see it as, oh my gosh, this is like, I'm going to be replaced. Why even do it? Why even learn?

Yeah, it's a great question. And like, I can say all those things and then, you know, I'll talk to a developer in person that can't find a job job. And so like, you know, my words don't mean much when I can't give them a job or help them land a job. Right now I again, optimistic on this as a whole. And I don't think it's going to be something where in the short term, you know, they're going to get a job to only get replaced a few years later. I think that you do have to lean into this, you know, a question that gets asked a lot or something people like to say is that AI is going to be like a Net job creator. That's fun to say and it's nice to say and people like to hear it. But when you zoom in and you look at the individual, you know, people are going to get displaced and it's going to change industries. And so you've got to be as the individual, you have to understand this technology and then use it as a differentiator. And that is what I tell the people that I meet in person, which is, hey, find a way to differentiate yourself in the market and use AI as a tool to help you do that. And then go explain to that CTO that is blown away by cursor and say like, hey, I'm a junior developer. Let me show you how this is helping me level up. Let me show you the some of the limitations I found in it and how I'm applying this as a junior developer who hasn't been out there in the market and seen a lot of things yet.

All right, so I think we touched on a lot of things, but, you know, for example, ChatGPT really in its, in its form now, it's been only around for two years and it seems like we've hit a wave. Do you see the wave continuing or do you see it coming down now? So in the next 6 12, three years, it's more now of how do we now leverage this or do you see it like, I guess I hate asking these questions because you're not a. You can't see in the future per se, but where do you see us maybe in three years? Do you see it just as now adopting to this world? Do you see evolving even more so in three years? You're like, I have no idea at this point where it's going.

Yes, it's a great question. So one way I think about this question is like, okay, how much are we going to see the models continue to evolve? And then kind of to the second part of your question there, what's the implementation going to look like? How is our workflow going to change? And I'm a believer that like even if we're like done from a model evolution standpoint, we have more than three years of work to do to figure out how to use this and really impact our workflows. I think there's a lot of interesting things going on right now. You've got things like Copilot, which are just extensions inside of your editor. And then you've got things like Cursor, which is a step up from that and that it's a full editor that has been AI enabled. And you start to get into these agents, these agentic workflows where they do more things for you without you having to be involved in every step of the way. We've got new tools like Bolt bolt new or V0 by Vercel where, you know, these are app builders, where you can build a full stack app in Depolia. These are just examples of some of the ways that we're using generative AI right now in the coding process. And I think that even if the models stop evolving at this point, we've got to figure out how to use these betters better, how to make it more functional for the developers. The things that developers hate about it, where it starts to hallucinate or you know, uses a class that doesn't even exist inside your code base. Like there's still so much opportunity to improve the workflow that we have right now. I personally don't think we're done from a model evolution standpoint. I'm interested in seeing what happens when we get more code specific models, if that's necessary. Like those are just some of the interesting things that I'm thinking about. But yeah, I think that really what we're going to see over the next few years is how does it just become a part of everyone's day to day SDLC software development lifecycle process 100%.

So you know, you mentioned dev clarity a few times. Obviously you were a CTO and then you started A company, I assume you started a company, but due to maybe problems you've been seeing and how it could solve AI. So I guess before I assume things, what is dev clarity?

Yeah, yeah, thanks for asking. Dev clarity really came out of my experience as a cto. So I went from founding employee and just a software developer to eventually being a small team lead of five developers and then 15 developers and finally a chief technology officer managing 35 developers. And I made a lot of mistakes in that process and luckily had a great leadership team and great developers to help me learn and grow from those mistakes. And I encountered a lot of pains as a development leader and really like thought, wow, it's amazing how much work I'm doing in Excel in OneNote. And so from that experience set out to create a platform to help development leaders make that transition from being a developer to being a manager, and then to help development leaders up the chain to better lead their organization. So today we help engineering managers, that kind of first level of software development leader. We help them better lead their individuals on their team with a dev specific one. On one note, that really creates structure. And oftentimes you've got a developer who's spent 10,000 hours talking to a computer and giving them commands and then the next day they're a manager. And now they have to talk to humans and try to get them to do things. And it's not easy. Humans have a hidden compiler where you can't always be sure that the code that you gave them, the instruction that you gave them, was processed in the same way that a computer was processed. And so we are really creating a platform to help in that journey and to give them structure and leading their team. We believe that with AI, developers are actually only going to become more important when companies realize that they can do 3x5x10x more and there's more opportunity, they can serve their customers better than ever before because they've got these large language models to help them better understand what's going on. We believe that dev leadership will only become more important because the output of developers is leveraged and managed by these leaders.

Very cool. Want to dive a little more about that? But as you was talking about that, like how does, as a leader or cto, how do I start measuring performance of or what metrics do I use for my team with AI now because you just mentioned 3x2x should I be looking at, okay, I have developers I know before AI, they were producing this amount of code. Now with AI, they should like, is that a metric you, you, you think people should start using or is there any type of metrics that you would think with AI now is really going to change?

Yeah, that's a great question. One of the things that we have really tried to do is understand how developers are judged and how development leaders are judged. If we're offering a product to them, we need to understand like how can we say that you're going to get a return on using dev clarity and pretty much universal. The thing that we've heard is that development leaders are just judged on outcomes and those outcomes are usually delivering on the roadmap. If you're creating any sort of product, any sort of service, you know, that's really what you're judged on. You've got things like DORA metrics and different tools like that, that kind of talk about the health of the organization and let you see things that, that happen. But it's just, it's really tricky to gauge developer output from a metric standpoint. And I think that's why so many leaders use outcomes such as, you know, you said we were going to do this by then and did we get that done? And so I think as AI comes out, there will be even more of a request from leadership, whether that's technical or non technical leadership, to say like hey, how can we measure our output? I think things like DORA is a good place to start. I am always very critical is not the right word. I think looking at metrics from an absolute standpoint is the wrong thing to do. I think looking at relative metrics and trends over time is the right thing to do. So if I can start measuring, you know, what is our committed versus completed over a sprint look like for how many story points we said we were going to do. If I can start measuring that now, I'm more interested in the trends that happen over the next few months, over the next six months as we adopt AI to give some sort of narrative around how this is helping as opposed to saying like I'm going to look at an absolute metric like PRs, you know, merged and make a judgment off that. I would much rather look at the trend and then you can look at things relative across particular teams or particular team members and you've always have to lean on the leaders to provide the context around those numbers.

Great insight. So I guess that implies not only to metrics, but is there a risk to over relying on say AI for interpersonal aspects of team dynamics? So you see a trend, but you have to have discussions, I guess with the developer as a human to see what's going on. Do you see people over relying on AI maybe to make these type of decisions?

Yeah, I think what we've seen is that really the only thing that's been available to development leaders have been quantitative things. It has been how many commits people are making, how many pull requests they're closing, how much, how many comments they're leaving, how many reviews they're doing. And so it's always been this quantitative thing. What we're trying to do with dev clarity is actually use AI to bring in some of the qualitative things that development leaders just don't have good visibility into right now, or that it's really time consuming and manual to do. And so if we can take your one on one notes over the past 90 days and if we can provide you context around trends, sentiment, different things like that, then that is something that's now a new tool to help you have a more complete picture. Don't just use the metrics and also, you know, don't just rely on an AI generated sentiment trend. But when you start to put that together, it begins to give you a complete picture and it builds confidence in yourself as a development leader that you understand what's going on, that you know where to drill in and where to spit your time.

Nice. So could it also help get insight to maybe a team that's not as forthcoming? You know how you have like every sprint, like how happy are you? Like thumbs up or thumbs down. But then if you have notes and everything, you can try to say, okay, the same thumbs up. But I'm noticing that there's an issue here that maybe isn't being bubbled up to me that maybe should, should help, not to be punitive, but maybe to fix before it becomes an issue.

Yeah, yeah, absolutely. You know, again, the thumbs up, thumbs down, other metrics are great. Oftentimes they're lacking indicators and not leading indicators. And so that's another great thing that you can get. If you have structure around talking to your individual developers and seeing that information, you know, oftentimes the developer in the project knows what's going to go wrong before anybody else does. And so if you've got a structure and a process to help get that information, you know, up to the development leader, then they can help that developer make the right call, make the right decision, unblock something for them. And then to your point, you can really marry the qualitative and the quantitative data. Like, okay, this sprint, we had 30% more thumbs down. Well, why now if I have qualitative data that I can put with that system, I can see what happened. Maybe somebody was off for vacation and we knew about it, but we didn't adjust the story points that we were going to assign to them. So that crunched the rest of the team members. Or maybe it's a problem vendor that, you know, just you always run into issues when you're working on that vendor's integration. That type of information can flow out of the system much more clearly.

Nice. I always love speaking to founders of solutions and products. So I'm just curious to know when you created dev Clarity and now that it's in the wild, is anything you've heard as feedback, like, well, I didn't even think of that. Or, wow, it really helped this team do X that. That people may get a sense for, like, all right, sounds cool. But like any case studies or any other feedback that you've been getting how it's helped teams.

Yeah, I think, I think probably the thing that surprised me the most is, you know, as a founder of a solution that you've built kind of for yourself, like, I experienced these problems. I was thinking so much about me as the individual. And we've got a company that we put out a case study for and they're managing about 35 developers, which is very similar to my team and my last company. This is part of an organization. And the thing that surprised me was how much they said it helped their dev leadership team better communicate with each other. And so a cool side effect of the platform is that if you're an engineering manager and you report to a VP or a director, you can actually tag them in a note. You can tag them in an issue that's come up with a particular developer or with a particular team. And they said like, that was a game changer for them when they were able to surface issues from the bottoms up and really say, like, hey, like, I know we've always kind of heard murmurings about this process or this issue that always runs on Tuesday nights that disrupts our sprints. And now here's like a very clear statement from a developer about that being able to share that or share amongst the engineering managers team. That was a really cool side effect that was kind of emergent in our platform. We didn't think too much in designing that feature. And then really we were led there by our customers.

Nice. All right, so here's a little switch up here. Just came to my mind here. All right, so AI, so you founded a piece of software, obviously it's very cool. It does a lot of things. I'm not saying it's easier now to create software, but as a founder or someone that's creating something, how do you not have be worried that someone's going to use AI to say, hey, build me an application like dev clarity and then you lose all your IP or something like that. Like how do you stand out if someone's like, I built software. Now not only as a developer am I afraid of it, but now as a company I'm afraid because it could just mimic what I'm doing and kind of recreate it.

Yeah, that's another good question. So I'm a first time founder, but I was founding employee in the last company and I was there day one. And that experience and over the past year of getting dev clarity off the ground, it just, it really shows you how much execution matters and that ideas don't matter that much. And not only does execution matter, but having clear purpose and a vision for the future and a mission that you're on is just required to be willing to do the hard work that it takes to build a company and to grow your customer base. And so you become less worried about people stealing your idea or running, running off with something. Sure, there's a threat out there, but you know, I don't believe that anybody else is on the exact mission that we're on and has the set of unique characteristics that we are putting together to form a unique group of values and mission. I don't think that there's anybody else out there that sure, somebody could come and copy what we're doing. But you know, six months down the road we are marching in a way, in a direction that's going to be really hard to continue to copy.

Great advice. Okay, well before we go, is there one piece of actual advice you can give to someone to help them with their AI DevOps efforts and what's the best way to find contact you or learn more about dev clarity?

Yeah, I think I always like to leave with a piece of advice which is use the tech, use the tech and that's how you're going to understand it. Do the work to form an opinion and maybe your opinion is we're not ready for it or this is not good for this reason and that's okay. But make sure that you've done the work to form an opinion and through that you will learn a lot. And I think it is the only way that you can really set yourself up for success in the future is by spending the time to use the technology to try it out, to see what's good about it, where the limitations are, and then to ask yourself, are those limitations always going to exist or are they today's problem that will be solved in a month from now. So that is my advice and recommendation. Yeah, and then I am active on LinkedIn. I post most every day and so that's the best place that you can get a hold of me personally. Should have direct messages opened up where anybody can message me. So it's LinkedIn.com in Will HBlackburn and then you can find more about DefClarity and also contact me through Deaf ClarityEvClarity.

AI and we'll have links for this and much more in the comments down below. All right, before we wrap it up, remember frustrated users quit. Apps don't rely on bad App Store Reviews. You use SmartBear's Insight Hub to catch, fix and prevent performance bottlenecks and crashes from affecting your users. Go to smartbear.com or use the link down below and try for free for 14 days. No credit card required. And for links of everything of value we covered in this DevOps Toolchain show, head on over to testguild.com p175 so that's it for this episode of the DevOps Toolchain Show. I'm Joe. My mission is to help you succeed, creating end to end full stack DevOps toolchain awesomeness. As always, test everything and keep the good. Cheers. Hey, thank you for tuning in. It's incredible to connect with close to 400,000 followers across all our platforms and over 40,000 email subscribers who are at the forefront of automation testing and DevOps. If you haven't yet, join our vibrant community at Test Guild, where you become part of our elite circle driving innovation in software testing and automation. And if you're a tool provider or have a service looking to empower our guild with solutions that elevate skills and tackle real world challenges, we're excited to Collaborate visit test guild.info to explore how we can create transformative experiences. Together, let's push the boundaries of what we can achieve O the Tescal Automation Testing Podcast with lutes and lyres the bards began their song A tune of knowledge, a melody of code through the air it spread like wildfire through the land Guiding tester showing us secrets to behold.