AI in DevOps with Chris Navrides and Jason Arbon

Get ready to discover some of the most actionable DevOps techniques and tooling, including performance and reliability with some of the world's smartest engineers. Hey, I'm Joe Colantonio, host of the DevOps Toolchain podcast and my goal is to help you create DevOps toolchain awesomeness. Hey, it's Joe and welcome to another episode of the Test Guild DevOps Toolchain podcast. Today we'll be talking with Chris and Jason all about rethinking the DevOps cycle with AI and probably a whole bunch of other DevOps AI type topics, if you don't know. Chris is the CEO of DevTools AI, working to help developers not having to fix broken and flake UI tests. Prior to dev tools, he was the VP of Engineering at Test AI, which built the first AI model based mobile and web testing solution. Before that, Chris worked at Google as an automation engineer for Google Play and Mobile Ads and Dropbox on their mobile platform team. Also, Jason is a repeat guest here as well and he's the founder of Test AI, author of the book How Google Tests Software, frequent speaker and all around AI and testing expert. He's one of the few people I actually really listen to when they talk about AI and testing because there's a lot of misinformation out there and I think we're going to set a lot of things straight. So you want to stay all the way to the end to hear all this awesomeness. This episode is brought to you by SmartBear. As businesses demand more and more from software, the jobs of development teams, business gets harder and harder. They're expected to deliver fast and flawlessly, but too often they're missing the vital context to connect each step of the process. That's how SmartBear helps. Wherever you are in the testing lifecycle, they have a tool to give you a new level of visibility and automation so you can unleash your development team's full awesomeness. They offer free trials for all the tools, no credit cards are required, and even back it up with a responsive, award winning support team show on your path to great software. Learn more@smartbeer.com today. Hey Chris and Jason, welcome back to the Guild.

Hey, thanks for having us.

Thanks Joe.

Good to have you back. I always botch bios. So is there anything I missed in your bio that you want the Guild to know more about?

Yeah, just the failure of my company.

Other than that, no real update. Yeah, so we weren't able to raise a B round. That's another podcast I'm happy to share about details, anything actually, but couldn't raise a B round. But I got really creative, started a new company called Testers AI.

Oh, Testers AI. Nice. Nice.

Yeah. So still focused on AI ML and the same old stuff. Sweet actually, same old stuff, all new technology.

Oh, nice. Perfect. Even better. How about you Chris?

Yeah, still plugging away and we have a new product as well that does AI based code reviews. And then we're just, we're kind of looking now with the advent of large language models, we're, we're looking at, you know, where we can apply Those within the DevOps toolchain and what kind of that means in terms of where are we going to be developing and how are we going to be developing in the next, you know, two years or so.

Nice. So I would think the code review fits in nicely to the DevOps toolchain helping developers. When you have the DevOps toolchain, a lot of time people think AI, they think of functional automation. Well, at least me because that's mainly where I'm from. So where do you see AI then now impacting actually DevOps or DevOps engineers.

Yeah, I think from a DevOps standpoint AI is amazing. The large language models are able to look at your logs, synthesize and write up reports for you on those. So, so traditionally you'd have to, you know, train a model specifically based on your data. Now it's able to just suck in the data, read it, you know, human like, and be able to then summarize and report on that. And so I think that's really the biggest impact that we'll see in the DevOps side with, you know, monitoring anyways, I think you'll also see where it can be applied with, you know, coding code review. I think it'll just kind of affect each of those chains individually and it'll be interesting to see over the next year or two where each of those gets kind of disrupted. But I do believe all of those areas will be disrupted with these large language models like OpenAI and Jason, what.

Do you think how it applies to.

DevOps faster than we realize? I think we'll see a lot of the roles that people play being automated by stuff powered by LLMs. Because the cool thing is that it can act like a person, it starts to sound like a person. These days it's not just a neural network that can classify. Cat, dog or hot dog can actually like think like a person and then people. So if you look around us, we're doing things like auto GPT Right. Where you can have like high level tasks broken down to subtasks, passed out and farmed out to other bots and they can roll up the results. That starts to sound like a DevOps team. It starts to sound like an engineering team and then things with like you can start to imagine, you know, you produce the code, right? Like the AI produces the code. Everybody laughs today and says, oh, it's not that great a code and there's bugs in it and stuff. But guess what you can do? You can now run that code via that chat interface, find the breaks, pass those breaks back to the original LLM and it can fix them. So that starts again, sounds like a DevOps test kind of loop there too. So I think we'll see the automation of the what traditionally is human handoff In Slack and API, hard coded APIs will be handed off to more natural language automation flows in the next 18 months.

All right, I'm just assuming when people hear that Jason, they're kind of freaking out, like what it's going to replace DevOps. So do you see replacing people, is it just replacing titles with new titles?

Which do you want for the clickbait? Clickbait.

I'll say both are true. There's this really good presentation I watched last week where someone talked about just automation in general. Forget AI and all this other buzzwords. But the funny thing is when you automate things like everyone thinks there's going to be this cost savings because once you automate it, there's no humans needed. The reality is in almost every kind of general automation product and there's like a formal study I think related to it, but is that once things become more efficient and automated, you do more of that. So I imagine in the next 12, 18 months it won't be like a sudden mass loss of jobs. It'll be people are just shipping five times faster, five times more often and with five times kind of more confidence is really what I think will happen in the near term. But ultimately, yeah, I mean you see where this is going. It's exponential, right? So I think anyone that predicts out more than 18 to 24 months is confused actually, or just indiscenuous at worst.

Yeah, I'd like to just add on to that as a valid point, but you know, but no, I think very similarly because ultimately the thing is these companies are profitable today. And so if you can have one engineer and 5x them and you're already profitable with an engineer who's 1x, you can get 5x, the features done now. And I don't think there's a real incentive for most companies if they're, you know, in a competitive space and not just trying to cost cut, to cut your engineers and if their competitors are doing the same then it's, you know, it just will amplify that gap that they would have to close. And so I think that you can't really fire an engineer right now and assume that like AI would take over, but you can start to assume that these engineers are going to be far more efficient. And that's going to be I think the key value add.

So I think that's key is being efficient. Maybe over optimistic is that I heard a presentation where if an engineer doesn't take advantage of AI, they'll be replaced, but an engineer with AI is only going to be more productive. And this is what I saw with automation in the early days. Testers that didn't invest in automation and they just did just straight up testing. Eventually they were replaced by people that actually use automation tools as well. I see this kind of similar, but I don't know, it seems like AI is a different kind of beast. Am I being too optimistic with my thoughts here?

No, I think, I mean there's still testers today, right? With all the automation tools, all the automation advances, there's still a lot of open positions for manual testers. And I think there's going to be some mythical hurdles or high level hurdles that'll have to be overcome that right now are not solved. And it's unclear how achievable it is or at what cost it would be solvable. But you know, things like being able to take in the code base and take in the whole context of the code base, right, That's, I think large enterprises, they have massive code bases that you have to understand before you can write a line of code. LLMs today don't have that ability to understand this specific code base. I think the other aspect of it is trust, right? So there's these articles like Samsung saying their code leaked because their engineers were using ChatGPT. And so there's a lack of trust of AI, where's my data going? So these models are also going to have to be able to be siloed so that an engineering or a company can trust it. And so I think you have to kind of overcome some of these hurdles. And to Jason's point, it's unclear right now, you know, how much context can be, you know, loaded in. Right now it's 8,000 tokens. It'll soon be 32,000 tokens. But like the cost is you know, I think double to just do 32,000 tokens. But the number of tokens you need to make this work for a large code base is going to be, you know, millions of tokens. And so it's like what does that cost and is that worth human salary? Unclear. I think a lot of what you see today is, is all started with new projects. So I haven't seen auto GPT with an existing project and add onto it. I'm sure it'll happen soon but they're going to have these obstacles to overcome to add that value. And so that's I think going to be the real challenge that we'll see. But when you see that I think then it starts to become a real question.

Yeah, I think I wasn't asked but I'll jump in is we'll see the reverse of what happened with before. Like there's you know, I don't even know if I think I tried to look at the data and I think it's real. I'm pretty sure yeah I verified it before once that there's more tellers today than there were when ATMs were invented and that's, you know, there's population growth and other stuff and other factors into it but it wasn't devastating. And there's also I think more horses today than there were cars. I believe that's at least the minimum. And so you know, cars were invented but I think it's going to eat the different type of people. It's going to eat the middle, the entry to middle level engineers. So I think what you'll see is and actually it started hard to tell things whether they're macroeconomic or not. But I think you'll see a lot. You'll see the slowdown in hiring of junior folks I think across the board regardless of discipline because the grunt junk work can't be handed off to you rather hand it to ChatGPT than an intern or than a freshie out of school because you can control it. It works whenever you are working and make it work in your style. And for example like I'm writing probably twice as many blog posts as I used to with ChatGPT and I also don't ask Chris for feedback on them anymore.

And the quality is going up.

According to ChatGPT. It is going up and then. But the other thing is I think, I think we're starting to still see though like Chris said, you know, when you get to that larger scale thinking like the design architectural kind of thinking. You know, you can't fit that many tokens into an LLM today, but they're getting, things are getting better with the reinforcement learning where. So if I ask it for like to analyze like a giant chunk of code, I can give it in a bunch of pieces and I'll add something to it.

Right?

It'll add to it, but it will truncate and abbreviate subsections so it can fit in the token box. Right. So and then I have to go manually merge that. Maybe Chris will figure out how to just merge that for me soon. But it can. There's still a lot of room, I think, to grow in terms of scaling up, not just with tokens, with methods and, and procedures to scale it up to pretty good size in the near future.

Yeah, just to add on like I think to Jason's point, I've already seen this. I've talked to friends who you know are at large tech companies and they were saying the amount of time they're spending with new college hires interns is dropping because they just personally don't see the value. It's like if they're asking a new engineer to hey, can you go do this function? They spend time explaining, explaining, explaining it. And then, you know, eventually this person gave up and was like, I'm just gonna see if I can use ChatGPT. And within 30 minutes they were able to get the function written that they were trying to describe and iterate to this intern. And it's like, I think those are the things that as a junior engineer, a junior person, you need that mentorship. And I think now with ChatGPT, if it's all about just functional export of code, how much code can I write? You're going to lose, I think some of that human side component for the junior engineers, while your senior engineers are going to probably get a lot more exposure and a lot more, you know, responsibilities because they can handle it now and, and execute at such a high level with ChatGPT on their side.

Chris, you did write an article, I believe, on DevOps with AI and I think you call this stacking. Is this what you mean when you allow developers to get higher level input by leveraging AI to write code and then doing other things? Could you explain stacking, I guess within the DevOps cycle?

I'm trying to recall the exact stacking aspect.

I can say one thing. While Chris is rereading what he wants.

I think that yeah, everybody needs to up level in this. Even if you feel secure and senior level engineer, you need upload. Because the weird thing is once you get beyond the simple functional things like Chris is saying, the you actually have to be when the thing breaks or gives you the wrong code. Usually if it gives the wrong code, gives us a code for an older version of some API or something. But if it can't fix itself or understanding how to merge it into existing larger code bases is actually a real high level skill. Like that's actually the hard part of engineering and making sure you're not incorporating new bugs and you can review and understand what this generated boat is doing. So it's a higher level function actually. People need the up level. So there's a lot less room at the. Even less. Far less room at the bottom rungs for junior engineers to be value add. They're dangerous. I think actually more so these days.

Yeah. So stacking's just a technical concept with that. I wasn't listening to what Jason was saying necessarily on that. I assume it was very, very deep and insightful.

Orthogonal.

Orthogonal. Same, same. Right. But yeah, so stacking is really just a technique where basically you can generate like a memory. Right. So traditionally with these GPT models, the memory is just whatever you've chatted into it in the past, but the memory can be these larger scale, you know, code bases, for instance. Right. And so people are starting to play in that space of like, hey, could you bring in context from the code base, search the code base, find the relevant context that might be in there and try and break the task down and you know, componentize it down to that level. It's early days on it. We'll see. I have not yet seen this work in demos and maybe if someone has it working, they're keeping it secret to, you know, because they've offloaded their job. But you know, I think that's going to be a concept where bi essentially the, you know, you give it a high level task and then you can kind of stack up the tasks within it and it can just kind of use this RL agent to just reinforcement learning agent I should say and use that to then execute GBT to figure out what it needs to do. Take the memory, understand what is already there in the code base or what's in this code file, pass that context into GPT and kind of keep this loop going until it completes.

And actually, can I add Chris, you get me all worked up. Is that the. I think that there's. We're also talking about what's happening like right now. By the time this is pushed to the web. This may be irrelevant. Some of it, like there's already papers coming out in the last couple weeks, even last week, where, you know, people are getting so like, you know, the complexity in CPU cost goes up exponentially sense, relative to the number of tokens. But some I've seen some papers come out where they're actually making it sublinear. So meaning you can actually maybe get a million tokens and maybe if you just waited instead of like, you know, two seconds, if you waited like 10 seconds, can you have a huge trade off in terms of the amount of, of tokens that can be processed by the LLM? So we might actually, I won't make a bet here, but I'm a little optimistic on this. I think that. And the model sizes are getting smaller actually too. So they'll fit on the phones soon. And that also means less memory and less compute to execute them. So I think that we'll probably laugh because maybe in 12 to 18 months when I say that it's probably six months that we'll be able to fit entire generally decently sized code bases in the tokens here pretty soon. Like it seems to be real engineering work to make that happen. And that's the huge bottleneck we're focusing on right now. And I think that it's. So we'll probably solve that problem in the next four months.

So in DevOps, does that mean then we'll have even more frequent releases? Because the amount of work that could be done for us automatically there's, I.

Think, an entire world where you could basically just have, you can imagine a world and I don't think it's crazy at this point to talk like this, but a world where you basically say you're optimizing for revenue or you're optimizing for clicks on an article or podcast, right? And then it just goes. And you know, you have a whole agent that's constantly generating new variants, testing those out in production without any human input. And just the entire optimization is how many clicks, how many people can I get to buy this thing, right? And they can, you know, configure the buy now button on a Amazon page for colors automatically. They can think of different ways to say add now or buy now. It can have. You can give it the flexibility of here's the entire code for the front end. Tweak whatever you want, you know, optimize the backend releases if it's. You think that it's slow on the database. Lookup time, right? Spend time, optimizing that. So yeah, I think releases are going to go faster, but I think that there's a potential in a world where the releases themselves could just be fully autonomous, where you just give it this task, you give it a goal and say optimize for X. And it's just a continuous release that's constantly testing itself. No humans in the loop at all. And as long as you give it the kind of parameters those guardrails of like don't crash, optimize for, you know, this thing, like it'll just go and it'll find the best path. And I think that's starting to become more and more of a reality or a probability.

So are you saying the only goal of the DevOps engineer then at that point is just setting parameters?

Potentially, yeah. Setting up logging, setting up the core infrastructure to actually allow it to monitor those parameters. Right. So if you gave it, you know, you integrate open telemetry, you integrate the, you know, all the API call times, you give it the access to the finalized metric of, you know, subscriptions or purchases or whatever, it could then handle a lot of those. It could look at, monitor the logs and say, hey, I did this push. Now I'm seeing a bunch of crashes on the back end. Let me roll that back. Like there's, I think a world where if you have enough of these agents, those agents can then, you know, self optimize. That's my personal thought anyways on the matter.

Jason.

Yeah, you can tell, you know me enough, Joe.

You know, I'm like, I'm confident, eyes are plugging.

I like.

Just, that's a problem I've had since.

The doctors were like, it's okay.

Good thing my kids, it's not, it's not.

In case they're watchless there. But no, I think two things are important. I think one is that I think Chris is right about this kind of autonomous new layer being introduced. And it's not that ridiculous because we used to have a bunch of DBAs, database administrators. Where are they? But it's because we've had higher level tools and these things become standardized and commoditized. And the things he's talking about, right, there are things that we used to worry about with databases. Right. It's just now it's more complex because it's compute storage, you know, memory distribution. So I think it's very plausible. And that stuff's happening right now. The people, there's companies starting, they're working on that right now. But I think the simplest, most tactical thing to think about it, to make this in terms of LLMs have an impact on things today is you used even like Chris said, this Chris just said, you know, oh, here's a bunch of. We've seen a bunch of crashes. Go try to fix it, right? That loop, I think, I don't think it's going to be the place where the DevOps, because there's a bunch of crashes. It's just going to look at the logs and there's a. There's a crazy thing because I think people are using lms like, this is worth at least a hundred dollars. I think this little meme here and that is that people are looking at LLMs to do what they do today, right? Like they're trying to reproduce the same flows that they have today, the same processes. And then you having a function that they have to code the heuristics into that looks at the data, right? And they're trying to put those heuristics into ChatGPT or these LLMs. You're saying, hey, if this and this and this, then this is bad. And that and then analyze this data. I give you, it's. I think it's funny. The LLMs are actually far more capable of reasoning than people give them credit for. I'm still discovering it myself. You can just say, here's a bunch of my app engine logs or here's a bunch of my SQL logs. Is there anything wrong? You can just ask it that and guess what? It comes back with intelligent analysis, even an arrest call, and just say like, is there anything wrong? You know, it'll do. It'll tell you things like, it looks like there's a password in the clear, right? And you go like, huh? So you don't even. It can tell you things that you didn't even know to look for. So I think we'll transition to that eventually. But we're in the mode right now where everybody's reproducing their functional code in LLMs, but I think they're just. They need to. They can skip a level and skip a step and just say, hey, what's wrong with this stuff? And it will give you very interesting information back. You don't need to hard code everything.

What do you think about the adoption of AI in DevOps, though? Because you're all so close to it, it seems like, yeah, everyone's doing it. But even with automation, I know there's still companies that have no automation in place for AI. How do you see the adoption of AI in DevOps? Is that something that you think the tool vendors will start incorporating and so but just by that people that already use tools will then have AI or is it going to require someone to initiate incorporating AI into the DevOps pipeline? I don't know if that makes sense.

Yeah, I don't know. Jason, do you want to go first this time? I want to be polite and a gentleman.

Yeah. Then you can correct me after. Yeah. I think it's the same story with. Yeah. The power of AI is this generalistic, it means generalized. And so the problem with test automation was that everything was hard coded to your one product. Right. The power of AI is that it can work on arbitrarily different number of products and it has context across products and can be reused. So I think the big advantage will be to the vendors, the datadogs, the splunks, that kind of the world and people will just get the magic of AI because they have a subscription and it will just appear in their tooling. I think similarly to testing, I actually advise people not to learn AI like testing Vendor world's another story for another podcast. But it's not something it's very difficult to learn, very difficult to deploy, very difficult and expensive to maintain. And the reality is only like the Google's dropboxes and those kind of people in the world can really deal with the sophistic, the complexity of maintaining an AI system I think in production because not only is it complex and outside of their core value add, but and that vendors can do it cheaper at scale but because you actually can't find these machine learning engineers on the planet like they don't exist. Even Google has a hard time finding enough AI engineers for all their products. Like literally they don't have enough. So the odds that some bank in the middle of America is going to have access to some great AI engineers to deal with all the complexity themselves is very low in the near future. So yeah, so short answer vendors I think is the major adoption and way to go.

Yeah, I think similarly I think the vendors are going to have the better location for it to exist. So I can imagine like a datadog or a splunk or something being able to add ChatGPT that's trained on your custom data automatically with you know, the click of a button, but it's already integrated and all of those components over say a DevOps engineer is supposed to learn AI, understand it, how to, you know, rebuild all the stuff that you know, say a spelunk datadog, what have you has done just integrate the AI. I think there's low hanging fruit today that people can use like LLM. Like Jason was saying, you can call ChatGPT with your app engine logs and it can tell you, hey, your password's in the clear. There's this might be an issue. That's awesome. But then there's the entire productization around that. And that I think is okay, well, it's easy to kind of get that mvp, but then to make it, you know, scale and make it stable, make sure it can handle the volume, make sure you can do it at an efficient cost, like those are sorts of things where it's like it starts to break down outside of that initial piece. And I think the vendors are going to spend the time to really figure out how to make this scale. I think the, the interesting part to me as I think about this and traverse everything is not necessarily the world of today, but kind of going back again what we were talking about earlier, like the world of tomorrow. So everyone right now is thinking, how does AI and LLMs and OpenAI affect my products services work of today? But it's like if you kind of map out from where we are today to where we're going in that world, like if everyone was already using AI to build their products, AI to maintain and scale their revenue and, and those aspects, what would the DevOps world need to look at for that? You know, so what does that world look like from a coding perspective, from product perspective? Right. What does that look like if now instead of a team having launching 10 features this month, they can launch a hundred features or a thousand features, what does that do to the whole, you know, people side of that? Like do you need a thousand PMs or will LLMs take over the PM job too? Do you need a thousand? You know, if you're 10xing the number of features, 100xing the number of features you can launch, does that mean you need to 10x the number of DevOps engineers that you launch today or make them 10x more efficient before you could even launch those 10x products? So I feel like it's a broad front that's going to have to move forward. But I think if you, you project out like things are going to have to change. If everyone's launching 10x more stuff and the ability to do 10x more stuff in the product world, then it means that, you know, you're going to have to have 10x more, you know, server capacity or 10x more, you know, monitoring capabilities. And those I think are going to be the, you know, next limiting factors.

So, you know, you both mentioned how junior developers, junior DevOps engineers seem to be getting replaced. So what skills then, should a DevOps engineer focus in on? And Jason mentioned, don't even think about AI because it's. It seems so complicated. What should people be doing then? And I know you said it's foolish to think 18 months out, but, I mean, are there any skills you see that will be. Will endure throughout the years as a DevOps engineer?

Podcasting?

I hope so.

You're a genius, Joe.

We laugh, but you'll be the guy that shuts the lights off.

You realize the only one in the, in the room.

You're like, well, one human and probably a thousand bots.

Right, right, right. Yeah, yeah, that's right.

It'll be a grid.

Grid of sycophants. Yeah. I think it's the same old story, though. People need to level up, I think, number one. But level up means again, not just like, get more languages, more skills, more Java, more frameworks. It's more of a. Because the building too, people don't really talk about it. I think much is if you can write Python, you can convert it to JavaScript, you can convert it to Swift, you can. So this whole. All these engineers spent all their lives, like, I've. I know seven languages. Well, for us, like, congratulations, it doesn't really matter as much anymore, but it's more around, you know, level up generally, but, like, understand the business aspects of what you're doing. Like in DevOps, like, understand, like, you know, do you spin up 10,000 new machines? Because you can. Because you can deploy 10,000 times faster. Like, does that help the business or not? What's the incremental value to the business or risk to the business in doing these things? That's. I think that's where people need to kind of level up. But it's. Yeah, we're going to see that across. But that's. And by the way, we're talking about DevOps, it's just kind of a universal theme across humanity, like, that people will need to do that. Like, what does a doctor do? Right? What does a lawyer do in the coming couple of years? I think it's the same thing, not just tied to DevOps or programming.

So everyone's in the same boat. So if for some reason it did replace all jobs, and if it replaces all DevOps jobs, it can replace lawyer jobs, I would think, and other jobs. So something would have to come up. Maybe some sort of universal wage or something.

I don't know.

Yeah, there'll be A lot of podcasts.

Yeah, no, I think that universal income stuff is somewhat rational. Remember when I was working early on test A.I. chris, remember this? I'm nuts. But like I was definitely. The lawyers didn't like it and the board didn't appreciate it but I wanted to give equity to the bots and then also have a tax on them that was refunded back to the humans through bonusly. So I think we need to figure out schemes, actually figure out some schemes like that event. But you know I think sorry, this is way above my pay grade but if you already look at the trend of history and technology like you know almost the majority of the US economy is services based. It'll just be more services. So everyone will get their nails done Joe, not just you. And you know, we'll watch a lot more Netflix auto generated Netflix with whatever characters you want in it. I think it'll just be gravitating toward a world of leisure and, and I believe it's a little bit nutty but I believe also towards surplus net net so. So people be more creative and have more fun.

Like.

Like rather than get up in the world morning and worry about those servers going down. Right. You might think about new topologies, new compute paradigms or new ways to, you know, maybe you spend all your day programming but you're programming your entertainment.

Right.

And making it personalized and personalizing all these services may be a chore, the chore of the future. But it won't happen overnight. But it's. I think it's already hitting industry today like that.

Yeah. I think somewhat similarly I think there's definitely a world where yeah it's the issues of today where it's like you have to have a job to you know, meet ends meet and it has to be a high enough paying job that's specialized might change, right. The more creative art type jobs, passion jobs might start to become real value add. I think there's you know, things that help the community and volunteer work might become a real thing as well, right?

Apocalyptic. For like my parents.

It might be a happy apocalypse.

Yeah.

But you know, I think the other thing to think about is like if you kind of look in, in the technology space, right. As things have been more democratized it also has allowed for smaller entrants and more niche areas to come up and sprout up. Right before like say AWS came out, you had to start a website. You had to have a server and pay for an entire server rack and all that. Now you can start website a Server running in a cloud that's always on for five bucks a month. So it's like, you know, it really lowers that barrier to entry. And this might do the same. If you wanted to go, you have an idea for an app that you want to go build that does X, Y or Z, you can go ahead and now build that with ChatGPT without having to spend, you know, months learning Android or iOS. To Jason's point, you know, you don't have to learn these new languages. You can kind of go ahead and execute that vision without having to have that resource and all of that, you know, amount of time that you would, say, spend to learn, learn that. So I think there's a possibility that maybe it's in the short or medium term, but it's, you'll see, I think, an influx of creativity and an influx of different services and apps that come out. So, yeah, the general kind of low level, generic type of positions maybe are less interesting, but that will allow people to specialize more in interesting and creative ways that we just don't necessarily know about yet. And so I think there's a bit of that. And so if I were, you know, kind of the initial question, if I were a junior engineer, it's, yeah, I think focusing on understanding the technology, understanding the business, understanding users and how to think through that, that process and that architecture level aspect of what's the roadmap, how to think, break a big problem down into sort of, you know, manageable chunks. That's, I think, still going to be keys to success, at least in the short to medium term. You know, once you get it down to a small enough level, right. AI can automatically handle that. So if you break it down to a method level and you write a description, it can generate that code for you at that method level. And I think we'll just start working up to, you know, next it'll be able to write a class for you and then it'll be able to write, you know, microservices and this and that. But if you kind of keep that mentality right now of, hey, how can I, you know, what is the core business need? What is the, you know, what's interesting to me? What is, what am I willing to spend time on and learn about? Those are kind of the key things. And then this is a powerful tool that'll let you, you know, accelerate that, take that idea into a product.

Great.

So there's one other question I want to ask. I think, Jason, you recently kind of touched on this in your latest post. You know, as AI becomes more involved in things like DevOps, how could companies make sure, you know, there's ethical considerations and human values are not compromised, that the models aren't biased, you know, that they're not doing bad things, they actually are doing what they should be doing. Last question. But like, if you could sum up, how can that be done?

Well, I've failed because my main takeaway from all of that is that my main point on that is actually that I would say is that all models are biased. Like, they just are. Like by nature they're bi. Like the only unbiased AI system is one that's not been trained yet. You're biasing it by training it. And so it's not just. But there's good and bad biases and that's an opinionated thing. It's also going to be specific, business specific thing often too, right? Like, and there's going to be business specific biases and stuff. So I think it's more about understanding what biases you want and do not want in the system. And I think time evolves here, just like, you know, open. So I think the right. This is actually, I'm very. We've only got like one minute, but like, to save the planet from AI and save DevOps from AI, the most important thing to do is, is around reinforcement learning, I think in terms of guiding the AI to behave the way you want to. And we'll see a bunch of this coming out in the next six months where you take the core models, then you personalize them or you customize them using reinforcement learning and other techniques to make them conform to your biases. But they're going to be biases. The question is, and this is part of that leveling up, right, the DevOps person, you know, today some of the smartest DevOps people are just going like, you know, I don't like crashes and I like efficiency and I like failover, right? Like, they need to uplevel and say like, well, what level of, you know, what cost benefit ratio is there for crashes or for bad connections to my infrastructure or, you know, what is tolerance of this business? For example, in different times of the day or different times of the week or you know, shopping, you know, Christmas shopping season or something? So it's going to be applying those policies and then making sure that the AI conforms to those is going to be a problem. But it's going to be, again, I think it's going to be a collaboration between that DevOps engineer and the infrastructure that they have at hand like their, their data dog and, and all these other things. It's going to be a collaboration between them. Like I think Kevin Powells actually said this last week, I think pretty well he just called it like it's AI partnership. So it's like it's not like AI or the human and it's not just even collaboration because really is AI collaborate? No, we're, we're still in charge but we're partnering with those AI systems and partnering with those vendors to impose our whim on them. Right. We want all of our DevOps tools to actually act like Netflix. Like they know we don't like horror shows and they know we like cartoons. Sorry, that was a little too much. But like my kids use my Netflix so it's awesome.

We got very personal there.

I know Especially yeah, bananas and pajamas by the way, is awesome. So. But you want to be personalized so there's a lot of work in that. But it's out of the drudgery and it's more as a higher level thing. But you have to come up with your biases that you do want can eliminate bias and then help impose that will on the AI.

Yeah. Human intelligence layer I think will still need to be applied to these where you can look at it, look at that system and actually say this is correct, this is incorrect. But there's a theoretical question I have which is like, is there such a thing as no bias? Because if you were, you know, in the DevOps world, right. If you were to optimize for, you know, the servers to be able to support this new feature that requires high latency in doing that and making that goal happen, you're now biasing against people who have lower latency, Right. Maybe more in the rural areas of the country, let's say, where there isn't, isn't as good of connection. It's.

Don't be rude to Joe.

Yeah, well, so I mean like if, but if you're on like a 3G, you know, driving, right. Like you're biasing against that person because you're optimizing for the user experience of someone who's on a WI fi connection. Right. And so I think that's something where it's like, yeah, similar to what Jason's saying and Kevin had saying as well is like, yeah, it's. I think you need to have a human layer, human in the loop on that to be able to then help guide it and figure out what the business needs are and what the, you know, core competencies of that Is.

Yeah. And if you don't need that human in the loop, you're commoditized. And a lot of stuff will be commoditized. Like you can say at some point the AI will just know, like, guess what? These are the optimal preferences for a streaming service or for an email service or for, you know, whatever kind of business you are. It's DevOps are needed to provide biases that are just non obvious and non standard and business specific.

Cool. Okay, guys, before we go, is there one quick actual piece of advice you can give to someone to help them with their DevOps AI efforts? And what's the best way to find our contact? You, Chris, will start with you.

I found the fetal position as you think about this and just rocking back and forth is helpful, but no, I think, yeah, it's a good question. To me, it's, don't get too freaked out about it. I've had, you know, the kind of joke. I've talked to a lot of founders and a lot of people in the AI space and it's, you know, someone's like, oh, I'm really freaking out having a panic attack about this. And the joke is, oh, is it your first one? Like your first panic attack about this? Because it's, you know, every single time, you know, something big comes out, you think, oh, that's the step. And then you kind of can calm it down and realize that there's these stuff. So I think, you know, don't freak out and just realize this is a tool just like any other tool. It can really help accelerate your career. It doesn't have to be, you know, you don't have to view it negatively. It has a lot of really good benefits to society, the world. You personally, you know, if you know how to use it, leverage it and it can, you know, allow you, if you have a side project that you've always wanted, it can help accelerate and make you overcome that obstacle of I don't have that resource to do it. It can be that resource. You know, if you wanted to write a book, it can help you write a book and, you know, really help that or a new app or this or that. It has that capability. So, you know, viewing it, I think more positively than fearful of my job's going to be disrupted because it is here. It's going to be here. And so, you know, trying to accept it and understand how to use it, I think a much better use case than trying to fear it.

Awesome.

Jason, you missed a book plug opportunity.

Yeah. Check out my new book coming on Amazon.

No, Joe's got a new book coming out.

Yeah, sorry, I was going to say what Chris said. So. But I think very technically I'd say two things. One is on a very technical front, the thing I think people are not doing today across just DevOps in general too is people are looking for anecdotal evidence of AI is good or bad, like it's Boolean, and they're looking for like anecdotal evidence. I would say if you're trying to measure the goodness or badness of any AI kind of based system at all, the AI itself or products built on top of it, if you're not doing sampling, you're very likely doing it wrong. And so think about, about how to do that. Like how to do. And that's really like, you know, measurement at scale. If you're not doing that, then you really kind of don't understand nuances of the system that you're, you're measuring. And I would add also, you know, the best way to push kind of AI and automation forward is to get an enterprise license for testers, AI and.

For links of everything of value we covered in this DevOps toolchain show, head on over to testguild.com P113 and while you're there, make sure to click on the Smart Bear link and learn all about Smart Bear's awesome solutions to give you the visibility you need to deliver great software. That's smartbear.com so that's it for this episode of the DevOps Toolchain Show. I'm Joe. My mission is to help you succeed creating end to end full stack DevOps toolchain awesomeness. As always, test everything and keep the good. Cheers. Hey, thanks again for listening. If you're not already part of our awesome community of 27,000 of the smartest testers, DevOps and automation professionals in the world. We'd love to have you join the fam@testguild.com and if you're in the DevOps automation software testing space or you're a test tool provider and want to offer real world value that can improve the skills or solve a problem for the Guild community, I'd love to hear from you. Head on over to test guild.info and let's make it happen.