Will AI Replace DevOps Engineers, BI and More with Mark Koelsch

Get ready to discover some of the most actionable DevOps techniques and tooling, including performance and reliability with some of the world's smartest engineers. Hey, I'm Joe Colantonio, host of the DevOps Toolchain podcast, and my goal is to help you create DevOps toolchain awesomeness. Hey. Today we'll be talking about what DevOps leaders need to know about AI driven business intelligence. Joining us today is Mark. If you don't know, Mark is the North American President and co founder of agiloway, a global software development firm blending Eastern European engineering with US leadership. With over 20 years of experience in data analytics and AI, Mark helps businesses turn complex tech into real world results, especially in high stake industries like healthcare. So if you're curious to know more about how AI compliance and innovation intersects, you don't want to miss this episode. Check it out. Hey, before we get into this episode, I want to quickly talk about the silent killer of most DevOps efforts, that is poor user experience. If your app is slow, it's worse than your typical bug. It's frustrating and in my experience and many others I talk to on this podcast, frustrated users don't last long. But since slow performance is as sudden, it's hard for standard error monitoring tools to catch. And that's why I really dig SmartBear's Insight Hub. It's an all in one observability solution that offers front end performance monitoring and distributed tracing. Your developers can easily detect, fix and prevent performance bottlenecks before it affects your users. Sounds cool, right? Don't rely anymore on frustrated user feedback. But I always say try it for yourself. Go to smartbeard.com or use our special link down below and try it for free. No credit card required. Hey Mark, welcome to the guild.

Oh, thank you Joe. I appreciate you having me on.

So hey Mark, before we get into it, I guess like it sounds like you have a lot of, a lot of experience in this area. How did you get into what you're doing now and finding your company? Why did you find found your company around this particular topic as well?

Yeah, yeah, certainly. How far back do you want me to go? So I was actually working as a chief architect with an AI. Well back then we called it machine learning company here in Austin and we had a team of about 45 developers from the Ukraine. And so later on after I had left that company and sort of my tech lead with that group was leaving, he reached out and we, we joined together with a couple other senior people from the Ukrainian company and we built Agil away and that's how we started 10 years ago now, I believe.

All right, so you mentioned it used to be, you know, obviously machine learning has been around. AI has been around for a while. What is, is there anything you were surprised by? By AI now being all the rage since ChatGPT came out maybe 3 or 4 years ago?

Why? Well, part of it is obviously everyone thinks it's new.

Yeah, exactly.

So I'm sure you know, as well as a lot of your listeners know, it's actually been around for quite a while. So I spent 10 years actually with machine learning, the AI company. Basically, we were doing price optimization for B2B companies who had negotiated sales. And so we would identify segments within the data that gave the best indication of sort of a price envelope, a range in which reps can attempt to achieve an optimal price. And the biggest thing is, I think the fear I find funny. You know, everyone's worried about the Terminator coming. I think you and I both know something like that's never going to happen in our lifetime. You know, my biggest concern is probably giving too much authority over to AI. I worry much more about, you know, an accident of some type involving human lives from a glitch essentially in the software utilizing the AI and not in the technology itself. I mean, anyone that's done software engineering before knows there, there are bugs, there are problems that come through, and your software is only as good as the developers and your QA team, the people that have gone in and tested it. So I don't worry probably about all the same fears that newscasters talk about every time they bring up AI. But I do worry more about how much we're already willing to hand over to AI applications to take on.

That's a great point. I know generative AI could be very persuasive, that it's thinking and it's doing everything correctly. How do you get by that bias then, that people. I'll just trust it because it sounds like it knows what it's doing, especially when it comes to compliance data, especially in healthcare and those type of areas.

Yeah, and first, don't get me wrong, I think AI is going to add a lot to industries, especially to healthcare. Right. There's a lot you can do with health with AI into analyzing MRI data, you know, looking for patterns and data that humans may not readily see to really assisting doctors and research in general. You know, I think a lot of it just comes down to learning more about what AI is and looking at, you know, the feedback and the Data that comes out of AI learning that it's not the Terminator. Right. It's handing over an MRI to an AI application and having it look for patterns or subtleties in the MRI results that maybe you didn't pick up on immediately and showing them how they can use that and teaching them that it's, it's an assistant. Right. It's not, it's not there to replace anyone. Right. I don't see AI generally replacing people in their jobs, but in, in making people better at doing their jobs and giving them the additional tools that makes them more efficient and makes them better at doing their jobs in general.

Beautiful. So once again I want to circle around health care just because I used to work at GE Healthcare and radiology.

And Good old centricity.

Yeah, yeah, right, yes, exactly. And the problem with that is anytime we try to do any type of innovation, they'd be like, oh, the FDA isn't going to allow this, I assume. Do you still have these conversations? Maybe I'm just older, I haven't worked in that in a few years. Where people are like, we can't use AI because you're going to be training it, we're going to be releasing data that we shouldn't be. Do you have those type of fears from your customers? I guess is of the big questions.

Yeah. I actually think that's an enormous concern right now. Right. Because AI works by being able to process very large sets of data and finding the patterns within that data. Well, unfortunately, a lot of the data in the healthcare world is phi. Right. And so how do you regulate who has access to that data? How do you regulate what happens with that data? How do you make sure that everything you're doing remains HIPAA compliant as you're utilizing these new tools, bringing these new tools into the market? Especially when there's so many startups that may have a lot of developers and leaders that don't have that healthcare background and haven't had to work directly with HIPAA requirements before to make sure that they're always, you know, within the bounds that they need to be.

So couldn't you train like an in house AI on regulations from FDA or hipaa, whatever, and just as you're trying to test it or develop it, that you say, okay, based on these requirements, am I meeting these regulations because I have all this data that it's able to analyze?

I mean, I think that's part of it. Right? I mean, just like you would with any secure data, you need to start by building out a plan in the beginning. Right. You need to define who has access to the data, what roles specifically. Right. A lot of HIPAA requirements come down to defining the roles of people that access the data and what they can actually access. Because the people that are, you know, potentially writing the data, the data scientists, aren't necessarily the same people that should have access to phi data. Right. You can anonymize data and still give that to your developers, allowing them to continue to do their job without essentially opening up protected health information to those same developers. Right. You have to essentially take out any known identifiers within the data to make sure they can't tie it back to a specific person. And so I think, you know, working up front to define your plan, to define who has access to data, define who the, you know, what the specific roles are, and definitely, obviously bringing in lawyers who could tell you exactly, you know, how to handle each of the scenarios, how to write your contracts, how to alert, you know, patients that their data is going to be used in specific manners, how to work with your baa. Right. Your business associate agreement to make sure that the people you're working with, the other tools you're working with, are also legally liable to protect the data in the same way that you are. And I think it's, you know, like I said, the lawyers are probably, you know, possibly one of the most key components of that, because I have my knowledge from working in that industry. I was a CTO with a healthcare company for a number of years, but I'm certainly not a lawyer. And I think at the end of the day, the lawyers of the background in healthcare information are really the ones that can protect you to make sure that your company is not taking steps that are going to get you in trouble or leave you liable to lawsuits down the road.

Absolutely. So circling around once again, business intelligence, you've been dealing with it for over a decade, I think, or for a while.

Several decades.

Several decades, yeah. So how has AI, and you've been using machine learning for a while, how has AI, or as we know it, fundamentally changed? Maybe how BI has been used in the last few years?

Yeah. From my perspective, a lot of what was out there maybe 10 or 15 years ago is just much more readily accessible to users today. Right. You no longer need a team of data scientists to go in there and identify the patterns in your data and to build the analytics around it. There's some great tools coming out now that can actually go in and they model that data for you. Right. They're proactively identifying those Segments identifying those points within the data and providing you the information without the building the infrastructure to do that work, you know, and to then provide it back to the end users. So I don't necessarily know, and I might get in trouble with some of your listeners on this. I don't necessarily know that we're doing anything more than we had done before. I just think we're, we are taking on a lot of that and doing it for the end user without having to have, you know, the full team analyze and preparing your data ahead of time.

All right, so you have a pretty good balanced view of AI, which I haven't been hearing a lot of people. Like you said on. My audiences probably hear a lot of things, especially from Microsoft. They just laid off, I don't know, 6% of their company and saying they're going to be replaced by AI. There's a lot of initiatives by companies saying they'll be replaced by AI. And you're saying AI really hasn't changed, you know, from machine learning for how you've been dealing with it all the years. So where's the disconnect then from what we're being told to what it can actually do and where's the middle?

Yeah, no, don't get me wrong. I do think repetitive actions are going to easily replaced by AI. Right. I think you're going to see this of all places, more potentially in DevOps in other areas. Right. Oh, you look nervous. You know, configuring new environments, setting up new accounts, these sort of things that obviously have some complexities to them as you move from one environment to the next or based on what you're doing. But you know, once you've gone through that, that process and you can create scripts to do a lot of this, that's something you could easily hand over to AI. But I think as you do that, obviously more development opportunities pop up for AI. Different, different jobs come about. But I do think when we talk about, outside of the software world, when we talk about doctors, when we talk about engineers, there are a lot of jobs out there that I just don't see we're ever going to remove from our society. I think AI is essentially just going to give them additional tools to give the them the ability to be better at doing their job, to whether that's faster or safer or just with more information at their side while they're doing it.

Yeah, even in with healthcare, I heard that, you know, AI is going to revolutionize or change the game, be a game changer. And it's going to replace doctors and it's going to be easy to diagnose you because, you know, it's been trained on all this data points. So once again, it sounds like maybe an overreach.

I think it's an overreach. I think, you know, for a lot of people that aren't in the tech industry, a lot of this is new to them. Right. You and I have seen AI and machine learning behind the scenes for 20 years. Right. And so we've seen it coming, we've seen how it's used in different environments, in different industries. And I think, you know, there is definitely, you know, the James Cameron fear factor that comes into play when we talk about AI. Again, I by no means think there won't be some jobs that are going to get replaced, but I think those are, I'm trying to find the best way to phrase it. I don't want to say mundane, but the repetitive tasks within jobs I think are much easier to replace with technology, such as AI or just technology in general. And I think, you know, these jobs that take a lot of evaluation and human interaction, they're not going to get replaced. I think what we're doing with AI is simply giving them the tools to do their jobs better.

Yeah, 100% agree. I don't think jobs are going to be replaced more than they're going to probably morph or change, obviously. And if people that embrace that are going to continue and people that don't are going to. Because I see a lot of probably DevOps and programmers that like to do the coding and the busy work where they put on their headphones and they go to town. But AI can probably do a lot of that for them now and they may not like the, the type of activities they have to do now.

Yeah, I guess.

I don't know.

Yeah, I mean, just think about coding, right?

Yeah.

The basic part of the code that you're writing before. Yeah, that's probably something you could throw over the fence. AI can spit out some code for you. You're still going to need to evaluate it, you're still going to need to look through it. But for me, the most exciting part about writing code, way back in my career when I was still doing it, was solving the complex problem problem. Right. Figuring out that complex angle in which to evaluate something differently. And in a way that's going to give you the functionality you're looking for that's not so obvious. And I don't think AI is going to be able to do that for you. It may be able to help you write the code once you figure it out. But the problem solving, it is still going to be there to a certain extent. You know, I think history repeats itself. Right. And neither of us are old enough to remember, but from our grandparents, when computers came out, these same conversations were happening. They were talking about, oh, we're going to lose all these jobs. Computers are going to do them for us. Right. Did some things go away? A couple things. Right. But a lot of industries just became stronger and better through the use of technology like computers. And we're just seeing the evolution of that, the next step.

I like your point of view because once again, I've been seeing articles, maybe they're just clickbait, like universal income has to become real. Everyone's going to be displaced. And I mean, yeah, I don't see it. But if, if one person is affected, then everyone would be affected. So what's the point at that time?

Yeah. And good luck and getting a lot of people to agree to universal. Universal income.

So great. So you know, I forgot to follow up. You mentioned there's a lot of tools out there to help with BI and AI. Any tools that you can mention that you use all the time or frameworks.

My favorite tool, and it's only because I've been using it for a long time, is Tableau. I think Tableau took PI to another level by making the whole interface so intuitive that non technical users could come in and build reports that they needed and complex calculations within those reports. That said, obviously there's a lot of great tools out there. So I don't want to diminish, you know, the quality of something like MicroStrategy or any of these other tools. But you know, we're starting to see BI applications and even agents starting to be embedded into tools like Tableau. Right. You know the next step, let's ask an AI agent, how do I do this specific task within Tableau? And it'll help you take it to that next step. But it goes back to our previous point. You know, that business analyst is still there using Tableau. They're just being able to do more complex analysis of the data based on the tools that they have.

Absolutely. So if AI machine learning aren't really going to replace us, they're going to enhance us. It sounds like what are some fundamentals people need to get right about BI then? So they need this solid base in order to even succeed with AI.

Yeah. And maybe not even just an AI. You know what's funny is I was looking at BI trends earlier just kind of stepping through some things that were out there. And when I asked Google, you know, what are some of the new BI trends? One of them that Google AI told me was no longer are we creating reports that just show data, but now we're creating reports that give businesses actionable items. And my first thought was if you Weren't doing that 20 years ago, you had no idea what you're doing within bi. So I, you know, it's, the more things change, the more they stay the same. I think, you know, maybe you don't have to have the same level of data science knowledge and mathematics to get as much out of analytics today as he would have 10 years ago. Because you know, like I said, a lot of the tools now are embedding these AI agents into it. They can do some of that analysis for you as long as you know what you're asking for. I honestly, I think that's going to be the biggest trend we're going to see in BI over the next couple of years. That and just the continued integration of unformatted data. You know, the same migration we saw from data warehouse to data lakes pulling in some of that unformatted, unstructured data more readily available for analytics and easily matching it to, you know, your already structured data.

Yeah, I mean there's gonna be explosion of all the being able to access all this data. But do people know enough about they have a BI problem, what they need to get? I guess if someone's a competent BI person they would. But like is there anything like all this data you can do reports on everything then like how do you know what to focus in on then? Because I feel people might get inundated with types of reports and things that they could look into.

Yeah, I mean maybe that's going back to why I don't think the BI analyst job is going to go away. Right. I mean you have people that have that experience know you start by building analytics by asking them what the question is that they're trying to answer. Right. Understanding where you're trying to get with that data first, then you step your way backwards to build the analytics, to build the answer in a way to provide you with those results. And so I don't think the BI analyst role goes away entirely. I just think it becomes possibly a little bit easier to do going forward.

Yeah, I guess it's still not old fashioned, but know your audience, right. Because I used to do performance testing and would spit out like a hundred page reports of all these technical Data and you hand it into your executives, they're like, I don't care about this. Be able to summarize it right to the right audience. Know your audience, I guess. Still a problem.

Yeah, exactly. It's just today you would ask ChatGPT, give me an example of what summarized tech specs look like and then you could just plug your information in from there.

Absolutely. Right, right, right. So when a company contacts you, what are they usually struggling with though? Like what's the. Maybe a little plug for what you all do for when people may contact you all for what they're struggling with. Bi. Like what do you help folks with?

Yeah, so glue. We're actually a custom software development company. So we have roughly 250 developers across Eastern Europe. And so we help small and medium sized software companies augment and grow their engineering teams. And so a lot of time when they come to us, they're just looking for expertise in specific areas, whether that's in building out their data warehouse, data lake, integrating additional analytics into their systems, or you know, developers to actually build out what their actual application is. That's wrapping, let's say, the analytics that's there. So really we provide experience, very experienced engineers, you know, with a strong educational background that can come in and help them with that process.

Awesome. So once again, AI. AI hasn't changed what you've been doing lately. Like has it disrupted anything or is.

There, you know, the biggest disruption is everyone now thinks they need to have AI integrated into their system.

Like, sure, yeah.

You know, I've said for years the biggest ROI in AI is just simply embedding it into the name of your application. You know, we get a lot of people that come in and tell us, I know I need AI, I just don't know what I need it for. And so I think helping end customers understand ways in which AI can help them can assist them. Like I said before, replace repetitive task. That's mostly what we're looking to do to help them. Right, that's what they're looking for. Beyond that, we work with engineering companies that already are building AI agents or they're already embedding AI into their applications and we simply help them with that development.

Is there any commonalities you see then with people trying to use AI agents that they're doing wrong, that they, that's why they need a company like yours to kind of guide them, like, hey, we know this isn't going to work or this is going to cause issues down the road.

You know, I think we're Quickly moving away from people that think they need to build their own LLM. You know, we're getting to the point where people recognize I can build an agent that works with one or many different LLMs and then use that to provide functionality to my end users. You know, I think we're at that stage now. Now we know we can do this. We've seen people build the basic, you know, chatbots or whatever it may be in their application to do this. Now we're really getting to the point where people are starting to come up with these great new visions of what else they could do with agents. Right. That's the exciting part. It's like, yeah, we've done, you know, what everyone else has done. Let me figure out this unique, creative new way to use an agent in my application. You know, are there things they're doing wrong, the things that I recommend they do differently? I don't know. I mean, I think it's still, we're all still adjusting to sort of this new approach of tools of integration. I think, you know, don't, don't rebuild what's already out there. If you, you can utilize somebody else's tool, that may be a first approach. There's so many companies out there building the same thing over and over again. You know, be aware of what's going on in the environment, be aware of what's in the landscape already and focus on, you know, how you can build it better or do it differently or don't build it again, don't rebuild it.

Can you give an example? Is there something that you does, a solution that's already out there and people keep building over and over again?

I will be embedded chatbot on your website. It's probably, you know, the biggest one. Everyone builds their own and it's, you know, I would promote any specific company, but there's so many companies out there that can already do this for you.

So because a DevOps show and you do a lot of development, custom development, then how do you test an AI agent? Is that something that's hard to do? A lot of times it's non deterministic. How do you know something that's non deterministic is doing what it's supposed to be doing? I don't know if that makes sense or if you even.

No, I know what you're asking. It's a great question. You know, the very first time I tried out ChatGPT, I asked it to schedule a vacation for me using Delta Airlines through very specific places and times. And it gave me A list of flights that just did not exist. So, yeah, I think interacting with it kind of above and beyond. You know, I am by no means the expert when it comes to, to testing what data actually acts the same, what data is being maintained that it should or should not. Right. I just, I understand the data that it should not be holding on to. Right. So. But yeah, looking for valid results, different scenarios, looking at the number of transactions. One thing that people don't think about is the cost model of this. Right. So if you're accessing somebody else's LLM and you're sending it 500,000 transactions a month, you're about to get a bill that's probably not going to make you too happy. Right. How do you limit some of those interactions and make sure that you're only going for the responses that you need? Yeah, I mean, I think there's, there are experts out there that can provide you with much better examples, I think, than I could. But, you know, that's kind of what I'm seeing. And then working on identifying hallucinations and figuring out how to better train and create better prompts so that you're not dealing with that.

All right, so I know people hate this question. Any predictions you see, you know, I know it's, people are hard to predict two to five years though. But knowing what you know about machine learning, what it can and can't do is anything you can see down the road that it may, people need to be aware of.

Maybe, you know, I, I look at it maybe like I've looked at other tools in the last 10 to 20 years. You know, in the first couple years that something really takes off and it's popular. There's 10 tools out there. Two years later, there's two or three. Right. The market is going to get slimmed down over time. Hopefully best of class surfaces to the top, but, you know, it's not always the case. You know, also I, I think the creative, unique cases for utilizing it, I think those are the companies that can identify those, those are the ones that are really going to stand out. And then I, I think hopefully some of the fear in the buzz will dissipate over the next couple years and people will focus more and more on how to use it and less on the fear that it's going to replace that.

Now you're doing a lot with AI agents. There's a lot coming over mcp. Now, are you seeing a rise in people trying to create an MCP for everything?

Yeah, that's a good question. I don't know that I've got a great answer for that. I think, you know, the big thing with agents, I see a lot of people focusing on very specific prompt creation and, you know, maybe we're going to see ways in which other tools can utilize a prompt creator essentially to help build that. Honestly, I don't know. I don't, I don't think I've got a great response.

Okay, Mark, before we go, is there one piece of actual advice you can give to someone to help them with the AI BI DevOps efforts? Whatever you think. And the best way to find, contact you or learn more about agiloway?

Yeah, of course, the best way to find us and contact us about agilia way is just go to our website, you know, the usual www.agiloway.com and you can reach out to us directly on the site. The best piece of advice, just learn more about the tools that are there, learn about, you know, the best way to create prompts that provide useful feedback from AI engines. Look at the ways people are utilizing agents or agents on top of agents on top of agents, and just play around. You know, a lot of these things seem like black boxes, I think, until you dive into the code and look at how it's actually being used.

All right, before we wrap it up, remember, frustrated users quit apps. Don't rely on bad App store reviews. Use SmartBear's Insight Hub to catch, fix and prevent performance bottlenecks and crashes from affecting your users. Go to smartbear.com or use the link down below and try for free for 14 days. No credit card required. And for links of everything of value we covered in this DevOps Toolchain show, head on over to testguild.com p188 so that's it for this episode of the DevOps Toolchain Show. I'm Joe. My mission is to help you succeed creating end to end full stack DevOps toolchain awesomeness. As always, test everything and keep the good. Cheers. Hey, thank you for tuning in. It's incredible to connect with close to 400,000 followers across all our platforms and over 40,000 email subscribers who are at the forefront of automation testing and DevOps. If you haven't yet, join our vibrant community@test guild.com where you become part of our elite circle driving innovation in software testing and automation. And if you're a tool provider or have a service looking to empower our guild with solutions that elevate skills and tackle real world challenges, we're excited to to Collaborate, visit testguild.info to explore how we can create transformative experiences together. Let's push the boundaries of what we can achieve.

With lutes and lyres, the bards began their song A tune of knowledge, a melody of code through the air it spread like wildfire through the land Guiding tester showing the secrets to behold.