CTO Insights on Gen AI's Impact on DevOps with Matt Van Itallie

Get ready to discover some of the most actionable DevOps techniques and tooling, including performance and reliability for some of the world's smartest engineers. Hey, I'm Joe Colantonio, host of the DevOps Toolchain podcast, and my goal is to help you create DevOps toolchain awesomeness. Hey, it's Joe. And welcome to another episode of the test skill DevOps toolchain. And today we'll be talking with Matt all about all things like coding the future, how to bridge the AI divide in software development, and I'm going to touch on maybe how to make your CTO happy. Or if you're a cto, this is definitely a must. Listen as well if you don't know. Matt Van Italy is the founder and CEO of sema, which builds a bunch of cool tools to help companies really capture the benefits and manage the risks that come with generative AI, probably with things you're probably struggling with now. And so you're going to learn how to bridge the gap between tech and and non target audiences and why this is so important, especially nowadays. You don't want to miss this episode. Check it out.

Hey, if your app is slow, it could be worse than an error. It could be frustrating. And one thing I've Learned over my 25 years in the industry is that frustrated users don't last long. But since slow performance isn't sudden, it's hard for standard error monitoring tools to catch. That's why I think you should check out bugsnag, an all in one observability solution that has a way to automatically watch for these issues. Real user monitoring. It detects and reports real user performance data in real time so you can quickly identify lags. Plus you can get the context of where the lags are and how to fix them. Don't rely on frustrated user feedback. Find out for yourself. Go to bugsnag.com and try it for free. No credit card required. Check. Check it out. Let me know what you think.

Hey Matt, welcome to the Guild.

Thanks so much for having me, Joe. I am so excited to be here.

I'm pumped to have you. Because I don't think I've ever talked about someone from more of like a CTO perspective. And I think we'll touch on a few of those as well. But you know, I'm just curious to know, before we do, is there anything I missed in your bio that you want the Guild to know more about first?

No. You did a great job. I do really like making peepshee. I feel like I Should bring that up. You want to guess what peepshee are or do you already know?

I don't know.

All right. Peepshee. It's the right season for this is an amalgamation of sushi made out of Marshmallow Peeps. Non denominationally you can take the marshmallow peeps and those that's like the fish rice crispy treats for the rice and then sushi rolls. The seaweed wrap is fruit roll ups. If you have anyone in your house who's a child at heart or is actually a child, it's the most fun thing to do in the spring.

Oh my gosh. I have to google this. This is totally new to me. That sounds awesome for sure. So speaking about new and things I'm just learning about and that's just coming out. Generative AI. Let's just jump right in. It's been a hot topic the past year and a half especially, but it seems like your company's been around longer than like, like chatgpt and things. So how did you get into generative AI? Did you start from the beginning knowing AI was going to be a really going to start taking off? Like how did you know? It seems like you timed it perfectly for what you're doing right now.

That's very kind of you to say. I'm not sure that's true, but I will gladly take it all credit to our great team and our great clients. We've been around for seven years and you're exactly right. Long before the rise, the rise of generative AI, we got started out helping provide, let's call it a CTO dashboard in an instant. We call it a comprehensive code base scan and it summarizes all of the key metrics that are most of the key metrics I should say about an engineering team in the form of report, kind of like a credit score report based on that and absolute honor to help evaluate over a trillion dollars worth of companies. We spent a lot of time thinking about, well, how do you summarize what's going on in the code and how do you summarize what's going on with coders in a way that is defensible to technologists in the detail but also is accessible to non technical audiences who we all know sometimes get lost in technospeak. And so we had built that tool and been working on it for, you know, for over six years when some of our amazing clients came to us and said AI is coming, AI Gen AI is coming. And it involves understanding the impact on code and it involves understanding the impact on coders and we're very fortunate to follow their guidance and help support companies making the transition to Gen AI.

So if I'm a cto, why should I care if the code's generated by Genai? Like what kind of insights would I need to know? Oh, this code was generated by Gen AI?

Yeah. Great question. The first and primary question is do you want your organization to be using generative AI tools to code or not? We at CIMA are extremely bullish. We think the return on investment measured by developers being happier as well as more productivity, you know, being able to build a roadmap faster really outweighs the cost. And the risks are mitigatable. So you need strong positive benefit, low cost, including low cost to implement and the risks need to be mitigatable. But that is a question that legal counsel needs to weigh in on. This is a powerful tool and used in, when used not safely, it can really create some risk for the organization. You know, we've seen many cases where organizations don't really have a formal policy yet and developers who want access to this are bringing their own LLM license to work. And we really, really, really strongly recommend that organizations make a policy that the answer is yes and then give every developer access to the enterprise grade tools with all the safety, et cetera. So step one, should we use it at all? And if you don't want to use it, then of course you should monitor to make sure that people aren't if you are using it, you know the two big reasons, additional reasons to monitor are to help support developers sensible use. So if you do it at a developer by developer level, you know, to some extent Genai is radically different, but in another way it's just an extension of copying from stack overflow of using open source code. Right? It's code you didn't write that could make your life easier and better. With all love to junior devs. It's a tool that needs to be supervised, right, because they don't have as much context and wisdom as some of the older engineers. So you want to look at it by developer group at least and help make sure that they're making good choices. And then finally this is a little bit into the future, but there are internal, there are, I guess I should say. And finally looking into the future, there are external stakeholders who will ultimately care about gen composition. So folks today you think about open source composition, you need it frequently in computer conversations with insurers. You need it for procurement, certain procurement processes, you need to keep track of it in diligence to show how much open source you're using because open source is very powerful but it comes with some associated risk. We believe Genai, which also comes with some of those risks, even though they're mitigatable, will also lead to companies needing to have a gen bill of materials along with their open source bill of materials.

Absolutely no. Open source bill of materials was a hot topic a few years ago. Become more important. I definitely agree with you. So I don't know if you can answer this. I guess it all depends. It depends on risk but like legalities that could come up. I'm just thinking like in the future someone's like, oh, you wrote your code, your application based on code that I can trace that was trained on mine unbeknownst to me. And so now you have IP impact and all that. So are those some of the types of risks that we have to worry about? Is that maybe a little too. You can't predict the future, but it could go that way.

That's a great question, Joe. For sure one of the major risks involved with using generative AI tools is potential legal risk. That's number one. The other three big ones at a broad category are data leakage. Your information is getting out into the world because you're putting into Genai to LLM and it's going elsewhere. Third is code quality. And fourth that the code may be less accurate. We all hear about hallucinations with gen. The impact of hallucinations in code writing is that the code looks right but is incorrect. And the fourth is code security. Despite the very strong investment that the LLMs are making in code security from the AI code, it still is not fully secure. No one would expect that. And so those are the four big risks. I promise I'm going to get to your specific question. The way that you solve data leakage is make sure that your team is is using the right corporate approved tool and all the data is in privately and there's enterprise grade products that can do that. You solve code quality issues just like every other code quality issue. Code reviews, automated tools where you have them and keeping track of its usage. You solve code security the same way. So again I don't think anyone thinks this but just to make sure, just because it comes from Genai doesn't mean you can skip your security gates. Absolutely not. You should put it through for SAS and DAS scans and CVE scans. So those are three of the four. Now we're left with legal. We're in legal and I should say besides peepshow I also love taxonomies, so apologies if I'm being too taxonomical. As we go within legal, there's a couple different forms of risk. The one that you said, Joe. Now I can get to it. What is the likelihood that creators of the training set down data that were used to create the LLMs will pursue legal action against the users of LLMs? So an open source community taking action against because that's where the code was trained on taking action against a user of an LLM. Did I capture your question right?

Absolutely. Yep.

Okay. Sima assesses this risk as very low and I promise I'm only going to say it one more time. It is not legal advice. This is why you need to talk to your lawyers.

Yep.

We think it's very low for a couple reasons. One, in many of the enterprise grade tools copilot as an example, you can turn off the wrong kind of snippet duplication and you should most certainly do that. And that's one of the reasons you should get enterprise grade level. Second, even without the toggle on, very little code is directly copied in. It's less than 1% of what comes through says some very trustworthy sources. Three, the enterprise grade LLM providers have indemnification clauses which say if you use our product the right way, you can rely on us to protect you if someone from the creators were coming after you. And then fourth, and finally, it's extremely unlikely in general for open source creators to take action against users of open source. And now it's even more attenuated. You should always, always, always be a good supporter to the open source community. If you're not, if your organization doesn't allow volunteer time, you know, paid volunteer time or buy licenses, please go do that immediately. It's absolutely the right thing to do on a variety of levels. But for all those reasons, we are not worried about the legal risk from the creators of the training set.

All right, cool. So I'm a cto. I don't want to have to go through tons and tons of reports. I have a policy that maybe says no gen AI should be released. Do I get bubbled up in insight or does it just send me a flag and say hey Joe, just checked in code that's generated by AI maybe because I don't think this is a DevOps tool per se, so I don't know if it lives in your CI CD world. Swat it down. So how does it work?

Yeah, so right now our product is a CTO dashboard and in beta is a VS code Widget so we can, so developers can see their results as they are coding. We will have alerts very soon. Right now it is by login you can see a dashboard with custom rules. So if you say I want zero, if it's more than zero, you're going to get the flag when you log in and traffic lights and all of that.

Nice. So I like to be optimistic, but is there a way like say you hire someone that's supposed to be a senior developer and you just want to say okay, I'm a cto, I just want to check this guy's code when he checks it in and you notice 100% of his code is always gen AI. Is that a concern? Is that something that could help you just know like oh, I better keep up on this or just check in to see what's going on?

Yeah, super good question. We like to, and you know, we've been understanding code bases and interpreting a credit score of code for you know, seven years and 850 organizations. So at SIMA we always like to start with curiosity about data, especially data about code, then judgment in part because there's so much context. Code is a craft, it's not assembly line work. Up isn't always better, down isn't always better. So we like to lead with curiosity. That's part one, part two. We are at the beginning, we the industry of this incredible journey of figuring out the optimal amount of pure generative AI code and blended generative AI code. Pure generative AI code is copy pasted in that any modification you know that can tell it from the names blended means developers modified it. The lesson from open source that folks have used this third party code that they didn't write that they can largely rely on is to not touch it. Okay, I found the right library, I'm going to reference it and I am definitely not going to modify it because that creates legal risk and that creates security risk. The opposite is actually true for generative AI. Pure code is more dangerous than than blended code. That doesn't mean you can't have any pure code. But if your code is, if literally it's a hundred percent untouched by humans, then it didn't get the security check, it didn't get the quality check. It probably doesn't have the context associated. So we like generative AI usage, generative AI usage in the sclc, but we want that blending. And again I can't. I've just spent too much time working with engineers. Not to say this. Any CTO who had this observation, the first step is let's go have a conversation with your teams. Engineers are so smart. Have them hear from legal. Here's the pros and cons. Here's what you need to think about. There's nothing worse than top down. Just cringe even thinking about it of judgment before you've given developers a chance to understand what great looks like and the why, the why behind it. And we're always bullish on engineers doing the right thing when they have all the information.

And this may sound low tech, but communication is so important in a culture and it seems like a tool like this, like you said, that can then open up communication between your CTO and your developers to say hey, we noticed this, why are you doing that? And get the flow of communication going between the teams, which is healthy, right?

Exactly right. And this is not a gotcha. It's not a surprise. It's a group exercise. And you know, hope this isn't too salesy because it's only a white paper. We wrote a paper about a month ago on unleashing developer productivity. When you're using a gen tool, you don't need SEMA's tool to help you do that. But it's things like let's set up a developer council with a range of perspectives and skeptics about gen AI and advocates of gen AI. Let's hear what the big goals are and let's just experiment. What would we expect to see this week? Do we want juniors to be using it this much? Do we want to have it blended that much? And just look at the data together. It's not a secret dashboard, not like joint discussion, developer contributing if not developer led. Any new technology, that's the right way to do it. But it's especially the right way to do it on something in their workflow as gen AI.

As you mentioned, it's not always black and white. Depends on different risks, probably different industries, different verticals, all that stuff. So how do you know as a cto what metrics then to look at to know that hey, we're on, we're doing well. Or is that once again subjective or is that depends on. Yeah, are there any. Go to CTO metrics?

I guess the question I would definitely build this with the team and look at changes with the engineers and look at changes over time and see if you're heading in heading in the direction that you would like as an organization to head. And I know that's a wishy washy answer, but we really do. When you set outside in numbers and hope to people achieve them, that you can have some paradoxical results. That said, if you are looking at your most valuable intellectual property, let's say the core product functionality, we'd be pretty nervous about more than 25% pure generative AI. So using in a sentence 25% of that code came straight in copying and pasting from Genai. I would worry about the correctness, I would worry about the quality and security. And a second one outside in if you decide your organization's going to use it, I would move towards at least 5% of your code has genai because that means 1 out of every 20 commits an engineer thinks it's useful to use Genai. And if that's not true, then I would unpack what is getting in the way of that because we do believe that Genai is useful in new code creation, in test writing, especially in test writing, in understanding legacy code, et cetera. So if you're below the 5% threshold of any use, I would, I'd be curious. And if I'm. If you're above the 25% pure usage, I'd take a closer look. And again, that's code base wide for important code. If you showed up tomorrow, Joe, with if a developer showed up tomorrow with a prototype with 100% pure gen AI, I'd say amazing. That's one of the most beautiful things about gen AI is it just lets you experiment. So it really does need to be contextual for what parts of the code you're measuring.

Love it. So usually when we talk about gen, you see like the hype curve. Obviously you're a company, you speak with a lot of companies, you're creating white papers. How many developers are actually taking advantage then of Gen AI? Have you seen in the last year, like 40%? We've seen a 40% increase in Gen AI and we only see it going higher. Or do you see developers tried it and they're like, I'll just code it myself. Like, where do we stand right now?

The best estimates we've seen is that over 90% of developers are using Genai tooling on a regular basis at home or at work or both. And at home is so interesting, of course, because it's such a tell. How you spend your free time is a real indication of your passions, passions and interests and commitment to tools. We do not think this is a fad. We think this is conservatively. We think this is like open source. And I don't have to be a genius to say that it is code someone else wrote that does some of the work for you and lets you do more of the fun stuff. In 2024, if you were trying to build a code base without open source, you'd be bananas. You'd be bananas because you're wasting time, you're not having any fun. Why would you not use it? Even though there are risks with open source, all those open source risks are manageable. A CVE detector, working with legal on the legal risks, et cetera. It is not bananas March 2024 to not use Genai tooling. It is. It is not bananas yet. But we predict by a year from now, max, in almost any possible circumstance, we would expect any CIO and legal counsel to deal with any perceived risks that they see again at CMS perspective, that they're mitigatable now. But within the next year, we think it will be at nearly full adoption. And it should be for all the same reasons that open source is at nearly 100% adoption. It's better for the organization and it's better for the individual developers.

Absolutely. So could you just go a little over now, if I'm a CTO and I have your solution, like, how does it work? Do I have a gen AI where I go, tell me, blah, blah, blah, or it generates a report for me, Is it a dashboard that visualizes things easy for me as a cto, how does it make it easy for me to handle all this?

Yeah, it's literally a dashboard. So this is a podcast. We'll just have to describe it. But get ready, listeners. There's big number widgets and so at the top of the page, just think of a sandbox example I can think of without naming who it is. 5% of the code was GenAI originated. That means that 95% was not GenAI originated. Could have been open source, could have been a developer, could have been copying from somewhere else. Of that 5%, 4% is pure, so people didn't modify it. And 1% is blended. One plus four equals five. Below that is trend data, how it's changing over time. Below that is groups of repos. We call them groups because sometimes it's product one versus product two, sometimes it's product versus tooling. So you create the groups you want of your repos and with that, custom rules. So let's say no restrictions on Genai usage for the prototype repos, which is definitely what we recommend. So you have a group called prototypes, and it's always green because anything you want is fine and it might be red if you're not using gen AI.

Right.

Other than that and then you have other custom rules for, you know, the highest priority intellectual property, et cetera. And then below that, developer by developer view of how they're using code, you know, pure, blended, not gen, AI, et cetera.

Awesome. So how hard is it to implement then? Do I need to have developers that do it for me? For. Is it easy? Yeah, like sits on top of something. How do you get it integrated?

It takes about five minutes to set up. It's a GitHub API connection for those using connected to GitHub API. And then if you're not using GitHub we have other methods. So a couple minutes of work.

So how do you get this in the hands of people then? I mean do you just target ctos or is it sometimes if a developer's listening, would you recommend developer goes, hey, let me open up, tell my CTO about this because I think it really will help them and by helping them it'll help us because then we have everything goes smoother or something. I don't know what are the benefits for a developer then to tell their cto, hey, you should check this out.

Yeah, I really believe this is valuable, full stop. I believe the faster that organizations adopt it, the more quality of work life there is for developers, the less time on, you know, just think of unit tests that the difference. If you've ever. If you're a developer and you're just try it. If you don't have a company tool, don't put anything proprietary in it. So find some open source code and try it. That's really important. Do not leak. Do not leak, but just try putting it into a chatbot and say appropriate chatbot and open source code and say write me a unit test on it. And you will not go back to the old way because it is so much better. If your team hasn't adopted it fully then of course, or even if they have and they want to know how adopted it is, of course a tool like ours, you know, we'd be delighted to help. You should feel free to contact us, of course and folks can try it out for a bit for free, see what they think. And you know, we're always, we're developers first, we're engineering first. So we're so excited for developers to get to try it out and give feedback and tell us what they think.

Awesome. So Matt, you know, I don't know if I gave you a chance to describe the product in more detail. I just jumped right into gen and you probably have multiple products and services. So is this just a dashboard for CTOs. You mentioned unit testing. So does this help developers write unit tests? Like yeah, tell me a little bit more about maybe the solution holistically.

For sure. The unit test example was more about. So if you think about the software development lifecycle, there are 15 different subtasks. It could include understanding a new code base, it could include prototyping, it could include bug fixing, it could include refactoring. Genai can help in different ways at different stages in that process. A Geni tool can help you write unit tests. A Genai tool can help you understand legacy code. Just explain to me what it does. And so those are things that Geni can help with. It hallucinates. You got to double check. This is humans must be in the loop. SEMA does not do any of those things. That is what the LLM is for. SEMA's gen AI code monitor helps understand the level of usage of generative AI in a code base where it is who's using it, allows organizations to set goals to work towards and to see if they're, you know, in and out of to the tolerance level and then keeps track of regulations related to generative AI. We do think at some point more regulations are coming, but there's some now for folks to think about and we keep track of it all in one spot so there's no big surprises.

You know, I used to work for a healthcare company and the FDA could come in and audit you having a tool like this. Would this help? Is this help you with an audit to say, hey, look, we have this under control and here's the proof. We have this dashboard that shows you.

It does, it does. And we're at the early stages. So in the European Union, there are audits coming for AI usage, not for used by developers, but in the product. If you have an AI engine that's deciding who should get credit or not or underwriting, you're now going to have to prove that usage is appropriate. We do believe just like open source, there will come a time when you have to prove the provenance of human code versus gen AI code. And so when that moment comes, a tool such as ours will be able to help. And some folks want to know in advance because they think it's coming sooner rather than later. And so this is getting the test results before you actually have to take the test.

Love it. And that's a big point about regulations. Then you keep on top of regulations and I assume you update the software frequently. So if I'm a cto, I don't necessarily I need to be aware of it. But it automatically will take the latest and greatest regulations and maybe highlight areas you may not be in compliance.

Exactly right. And so probably the biggest one for folks who work at organizations that seek copyright protection. So there's a couple different ways that you can protect your code. In general, we use trade secret, which just means if it's not open source, you keep it private, you don't let anyone look at it. Copyright is actually protecting the words that are written themselves. Not every organization does it. It's usually for the larger ones. But there is already a disclosure requirement. So if you are trying to copyright work that was created in part with Genai, you have to acknowledge that Genai was used. And our advice all right, third time's a charm. Not again, not legal advice. Ask your lawyers. But our advice is before you say yes or no, you should actually know what the answer is. So for in that situation, that's already a standard for disclosure. And you're exactly right, we don't ctos are busy enough, goodness sakes, lawyers are busy enough, counsel are busy enough. So our job is to keep that up to date and keep those standards front and center for them.

Awesome. So definitely a must needed tool I think now in the age of gen AI if people don't have it, I think it's this is brilliant. So Matt, before we go though, is there one piece of actual advice you can give to someone to help them with their CTO Gen AI development efforts and what's the best way to find contact you and learn more about SEMA?

Sure. So you can find me@mviimasoftware.com Matt Van Italy you can also find me and my colleagues@aiima software.com they both get to me eventually. The one piece of advice this is a little bit general and had the good pleasure of thinking about code deeply at CMON for the last seven years to engineering leaders I would say and you got to believe me, I love thinking about code so much so I say this from a place of love. When you are talking to non coders you need to express the benefits in terms of if you're in a business, business outcomes or organizational outcomes or technology outcomes that directly impact organizational outcomes. I love clean code. I love figuring out sticky legacy code bases and what a journey it is for developers to figure that out. That is a grand journey but it doesn't matter to the business and isn't a worthwhile investment to the business unless there is a business outcome. So are we going to get more customer retention because performance is higher, are we going to get greater developer productivity which then leads to faster roadmap because they're spending less time getting up to speed in the code base? Getting up to speed in the code base faster is a technology benefit that then leads to a business benefit of faster throughput, faster performance because the code's been refactored is a direct business outcome. So before you talk to you, go to your C suite or you go to your board of directors and you want to ask for something, can you frame it in terms of a business outcome? Second choice, can you frame it in terms of a tech outcome that will directly benefit a business outcome? And if you can't, I would say as a cto, you shouldn't bring it up because they're looking to you not just for technical leadership, but to bridge that gap between the tech world and the not tech world. It doesn't mean you might not want to ask for help on it, or I think we might be approaching this as a need. But think what does? Does this investment in the code actually serve the organization? And if you don't have a great answer for that, work on collecting data and support that claim until you do.

Remember, latency is the silent killer of your app. Don't rely on frustrated user feedback. You can know exactly what's happening and and how to fix it with bug snag from SmartBear. See it for yourself. Go to Bugsnag.com and try for free. No credit card required. Check it out, let me know what you think.

And for links of everything of value we covered in this DevOps toolchain show, head on over to test guild.com P140 and while you're there, make sure to click on the Smart Bear link and learn all about Smart Bear's awesome solutions to give you the visibility you need to deliver great software. That's smartbear.com so that's it for this episode of the DevOps Toolchain Show. I'm Joe. My mission is to help you succeed, creating end to end full stack DevOps toolchain awesomeness. As always, test everything and keep the good. Cheers. Hey, thanks again for listening. If you're not already part of our awesome community of 27,000 of the small artists, testers, DevOps and automation professionals in the world, we'd love to have you join the fam@testguild.com and if you're in the DevOps automation software testing space or you're a test tool provider and want to offer real world value that can improve the skills or solve a problem for the guild community. I'd love to hear from you. Head ON over to testguild.info and let's make it happen.