Podcast

Get ready to discover some of the most actionable DevOps techniques and tooling, including performance and reliability with some of the world's smartest engineers. Hey, I'm Joe Colantonio, host of the DevOps Toolchain podcast, and my goal is to help you create DevOps toolchain awesomeness. Hey, it's Joe. And welcome to another episode of Tuskill DevOps Toolchain. Today we're going to be talking all about the API vector and a bunch more with Jeremy Snyder. You know, Jeremy is the founder and CEO of Firetail I.O. and he thrives on interesting challenges in the cloud, cybersecurity, data robotics in the domain space, and a whole bunch more. Also, as a cloud security expert, he really knows his way around digital security and he's a thought leader who covers topics on API security, cloud formation, challenges and risk. And he also hosts the Modern Cyber podcast chat with CISOs from all over the world about cybersecurity and IT issues. If you want to learn more about security, you're in the right place. You don't want to miss this episode. Check it out. Hey, before we get into this episode, I want to quickly talk about the silent killer of most DevOps efforts, that is poor user experience. If your app is slow, it's worse than your typical bug. It's frustrating and in my experience and many others I talk to on this podcast, frustrated users don't last long. But since slow performance is as sudden, it's hard for standard error monitoring tools to catch. And that's why I really dig SmartBear's Insight Hub. It's an all in one observability solution that offers front end performance monitoring and distributed tracing. Your developers can easily detect, fix and prevent performance bottlenecks before it affects your users. Sounds cool, right? Don't rely anymore on frustrated user feedback. But I always say try it for yourself. Go to smartbeard.com or use our special link down below and try it for free. No credit card required. Hey Jeremy, welcome to the Guild.

Hey Joe, so happy to be here.

Great to have you. I guess before you get into this, you know in the pre show we talked about types of things you want to talk about and you mentioned the API vector. So why, why did you choose this particular topic?

Yeah, you know, look, I've been in the cloud space for a long time. I used to work for one of the big cloud providers and then after I left them I've kind of stayed in generally in the cloud ecosystem and what I saw through like the mid 20 teens was more and more customers were really moving to the cloud with serious scale. And what would generally happen is they'd move to the cloud and after like a year, year and a half, two years, they'd start to find a little bit of frustration with their cloud adoption. And they would generally say things like, I don't understand, we're not getting the scaling benefits, we're not getting the flexibility benefits, the agility benefits, like they promised me that we scale up, scale down. And what they finally realize is that that can work. Absolutely. That is available to you, but you actually have to change your architecture and your operations model to make that happen. And so when they look at changing the architecture, what ends up happening is they realize their applications are often too monolithic and they need to be like broken down into more composable decoupled services. One way or the other. They might decide, oh, we're going to do that with let's say serverless functions, or we're going to do that with a microservice architecture and containerization or whatever it is. But one way or the other, they end up with sensitive data crossing APIs that might be internal, that might be external, that might be a combination. Right. They might have like services that are calling each other internally and they integrate Twilio or Stripe or whatever it may be that they're integrating from a third party API perspective. But all of this sensitive data is crossing over APIs. And we started to dig into that and realize like, oh, okay, so you've got sensitive data in transit. Is that data being breached? And the more we started to scratch that or like pull that thread, the more we realized, oh crap, not only do we have sensitive data being breached by APIs, but, but we have a lot of sensitive data being breached by APIs. And so that really made me and my co founder Riley, we really wanted to try to like solve that problem and help organizations do a better job with securing their APIs so they wouldn't get their data breached. Plain and simple.

Absolutely. And so I guess first question that comes to mind, kind of random, is you mentioned a lot of third party APIs.

Yeah.

How many of these APIs have been compromised in a way that do you need to worry about third party APIs or do you only need to worry about testing your own in house microservices and things like that?

The vast majority of breaches are in house microservices. Vast, vast majority, like 90 plus percent. So like there are certain third party APIs, like, don't get me wrong, like Never just kind of trust and don't verify. But the vast majority of the risk is really on the first party side. And so it's, you know, it's software companies, it's mobile apps, it's IoT companies on their own that are developing their own services. And typically what ends up happening is they have one or more design flaws in the APIs or a combination of a design flaw in the API and then let's say a misconfiguration on the cloud architecture where the API is running, that in conjunction leave it vulnerable to data scraping or exfiltration. So yeah, that's really the concern.

All right, so that's interesting. So someone maybe may be testing their API, but if they're going to the cloud, they're not testing it in the cloud environment, could that expose them then because they think they're covered because they did maybe unit tests, but not necessarily kind of integration tests as it goes to the cloud?

Yeah, I mean that's one area of risking. And then the other, let's say some of the big ones that I would call out is like if you do unit testing on a one off basis, I think that's all well and good, but actually if you had to extract out of all of the data all of the breaches that we've seen, the billions of records that we've seen, the one specific unit test that I don't see a lot of organizations doing is really can user A access user B's data? Because that ends up being the number one issue. It's something called broken Object level authorization. And it is effectively a check whether like, you know, can Jeremy logged in as Jeremy also pull up Joe's profile and you know, like, maybe I'm allowed to pull up your profile to view it, but I'm not allowed to edit it. So it's like that unit test has to take into account not only let's say another person's data, but all the actions that I might want to do with that data. And so a lot of unit tests won't go to that level. They'll go to the level of like, hey, can Jeremy render Jeremy's data? Yeah, great, okay, pass, let's go and let's keep moving. And so it's that level of complexity on the API side that I think gets often overlooked. The other thing with it is that I think a lot of the times the unit testing is very much driven by intended use cases, let's say using my client or using my web front end or whatever that may be, because Maybe I'm pushing an authorization check into the front end, whether that's a mobile app or a web ui. The problem is, by the way, that, like, hackers don't play by those rules. So if they're going to try to break into your API, they're going to like, yes, they're going to use your web UI to understand all the API calls that it's making to the backend, but when they try to attack your API, they're going to go straight to the backend API. So any kind of, let's say, checks that you're doing, whether it's authentication, authorization or data inputs or sanitization, anything that you're doing in the front end, again, mobile or web ui, that's just going to get bypassed by a bad actor who's going straight to the backend API services. So unit testing that doesn't take into account the way an attacker looks at it is also going to miss these problems.

All right, so is this a education issue where a developer may not know, or is it more like they're not thinking about it or they think they have a security team that's going to take care of this for them?

Yes, yes and yes. I mean, honestly, like, it's kind of a combination of all of those things. Right. This isn't an attack surface that has gotten a lot of attention up until the last couple of years. And I would say, honestly, like, you know, lockdown and the shift of services to digital has accelerated the risk around APIs quite a lot. And by the way, right now, the current wave of AI adoption is all driven by APIs as well, so that's accelerating further. But I do honestly think that to the first part of your question, there is a lot of education that still needs to get done about what is good API security design. And then also for security teams, like, what do they need to understand around it? I will tell you, by the way, just as a point of kind of curiosity around this, every security team that we talk to, the first question that they bring to us is, we don't know all the APIs we have. So there's still like that constant kind of classic disconnect between developers who are just, like, doing their job as far as they know, and security teams that are left kind of chasing after them, picking up the pieces.

That's a good point. They may be working on an API and have no idea that down the road it has a dependency on another API that wasn't even written by them. Yeah, so, you know, we talked about AI making things more complicated. Can AIs AI then help, maybe them, let them know, hey, this API has a dependency on this other API, so you may want to test that as well.

Yeah, for sure. There's a lot that AI is actually good at, and we're doing a lot of this testing in our own product in our own platform today, as I'm sure you can imagine. And there's one, like, really kind of good thing about a lot of the current AI stuff, all the LLM models, one is that they're all trained on code, so they're generally pretty good at reading code and figuring out, like, what does this code do, or extracting from code, this code calls this other code. So all of that kind of, let's say dependency mapping or connections to other services, that is something that the AIs are pretty good at. And then when you combine that with, let's say, like static code analysis techniques, you can usually figure out, okay, this API over here is part of a service that talks to this API over here, and I've got this vulnerability over here and this design flaw over there that together can lead to a particular risk scenario.

Absolutely. So you mentioned broken object authentication earlier.

Yeah.

And you made it sound like it was a more advanced kind of maybe testing that you need to do for that. So is it a lack of environments then that would make someone not go to that next level? Or once again, is it yes. And then also education.

Say yes. And then also education. It's again like, you know, you brought up the unit testing aspect of it earlier and I think that's like a, certainly a super valid place to start. But like in that unit test, don't just test one user, test two users and then test those two users accessing each other's data. Right. And I think that's again, a level of testing that a lot of people maybe just aren't going to. And you know, we talk about how you're running the test also being super important on that regard. But I think also, like, generally speaking, the understanding that something that you've taken for granted as a development practice in the past actually can lead to a vulnerability. And, you know, just to call out one aspect of it, using a primary key column on a database as the numbering scheme for data records is super predictable. Right. So if I start with user ID equals 1, I'm going to guess that the next user ID is going to be 2 and the one after that is going to be 3. And you know, just using sequential integer numbering, super easy, super straightforward. We've all been doing it for decades at this point, right? Some of us, Those of us been doing this for decades, right?

Yeah, yeah, right.

And it's a super logical approach and it makes a ton of sense. And it's efficient for the database and it's efficient for the developer and it's kind of efficient for everybody, including the hacker who wants to enumerate your entire data set. It allows them to just like write a script that says, oh, okay, I've got an authorization flaw right here. Let me just request the next record and the next record and the next record. And I can use a for loop and I'm just going to get a whole data set. Right. And so like those little design elements that you maybe take for granted as part of a, a standard, pretty well embedded or ingrained process are things that actually have inherent risk or can have inherent risk to them in an API driven world.

All right, so I speak with a lot of enterprises. They have really older technology places like healthcare insurance. They probably have these design elements. They want to go towards microservices. Like, is there like a, like a middle part you can add that kind of makes it so that the API doesn't have these design element flaws?

Yes and no. To some extent. Like certainly you can do things like add a web application firewall, you can add an API gateway and those can provide some layers of defense. You know, like an API gateway can do rate limiting and you can, you know, kind of slow down an attacker. Right. From that perspective. And a waf, if you give it certain signatures of particular types of behaviors to look for, it may be able to block them. Clever attackers can usually find ways around those. So really, like a lot of it has to come back to making sure that you're doing, you know, following good design practices on the back end with your API.

Absolutely. So we mentioned AI a bunch of times. Do we just throw our hands up in the air now and think, hey, you know, AI is going to be a better hacker than I am, probably is a good developer as well, how do I defend then against a lot of more sophisticated. We thought these kitty scripts were kind of tough. When we have an AI that's built to do hacking, especially for API, and it's been trained on that way, how do we get around that?

Yeah, look, there's, I've had a lot of discussions. We've got two guys on our advisory board who are both kind of quote unquote, thought leaders, and I use that in air quotes, but they really are like, both are Published cyber authors. Both are like longtime researchers and practitioners who have, like, you know, they have real substance behind them. And one of them in particular, I've been having this exact debate with him about, like, okay, who gets the upper hand with this? You know, do we throw up our hands, as you said, or do we go all in and we're like, you know what? We're going to throw AI at everything we're doing on the defense side. And who has the upper hand in this? And he said this really is a case of first mover advantage. And that's his viewpoint. And he's run a bunch of lab simulations and research work on this, and I think it's really interesting to think about it. Right, so, like, from the perspective of defending APIs today, we talked about, you know, this design aspect of it. We talked about putting in, you know, some network layers like the WAF and the API gateway. But really writing a more secure API is honestly the best thing that you can do. But again, AI is trained on code. So with a little bit of a nudge and a little bit of guidance, you can actually use that AI model to tell you, hey, Joe, this is what a good API looks like. Or to tell you, hey, Joe, you wrote this API, it currently looks like this. If you were to enhance it with these 10 changes, it would be more secure because it would remove the following risk. And so like that level of kind of human plus AI augmentation where you're writing the code, designing your business logic, designing your application, running it through in a security assessment, and then using an AI help you to improve it, I think is where we have to move towards. And that's something that we're actively doing at Firetail. We're not the only company, by the way, like, that's doing this. I'm seeing this approach more and more from the leading organizations that are trying to get ahead of the problem. But I think, like, that's the approach that we should be thinking at instead of throwing up our hands.

Great. You know, my bad host, we're already almost 15 minutes in. I haven't even asked you more about Firetail IO, so maybe that will set some groundwork for the. Yeah, the second half of this. What is Firetail IO? You're actually passionate about APIs, you're passionate about security. Why did you create it and what does it do then to help us in an AI API world?

Well, we, we talked a little bit about kind of like the origin story for why we started looking at this problem. I will tell you A little bit just as a tangent about why I'm actually passionate about cybersecurity. So my background is I'm a computational linguist by training. I actually thought I would be a software developer. Turns out I sucked at writing code. I got a job with a linguistics software company. Couldn't be more perfect. Early in my career I'm like, this is awesome. This is my chance. You know, all the work I did writing my honors thesis around generating language models, et cetera, this is back in 97 by the way. Like so generations of natural language processing ago, multiple generations. And I got into the real world and turns out, like I said, I sucked at writing code and I said, okay. So the company was growing really fast at the time and they're like, hey, go do it and cyber stuff for us. We just need infrastructure, we need security as we grow, et cetera. So I did that, did that for like the first half of my career. Started moving into more kind of customer facing roles over time. And then I actually was pretty burnt out. And in the years like 2016 to 2019, I was going like really, really hard, you know, more than a hundred thousand miles a year on a plane, working with customers all around the world. It was all exciting, but it was also like heavy burnout. And then we hit Covid and I was like. And then the company I was with got acquired and we had a nice little outcome, trying to figure out, hey, what do I want to do next? And my wife and I were talking about like, okay, maybe this is the chance to get out of tech and go do something different. It so happened at the time that a couple of our customers were working on things that helped get us out of the pandemic, things that, that were really like super beneficial to humanity. And I realized something, which was that they were under constant attack, like constant constant attack, nation state actors trying to get their intellectual property right. And I realized like, okay, I'm never going to be that scientist, but I actually have a level of understanding and a level of expertise in cloud and cybersecurity that can allow companies like that to do the work that they do and can help them to protect their ip, protect their assets, protect their research, et cetera. And so I decided to stay in cybersecurity. So that's where the passion comes from. I realized it's actually like a good use of my time and a good use of my skills. So that's the passion side of it and kind of figuring out why APIs really comes from the other side of the cloud. Experience that I mentioned before. So Firetail, what are we doing? So, like I said, the first question that customers always come to us with, and it's mostly security teams that we work with, a little bit of developers, but mostly security teams. They always come with the question of like, hey, we've inherited this environment. It might be a cloud environment, it might be on prem some combination thereof. We're getting attacks. We don't know all the APIs that are getting attacked. So that's the first question. So this question of like, visibility is always the first challenge in a cybersecurity problem, especially when it comes to new technology. Pretty much the constant struggle is that, like, developers adopt new technologies before security teams do, and so they get out ahead. It was the same with cloud. It's been the same with, you know, it's the same with AI. Right now, developers are implementing AI integrations faster than security teams can approve them. Right. Pretty much a pretty well known cycle and pattern at this point. So we help them discover APIs wherever they may be, in cloud, in code, using a variety of different methods, that builds an inventory of APIs that's constantly updated, looking for change across all those environments. So new APIs, APIs that get updated with new endpoints, et cetera. We assess that inventory with two types of different tests. We do a code and design analysis. We also run bad traffic, intentionally bad traffic, against the API to look for flaws from the outside so what an attacker would see of your API. And then we put in place monitoring with detection and response capabilities so we can detect certain types of attacks right out of the box. We can also help customers build their own detection rules for different types of behaviors and malicious sequences on API calls. And that's what we do at Firetail.

All right, I want to unpack that a little bit.

Okay.

When it's monitoring, yeah, it has an inventory. But are you able to say, wait a minute, this API usage for this API that's pretty obscure has gone up 100% in the last month. Probably want to look at it. Is that a common use case?

That is a common use case. And it can be up or down, or it can be like the usage is all normal, but let's say all of a sudden the size of the response payloads has dramatically increased. You know, that would be an example that indicates, like, more data is being returned than in previous scenarios. So any kind of change in direction, change in amplitude, all of that is common scenarios.

Nice. And I assume someone could introduce a malicious API, maybe that helps do hacking and I guess this would alert it if you had taken inventory and say, hey, what's this? New API? Or what's this? What? What's it.

Yeah, new APIs is definitely a common scenario. Also changes that make existing APIs worse is a common scenario. So like an API is all, you know, let's say all blessed and good and whatnot. And then all of a sudden authentication gets removed from a particular endpoint or something that was sanitizing inputs before is no longer sanitizing inputs, or there's no longer controls around request parameters. All of those types of changes are definitely things that get monitored for.

Nice. So I assume this is built into the CICD pipeline. Someone checks in code of their API and you alert and say, hey, this actually got worse than before. You may not want to go and check it into your main code base, I assume.

Yeah, absolutely.

All right, so stupid question, off the wall. It is election day.

Yeah.

And you were talking about how you talk to people and they're talking about nation state hacking and the pandemic.

Yeah, yeah.

And when I talk to people, security experts, a lot of times they tell me of stories of how companies don't even know they were hacked until like six months later. Take elections, for example. Like these machines. How do we know how the things are on the up and up that haven't been hacked?

Well, complex question. I don't know that we have enough time to answer it in full. But look, I think like, generally speaking, I'm no expert on this, on the voting machines themselves, but all of these kind of, let's say, machines and devices and whatnot, they do grow through testing sequences and they have network design controls also built in. So I have a high level of confidence in the fact that the machines themselves can't really be messed with while they're in operation. The bigger risk is more, in my opinion, is more up the supply chain. So if something were to go wrong with the machines, it's something that was planted into the machines, likely weeks or months ago. Whether we, to your point, whether we know about that right now or we know about it six months from now, I think is. Is a question. I don't think, though that there is a ton of risk around the machines. I think there is a much bigger risk around misinformation relative to the election. You know, misinformation that's been on social media feeds for, at this point, years. Right. Like as long as the election cycle. As long as we've been talking about this election going back to what, like 20, 23, probably when the candidates were announced. You know, that's a much bigger risk in my mind than anything related to, let's say the actual vote counting or the actual equipment used in the election.

Makes sense. All right, changing gears again. So we, you talked a little bit about design elements of older, say older code, legacy code with the firetail, I guess is, does it able to, you check, it's able to read all your APIs and then it can tell you like maybe prioritize, hey, these APIs, design disaster. May want to look at these first before you get to these other, other new ones or whatever you're working on.

Yeah, so we use a risk scoring methodology around the APIs themselves to try to help you prioritize that. We do a couple of different things. We look at the design of the APIs, we also look at some of the in place scenarios. What I mean by that is like monitoring the traffic against the API. We can also then like provide you that risk score in conjunction with another system so that you can prioritize better. So like for instance, right now let's say that we find an API that has a design flaw and you happen to know that it's in a staging environment. Great. Well, you can deprioritize that. But let's say we provide you the risk score around the API. You pull that over into something like your cloud security platform and you realize that actually the crown jewel data set is sitting behind that API. Well then you can prioritize that as well. So we use our own risk scoring methodology, but we also make that available for you to contextualize in other security tools that you might be using.

Nice. How do you get above the noise anytime I once again speak with people, developers with security. Oh, we have like 10,000 failures on check in. Yeah, they just ignore it. And this is just another layer. Now how do you get around that? I guess.

Yeah, look, it's always a challenge in security and like there's this funny kind of trade off because when you're actually like selling a security product to a customer, you kind of want the security product to find as much bad stuff as possible. So like if the customer is testing out your product in a proof of concept type environment, you want it to show the customer that it's like the terrible like worst environment you've ever seen. And oh my gosh, you know, like, but the reality is exactly what you said, like you then don't want to be this product that as soon as it actually gets into the customer's live environments, it's just going to be like this spam bot that is immediately alerting you about a gazillion things because everybody will tune that out like super fast. Right. So what we do, aside from the risk scoring that I mentioned, is we try to contextualize in terms of everything is relative prioritization within the product. So there's like, you might have an 84 on a scale of a hundred and you might have an 81, but when you then understand the ongoing usage of that API. So let's say like that 84, it scores higher in terms of the design flaws. But the 81 is also tagged as processing PII. Like we provide you these additional signals in addition to your risk score that allows you to prioritize accordingly based on that. So it's gotta be a combination of not only the design, but the design plus the real world usage. And again, like make that available. The other thing that we do is out of the box. We don't actually like turn on any of the alerts. We make all of our alerts, opt in for the customer. And that does actually introduce a little bit of friction as they're going through the adoption cycle. But our logic is again, like, you don't want to be that thing that is immediately tuned out so customers can kind of work their way down an adoption path of, okay, well, now I'm going to turn on this one, now I'm going to turn on the next one, now I'm going to turn on the next one. But it's at their own speed how they choose to turn on the different components within, like the design analysis or the monitoring and alerting and so on. And so it's a trade off because like out of the box. There's an argument that you want to be showing everybody everything right away, but I think that's too overwhelming.

Right, for sure. So, you know, earlier we talked about education being one of the roadblocks. So I do get, say I get a risk score. Okay. It's high. I need to fix it.

Yep.

How do I know how to fix it? Is that a dumb question?

No.

Do you give them hints on how to fix it?

Yeah, we do. We do. We give as much as we can and sometimes that's down to the level. There's about, from the design perspective, there's like 60ish things that we look at. And as much as possible, we will try to take that down to the line of code, whether that's in the software code or the API specification, which is, think of that as like the design document for the API, we try to get as specific as possible. Sometimes we're able to actually give you a suggested code snippet to replace what's in there. But more often we're able to just give you like kind of text guidance that tells you, hey, you know what you've got here, Joe, is you're, I don't know, you're passing the API key directly in the URL. What you need to do is move that into the header of the API so that it's passed in the header of the request and not, not logged directly as plain text. So we'll give you as much as we can there. We also make it so that you can feed that directly into like a ticketing system that a developer is using. So whether that's like JIRA or whether that is, you know, some other kind of task or ticket management system that they're using with that full context. Here's the problem, here's the recommended remediation, here's the line number of the code.

Nice. How hard is to integrate Firetail into an existing environment?

We see customers getting live in as little as about an hour and a half, two hours. I do think like, you know, you can get started in less time than that and you can discover your APIs in less time than that, but to get it implemented in a way where it's going to be more useful to the organization, you typically need a combination of, let's say, some discovery mechanism, some logging mechanism, and some alerting or ticketing integration. And so across those three things, I think it's more likely an hour and a half, two hours to get up and running and have some valuable information for the organization.

Does someone need to have something in place to make this work? Well, for example, if it's all just design disasters to begin with.

Yeah.

Do they have bigger fish to fry before they get into this? Is this like probably the first step to maybe bubble up? Hey, these are real issues. Now we have data to actually show. We need to invest money in it.

Yeah, it's a great question. I mean, look, security is one of those hard things where like every security practitioner will tell you that it's never done. I have a friend who likes to say the most secure computer in the world is the one that's turned off, enclosed in a block of concrete and at the bottom of the ocean. And even then, like, I'm not too sure. The challenge is to your point. Like, look, APIs may not be the number one risk for every organization. I think They're a massive risk to most organizations, and I especially think they're a huge risk to modern companies that are kind of born in the cloud that are doing, you know, mobile apps, IoT, whatever. I will not be one to tell you that, like, APIs are the only thing you need to pay attention to. It may well be the case that you have bigger fish to fry in terms of, like, I don't know, identity or endpoint security or just general security awareness, or like, maybe your baseline cloud security practice isn't very good. All of those might be higher priority to you and your organization. But what I will tell you is, like, assuming that you are the type of company that is processing a lot of PII, or sensitive customer information over APIs, or invoking critical business functions over APIs, your risk is likely higher than you think it is. Now, from that perspective, if you start down an API security path, I will always tell everybody you have to start with visibility. Just discover all your APIs, and then typically your next step is to assess the design. And that is, you know, a combination of code analysis and design analysis. With that together, like, that's the starting point. That's how I would recommend everybody to really get started on this process. And if, by the way, and like, if you're not ready to start that yet, like, start by just getting educated. Like, we have a ton of free resources on our website. We have an API security report from 2024 that we did earlier this year. You can find many of our talks about understanding what the risks are, understanding, like, what the potential implications are. You can find any of that on our YouTube channel or on our website. And it's all free and nobody's going to hassle you. You can, you know, sign up, get that stuff without even talking to a salesperson. All good. No problem. Start with getting educated. Right. And figure out if this really, you know, is potentially a risk for you and your organization.

Nice. So you have your own podcast. You talk to CISOs all the time.

Yeah.

Is there a certain hurdle you need to, to get over in order to convince your CISOs that this is an issue? Or your developers, like, how do you, the developers know this is an issue? You need investment. It. How do you convince your CISOs that this is something you definitely need to like, we have the sensitive data. If we get breached, we're going to get slapped with this huge fine or something like that?

Yeah. It's been interesting to watch how that's played out over the last couple of years as we've been Getting up and running. You know, in the beginning, there was very much a do we need to worry about this? Is this actually a risk? That was about two years ago. What we've seen is that as the incidents have kind of racked up, the volume of inquiries that we're getting has also kind of gone up as well. So. And in particular, what we see is things that are pretty classic, like not your. I mean, maybe your organization got breached on an API or maybe one of your competitors did, or somebody, you know, at a friend's company that you know about. Like, those are classic kind of things that generally trigger people to go take a look at a new emerging technology space where they need to think about what the risk might be. We have also started to see some regulatory intervention into API security. Not too long ago, there was a telecommunications provider who got hit by a, what's called a consent decree with the Federal Communications Commission. And this is, you know, if you're in a regulated space like telecommunications, your regulator has the right to shut you down. This is an organization that happened to have three breaches, two of them against their APIs in a brief period of time. So the regulator stepped in and said, hey, you guys want to stick around? You're basically going to have to agree that you've screwed up. And part of this is you also have to get better at it. So we're going to give you this homework assignment to get better at your cybersecurity, in particular API security. You've got six months go. And so we're starting to see. And I, you know, from a lot of my experience and a lot of people that I know who have worked in regulated industries, once regulation starts in one space, it tends to spread to others. So in this case, it started in telecommunications. But I do think it's probably a question of time before it spreads to financial services and healthcare and whatnot. And you'll start to see that those annual SOC 2 audits are going to start including APIs as part of the scope. There's. Right. And when they come around and they want to run a pen test, that pen test is going to test your APIs increasingly going forward. So I think we're in the early days of it. This consent decree was just a few months ago. I think it was in June or July, if I remember right. And like I said, I think we're just going to continue to see more of this going forward.

Absolutely. All right. So I guess as a tradition here, we're always looking for new trends. Are there?

Yeah.

Any Other trends you see in 2025, maybe people listening, like, right, I know by AI, I know maybe my API is not secure, but I better watch out for this coming down the road at me.

Yeah, I think the thing that I am watching for the most is more kind of, you know, I'm obviously thinking from a security lens. So my predictions are going to be pretty much security related. One of the things that I'm looking at right now that we're starting to see on the AI side and I think we will see increasingly more of is, is an increase in kind of repeating some of the problems of the past. What I mean by that is, is like this. So if you think Back to like 2016, 17, 18, this was around the time that organizations were like really starting to move to the cloud in a more serious way, moving more production workloads, production data, et cetera. And what you saw was people move really quickly. They didn't understand the cloud platforms very well, so they made a lot of human mistakes. And those are things like misconfigurations, common stuff. If you've had, if you paid any attention to the cloud security space, you would have heard the whole thing about like S3 buckets open to the public or databases left open to the public, and these like common misconfigurations that don't really apply to traditional infrastructure environments. What I've started to see already, and I think is going to get worse before it gets better, is the repeat of that symptom applied to AI. And so we're going to see organizations who are like, yeah, I'm sending this huge volume of training data over to whatever AI provider, like insert AI provider name here, and when they move that data over for training purposes, they'll make a similar kind of misconfiguration or mistake where they're accidentally exposed that data in a rush to get a model trained up or in a rush to implement rag, you know, and augment and third party LLM with their own proprietary data. We've seen this play out again and again with every wave of new technology adoption. We rush to move really, really quickly. We make human mistakes because we don't understand the risks of the new platforms and boom, we end up with the same kind of effects. So that's what I'm seeing right now and that's what I predict will get, like I said, worse before. It gets better in the 2025 timeframe.

Okay, Jeremy, before we go, is there one piece of actual advice you can give to someone to help them with the AI API security, DevOps efforts, and what's the best way to find or contact you?

Yeah, okay. So the one piece of advice that I give to anybody who's working on anything related to APIs, and in particular like third party integrations number one, is make sure you actually generate API spec documents. It is the single simplest way to both catalog and assess the security of your APIs. So whatever you're doing right now, by the way, there's lots of open source projects that can help you generate this from code or from documentation, but that is a huge, huge starting point. Everything that you do, you will serve yourself and your organization so much better by taking that step. And then the best way to contact me, honestly, just JeremyIresearch IO just J E R E M Y at Firetail IO is the easiest way. I'm not active on that cesspool of a microblogging site that used to be known as Twitter that I am not a fan of. As you can probably gather from my description of it, you can find me on LinkedIn. But honestly, email is the easiest way to get in touch with me.

Absolutely. And we'll have a link for all these in the comments down below. All right, before we wrap it up, remember, frustrated users quit apps. Don't rely on bad App store reviews. Use SmartBear's Insight Hub to catch, fix and prevent performance bottlenecks and crashes from affecting your users. Go to smartbear.com or use the link down below and try for free for 14 days. No credit card required. And for links of everything of value we covered in this DevOps Toolchain show, head on over to testguild.com p180 so that's it for this episode of the DevOps Toolchain Show. I'm Joe. My mission is to help you succeed with creating end to end full stack DevOps toolchain awesomeness. As always, test everything and keep the good. Cheers. Hey, thank you for tuning in. It's incredible to connect with close to 400,000 followers across all our platforms and and over 40,000 email subscribers who are at the forefront of automation testing and DevOps. If you haven't yet, join our vibrant community@testguild.com where you become part of our elite circle driving innovation in software testing and automation. And if you're a tool provider or have a service looking to empower our guild with solutions that elevate skills and tackle real world challenges, we're, we're excited to collaborate. Visit testguild.info to explore how we can create transformative experiences. Together, let's push the boundaries of what we can achieve.

With lutes and lyres, the bards began their song A tune of knowledge, a melody of code through the air it spread like wildfire through the land Guiding tester showing us secrets to behold.