Are AI Agents Replacing Contract Testing?

Hey, if you want to learn all about API contract testing and API observability, you're in the right place. We have the man in the house, Matt, who is the core maintainer of the popular API contract testing framework pact, and the co founder and principal product manager of the enterprise contract testing platform PAC Float that you probably all heard about, which was recently acquired by SmartBear. I guess I don't know about recently now. I think it was 2022 prior to Smart Bear. Matt really has consulted with all the top products or developed products of some of the top Australian largest brands, specialize in distributed services, cloud computing, DevOps capabilities and continuous delivery. He really knows his stuff. Really excited about this episode. Do you want to miss it? Check it out. This episode is probably sponsored by the folks at SmartBeer empowering teams to build, test and deliver great software faster. Explore how Smart Bear is bringing AI to the forefront of DevOps quality engineering, helping organizations accelerate test creation, optimize automation and to drive smarter decision making across your software development lifecycle. Check them out. Learn more using that special link in the comments down below. Hey Matt, welcome to the Guild.

Hey Joe, thanks for having me. Really excited to be here. I've watched a lot of your episodes over the years, so it's great to be finally on it. Thank you.

Yeah. You are the co founder of PacFlow. How did you get into this? Like did you plan on this? Like how did you get into testing? How did you get into contract testing and how did you get acquired by such an enterprise company?

I guess, yeah, it's a great question. I don't know how far back you go, but like I think like a lot of people you kind of get into, you know, your own role in all kinds of strange ways. Back in school I was a pretty sporty guy actually. So you know, I was going to do sports science or exercise and sports science and you know, kind of had a change of heart, you know, in the last, the last moment and sort of pivoted towards computer science. But I did play a lot of video games in high school. I did a bit of basic coding. I remember, you know, a friend of mine gave me a Debian disc, a physical disk of Debian. And I was really excited to install it. And I got home and put it in there and I'm like, what is this terminal window thing? I was so used to using Windows at that point and so yeah, I kind of got interested in computers at that point. So I was kind of a weird growing up, like very sporty but Also kind of a bit nerdy in that sense. And then, you know, I last minute, like many people, you didn't really know what you wanted to do at that age. I'm sure it's, that's still common. So I ended up doing a double degree in computer science and cognitive science. So a bit of a strange one back then, this is 20 odd years ago that was sort of trying to put together effectively artificial intelligence, right? It's like take the human brain, take the computer capabilities and try to make something intelligent out of it. Of course things have moved on a lot since then. But then straight out of university I got a job doing web development and did that for quite a long time, about 10 years there. And I really just wanted to get back into you eventually get far away enough you become a manager. And I really wanted to get my hands back in the weed. So I did some consulting and that's where my first consulting job actually I came across Pact. It was a pretty new thing and I'd use it on this client and it was a very transformative project actually it was out at Australia's largest job board over here called Seek. Fantastic company. And we sort of did this called DevOps. Transformation went from monolith to microservices, this is circa 2014. And we took their first product and took it to the public cloud onto Amazon and went from deploying once every couple of months really painfully to literally we could hit a button, push the code, seven minutes later it was in front of a customer. And I still remember in Melbourne, the sort of head of the Australian division was walking around and we did a bit of a walkthrough of what we did and we'd like to build lights, the old school things. And we got him to actually sit down and make a change to the homepage of the application and he just had no idea what was going on. Like seven minutes later he's like, I don't understand, is this like a test environment? Like, no, that's production. You're in production. And so that kind of really helped cement, you know, fast, continuous delivery type activities in that company. And it was very formative for me in my consulting journey. So that's when I got involved with Pact. I thought it was a great tool, but of course like many open source things, it needed a lot of work. It was pretty rough around the edges. So I started maintaining it at that point, so about 10 years ago and have been involved with Pact ever since. In terms of the question, how do we build the company and how did that happen? I liken it a little bit to boiling a frog. It sort of gradually happened again. When I was at the consulting company we did a lot of product development, very product focus, lean startup, that kind of thing. Iterate, get things in front of users, get feedback and go from there. And I guess you work for enough customers and companies and you get frustrated at some of their decisions. You're like, I want to have to make that decision. I want skin in the game. And so we worked with the directors of DS was a company and they're a great bunch of guys and really good company, very supportive of this. So we kind of gradually introduced just a free, a free plan where you could sort of host these, excuse me, these contracts and yeah, it's kind of. The rest is history. A lot of people started using it. We got a bit more investment from Deers to help us to build it. And yeah, in 2022 we got a knock on the door from SmartBear and as we say, yeah, the rest is history.

Love it, love it. A lot of pivots there from sports science, artificial intelligence to contract testing. How'd you know artificial intelligence was going to be a thing? Or did you not know? You just was interested in it?

Yeah, it's a really good point. Like I didn't. So I didn't actually think too much about the artificial intelligence side of things when I joined the course in the sense. I actually just really enjoyed the psychology side of things and human computer interactions and all this. It just happened to be that the course had a lot of philosophy and morality sort of centered in the middle. But I think also just I didn't know what to pick at the time. Right. Like I found both computer science really interesting and I really like the cognitive science of things. It had neuroscience in there as well. So it was just a really interesting course at the time. We had to build our own neural nets and these things from scratch and perceptron networks, et cetera. And so I found that again, very technically, scientifically, theoretically fascinating in terms of real world applications at that point in time. Definitely we weren't thinking of artificial general intelligence. Generative AI was definitely not a thing. It had its roots in what deep learning would be these days in terms of those really deep networks. It was more about classification and identification of things and solving those very specific point problems. It definitely wasn't about what we're seeing today. And obviously we've always had a view to the AGI world at some point. It's always been somewhat this futuristic Thing. So, yeah, in my particular case, I just really enjoyed it from a theoretical, interesting point of view. But then it kind of went away. You get a job and all of a sudden the practicalities of life, all the practicalities of working in a software company kicks in. Right. And you have to deal with tests and all these other things and all of those theoretical things go away.

Absolutely. I'm sure it gives you insight though, into generative AI and where things are going, because you have a solid background, it sounds like. For sure.

Yeah. I mean, we spend a lot of time doing fuzzy logic and building our own perceptual networks. And I actually remember, I was thinking of the other day when I was doing something around the agentic AI is that I remember doing this thing called agent orientated programming. And I was thinking, I should probably go back and look at that. I wonder if there's anything in there that would be, that would be interesting or that would be relevant for today's world with what is now what we refer to as agentic AI. So I should probably go back. I've got some textbooks somewhere that probably have something about it. But yeah, in terms of that background, I think, you know, you learn about, you know, obviously Turing. You spend a lot of time thinking about Turing and you think about the morality and all of this. So I think that. I think a lot of that grounding has been helpful with some of my thinking today. But of course, you know, the world moves very quickly and particularly in the last few years, Gen AI has moved very, very quickly for sure.

So I didn't mention your bio, but it is part of your bio. You describe yourself as someone that I think really enjoys the intersection of tech, humans and ideas. Maybe. How does that mindset help shape your approach to product and open source and how you see the future with AI being more and more involved in everyone's lives?

Yeah, that's a good question, I suppose. I think like a lot of people who've been around for a while, you sort of swing a little bit between being a practitioner and getting really deep in the weeds. And then sometimes you spend so much time that you become disconnected from the thing you're trying to do, the customer or the problem or what have you. And then you move up the chain a little bit and then you feel like you're too far away from the code again. So there's this manager, engineer, pendulum, there's a few really good articles out there from charity majors on that topic that really resonated with me. So I think on the one Hand, I really like the problem solving aspects of things. I love the coding aspects of things and all that as well. But from a product point of view, you do need to make sure that you're connected to the humans at the other end of it who are going to use it. Maybe not so much these days, but at least traditionally, someone's going to be consuming your product. And so I think being at that intersection is really interesting. And ultimately, again, going back to that sort of formative experience I talked about earlier as software engineers or as developers, testers, what have you, you want to make sure the thing you build gets in front of people and gets out to the real world. Because yes, there's something interesting sometimes in doing something just purely technical, but if, you know, if no one's using it, is that fulfilling? Not really. So I think being able to connect real world problems, real people to the things you're building is really important. So that's probably the first part of it. I think the second part you're asking about sort of how AI is coming into that. Was that the question, Joe? Yeah, I mean, it is interesting. Obviously there was a survey from the Stack Overflow survey came out yesterday. I'm not sure if you've read that yet, but yeah, it just came out yesterday. So I look at that for a few reasons. It's a really good insight into the minds of those who are consuming our types of products. As a developer tooling company, who's using it, why they're using it, what other tools they're using. There's a whole section on AI, unsurprisingly, because it's such a big topic at the moment, I think it's okay to summarize it like this. That sentiment has actually gone down in terms of trust and frustrations with AI. Getting it close, but not close enough. It's kind of right. But then you have to spend more time debugging it, right? And I think what they said is compared to last year, those numbers have gone down a little bit. That being said, the usage is continuing to go up and everyone basically says we expect to do more of it next year. So I think that's really interesting. It tells us maybe the hype is. Hype is the wrong word, but it is very hyped up, right? It is very much in everyone's face. If you look at. There was one other statistic that I found really interesting was when they evaluated tools, which tools to use, they ask a whole bunch of what is the most important to least important set of priorities for you when you're using these tools. And out of 10, AI capable was number nine on the list. So I think number one was ease of use. I think it was. And two was basically the capabilities. So I think as tool builders and as really as anyone. But if you're building tools, that tells you some things like AI is interesting, but we still fundamentally care most about is it easy to adopt and use and does it have the features we need to do the job? Yeah. So I found that a really interesting data point and obviously we'll have to unpack the entire survey, but that was probably one of the headlines for me. So coming back to how it changes everything, I think it's obviously going to change things like the agentic movement into you're working with a pairing partner that you can give jobs and it can off and do in the background for you, or you're pairing with it in real time. We're going to see more of that, obviously, but ultimately, maybe I'm a little bit too optimistic, but the human in the loop design, I think we still want to retain that. Humans are still the ones making those decisions. We're still the ones getting paged in the middle of the night when something goes wrong. So do you want a machine creating code that's going to then wake you up in the middle of the night? I don't think so. All those factors, I think still put the human at the center and the arbiter of that quality. And so, yeah, I think more augmentation is going to be part of the future. I don't think it's going to solve the biggest problems though, just yet. To go off on one more tangent, I'm not sure if you're familiar with Goldratt's work on the book called the Goal, but it's a theory of constraints is what we're getting at here. And if you look at productivity at the individual level, AI is helping us all do things more quickly. I think that's really clear. And that was also. That came out in the survey. But at the company level, a lot like that hasn't seemed to have filtered through just yet in terms of companies being much more productive. And I think there's a bunch of reasons for that. One is that if you're just optimizing at the individual level, you haven't necessarily solved the systemic problems. And often the constraint isn't the individual performance, it's actually the system itself. So I think we're going to have to address that before it really makes a big difference and the second factor, I think, is that a lot of companies are still actually experimenting and trying to work out how to get the most out of these types of tools. There's some good public examples like Shopify, who have gone pretty hard at this, and there's a bunch of others as well. But that's going to take some time, I think, to filter through as well.

Absolutely. All right, so let's level set before we go a little bit further, we mentioned contract testing a few times for the folks. I don't know what is contract testing and has it pivoted in the age of AI or is it still the same? Do you see AI impacting it where it's not a thing anymore? I don't know if that makes sense.

Yeah, that's a really good question. So I guess, coming back to basics, what is contract testing? So, by analogy, my colleague came up with this years ago, which is, do you test that your fire alarm works by setting your house on fire, or do you press the button on the fire alarm to check that it's working? And of course, most people would answer, I don't set my house on fire. So contract testing really is a way to ensure that two different systems, such as microservices or a mobile app and an API, are compatible with one another. And it's going to capture the interactions that occur between the two of those systems into a contract. And then we can use that artifact to check that both those systems adhere to it over time. And so that's very abstract. Pact is one version of that that allows you to do this kind of testing. It's specifically a code first way. So you write a unit test against your API client that spits out at the end of it all the interactions you made with it, and some other things like matching rules, et cetera, to help make it more flexible. And then that contract is what gets shared between its provider, who then verifies it. So it's a bit like record and replay in that sense, but it's got matching rules and these other examples attached to it to help make the interactions between them very concrete. We would also refer to that as a specification by example. You could do that without the need for a specification. You've got all the examples that explain how the thing works. It's often compared to end to end testing, and we're responsible for a lot of that. End to end testing is where you stand up the entire system and you run tests from the top to bottom. And I'm sure your audience is more and more Very familiar with all that. So contract testing differentiates itself a little bit from that in the sense that you want to isolate the two components that are talking to each other and take it out of the broader system. And so that does enable you to do that in terms of the way you scale that up across the different systems. With an end to end system, if you're running the test from the outside, the number of tests you need to cover it becomes quite significantly large. And so you have to be quite tactical in terms of what endpoints you test. Whereas with contract testing it scales linearly. So you can keep adding as many contract tests as you want between those individual services and it's an N plus one as opposed to constantly having to combinatorially work out all the interactions. That's a bit complicated. So coming back to basics, your question is where does contract testing go in the age of AI? Well, yeah, that's the million dollar question. Of course, the product managers like myself are thinking, I guess there's a few parts to that. So the first answer to that question is, well, we know AI adoption is going to continually grow. That's not going to stop anytime soon. We went out from the survey, but I think we all just know from our own daily use of it that there's more and more tools coming up that are doing this. So I think that's the first point. I guess the second point is we know that the LLM is going to get more and more powerful, the frontier ones in particular, and probably more Frontier LLMs will start to appear soon too that have very specific use cases. I saw one the other day specifically for note taking and I think we'll start to see more of that as well. So I do think running test pactest might be a thing of the past eventually, for example with pact, but more generally. So I think the future is moving towards autonomous contract testing with that human in the loop design where effectively those things can be written from your existing code or other artifacts. What I will say is, and then everyone's going to be grappling with, to some extent, as a community, one of the things. So we know that tests are there for regret. Often they're used there for regression, right? They're used there to make sure behavior doesn't break, things keep working. That's an obvious use case for tests. But the, you know, the thing that I think we often miss in this conversation is that tests aren't there just for that one purpose, particularly unit tests. Unit tests are there. And in the case of pact, the Reason I mentioned this is PACT is often written as a unit test or should be written as a unit test or close enough to. By automating unit tests. That's. That's an interesting thing for me as someone practitioner of XP and these kinds of things. Right, because unit tests exist to not just check behavior, but to help you refactor your code, to help you build higher quality code. Right. It's testing the system design itself. And so if we auto generate all those tests, are we losing something by not doing that? Like, are we making. Are we making our systems less good? Are we making them worse? Less quality?

Are we. If we do that?

I don't know. I think the jury's still out on that a little bit. But it is absolutely certain that we're Vibe coding all these things. We're not necessarily producing better code. With AI, I think there's definitely a risk of producing that. And again, the Stack Overflow Survey has some little insights into the differences between those who are learning to code versus those who've been professionally using it. There's a few data points now I won't remember off the top of my head, but what I worry about, of course, is that the more and more things we throw at the wall through Vibe coding and don't check ourselves, we're going to be creating more and more bugs before. So I've gone blank on the old saying, but for every line of code I write, two bugs, I'll have a job forever. Well, I don't know what the equivalent of that is for Vibe coding, but I suspect it's more than what you're doing if you're running by hand. So I do worry about us producing worse code. And by definition, if we're doing the same thing to produce tests, are we now making our test worse as well?

I have the same problem. I've been Vibe coding for a few months now, and it's just pumping it out to production. And just today, AI wrote something crazy and brought the whole site down. And this is just my little application. I can imagine how many people actually doing this. I haven't even looked at the code. And I'm a tester, so it can be kind of dangerous just putting your full trust in it almost without having that sanity. And every time I bring this up to an AI expert, they go, oh, just early days now, though. So in a year or so, two years, it'll be even better where you don't have to worry about it. But I don't know if we're ever going to get there?

Yeah, absolutely. I think it will get better. I would agree with that. But if we expect AI to get to human level intelligence, well, we know humans make lots of bugs, so then we expect at least the clones of us to do something similar. Maybe they'll make the point that AI will exceed human performance for some of these things. And maybe that's true as well. Will they be bug free? I don't know. I don't think so. And then also that bug's only one dimension as well, Right. So there's performance and all these kinds of things. And of course you can go, we'll go get Claude Flow or something like that out and we'll go put a swarm of agents on this problem and make it performant. And so maybe that will be part of the future. But I do think who checks a checker type thing? I don't know if we're going to live in a world without that. And again, going back to all the other practical limitations, at least for now, if the banking system goes down, mostly humans still care about it and so it would want to come back up pretty quickly. Whereas if you start optimizing for machines, certain end goals might fall by the wayside. And so, you know, they might fix it, but they might fix it really slowly because they want to re optimize the system. So I think it's going to be hard to pull the humans out of the loop, at least for some while. And as soon as humans are there as part of an important part of that value chain, I don't know, I worry about putting a full confidence in AI at this point.

Definitely agree. All right, so I keep going back again, tripped up on AI. But I also want to talk about API observability. Where does API observability come into play here?

Yeah, that's a good question. I mean, it's not something that's my expert area, but you know, you're talking about from a, from a contract testing point of view or from just a.

Yeah, so if you're doing AI's pushing code, I'm sure you have some sort of alerts that humans get maybe that are monitoring just to verify that whatever's being pushed is correct, I assume.

Yeah.

Is that something SmartBear is building into?

Yeah. Okay. Yeah, from a smart bear point of view, that's a good question. It's probably not something I'm as close to. There's a separate set of product group there called Insight Hub and we have bugsnag in there and we use that and yeah, from an API observability point of view, definitely that would be one of the tools and areas that we would be continuing to invest in. I'll be honest, Joe, I don't have a whole lot of great answers for that on this point at the moment. So.

All right, so as you mentioned, we see AI generating more and more code. Could AI generate reliable contracts and mock automatically? Which it sounds like maybe. And does that introduce new risk, which I think we touched on a little bit more?

Yeah. So specifically on that question, we in PacFlow will definitely be moving more in that direction, for sure. So I think our customers tell us pretty clearly, and it's no secret, we know the, the pros and cons of tools like PACT and the challenges that come with writing PACT tests. You know, we get told by our customers and competitors that, you know, writing PACT test is a, you know, is a maintenance burden. Before I address the AI question, what I will say is I was talking to a customer a few weeks back and we're exploring some different options in terms of how we're looking to, you know, look at like a pack 2.0 in the future. And this person made the point that they really liked the way the PACT works today in that you have to write a unit test and you have to slow down to write it, which was interesting. It's not something that's totally surprising, but it's interesting to get that kind of contrarian view. And their point was you have to think through what you're doing to make sure that your code is doing what it's supposed to do. And I don't necessarily want to be in a world where those tests are generated automatically for the reasons we discussed in that. How do I know that it's really stressing the contract, stressing the system? It's high quality. Our code is actually doing what it should do. Is our API client supposed to do that? Maybe it's not. And so of course, you run the risk then of automating tests for a part of your system that doesn't get used. And so now you've just got these wasted tests. So I think that's one angle. But in terms of where does AI go? Go here? Yes, I do think we'd be moving more to that sort of autonomous testing with the human in the loop design. So you'd imagine in the case of pact, for example, we would produce those consumer contracts that fit to the API client, and we would make sure that they would align to best practices, et cetera, and then the contract would be able to be generated from that. And conversely, on the provider side, we would set up the provider side, those tests as well, pull down all the things you need to test and, and point that against the running version of the provider. Some of those things are tricky and complicated and that's where tools like Copilot and Claude, et cetera, so pick whichever one of your agenti tools that you like can do a really good job of that because they have really good awareness of your code base. They can index things quickly, they can do that. That's something that those tools are really good at. So I think products like us, developer tools like us, will spend more and more time investing in things like mcp, which is something we're doing right now to not only provide capabilities to do certain things like generate a test or review a test, or provide access to existing APIs so that you can introspect data or introspect contracts and things to help write them and verify them, but also to provide specific prompts that enable those copilots and things to do specific use cases. For example, establish PACT on a new project. That's a very complicated thing to try to automate. If you're not a Frontier LLM, you don't have that kind of capability, you would need to call into those really powerful models, whereas the agents like Copilot, Claude, Cursor, etc. They can do this. And so if we can give them the right prompts with right information and context, they're going to do a much better job of automating some of those. So that's probably where we're headed, at least from the existing point of view. And of course the challenge we want to make sure is that we don't have tests that just check fit what they are currently doing. And so effectively you're just testing the bugs as opposed to finding the bugs. And that's probably part of the challenge. Look, it's entirely possible that in some future world you don't need something like Pact or any of our alternatives, because you can just statically look at code on both sides or something and go, actually, we know what this API client can do on this side, we know what the API can do and we can just tell you, yep, they're going to work. And maybe that's where we eventually get.

Yeah, when you bring up MPC like Playwright, you don't need to know what the clicks and everything is. It just knows. You don't even know. Just, hey, write me an automated test, I may call that server and know what to do. So it's almost like contract testing. And I'll know, like I just know what they're expecting and what, what I should send it sounds like.

Yeah, definitely. I think the paywright example is a really good one. Even the Cypress, I think that those analogies work pretty well for us because you can't really write those types of tests without access to the code base. Like you can't do it from, well, you kind of can do it from a UI I suppose, but you need to run it in some real environment. It can't just be done through a GUI somewhere else that doesn't have access to your code. And so same with pack testing or contract testing. We definitely want to make sure that we check the real behavior of your API client and not just check the API itself. We want to make sure, okay, if you've got a mobile application and it's calling these different APIs that we actually check the mobile client calls those various APIs to make sure that we've got visibility. And this is something I didn't mention earlier, but I think one of the benefits of contract testing over just purely API focused testing, I would say schema based contract testing, where someone's just checking the API matches its specification. One of the benefits of contract testing is you get visibility into what your consumers are using of that API, what the surface area that's being used. In the case of Pact, you can get that down to the field level. That's really powerful because now we can know, okay, we've got these 10 consumers, but no one's using this endpoint or no one's using these properties, so we can refactor those out later on without damage. That's a really powerful thing to have. So bit of a sidebar there, but coming back to the contract testing AI side of things, we obviously want to make sure we're testing the clients as much as we can and ensuring that we're testing real behavior. And then of course, once we've got the test for real behavior, then we've got the confidence. So I would imagine the tons of sources I can think of that we would be able to use to generate those because all of those tests happen API to API. So some of our competitors, for example, they don't do contract testing, but they do testing through effectively kind of an end to end test by using traffic analysis. So hypertest and Signadot are examples of companies that do that kind of other ephemeral environments or mocking using traffic inspection and effectively record and replay, but at scale ebpf is another promising kind of technology. If you guys come across that, and a bit esoteric at the moment, doesn't quite work fully cross platform. But you know, I think that's, I think a lot of really cool things will happen in that space in the future and obviously a whole bunch of things have already happened. You know, OTEL opentelemetry might be another source to pull that kind of information from. OTEL by default doesn't capture full request response payloads, et cetera. So if you want to get the full details, you'd probably need to modify your instrumentation. But there are a few kind of alternative, promising ways that probably could be pulled into, we'll call it quote unquote, Quote unquote contract testing workflows, whether that's with PACT or otherwise, to check that both sides are working together.

This might be a dumb question. Do these agentic, not these agentic, these code creating things like cursor and copilot, do they know about contract testing? But is the person like it knows, okay, I'm going to create a contract test for this. So you have to prompt it or actually have it in the, in the instructions. Every time you write a code, make sure to write a contract test for it.

Yeah, they, for the most part they do. They're usually built on top of the frontier models. So the frontier models, you know, like OpenAI and Claude etc. Or anthropic and all their models, they, they obviously index the Internet at a certain point in time. So one of the challenges is they often have outdated, you know, DSLs and things like this. So what was it? I think it was OpenAI. Oh no. So I think it was, I can't remember now actually, I won't name it and I won't name one, but one of them was constantly using a very old DSL and we couldn't force it to use a newer one. And so that's where additional work on our side to get it to generate tests is required to do that. And also sometimes, you know, because the APIs and the DSLs that exist out there, at least in packed world, there's lots of them and there's like, you know, there's a generation of changes, it doesn't necessarily know which one is the right one to use and it often will use the one that has the most examples on the Internet. So you know, something I noticed, for example, is in, in the JavaScript land, a lot of the examples I wrote, because I wrote, I manage pack JS and I in my test, I still use Mocha and I still use Chai as my testing framework in the assertion libraries. And it's really interesting because if you look at the kinds of tests that when you generate with OpenAI, for example, it always with JavaScript defaults to using Mocha and defaults to using Chai as the assertion language, which, pardon me, probably.

Trained on your work.

It's trained on those examples. Exactly. And they're probably in some ways not the best because most people probably use Jest these days. And so having it default to jest again, we've had to put work in there to make sure that we don't, you know, do that. And so I think that's a good reminder that, you know, they are built on our own work and, you know, so we need to think about kind of. I was talking the other day to our dev teams, like, do we need to think about effectively SEO? But for AI now, like, if you think about in the old days, you needed to get a lot of content out there so that the search engines would find you. It's kind of the equivalent of that now with AI, right. We want to make sure we've got really good examples and documentation and LLM, Txt, et cetera out there so that the agents index your code and your examples in a way that you want them to and to make them visible, to then have them used in the tools. So it's going to be an interesting world there. But yeah, to come back to your question specifically, they do know about contract testing. They occasionally just need, well, they usually need a bit of prompting and steering to get the right kind of output. And so in some of the tools we built, there's a fair amount of prompting, prompt engineering we've got in there to. To steer it away from bad practices, to steer it towards good practices, to review the code in a way that we would like it to as a humans. Like we just released code review this week actually to help anyone who's got existing tests to make sure they're good, but also when we generate tests to make sure that the generation itself follows those best practices. So there's a lot of little things in there just to improve the usability of that, but for the most part they can do a pretty good job if you prompt it really well.

So can you share anything on maybe on the roadmap for SmartBear where you see contract testing, I know you're not a, you know, you could see in the future, but three to five years. I know you mentioned the stack overflow where there's some people like AI was 10th on the list. So how do you take that with, you know where your roadmap is, which I assume is just AI heavy to begin with?

Yeah, it's a really good question. So I've been thinking about this a lot recently, as I'm sure pretty much every product manager has for a while now as well. I think the first thing to point out is, yes, we need to enable our tools in the AI ecosystem. So that would be what we've done initially is built some very base capabilities in the AI space. So generating pack tests, reviewing PAC tests and across multiple languages, et cetera. But we knew we're never going to be able to do that and compete with the frontier models. Right. All we can really do is work with them. Because unless you've got thousands of engineers with PhDs and giant data centers, that's a very difficult thing to do. So I think the best thing for us is to partner with those types of models and leverage the best we can out of them. Where we need to move to, where we are moving to now is supporting work within the agentic system. So how do we make our data, how do we get unique sources of data into our products? Because that's actually what's going to be really helpful to provide support for those agents when you're running tests and you're fixing bugs, hey, we've got a bug Snag integration. We had this error in production. Can you help me fix this error in your agent? You give it the error, it goes to bugsnag, finds the error, finds a line of code that is relaxing, and then it tells you, okay, you need to fix this line of code. And so those kinds of cross product journeys within SmartBear, for example, they're going to be really powerful in terms of solving really big issues. Especially if you've got contract tests there and you've got your spec and you've got your other tests all in one place. Now the agent can query all that data from those various systems within SmartBear, piece together the puzzle and then suggest the next step to help either fix a bug or modify that change to the API. If you go a little bit further into the future, obviously three to five years is like a millennia in terms of AI in terms of. But I think, I think we all need to be at the point where we've got really good support for those agentic workflows. That's the first thing. But beyond that, again, going back to the Stack Overflow survey, what everyone tells Us more breadth and depth of feature support is number two and ease of use is number one. And so I do think getting back to bread and butter stuff is actually going to be where we win in the future. So we'll need to better support the models. We'll need to be in that ecosystem through MCP or some of the other protocols that exist in there. But ultimately it's actually building really good products that have the features that solve the problems, and then humans and agents hopefully will then choose those features that have those products that have the most capability.

Love it. The first part, you did mention writing for agents instead of humans almost. It's the same thing with SEO. Now, I heard you have to write for be able to have the LLMs be able to understand it rather than a human. It's a hard mix. I would assume being a product owner and product manager of that for sure.

Yeah, I certainly worry about that in some ways, but it makes sense. At least in the near term. It makes sense to ensure that you're giving your product and your capabilities the best chance of being used in those tools. I still think humans are first, but I still use please and thank you when I talk to the agent, just in case. You never know.

I heard that. Waste tokens and it wastes energy. Awesome. Okay, Matt, before we go, is there one piece of actual advice you can give to someone to help them with their API contract testing, DevOps efforts, and what's the best way to find or contact you or learn more about PACT and contract testing and all the goodness at SmartBeer?

Yeah, I mean, the simplest advice for me is just start simple and get started. So if you want to look at contract testing, don't overthink it. Just literally get in there, write your first test, make a few mistakes and then move on. But keep it small and constrained. I always think that's the best way to get started. I'm a very kinesthetic kind of learner. I like to just get my hands on things and figure it out that way. That's how I like to do it. In terms of getting contact with me, I think I've got all the usual things. I'm on LinkedIn. You better find me under my very interesting name of. I think it's digital Matt, actually. And then on Twitter you can find me under a less interesting name, which is Matthew Fellows.

And for links of everything of value we covered in this DevOps Toolchain show, head on over to test guild.com p19e. So that's it for this episode of the DevOps Toolchain Show. I'm Joe. My mission is to help you succeed creating end to end full stack DevOps toolchain awesomeness. As always, test everything and keep the good. Cheers. Hey, thank you for tuning in. It's incredible to connect with close to 400,000 followers across all our platforms and over 40,000 email subscribers who are at the forefront of automation, testing and DevOps. If you haven't yet, join our vibrant community@test testguild.com where you become part of our elite circle driving innovation in software testing and automation. And if you're a tool provider or have a service looking to empower our guild with solutions that elevate skills and tackle real world challenges, we're excited to collaborate. Visit testguild.info to explore how we can create transformative experiences. Together. Let's push the boundaries of what we can achieve O the Teskill Automation Testing Podcast with lutes and lyres, the bards began their song A tune of knowledge, a melody of code through the air it spread like wildfire through the land Guiding testers, showing the secrets to behold.

To not just check behavior, but to help you refactor your code, to help you build higher quality code. Right. It's testing the the system design itself. And so if we auto generate all those tests, are we losing something by not doing that? Like are we, for example, we would, we would produce those consumer contracts that fit to the API client and we would make sure that they would align to best practices, et cetera, and then the contract would be able to be generated from that. And conversely, on the provider side, we would set up the provider side, those tests as well, pull down all the things you need to test and point that against the running version of the provider. Some of those things are tricky and complicated and that's where tools like Copilot and Claude, et cetera. So pick whichever one of your agentic tools that you like can do a really good job of that because they have really good awareness of your code base. They can index things quickly. They can do that. That's something that those tools are really good at. So I think products like us, developer tools like us will spend more and more time invest making are we making our systems less good? Are we making them worse? Less quality?

Are we if we do that?

Well, I don't know. I think the jury's still out on that a little bit, but it is absolutely certain that we're vibe coding all these things. We're not necessarily producing better code with AI. Right. I think there's definitely a risk of producing that. And again, the stacker thing in things like mcp, which is something we're doing right now, to not only provide capabilities to do certain things like generate a test or review a test, or provide access to existing APIs so that you can introspect data or introspect contracts and things to help write them and verify them, but also to provide specific prompts that enable those copilots and things to do specific use cases. For example, establish PACT on a new project. That's a very complicated thing to try to automate. If you're not a Frontier LLM, you don't have that kind of capability, you would need to call into those really powerful models. Whereas the agents like Copilot, Claude, Cursor, et cetera, they can do this. And so if we can give them the right prompts with right information and context, they're going to do a much better job of automating some of those. So that's probably where we're headed, at least from the existing point of view. And of course, the challenge we want to make sure is that we don't have tests that just check, just sort of fit what they are currently doing. And so effectively, you're just testing the bugs as opposed to finding the bugs. And that's probably part of the challenge. Look, it's entirely possible that in some future world, you don't need something like PACT or any of our alternatives, because you can just statically look at code on both sides or something and go, actually, we know what this API client can do on this side. We know what the API can do and we can just tell you, yep, they're going to work. And maybe that's where we eventually get.

Yeah, when you bring up MPC like Playwright, you don't need to know what the clicks and everything is. It just knows. You don't even know. Just, hey, write me an automated test. It may call that server and know what to do. So it's almost like contract testing. And I'll know, like, I just know what they're expecting and what I should send it sounds like.

Yeah, definitely. I think the paywright example is a really good one. Even the Cypress, I think that those analogies work pretty well for us because you can't really write those types of tests without access to the code base. Like, you can't do it from. Well, you kind of can do it from a ui, I suppose, but you need to run it in some real environment. It can't just be done through a GUI somewhere else that doesn't have access to your code. And so same with pack testing or contract testing. We definitely want to make sure that flow survey has some little insights into the differences between those who are learning to code versus those who have been professionally using it. There's a few data points now I won't remember off the top of my head, but what I worry about, of course, is that the more and more things we throw at the wall through vibe coding and don't check ourselves, we're going to be creating more and more bugs before. So I've gone blank on the old saying, but for every line of code I write two bugs, I'll have a job forever. Well, I don't know what the equivalent of that is for vibe coding, but I suspect it's more than what you're doing if you're writing by hand. So we check the real behavior of your API client and not just check the API itself. We want to make sure, okay, if you've got a mobile application and it's calling these different APIs that we actually check the mobile client calls those various APIs to make sure that we've got visibility. And this is something I didn't mention earlier, but I think one of the benefits of contract testing over just purely API focused testing, I would say schema based contract testing, where someone's just checking the API matches its specification. One of the benefits of contract testing is you get visibility into what your consumers are using of that API, what surface area that's being used. In the case of Pact, you can get that down to the field level. That's really powerful because now we can know, okay, we've got these 10 consumers, but no one's using this endpoint or no one's using these properties. So we can refactor. Those do worry about us producing worse code. And by definition, if we're doing the same thing to produce tests, are we now making our test worse as well?

Yeah, no, I have the same result.

Later on without damage. That's a really powerful thing to have. So bit of a sidebar there, but coming back to the contract testing AI side of things, we obviously want to make sure we're testing the clients as much as we can and ensuring that we're testing real behavior. And then of course, once we've got the test for real behavior, then we've got the confidence. So I would imagine the tons of sources I can think of that we would be able to use to generate those, because a lot of those tests happen API to API. So you know, some of our competitors for example, do. They don't do contract testing but they do testing through effectively kind of an end to end test by using traffic analysis. So you know, hypertest and Signa Dot are examples of companies that do that kind of other ephemeral environments or mocking using traffic inspection and effectively record and replay. But it's problem.

I've been vibe coding for a few months now and it's just pumping it out to production and just today AI wrote something crazy and brought the whole site down and this is just my little application. I can imagine how many people actually fail.

EBPF is another promising kind of technology. If you guys come across that, and it's a bit esoteric at the moment, doesn't quite work fully cross platform, but I think a lot of really cool things will happen in that space in the future and obviously a whole bunch of things have already happened. OTEL opentelemetry might be another source to pull that kind of information from. OTEL by default doesn't capture full requests response payloads, et cetera. So if you want to get the full details you'd probably need to modify your instrumentation. But there are a few kind of alternative promising ways that probably could be pulled into, we'll call it quote unquote contract testing workflows, whether that's with PACT or otherwise, to check that both sides are working together.

This might be a dumb question. Do these agentic. Not these agentic, these code creating things like, like cursor and copilot. Do they know about contract testing? But is the person like it knows, okay, I'm going to create a contract test for this or you have to prompt it or actually have it in the, in the instructions. Every time you write a code, make sure to write a contract test for it.

Yeah, they, for the most part they do. They're usually built on top of the frontier models. So the frontier models, you know, like OpenAI and Claude, et cetera or Anthropic and all their models, they obviously index the Internet at a certain point in time. So one of the challenges is they often have outdated DSLs and things like this. So what was it? I think it was OpenAI, I think it was. I can't remember now actually I won't name it, doing this, I haven't even.

Looked at the code and I'm a tester so it can be kind of dangerous just putting your full trust in it almost without having that sanity. And every time I bring this up to an AI expo, they go oh, just early days now, though. So, you know, in a year or so, two years, it'll be even better where you don't have to worry about it. But I don't know if we're ever going to get there.

Yeah, no, absolutely. Like, I think it will get better. Won't name one, but one of them was constantly using a very old DSL and we couldn't force it to use a newer one. And so that's where additional work on our side to get it to generate tests is required to do that. And also sometimes because the APIs and the DSLs that exist out there, at least in Pact World, I would agree with that. But if we expect AI to get to human level intelligence, well, we know humans make lots of bugs, so then we expect at least the clones of us to do something similar. You know, maybe they'll make the point that AI will exceed lots of them. And there's like, you know, there's a, there's a generation of changes. It doesn't necessarily know which one is the right one in human performance for some of these things. And maybe that's true as well. Will they be bug free? I don't know. I don't, I don't think so. To use. And it often will use the one that has the most examples on the Internet. So yeah, something I noticed, for example, is in, in the JavaScript land, a lot of the examples I wrote because I manage pack JS and in my test I still use Mocha and I still use Chai as my testing framework in the assertion libraries. And it's really interesting because if you look at the kinds of tests that when you generate with OpenAI, for example, and then also that bug's only one dimension as well. Right. So there's performance and all these kinds of things. And of course you can go, we'll go get Claude Flow or something like that out and we'll go put a swarm of agents on this problem and make it perform it. Always with JavaScript defaults to using Mocha and defaults to using Chai as the assertion language, which, pardon me, probably trained on you. It's trained on those examples. Exactly. And they're probably in some ways not the best because most people probably use jest these days. And so maybe that will be part of the future. But I do think who checks a checker type thing, having it default to Jess again, we've had to put work in there to make sure that we don't do that. And so I think that's a good Reminder that I don't know if we're going to live in a world without that. And again, going back to all the other practical limitations, they are built on our own work. And so we need to think about kind of. I was talking the other day to our dev teams, do we need to think about effectively SEO? But for AI now, if you think about in the old days, you needed to get a lot of content out there so that the search engines would find you. It's kind of the equivalent of that now with AI, right. We want to make sure we've got really good examples and documents, at least for now. If the banking system goes down, mostly humans still care about it and so it would want to come back up pretty quickly. Whereas if you start optimizing for machine imitation and LLM, txt, et cetera out there so that the agents index your code and your examples in a way that you want them to and to make them visible to then have them used in the tools. So it's going to be interesting world there. But yeah, to come back to your question specifically, they do know about contract testing. They occasionally just means certain end goals might fall by the wayside and so they might fix it, but they might need, or they usually need a bit of prompting and steering to get the right kind of output really slowly because I want to re optimize the system. So I think it's going to be hard to pull the humans out of the loop, at least for some. And so in some of the tools we've built, there's a fair amount of prompt engineering we've got in there to steer it away from bad practices, to steer it towards good practices, to review the code in a way that as soon as humans are there as an important part of that value chain, I don't know, I worry about putting our full confidence in AI at this point. Definitely agree that we would like it to as a humans like we just released code review this week actually to help anyone who's got existing tests to make sure they're good, but also when we generate tests to make sure that the generation itself follows those best practices. So there's a lot of little things in there just to improve the usability of that, but for the most part they can do a pretty good job if you prompt it really well.

Love it. So can you share anything on maybe on the roadmap for SmartBear where you see contract testing? I know you're not a, you know you could see in the future by three to five years. I know you mentioned the stack overflow, where there's some people like AI was 10th on the list. So how do you take that with you know, where you're all right. So I keep going back again, tripped up on AI. But I also want to talk about API observability, whereas API roadmap is. Which I assume is just AI heavy to begin with.

Yeah, it's a really good question. So I've been thinking about this a lot recently, as I'm sure pretty much every product manager has for a while now as well. I think the first thing to point out is yes, we need observability come into play here. Yeah, that's a good question. I mean it's not something that's my expert area to enable our tools in the AI ecosystem. So that would be. But you know, you're talking about from a contract testing point of view or from just a. Yeah, so if you're.

Doing AI's pushing code, I'm sure you have some sort of alerts that humans get.

Maybe what we've done initially is built some very basic capabilities in the AI space for generating PAC tests, reviewing pack tests and across multiple languages, et cetera. But we knew we're never going to be able to do that and compete with the frontier models. Right. All we can really do is work with them. Because unless you've got thousands of engineers with PhDs and giant data centers, that's a very difficult thing to do. So I think the best thing for us is to partner with those types of models and leverage the best we can out of them. The what where we need to move.

To where we that are monitoring just to verify that whatever's being pushed is correct. I assume.

Yeah, that's something we are moving to now is support supporting work within the agentic system. Right. So how do we make our data. How do we get. How do we get unique sources of data into our products? Because that's actually what's going to be.

Really helpful is building into the product.

Yeah, okay. Yeah, from a smart bear point of view, that's a good question. It's possible to provide support for those agents when you're running tests and you're fixing bugs. Hey, probably not something I'm as close to. There's a separate set of product group there called Insight Hub and we have bugsnag in there and we use that. And yeah, from an API we've got a bugsnag integration. We had this error in production. Can you help me fix this error in your agent? You give it the error it goes to bugsnack, finds the error, finds a line of code that it's relaxing, and then it tells you, okay, you need to fix this line of code. And so those kinds of cross product journeys within SmartBear, for example, they're going to be really powerful in terms of solving really big issues, especially if you've got observability point of view. Definitely that would be one of the tools and areas that we would be continuing to invest in. Contract tests there and you've got your spec and you've got your other tests all in one place. Now the agent can query all that data from those various systems within SmartBear, piece together the puzzle, and then suggest the next step to help either fix a bug or modify that change to the API. If you go a little bit further into the future, obviously three to five years is like a million. I'll be honest, Joe, I don't have a whole lot of great answers for that on this point at the moment. All right, so in terms of AI, in terms of. But I think, I think we all need to be at the point where we've got really good support for those agentic workflows. That's the first thing. But beyond that, again, going back to the Stack overflow survey, what everyone tells us, more breadth and depth of feature support is number two and ease of use is number one. And so I do think getting back to bread and butter stuff is actually going to be where we win in the future. So we'll need to better support the models. We'll need to be in that ecosystem through MCP or some of the other protocols that exist in there. But ultimately it's actually building really good products that have the features that solve the problems and then humans and agents hopefully will then choose those features that have those products that have the most capability.

Love it. The first part, you didn't mention writing for agents instead of humans almost. It's the same thing with SEO. Now I heard you have to write for be able to have the LLMs be able to understand it rather than a human. It's a hard mix, I would assume, being a product owner and product manager of that for sure.

Yeah, I certainly worry about that in some ways, but it makes sense, at least in the near term. It makes sense to ensure that you're giving your product and your capability.

As you mentioned, we see AI generating more and more code. Could AI generate reliable contracts and mock automatically, which it sounds like maybe. And does that introduce new risk, which I think we touched on a little bit More.

Yeah. So specifically on that question, we in PacFlow will definitely be moving more in that direction for sure. So I think our customers tell us pretty clearly and it's no secret, we know the pros and cons of tools like PACT and the challenges that come with writing PACT tests. We get told by our customers and competitors that writing PACT test is the best chance of being used in those tools. I still think humans are first, but I still use please and thank you when I talk to the agents, just in case. You never know.

I heard that waste tokens and it wastes energy. Awesome. Okay, Matt, before we go, is there one piece of actual advice, you continuous.

Burden Before I address the AI question, What I will say is I was talking to a customer a few weeks back.

You can give to someone to help them with their API contract testing, DevOps efforts and what's the best way to find or contact you or learn more about PACT and contract testing and all the goodness at SmartBeer?

Yeah, I mean the simplest advice for me is just start simple and get started. So if you want to look at contract testing, don't overthink it. Just literally get in there, write your first test, make a few mistakes and then move on. But keep it small and constrained. I always think that's the best way to get started. I'm a very kinesthetic kind of learner. I like to just get my hands on things and figure it out that way. That's how I like to do it. In terms of getting contact with me, I think I've got all the usual things. I'm on LinkedIn. You better find me under my very interesting name of. I think it's digital Matt actually. And then on Twitter you can find me under a less interesting name, which is Matthew Fellows.

And for links of everything of value we covered in this DevOps Toolchain show, head on over to test guild.com p so that's it for this episode of the DevOps Toolchain Show. I'm Joe. My mission is to help you succeed with creating end to end full stack DevOps toolchain awesomeness. As always, test everything and keep the good. Cheers. Hey, thank you for tuning in. It's incredible to connect with close to 400,000 followers across all our platforms and over 40,000 email subscribers throughout the fourth forefront of automation testing.

And we're exploring some different options in terms of how we're looking to look at a pact 2.0 in the future. And this person made the point that they really liked the way the PACT Works today in that you have to write a unit test and you have to slow down to write it, which was interesting. It's not something that's totally surprising, but it's interesting to get that of contrarian view. And their point was like, you have to think through what you're doing to make sure that your code is doing what it's supposed to do in DevOps.

If you haven't yet, join our vibrant community@test guild.com where you and I don't.

Necessarily want to be in a world where those tests are generated automatically for the reasons we become part of our.

Elite circle driving innovation, software testing and automation.

And kind of discussed in that how do I know that it's really stressing the contract, stressing the system?

If you're a tool provider or have a service looking to empower our guild with solutions that elevate skills and tackle real world challenges. We're excited to Collaborate visit test guild.info to explore how we can create transformative experiences together. Let's push the boundaries of what we can achieve with lutes and lyres. The bards began their song but tune of nothing.

It's high quality. Our code is actually doing what it should do. Like, is our API client supposed to do that? Maybe it's not. And so of course you run the risk then of automating tests for a part of your system that doesn't get used. And so now you've just got these wasted tests. So I think that's knowledge, a melody.

Of code through the air it spread like wildfire through the land Guiding tester, showing the secrets too.

That's one angle. But in terms of, you know, where, where does AI go here? Yes, I, I do think we'd be moving, moving more to that, that sort of autonomous testing with the human in the loop design. So you'd imagine in the case of pact, for example, we would produce those consumer contracts that fit to the API client and we would make sure that they would align to best practices, et cetera, and then the contract would be able to be generated from that. And conversely, on the provider side, we would set up the provider side, those tests as well, pull down all the things you need to test and point that against the running version of the provider. Some of those things are tricky and complicated and that's where tools like Copilot and Claude, et cetera, so pick whichever one of your agentic tools that you like can do a really good job of that because they have really good awareness of your code base. They can index things quickly, they can do that. That's something that those tools are really good at. So I think products like us, developer tools like us, will spend more and more time investing in things like mcp, which is something we're doing right now, to not only provide capabilities to do certain things like generate a test or review a test, or provide access to existing APIs so that you can introspect data or introspect contracts and things to help write them and verify them, but also to provide specific prompts that enable those copilots and things to do specific use cases. For example, establish PACT on a new project. That's a very complicated thing to try to automate. If you're not a Frontier LLM, you don't have that kind of capability, you would need to call into those really powerful models. Whereas the agents like Copilot, Claude, Cursor, et cetera, they can do this. And so if we can give them the right prompts with right information and context, they're going to do a much better job of automating some of those. So that's probably where we're headed, at least from the existing point of view. And of course, the challenge we want to make sure is that we don't, we don't have tests that just check the, just sort of fit what they are currently doing. And so effectively you're just testing the bugs. Right, as opposed to it finding the bugs. And that's probably part of the challenge. Look, it's entirely possible that in some future world you don't need something like PACT or any of our alternatives, because you can just statically look at code on both sides or something and go, actually, we know what this API client can do on this side, we know what the API can do and, and we can just tell you, yep, they're going to work. And maybe that's where we eventually get.

Yeah, when you bring up MPC like Playwright, you don't need to know what the clicks and everything is. It just knows. You don't even know. Just, hey, write me an automated test. It may call that server and know what to do. So it's almost like contract testing. And I'll know, like I just know what they're expecting and what I should send.

It sounds like, yeah, definitely. I think the paywright example is a really good one. Even the Cypress, I think that those analogies work pretty well for us because you can't really write those types of tests without access to the code base. Like, you can't do it from. Well, you kind of can do it from a ui, I suppose, but you need to run it in some real environment. It can't just be done through a GUI somewhere else that doesn't have access to your code. And so same with pack testing or contract testing. We definitely want to make sure that we check the real behavior of your API client and not just check the API itself. Right? We want to make sure, okay, if you've got a mobile application and it's calling these different APIs that we actually check the mobile client calls those various APIs to make sure that we've got visibility. And this is something I didn't mention earlier, but I think one of the benefits of contract testing over just purely API focused testing, I would say schema based contract testing where someone's just checking the API matches its specification. One of the benefits of contract testing is you get visibility into what your consumers are using of that API. What the surface area that's being used. In the case of pact, you can get that down to the field level. That's really powerful because now we can know, okay, we've got these 10 consumers, but no one's using this endpoint or no one's using these properties so we can refactor those out later on without damage. That's a really powerful thing, thing to have. So bit of a sidebar there. But coming back to the contract testing AI side of things, we obviously want to make sure we're testing the clients as much as we can and ensuring that we're testing real behavior. And then of course, once we've got the test for real behavior, then we've got the confidence. So I would imagine the kinds of sources I can think of that we would be able to use to generate those because all of those tests happen API to API. So some of our competitors, for example, do. They don't do contract testing, but they do testing through effectively kind of an end to end test by using traffic analysis. So you know, hypertest and Signal Dot are examples of companies that do that kind of other ephemeral environments or mocking using traffic inspection and effectively record and replay, but at scale. EBPF is another promising kind of technology. If you guys come across that, and it's a bit esoteric at the moment, doesn't quite work fully cross platform, but you know, I think that's, I think a lot of really cool things.