What Most People Get Wrong About Simulation with Chris Colosimo

Get ready to discover the most actionable end to end automation advice for some of the smartest testers on the planet. Hey, I'm Joe Calantonio, host of the Test Guild Automation podcast and my goal is to help you succeed with creating automation awesomeness.

Hey, it's Joe and welcome to another episode of the Test Guild Automation Podcast. Today we'll be talking with Chris all about what most people get wrong with simulation. It's a hot topic that I think more people need to know more about and I think there's a lot of misconceptions out there that Chris is going to address today. If you don't know. Chris is a service virtualization and API testing subject expert focused on continuous testing at speed. Currently he's a product manager of service virtualization and microservice testing at Tricentis. Chris primary role is enterprise testing strategies and deployment of continuous feedback across the software testing lifecycle. So he really knows his stuff. He is obsessed with agile testing, acceleration through automation, so he's the right guest for the show. He also works closely with developers and testers to discover constraints, eval processes and facilitate the change management required to adopt DevOps principles throughout the organization. Really knows his stuff. You don't want to miss this episode. Check it out. Hey Chris, welcome to the show.

Hey Joe, glad for you to have me on the show.

Great to have you. Chris, before we get into it, is there anything I missed in your bio that you want the Guild to know more about?

No, that's everything. Yeah, I've been in the service virtualization space both as a product manager, a practitioner, somebody who's been deploying it at customers for the last 12 years. And I've seen a lot and I've learned a lot through the course of that.

Nice. So actually in the pre show we talked about how long you've been in this area, but also how almost popular so popular service virtualization has become. I'm already off topic, but any reasons why you may think that is why service virtualization I've been hearing more and more about for some reason this year?

Yeah, absolutely. I mean there's a multitude of reasons. One thing that should be, you know, not very surprising is that service virtualization is something that comes and stands in when you don't have access to dependencies, don't have access to the real systems. You want to mock something that's evolving and you know, the last two years have been pretty transformative to a lot of the software that's out there. Folks are delivering software in ways that they, they didn't anticipate at all that they would have been, you know, two years prior. And what that calls for is a lot of change of architecture, a lot of change of services, rapidly deploying services so you can get new product delivered into a customer's hands the way that the modern world now needs that software. And it's all happening at the same time. So I think we've seen an explosion of the service virtualization concepts coming into play, because when things happen in parallel and they're not ready, they're not available, they're not stable, I mean, these are the hallmarks of service virtualization. The other thing that I'm seeing a lot is that a lot of organizations took this as an opportunity to sort of retrench and say, are we delivering software with the best possible architecture? Whether that means a transformation of technology to use something a little lighter, a little faster, or if it means let's literally pull off of our on prem and go into cloud and deliver software that way, all of those are catalysts for service virtualization. You know, seeing the explosion that we've seen lately, now, I was just going.

To ask, is it. Because I know cloud native, it's almost a buzzword, but it seems like this is going to change the approach people use for testing also ephemeral environments being turned on and off, kubernetes, all these types of things is probably adding on to the need for the service.

Absolutely. And the development of the code in the first place, we're seeing a lot of that shift into these cloud environments as well.

So because of these changes, like I said, you've been in this area for a while, especially service virtualization. What's happened to service virtualization? Is it morphing or what do you see? How do you see it evolving?

Well, it's really interesting. You know, I've worked on kind of both sides of the fence, both from the software vendor side making the product, and then from the practitioner or the deployer who actually sees that moment where somebody deploys a virtual service into an environment with something that had been notoriously complicated in blocking the testing of an application. And I've seen it from both sides. And, you know, one thing that I've seen change, which has been really interesting, is the Persona. You know, a long time ago, when service virtualization first came about, the concept was there's a bunch of developers sitting around, they're creating code, that code in many cases is calling out to APIs. It was sort of the advent of soa. There was this explosion of let's break everything apart. And so it was sort of a developer activity that was let's create a mock, let's create a mock API. And what kind of became interesting was that that evolved into a testing tool. Now here comes a shocking statement. I don't think service virtualization is a testing tool. I think service virtualization is a testability tool. It is a tool like a vice, a vice grip on your bench, your workbench, right? You don't, when you're building a table, you don't say, and then in step five, let's use the vise, right? You bring the vise in when things get tough and you need to hold that object. And I think service virtualization is sort of the same thing, but it was relegated to this testing tool. And a lot of the solutions that came out were marketed towards how do we create this tool that allows non technical users to simulate APIs. Now let's think about that for a second. That is sort of a mismatch because the non technical tester, yes, they understand when the system is broken, but they might not have the domain expertise. They certainly don't have the ability to crack open the environment, put a proxy in place, record traffic. A lot of those responsibilities are more on the developer, the technical tester side of the house. And so what I think happened was we started to see that the tool put more and more energy into the creation of the services for a non technical frame. And we lost the spirit of what service virtualization was supposed to be, which was a lightweight solution that came in when things went wrong. And so the renaissance and how I'm starting to see more and more people using the solution is they're starting to separate the creation and the deployment aspects of the simulation. And they're saying, okay, the creation is done by those who are right there at the pulse and understanding, you know, what's going on with that. It's a side effect of observability in a lot of cases because it's already there, it's already seeing the traffic we have, things are already instrumented and those services are becoming available before the test environment breaks. And so a phrase that I'm seeing a lot is I'd like that simulation to be available before I need it. And so that's sort of the shift that I've seen is breaking the creation and the consumption aspects of simulation. And I don't really like the term service virtualization. I'm starting to hear more and more API simulation because that speaks to the second Aspect, which is a split in desire. You've got the shift left, early stage feature testing and prototyping, and then you have the failover. And those two activities have been mixed for a really long time. But by breaking those apart and saying, well, the devs and the tech testers are going to use it for creating mock APIs, simulating APIs as they're building them in their framework right there in the code. And then at the same time, those could be leveraged by testers. Down the road, we're starting to see more and more people go, oh, okay, I understand this is a lightweight, punchy solution that we can use right now to get out of trouble, and then it can be rolled into our larger DevOps regression solution cycle. And so all those things are sort of coming together to create this renaissance.

So funny, I was just thinking this, I was having this discussion with another vendor and they were using API simulation. I'm like, no one I know uses API simulation. They really said on this, I'm like, service virtualization is a term everyone knows about. So I understand the reasoning. Why, after you explain that. So you think is API simulation then a better term that when we're talking. Why is that, though? I know you gave some examples. Service virtualization. I think people maybe think of environments rather than, like you said, the API interaction that we're looking for, for testability.

Yeah, I mean, the first thing was just in most conversations that I would have, you'd say service virtualization and they'd say, what, like VMware? Right. Because there's the virtualization means something. And the API can be a lot more than just what people traditionally think of as a web service, ras, JSON. An API could just be Kafka topic that has a schema associated with it. You know, an API could be something that we have homegrown on TCP socket. It's an interface. And so I feel like API simulation is a better term to use first because it's just immediately recognizable. Oh, all those APIs that I have out there, any of the ones that are causing problems, or maybe I'm just creating one on the fly, it could be simulated. And so I feel like it gets closer to the challenges that people are having than having to over explain the service virtualization term.

You also use the term roll into DevOps. How hard is it to integrate API simulation? Or roll it, as you say, into DevOps.

Right. So this is one of the keys of the things I've been thinking about over the last couple of years is that by Breaking apart that creation and consumption rule, you say, okay, the creation, we think about understanding patterns, looking at observed traffic, understanding the use cases, the metadata associated with a code check in that got us to this point, okay, fine, we have our service. Now what? Now I think the golden state that most organizations want to get into is that the simulation is deployed as a post commit trigger as a reaction to some build pipeline. And so in order to do that, the consumption or the deployment of service virtualization has to sort of be rethought. Instead of having these big hardcore dedicated servers where everybody sort of crowds in and you know, you create logical separations, why not much smaller servers? Why not micro instances, docker containers? Take your services, associate them via metadata to the code that you're currently working on and say, you know what, for this API, my blast radius is maybe these 10 microservices and then those services can then be deployed ephemerally to a docker container as a part of the build pipeline. You use it, you get the test results that you need by creating that bubble around the application and then you throw it away. And that's a much more DevOps appropriate approach to using simulation now as opposed to, and I always think this is really funny, creating this highly governed superstructure around what should be a band aid. You know, if you don't have the service, that's because something has happened where that service either is unavailable, unstable, doesn't have the right test data. Do we really want to have a 12 step process to get a virtual representation of that service deployed? No, it should be a reaction. It should be automatically generated programmatically, it should be code, it should be spun up on the fly. We get our value, we tear it down, we throw it away. And so I think that's how service virtualization really gets rolled up into a DevOps process.

I love, love, love this concept. It's a modern approach to what used to be service virtualization. I think for an old person like me, you know, I love this new paradigm almost. So, you know, I talked about how I'm hearing more about it than normal, but I know it's not as based on all the, you know, factors. You just talked about how it can help folks. Why aren't more people talking about it? Or maybe more people are talking about it for your customers? Or are they not? Or is it the terminology maybe that they're not using correctly and that's why it doesn't seem like a lot of people are talking about it.

It's a good question and I think it is the upfront cost. You have this idea of, yes, we have problems with our services. And by the way, a great way to discover that is to watch your slack channels for IT operations. Because all the time somebody's going to throw in there, oh, okta is down. Oh, you know this service is down. Oh, sabre is down. Right. It's a great opportunity to find those problems. But then, you know, what do you do about that? If you're just a normal person or you're testing an application and you're all the way at the front of it, you know the requirements, you don't know all those deep inner workings, it can be a little intimidating to say, okay, service virtualization is supposed to solve this problem, but where do I even get started? And so I feel like putting in place a mechanism where you're always watching. And whether that's through observability or hacking into your APM system or leveraging, you know, the traffic logs that are a part of your contract testing, you have all of this interesting data that's available to you. And if you have a solution that allows you to programmatically generate at least the baseline simulations as a side effect of just your normal work, then the barrier to entry or that intimidation factor goes down. Because instead what you're doing is you're saying, oh, I heard that, you know, PayPal is offline today. Let's go search in my library and see if I have a virtual PayPal. And if the mechanism by which you get that PayPal into the environment that you're testing today is easy, then we'll see more people using the solution. But right now it's a very tedious hand, cranky, you know, those who know know type situation where you have to go and find a team, usually a center of excellence put in a request. And sometimes it can take longer to build and deploy the virtual service than it can to fix the real service in the first place. And that kind of defeats the whole purpose. So what we're seeing with a lot of our users is putting in place a mechanism where, and this is the second time I'm going to say it, the services are ready before you need them and that they're deployed as a part of the process if something goes wrong.

So what do you mean by that? Do you mean it's sniffing the traffic already and then it's creating a library of commonly used interactions before, like you just explained. So someone goes, oh, it's down. Oh, wait a minute, it's been sniffing my traffic. It's been Creating these other, this library almost for me, I'm just making this up. Is that what it's like?

No, no, no, it's. That's pretty close to the workflow that we're seeing with a lot of users. We see two primary mechanism used. One is the contract tests. Everybody should be contract testing. I know this is a topic you've discussed before, but if you really think about a contract test, it's this very lightweight interaction between a producer and a consumer. And within that are a series of rules that you've put in place that says I expect your API to work this way. Well, think about that. Actually the opposite of that is a virtual service. In fact, in a lot of contract testing, you want to deploy a test double anyways so that you can verify against your system the test double, the partner system, that traffic is golden. And a lot of people are just throwing that away. A good solution would allow you to not only build the contract test, because that's your first line of defense to understand something's gone wrong. You know, smoke tests with information about where and why something broke down. And then you can take that either that direct contract test, read it in and say, hey, what are you doing? What are your triggers? What are the things that you're doing? To associate a request with a response, flip it on its head, you have your virtual service. The other one is the thing that you said, sniffing traffic. Quite often what you see in a lot of these microservice architectures is some sort of telemetry, whether it's an open telemetry, some sort of distributed tracing, some aspect of observability. In a lot of cases it's used to help understand the map, the topology of the environment and to understand, especially in like an APM type solution where things have gone wrong. But at the same time you have a one to one relationship between your producer and your consumer that could then be leveraged through an open standard like har, for example, to create a virtual service. And you know what, the contract test at the same time. And so that's the technique that we're quite often seeing people use. And it's a much better conversation than saying, hey, that service is down, let me record some traffic. Right? That's typically what people are doing with service virtualization. Oh, it's down, let me record some traffic so I can create a virtual service. Well, hold on a second. It's down. How can we record traffic if it's down? And also these guys are already putting out a fire. Do you really want to tell them to crack open their environment and put a proxy in place. And I feel like that high barrier to entry kind of erodes the value if instead you just get traffic and if it's not valuable until you throw it away. But if you find that piece of information that you could then later use for a virtual service, this is a much better way to approach the problem.

Love it. So we talked almost different layers of the software development life cycle, different parts of the pipeline. So most my audience I would think are mostly testers, QA type folks. So I know they're going to be thinking is an API mocking more of a developer activity or a lot of this stuff more of like an SRE type activity when we get into production. Why should testing and QA be involved in API simulation?

Absolutely. So I there's a spectrum of what comes in, at what time. And for me, most organizations, developers are building their API, they build the service spec, they have a conversation about its functionality, and at that time it's a really good time for a developer to use a local API mocking solution, not necessarily for testing, but to create the circumstance where they can then build their API to call something to verify that it's doing what they want. And at the same time, this is a perfect application for API simulation because then they can start putting different payloads coming to them. They can verify what happens if their service behaves in a certain way. They can start building those early stage contract tests. And what they're doing without knowing it for test is they're building the baseline service because the logic is the hard part. How do I know which element in the request ties to some data value that then triggers my response? The developer is going to put a lot of that in place. The second thing that's going to happen at that point is it's going to be associated to something, there's going to be metadata and what the developer is working on. Oh, today I'm working on, you know, an integration with this service service A. And so when I check this code for this for the virtual service in, I'm going to associate it with a as a part of the git comment or some tagging. Now how that benefits test is what test wants is test wants an association between an environment that they're testing, a specific requirement to and all of the associated services that could be required for that environment. And by making that initial commit with metadata, test can then say, okay, I have a problem. Today the baseline logic is in place and there are tags all over the place that I can associate as a part of my build and as a part of my run. And when I go to test this environment, if it goes down, oh, I see over here that involves service A. So I'm going to deploy that. They don't know necessarily how it was created. They don't necessarily care. They just know that there's an association with the environment and the test that they're trying to execute. And that metadata is the magic part.

So it almost sounds like it's always a concern for testers when it overlaps with development, when they take over. So it sounds like they check it in and because of these tags and almost they get an alert or they know I'll just focus in on these tags that were just checked in based on the metadata. And that's how I know it's time for me to really stop more of my testing activity. Is that correct?

Yeah. The tags help them understand where it is in the environment. It's not just a name, it's not just PayPal. It's a service that's associated with this environment, and this is the current version of it. And when the code was checked in for this environment, this service was a part of that. And so we can make that connection. Now, I'm not sitting here claiming that's going to be super easy. It takes a little bit of discipline, but it's much better than oh no, there's a fire. Let me guess which service I need to deploy and where.

Absolutely. So you also have microservice testing expert. This just came to my mind. I would think with API simulation, it makes it even more complex because there's other dependencies probably that you need to test that you may not be aware of, because the microservice can be consumed by multiple other microservices. And then you get lost in that chain. Does API simulation help you with this? Like the sniffing of the traffic or the production data? The observability that you could then bring into your testing and say, okay, I know typically this particular microservice interacts with these other microservices that you may not even known about if you weren't, like you said, be more proactive in the production piece of it.

Yeah, the actual environment discovery I like to see handled by an organization like an APM or a tool whose job it is to instrument each one of these applications, or as a part of the Kubernetes process, or even bringing up those containers, registering back to the mothership and saying, I'm here and I'm connected in this way. But when things go wrong, it is really hard for a tester to understand where in a sequence something went wrong. And this is why I say that the simulation and the contract testing are very similar to each other. And by having those little snippets of data between the producer and consumer you can use. You basically think about your smoke tests as just a series of connection producer consumer tests. This is a packed concept right here and you're basically saying, okay, I've deployed this thing, let's run all the contract tests. Pass, pass, pass, pass, fail. Okay, now I have a better idea of the dependencies that this service had. What changed and now which endpoint is not behaving the way that I want. Now, this is far before you're doing any of the end to end or regressive testing. This is the very beginning. Just to understand the blast radius of a code, check in. And at that point you either take that information and say, hey, this endpoint's offline because maybe it's valid, maybe you found a result they don't want to immediately just stuff in simulation. Now you're going to mask the problem. But if it turns out that it is an automation failure due to an environmental constraint, you know which one and what to deploy. And so I think it's using contract testing and simulation together in concert, but getting the information about what needs a contract from your APM solution.

Great advice. Great. So, you know, I forgot the wording you used, but you said you were working on like the next generation, the next approach to service virtualization, something like that. So it just triggered something. You know, we're heading into 2023, this might be even released in 2023, I don't know yet. So where do you see us heading or where are we going with this, all this?

Well, I've taken all these thoughts. Yeah. And I've rolled them into the next evolution of Tricentis technology called Tricentis Test Automation. It's going to be our SaaS offering that will incorporate a lot of the concepts Tricentis is well known for and it will have some special capability in there for API simulation and contract testing. It's in its early phases, so we're looking for a lot of beta feedback right now to really learn, you know, what folks are using, what standards are they using, what would they expect to happen after a code commit where they got a contract test failure and all sorts of those things. It's actually in beta right now, so folks can try it out if they want. And yeah, so I'M trying to kind of pack it all into that and learn as we go. It's an exciting time.

Yeah, for sure. I spoke to a few folks about on Tricentis about this new SaaS offering a few weeks ago and it seems like they were focusing more on almost the UI piece. But I didn't know you actually building a now simulation API piece. So it seems like it's going to be a pretty complete solution.

Yep, there's a lot to it.

Is it gonna be a workflow built in that you can call like, are you part of that to say, hey, let's not forget about the API piece when you're thinking of the ui?

Absolutely, yeah. There's. Within the solution there's a concept called a playlist. And as a part of the playlist you'll be able to say, hey, what if, if this, then that, you know, if this test fails, then let's deploy some simulation or if vice versa, if contract test maybe I don't run this test. And there's all sorts of associations that can be made because we're sort of bringing all the tech together to say what do we see and what can we do about this? You know, and if folks are interested, they can roll into the beta by going to tricentis.comttabata and taking a look at it.

Absolutely. And I'll also have a link for that in the show notes for the folks as well because they definitely should check it out. Like I said, it seems like I've been hearing more and more about it and so curious to see people's thoughts on it. So definitely getting the beta. So the beta would be free. Is it a paid offering right now or the beta?

Nope, free. Come in, give us some feedback.

Nice.

And it's rolling now.

Love it. Like I said, definitely. Everyone listening should check that out. I'll have a link to it once again in the show notes. Just something else came to my mind. You didn't mention something about testability, which I really enjoyed. Another thing I've been hearing about is robustness or, you know, chaos engineering, but not necessarily chaos engineering, more like resilience type testing. And it sounds like your vision almost bakes into this as well. API simulation. Is there any type of like chaos or resilience type testing baked into this new Tricentis product? Or is that just part of the API testing that you think people are simulation people could take advantage of as well?

Well, I would love to tell you that simulation could be a chaos testing tool and I have seen it used that way where you. In fact, I was just working with somebody the other day where they were building an API sandbox so that their developers could access the APIs without having to use real test data. And for fun, they baked in a one out of every thousand calls to this particular service, which was a common service and a lot of people used it. It just would randomly respond with a 404. And it was really interesting because. Or, excuse me, it was a 500. And it was. Because a 500 is actually harder than a 404. Right. A 500 means that the servers there. But you had an error. A 404 is. You unplugged it. And they just wanted to see. But the big difference for me between API simulation and chaos testing is that chaos testing has auditing, it has traceability. It kind of tells you what happened and when. And yes, you could do that with your API simulation, but I feel like it's a little overloaded. But if you want to try it, like these folks tried it, simulation's a great way to throw a little chaos into the environments.

Love it. And I like how you say it's fun because it is fun. That's what I love about this industry. What we do is really fun. But anyway. All right, Chris, before we go, is there one piece of actual advice you can give to someone to help them with their automated API simulation efforts? And what's the best way to find or contact you or once again, learn more about the new Tricentis test automation product?

Yeah, I just. For me, the actionable piece would be to take a look at any number of the free simulation techs that are out there. Simulation has so many uses, and I think that I would like to see more people, you know, pick up one of the free tools, no matter what it is, maybe try it in some of their development, some of their testing activity. Just see if it fits, see if it feels good. I think you will be pleasantly surprised. And then as far as action for us, the TTA Beta obviously will have that simulation component, which will have some free aspects to it. And then, as always, they can reach out to us at Tricentis to get more information on all of this.

Thanks again for your automation awesomeness. The links to everything of value we covered in this episode, head over to test guild.com forward/A 4, 2, 8. And if the show has helped you in any way, why not rate it and review it in itunes? Reviews really help in the rankings of the show, and I read each and every one of them. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack automation awesomeness. As always, test everything and keep the good. Cheers. Hey, thanks again for listening. If you're not already part of our awesome community of 27,000 of the smartest testers, DevOps and automation professionals in the world, we'd love to have you join the fam@testguild.com and if you're in the DevOps automation software testing space, or you're a test tool provider and want to offer real world value that can improve the skills or solve a problem for the Guild community, I'd love to hear from you. Head ON over to testguild.info and let's make it happen.