Service Virtualization Microservices Testing with Brandon Byars

Five, four, three, two, one. Drop that. Welcome to the Test Guild Automation Podcast. We're all getting together to learn more about software testing and automation with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Automation Podcast. Today we'll be talking with Brandon Byers, all about his new book Testing Microservices with Mountbank. Brandon is the technologist and principal consultant at ThoughtWorks and chief maintainer of the open source virtualization tool Mountibank. He has a wide range of experience in multiple technologies and business domains with interest in applying system thinking to software development. Brandon has a particular interest in exploring the social and organizational aspects that allow software developers to scale to large teams. You don't want to miss this episode. Check it out.

The Test Guild Automation Podcast is sponsored by the fantastic folks at Sauce Labs. Their cloud based test platform helps ensure your favorite mobile apps and websites work flawlessly on every browser, operating system and device. Get a free trial. Visit testguild.com sourcelabs and click on the exclusive sponsor section to try it for free for 14 days. Check it out.

Hey Brandon, welcome to the Guild. Thank you.

Go. Glad to be here.

Awesome. So really excited to have you. This has been pretty a hot topic for a while, but it seems to be picking up steam again for some reason. Microservices. But before we get into it, is there anything I missed in your bio that you want to tell us a little bit more about yourself?

No. That sounds about right. I've been at Thoughtworks for 12 years and I've been maintaining Mountie bank for about six.

First question I always ask authors, since writing a book is tedious and very time consuming, is why? Why would you do this to yourself? So why did you feel a need to write a book on testing microservices and specifically with the mountebank?

Well, part of it was just ignorance. I think there were plenty of people who tried to warn me about how tedious writing a book was, but I chose not to believe them until it's fun to have written a book. I wouldn't say it's fun writing a book, but it was good. So the publisher Manning actually reached out to me. It's obviously great when an open source product that you've built has taken off enough where there are people reaching out to you for that kind of activity. So as you know, as big of a project as it was, I was happy to have gone through it.

The book is broken out into three sections. You have the first steps. Second is using Mountbank and the second one is closing the loop. So I just thought we'd follow that format. So, first steps, folks that don't know. I don't know why they wouldn't, but say they don't. What are microservices and why do we want to test them?

Yeah, and it starts off in the first chapter with a little bit of an overview. I assume that most people nowadays have at least come across microservices, but I think it's always good to get a little bit of a perspective of why you're doing microservices and the book. I argue that really it's a scaling mechanism. If you look at some of the history of organizations like Netflix and Amazon, once they got to a point where they had monolithic software system, but they had so many engineers that they were stepping on each other's toes trying to make changes to it, those organizations found that they were able to bring more autonomy to teams by separating out the services with a bit more of a fine or granularity than what they had previous. So while there are arguments to be made for microservices for different scaling mechanisms, from a runtime performance standpoint, I think that the primary argument is from an organizational standpoint. It allows you to add engineers and still maintain a good velocity across your organization.

That has some consequences for how we do things that aren't just writing the code. And a large part of the book was on obviously testing, because I think I've seen anti patterns where we end up with the worst of both worlds. A distributed monolith, effectively where you've broken out a monolithic application into multiple separate deployable binaries. But then because of coupling at the testing layer or the release management layer, you still end up releasing them all together. And so.

A large part of the argument for the book is that if you really want to move fast, you need independent releases across those binaries and there's testing tools and processes that help you achieve that.

Nice. So for testing, are there any misconceptions on how one goes about testing microservices or anti patterns you've seen around this as well?

Yeah. So there's a couple that come to mind. One is treating microservices and some of the APIs as implementation details of some top level project. Maybe you're building a mobile project and so all the testing comes together as part of the mobile application itself. And while you certainly need some level of that type of testing that destroys your.

Treating each microservice as a product in its own right, each API and really thinking about the user experience. But the users of services are obviously developers, so it's really developer experience and you lose some of the benefits in terms of reusability with that approach as well. And the other, and it's not mutually exclusive, is what I just talked about where you, because there are service dependencies on each other, oftentimes when you're writing synchronous services, sometimes you can get some coupling between them. You organizationally tend to fall back into the habit of testing them and releasing them as a batch rather than treating them as independently deployable components of your ecosystem. So I think there's more than testing that goes into that. You need to think about backwards compatibility patterns, but there are key testing practices that allow you to make sure that you can release each of those services on its own timeframe.

Like you mentioned, you do go into certain, like the problems with end to end testing. A lot of times, you know, this doesn't have as much as it did back in the day. But you talk to a developer or a tester and they just say, well, because the front end is consuming the microservice on the back end, if I test from the front end, then I have everything covered, so why do I even care about not testing directly against the microservice itself?

Yeah.

I'd be interested in your experience, but I've seen in a couple of cases multimillion dollar automation test suites effectively abandoned because they were doing exactly that testing at the front end and just became so cumbersome to maintain and the signal to noise ratio was so low. You know, obviously those tests are hard to write when they're working making assumptions about the entire end to end ecosystem. So they could fail for environmental reasons or for application reasons, but they're also not very good at pinpointing where the error is and generating a lot of maybe false negatives as well.

You know, when you, you just get a test at the application level and then it's a murder mystery to navigate inside the different microservices to figure out if one of them is broken or if it was the application or just the infrastructure that you're working on 100%.

And that's the beauty of microservices is like you said, you create a test specific for that microservice and follow it all the way through the development life cycle. So when it is released, if it does fail, that tests almost one to one mapping, I guess.

Right. And again, you know, I emphasize this in the book. It's not the point of it but testing is part of a broader quality strategy that goes all the way into the production runtime. So a lot of observability practices intersect quite heavily with testing. That's not the point of mountibank. But I think it's important to make sure that you, you don't walk in thinking that one strategy is the solution to solve some of the architectural challenges that microservices bring with it.

All right, so this brings us to Mountibank and first off, ThoughtWorks. I don't know, is it part of your job description that everyone that works at thoughtworks has to come up with a really cool open source technology or tool? Because it seems like a lot of things come out of ThoughtWorks. So what was the genesis of MounteBank and why did it happen to occur when, when you're working at thoughtworks?

Yeah, well there is certainly some truth to that. That's actually why I joined ThoughtWorks. I was using a lot of open source tools that came out of the organization before I was an employee. Things like Selenium and water and Endunit in the early days. And there's an ex colleague of mine, Pete Hodgson, who made a joke one time that you never really get your badge as a ThoughtWorks developer until you've created your own mocking framework. And so I was working at an airline company who was looking to retool their mobile application. And this is a multi billion dollar business.

The website used a services layer that was not HTTP. It was binary serialization of Java logic that was very coupled to the web layer. And we were trying as we were building out APIs for the mobile application to not to boil the ocean. So we had to reuse some of that service layer. But it made testing very difficult because we were basically exporting Java objects over the wire. And we went through all sorts of crazy gyrations to try to figure out how to test it, including using some Java tool that actually exported Java logic to the remote application to see if we could inject something into the the response from the service layer. Everything was too hard.

We were well aware of HTTP stubbing tools at the time, but none of us were aware of anything that worked at a different protocol. And so over Christmas that year I remember having bottles of wine and watching the Daily show with Jon Stewart and just coding with this crazy idea to solve that specific problem and brought it in that January and tried it out and it worked and it just sort of took off from there.

So when would someone use MOUNTI bank then if they're Testing microservices, when do you think they'd have an aha moment to say if I could only have a tool that did this?

Yeah, I think that the moment is when you want black box testing of your service with determinism. So it's not for what I would consider, you know, sort of classical unit test. I think classical unit tests are in process pre deployment tests. And so you would write that with, you know, junit at the object level. It's not end to end integration testing because then you're wired into the real services in your ecosystem. But what it is is if you are a team that's owning microservice or an application that depends on downstream integration points and you want to test your service or your application over the wire like a user would black box and you want to automate that. But you need to stand up some deterministic services within your environment so that you're not dealing with random test failures. So if you depend on a pricing service, you would deploy mountibank and it could stand in for a real pricing service, you know, it could send back very deterministic responses for you. You're making assumptions that it's behaving like the real pricing service, that you need to account for those assumptions in your overall testing strategy. But then you could wire that into your continuous integration environment and the tests would never fail due to the environment. If they failed, it's because of either the assumptions are wrong or because your application has a bug.

So I guess a lot of times people always afraid of doing this because.

It'S like anything, they always say it's not the real system. But there are obviously issues where you're not able to test everything like you would in production. Are there any anti patterns someone could do using this tool that could hide a real issue or cause issues later on?

Yeah, so there's a couple I think. One is, as I just mentioned, you're making assumptions about the real systems if you never account for those assumptions downstream in your test strategy. So in the book I give some light coverage to this notion of contract testing which says.

Behaviorally testing a dependency that is not in my purview. If I'm a team that owns an application or service that has a dependency, the team that owns that dependency should behaviorally test it. But I need to validate that it is returning responses shaped in the way that my application expects them to be shaped. So while I probably don't want to have every single edge case tested through a real service because the costs of the environment to set that up. And the number of false positives you get in your tests or false negatives you get in your test starts to skyrocket when you introduce real environmental dependencies. I still need to validate those assumptions in a lightweight way. So there are techniques to do that. The other approach, which a number of organizations have used. So I'm a little bit nervous of calling this an anti pattern, but it's something that I'm still skeptical of, which is to say that one team owns all of the virtual services for all other teams or that the producing team owns the virtual service. And the reason I'm skeptical of that approach is because when I am writing tests for my application and it has a dependency that I want to virtualize for test determinism reasons, the only group inside the entire organization who knows the test cases that I want to set up and the data requirements of those test cases is my team that owns the application. And so I can set up the test data very specific to what my application needs to give me the confidence that it's behaving correctly. When other groups who don't understand my test cases try to do that, they make all kinds of assumptions about the test data that's needed. And what generally happens is you get quite a bit of complexity in the test data that actually reduces or it adds friction to my ability to write tests if some other group is running the virtual services. So those two patterns are ones that I think are important to watch out for.

Nice. I'm not sure if this is a bad practice, but do you think this is a good use of this as well as you have multiple Sprint teams and you have Sprint Team A that's working on a feature, but it depends on Sprint Team B completing their feature, which isn't going to be done. So they mock it out somehow using a tool like this to get them to be able to test earlier and just to prove out their work. Is that a common use of something like this?

Yeah, absolutely, with or without this type of tool. So a lot of API gateways, for example, will want you to author your specification in their tool and they'll create just a simple stub service of that. For the same reason, I think a much more robust solution is to do exactly what you proposed, which is to stand up a virtual service as a way of getting ahead of a dependency so you're not, you don't have to work as sequentially as a naive approach would indicate. And I think that's a very common use case when you're Building again a mobile application that has five service dependencies, but they're working on different time frames. Sometimes you just need a service and you can work out what you expect the service to return from a contract standpoint with the providing team ahead of development and then use that so that you're not completely dependent on them. There's other scenarios as well, like performance testing and some others, but there are, you know, I think the idea of decoupling your.

Development life cycle from the teams that you depend on is, you know, a key need in a lot of circles. And service virtualization tools in mountibank fulfill that need.

How hard is it to stand up something like mountebank? Do you have like an example of how some would go about doing it? How difficult is it?

Sure, it's meant to be pretty easy to stand up. So there are across the ecosystem of service virtualization tools, sort of loosely speaking, there's a couple of categories, one that are very language specific. So maybe you're in the Java ecosystem and there's tools that are all about written for Java users and there's tools like Mounting bank which are meant to be platform agnostic. So it's a REST API. So there's a command line tool, you spin it up, it exposes a REST API and you can.

Either use the API or a config file that's loaded from the command line to give it the test data that you expect.

The first use case and the one that I had in my mind when I wrote Maunibank was every single test would set up the test data specific for that test. So for example, the test day would be, hey, when I call this dependency on this port with.

Prices, I get this JSON response back in the HTTP body.

But that may change based on, you know, various test scenarios because sometimes I want to, you know, test, you know, prices when there's a lot of spike. So they're going to have a jump, I think, you know, you know, when Uber charges you more money during busy traffic. And that's a test scenario that will be specific to one automated test case that, you know, my application I'm building depends on, but, but another test may be it's low volume and so I want it to just average pricing. And so I'll have a separate set of test data for that test case. And so every test case we call the API, set the test data, execute the test itself in a black box fashion against your application, and then shut down the virtual service that was running. So you get a clean slate for the next test, and that is you. That pattern is used in a number of cases, but what I find is a lot of users also like to set up all the test data up front. And so in that case you probably just set up a configuration file with it all, you load it on the command line and then you're never.

Stopping any of the virtual services that your tests are on. You just rely on test data that was pre created.

So it's a good point. I didn't realize it was agnostic so you could use any programming language. You'll be able to interface with Monte bank because it is a REST API.

That's right. And there's, you know, if I had more bandwidth, one of the things I would love to do would be to write SDKs for the common languages out there. Since I don't have that bandwidth, this is where I rely on the community quite a bit. And if you search for Mount Debank on GitHub, you'll find pages and pages of language SDKs or build plugins. Many of them I try to list on the website itself. But there's definitely a number of organizations who've created their own client specific SDK.

I don't know if this is a major issue or have you seen this? In my experience, when people are testing APIs, it always seems to be they're using positive data or happy path. Using a tool like mountibank, can you test things or would it be appropriate to test things that maybe aren't formatted correctly? So rather than saying you expect something in a certain format, use it almost to help you with negative testing.

Yeah, and that was one of the inspirations for the tool to begin with. I mentioned the airline project that I was working on.

A negative test scenario was a canceled flight where you were trying to maybe check flight status or whatever we were building. And it turned out that through the services layer, a canceled flight put knolls where, you know, it really shouldn't have. It was probably a bug, but the website knew how to deal with it. And so everything worked fine end to end. We had bugs in production because we weren't testing that. And to test it, really the only way that we could test it was to go not just to another development team, but to an entirely different department in the IT organization who managed operations and asked them to add some data to the test version of the system that we manage flight cancellations and then for that to flow all the way through to our application. And so we had scenarios like that where it was just so much Friction that of course we had some bugs because of it. So mounting bank was a way of saying, hey, it's now trivial. Once we understand the shape of data under some of these edge situations, you have to go through it once to understand that. Then we can automate it. And we don't need to depend on either team to set up the test data because we can do it ourselves. So whether it's malformatted data or data that returns very slowly because there's ways of telling mounted bank to return slowly or just error conditions, you don't need to rely on some other system to generate those. Once you understand what they look like, you can create them yourself.

So another kick that I've been on recently is security testing and security. And I noticed there is something in here with a word of caution. Security matters.

I haven't read this piece yet of the book. Is that something that you go into details about security testing or things to think about security wise when testing or developing for microservices?

That specifically is around securing mountibank itself. One of the challenges with standing up a virtualization server is that your treating it, you know, as a lightweight test asset. Other people, you know, malicious users.

May treat it as a remote execution engine. And so there's just some techniques that you know, some good practices saying, hey, when you stand it up, you probably want to, you know, do these things to make sure that you're not setting yourself up for a vulnerability. It is possible, for example, for mounting bank to run where it accepts it has what I hope are.

Behaviors that let you do most things that you need to do out of the box where it can't do what you want to do out of the box. It allows some programmatic injection of capability, but that is opening up to remote execution attack. And so it doesn't run that by default you have to on the command line you have to say, I want to allow that remote execution. But then when you do that, you need to make sure that you are thoughtful about the security implications of it.

All right, so the last part of the book is closing the loop. And I think it touches on two very important topics. The first one is continuous delivery. Just tell us a little bit more about Mountie banking and continuous delivery.

This is where deterministic tests matter. If you're going to create a continuous delivery pipeline for your application or your service, and it has dependencies on other runtime services.

You need to be able to create a very lightweight environment where you could run automated tests against your application.

That are not End to end, but that deterministically return responses that those dependencies would return. Multibank is tailor made for a continuous delivery pipeline because it's a very low friction way, easy to automate way of creating that lightweight environment with the test data specific to your test cases in a way that doesn't rely on any other group inside your organization to set up for it. You can do it yourself. That is kind of what a functional or service or component level testing.

You'll have a stage in your pipeline presumably around that. Once you've unit tested and packaged and deployed your application, there may be downstream uses for it as well. It is possible to use MoneyBank for example, in performance testing.

And whether you put that into your as a gate in your continuous delivery pipeline or as sort of an adjunct that happens in parallel. I've seen both strategies, but it's a common use case as well.

And that is the last piece which I love. Performance testing. So can you expand on a little bit more how or why you would use Mount Ebank for performance testing?

Yeah, so it is unfortunately common just because of economics that while a production end to end environment and a number of organizations will scale to meet the load because the business depends on it, getting that same scale outside of production is difficult and stability in the environment is difficult. So one strategy that people can use for performance testing is to set up. You usually want to test your application or your service, assuming that the environment behaves a certain way. And so Maui bank can make those assumptions for you and you can include in those assumptions how long it takes downstream services to respond, maybe do some log analysis and you say this usually takes 700 milliseconds on average. I can add some fluctuation if I want and you can code that into mounted bank so it will on average return 700 milliseconds from virtual dependency it's providing for you. And that way you can test your application again deterministically without having to rely on a production like infrastructure for performance or load or stress testing. In some cases you might have to run multiple instances of mountibank if it itself becomes the performance bottleneck in your performance testing rout routine. And actually what I'm what I'm working on right now is much more robust mechanism for how to do that. Everything traditionally Moni bank has been in memory, which works very well for most use cases. But if you're looking for extreme scale needs to lay out some of the configuration on disk. And so I'm just wrapping that up right now for improving the performance testing capability of it, but it's an easy way to again to control your environment with appropriate assumptions about what production looks like and test your application isolation to see what the performance characteristics of your single application are okay Brandon, before we.

Go, is there one piece of actionable advice you can give someone to help them with their microservices testing efforts and what's the best way to find contact you or get our hands on your book? Testing Microservices with Multibank yeah, so the.

One bit of advice I give people for testing microservices is the only scaling benefits you get is if you can force release independence of your service from everything else in the organization. Testing is a critical component of that. You need to be able to test your service independently of anything else so that you can release it independently of anything else. And service virtualization is a great way of doing that. Money bank is a great service virtualization tool. I definitely recommend you test it out. The website is mbtest.org you can go there. There's a Google link to Google Groups where I monitor it pretty regularly for support type questions. You can reach out to me there. You can reach out to me through the GitHub page as well. I try to manage the issues through there and obviously welcome an expanding community and would love to hear your feedback.

Thank you Brandon for your microservices testing awesomeness. For links to everything of value we covered in this episode, head on over to testguild.com a293 and while you're there, make sure to click on the Try it for Free Today link under the Exclusive Sponsor section to learn all about Source Labs awesome products and services. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe and my mission is to help you succeed with creating end to end full Stack Automation awesomeness. As always, test everything and keep the good. Cheers. Thanks for listening to the Test Guild Automation Podcast. Head on over to test guild.com for full show notes, amazing blog articles and online testing conferences. Make sure to subscribe to join the Guild and continue your testing journey. This has been a Joe Calantonio production.