Modern Integration Testing with Kevin Wittek

Welcome to the Test Guild Automation Podcast where we all get together to learn more about automation and software testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Automation Podcast. And today we'll be talking with Kevin all about modern integration, testing using test containers, and probably other topics around testing and development. Kevin is one of the test containers co maintainers. He's really passionate about floss containers in Java Oracle. Groundbreaking ambassador, so he really knows his stuff. He likes to write Python programs to support his wife in performing behavioral science experiments with pigeons, which I'm sure I have to touch on. He also plays the electric guitar and is a musician in his second life. After many years of working in the industry as a software engineer and a short detour into the world of academic blockchain research, Kevin is now working at atomicjaw trying to make developers more happy and productive one container at a time with some what I think is some really cool technology that everyone should be using. Hopefully after this interview you'd think the same. So you don't want to miss this. Check it out.

The Test Guild Automation Podcast is sponsored by the fantastic folks at Source Labs, the cloud based test platform that helps ensure you can develop with confidence at every step and from code to deployment for Every framework, browser, OS, mobile device and API. Get a free trial. Visit testguild.comSourceLabs and click on the exclusive sponsor section to try it for free today. Check it out.

Hey Kevin, welcome to the Guild.

Yeah. Hi Joe. Great to be here.

Awesome to have you. I was really excited you actually spoke at Automation Guild this year. I got a lot of good positive feedback from it. So that'd be great to have you on the show just to talk a little bit maybe about your session and probably other things. But before we get into it, is there anything I missed in your bio that you want the Guild to know more about?

Oh, no, I would not know what to answer.

Cool. All right. So Kevin, I guess the first thing is I have to touch on this. You talked about writing Python programs to help your wife with behavioral science experiments with pigeons. I guess other people want to know more. Like what kind of experiments are these? What kind of programs?

Yeah, it's actually pretty cool. So my wife did PhD in neuroscience and behavioral psychology, and during her PhD, but also in general, we sometimes collaborate on some of her research and experiments. So it's like pigeon, the bird, the pigeon. She's looking into cognition and evidence accumulation processes in pigeons. So how do pigeons and birds in general make decisions and she does cool experiments. For example, how pigeons react to mirror, for example. So this is. Mirror experiments are a classic experiment to see if a animal or any kind of creature has a perception of oneself. And only higher intelligence animals understand the concept of a mirror, that it reflects themselves. And in those experiments we also with the pigeon found out that. Yeah, so pigeon when confronted with a mirror reacts differently than when confronted with a con specific. So this is kind of suggestion that they see something in a mirror which is maybe not self recognition, but something similar. Okay. And where the Python stuff comes in. So very often we have to analyze the videos and classically in behavioral sciences you just do a video recording and then the poor students have to analyze everything by hand. And when I saw this, so it was actually when we started dating, I saw her like just counting packs in videos and doing many stuff by hand and I got interested by this and so like okay, there are things that we should be able to do automatically with some cute computer vision stuff and so on. So yeah, that's where I got in. And sometimes doing like tracking like the pigeon goes through a labyrinth and we track the pigeon movement or recently there is a really awesome open source Library, it's called DeepLab Cut. It uses deep learning based pose estimation to then track animals in videos. And it works amazingly well. But then still you have to analyze the data afterwards. And yeah, Python is a language of data science and so on. So this works well.

Nice, because I think you're a Java.

Expert, so I'm a Java developer. Yes, yes. But Python is also pretty cool for some stuff.

Okay, I was curious to know that. So do you ever use test containers with the Python program?

So we have test containers Python also, which is like a fork, it's not as feature rich as the Java variant, but for many of the basic use cases it's also work. So I haven't used it in this scientific context, only for trying out things myself, like when interacting with database and so on just to see. Okay, can we also do it with test containers Python. But in case of this scientific programming, it's like a little bit different than normal software development. Like you have this jupyter notebooks or whatever, it's this very explorative interactive thing. And while I acknowledge that it would benefit a lot from more testing, much, much more testing, it's often not really done because you are integrating against libraries on a high abstraction level and so on. And you're kind of just assuming you do everything right and hope that the results are correct. In the end.

Nice. I guess, before we actually go any deeper, I know I did a podcast earlier with the on test containers, but people that may have missed that. In a nutshell, how would you explain what is test containers?

So it is a library to. From within your test. From within your test code instrument and create your test environment following an infrastructure as a code approach. And really the specific thing is that this infrastructure code resides directly next to your tests most of the time in the language in which you're writing your test. Like when I'm having Java Test in Junit, I will create those environments using test containers, also using Java code as part of my Junit test. So, like, executing the tests and spinning up the test environment becomes kind of a really atomic unit. And that's very cool and very useful. And on a technical level, how do we do it with test containers? So we start Docker containers. Of course, you could start Docker containers before. You could use Bash scripts or whatever. You could use Docker Compose. But the difference then is that you have had to do it in an outside process most of the time, like a process before you run your tests that would spin up your environment and then the test would run against it and would need to do certain assumptions or whatever, and then afterwards you need to clean it up. But having this possibility really in the code in which you write your tests just gives you much more flexibility. Make some patterns very robust. Like, we do a good job in cleaning up afterwards, we do a very good job in detecting your Docker environment. We do much more advanced things like waiting for the application within the container to be ready. You don't have this if you just start a container with Docker, there is no knowledge of the database running in the container. But test containers can have this knowledge and can be extended to have more knowledge about the applications in the containers using Java and so on.

So almost sounds like infrastructure as code type of approach. Been around for a little bit, but I don't see many developers necessarily embracing it. So you being a hardcore developer, if someone's listening, what benefits? Obviously, I know a lot of benefits, but like, a lot of people are like, oh, it just seems like a lot of overhead for me to write a few unit tests. Would you say to someone like that?

Yeah, so now that you mentioned unit tests, that of course makes it already, like, a bit hard to define. Okay, how do we define unit tests? So we often call them integration tests, but some people call them actually unit tests. And especially in the developer community, it's completely fuzzy about the Definitions those different layers. And I recently read an article from Kent Beck and Eric Gamma. They wrote in the 90s after they did the Junit framework. And they said through this Junit framework that allowed the developers to write the tests as part of the Java code and make test driven development much more accessible. It became in general more accessible to do testing for developers and in my opinion with test containers with a way to very easily spin up like your integration testing dependencies, like your database. Now it suddenly becomes much more easy for you as a developer to write integration tests and this can change the way you think about composing your test suite. So before you would maybe do a lot of mocking, you don't want to do integration test against the database because it was, I don't know, messy to set it up or it was flaky because you are sharing the database with someone, whatever. And now you just basically instantiate a Java object that is your abstraction. This will then spin up the container with the database and there you go. There you can run integration tests against it, like test real SQLs. And this just changes the way you test and develop. And it's not a lot of work. So sometimes it takes a little bit more time to spin up stuff like in the area of seconds. So you have to be mindful of this when you want to do it and what not. So it's still benefiting to decompose your test. Like if you have pure functions, okay, test them as a pure function. Don't do a big integration test. But if you are testing your integration with another technology, like with the database or with Kafka or whatever, and you want to use the full power of this other technology, I would suggest then, okay, embrace integration testing instead of building additional abstractions that then does not allow you to use the full power of the service in your application anymore.

Yes, I definitely agree. A lot of people have different definitions. What is unit testing? What is integration testing? What is functional testing? Almost. So a lot of times people say I'll just market a stub it like I don't care about the service. Like this seems like it's going to add an extra overhead. So are there different points in your pipeline where, okay, maybe I'll do some quick mocking stubbing, but then for the other part of the pipeline, I want to do this other test. I definitely need to make sure I'm integrating with a real service or something.

So you can do this. Like it depends very much on the application and on your needs there and what you're using. Like for example With Gradle you can have different phases in the tests. Like since the last version they have like a built in support in standard Gradle for this. With Maven we always had it. We had a surefire phase and then the fail safe phase. Or of course in CI you can have different stages in your pipeline. I personally I just have the tests in this case my integration test using test containers, I have them in the same project basically next to the unit test, like the pure unit test that would not interact with external processes. And if I'm developing something that will basically use this component against which I'm integrating, then I'm constantly running this test as part of my development lifecycle. So I'm a little bit of a TDD guy. Not 100%, like I'm not doing TDD by the book, but I'm doing kind of a TDD style development. And with test containers I can do a TDD style development with integration tests. And Ken Beck also never said in his definitions of test driven development that it has to be purely unit tests. So he himself admitted that a lot of parts of junit which were developed TDD style were developed using functional tests. And okay, functional test is again like a step bigger or whatever. So doing TDD doesn't mean I also can't do integration tests at the same time. And for me it's a great development tool. And yeah, so we have an experimental mode, for example, in test containers, reusable modes that will reuse the container that was already started before. And if using this in my local development experience, then I have a very low overhead, like the startup overhead of starting the container is basically gone. And then I can have very fast iterative development cycles.

That's a good point. A lot of people say, oh, this is just going to slow down the pipeline, which already takes forever to run tests. Sounds like it doesn't necessarily have a lot of overhead. You get more benefit even if it does have a little bit more time to start up, I would think.

Yeah, so if I'm a consultant, I would say like it depends, like everything depends, everything is a trade off. And here it's a question of trade off with more confidence versus maybe a little bit of overhead. Containers of course are sufficiently lightweight. So I think that's why we couldn't really do this widespread with virtual machines before. So logically it's very similar to what you could have done with virtual machines before, but just the startup would have been quite painful. And with the containers it's sufficiently lightweight, I would say Cool.

So we talked about unit testing, integration testing. I think a lot of people listening to this are probably more tester focus, kind of functional. Can you use test containers for scenarios where you need to do like an end to end test?

Yes, you can do it for all scenarios or all layers of the testing pyramid, so to say. So you can also use it for kind of out of process testing where you just spin up all the services and also your own application under test together and then just test from the outside with maybe something like a browser with Selenium and so on that works well. Or you test against the public API with Rest Assured or whatever. Rest API. So you can do this. Currently one kind of limitation is that all containers you're starting have to be able to run on a single host, which can be as big as you want. But yeah, that's the way test containers works currently.

Cool. In your session automation guild, you gave an example of a lot of times someone will be testing a technology and like staging and testing, but then when they go to production it's a different backend. You give an example of a database, how many bugs do you think people would find if they actually use test containers to actually test using the exact technology that they're using in production? Does that make sense?

Yes. Yes. So I definitely know this is the case sometimes with H2 like H2 in memory database and a certain emulation mode versus than the real database. So I can't give you numbers or percentage. But anecdotally I have heard this from users and I have heard this also from other open source maintainers that using nowadays test containers for doing a lot of the integration testing. So especially around this JDBC relational database domain, I think it really helps you to find some of the more weird idiosyncrasies of certain technologies. And Also so regarding H2, sometimes you had to then maintain two different schemas because H2 or two different set of SQL scripts because H2 was not able even in emulation mode to run all the features of postgres or whatever. So I think it's also can help you with making your code more maintainable.

Nice. So I know another trend that I've been hearing a lot about is server liability engineering and chaos engineering. And that a lot of times now when we build software, because we use so many third party applications and services, like there's a lot of things outside our control. Can test containers help us get some sort of like experiments of what happens when we rely on a third party service that we have no Control over if it goes down or if it starts acting funky.

Yes. So you can do this very nicely because you in your tests have access to the infrastructure components basically as Java objects in case of test containers. Java, so you can experiment and just see, okay, what happens when I stop the database container while I'm in the transaction, what happened then? And we have even a special module from the Chaos Engineering community that it's called Test Containers toxyproxy and this toxyproxy. So it's from Chaos Engineering, it's I think, a TCP proxy kind of that you put in between your application and the service against which you want to test. And then you can start to simulate all kind of failure modes. Like you can start to drop packages or add a lot of latency, disconnect and reconnect. And yes, so we have a real module even to support those kind of experiments.

So how does test containers work? Do you have to have an existing module in order to use the technology? Because I assume a lot of companies have using a bunch of services that you don't have a module for. How does that work with test containers?

Yeah, so in its core, you don't need special modules. So we have this one basic container abstraction, we call it generic Container. It's an object oriented abstraction over Docker containers. And you can put any Docker image in there. So anything you can run as a Docker container, you can also use with test containers. So those additional modules we have, they give you some more comfort features like we have for the relational databases. They have APIs to get you directly the JDBC URL with the map port. Or we have certain wait strategies already configured. So the wait strategies are checks that ensure that the application in the container has started. But if you are using your own image, like the one that is only in your company, you can just throw it in there. You can configure the generic container object to wait for some, for some useful indicator. Like for example, if your Service is an HTTP service, you just wait until you get an HTTP Status 200 from a certain endpoint or maybe a certain JSON response. You can do this kind of things. You say, okay, I want this container to expose Port 8080, whatever, then test containers will dynamically map it to a free port and then you are good to go. So if you know how to run an application with Docker, you basically know how to run it with test containers as well.

Nice. How hard is this to integrate right now? Say someone's listening to us. I want to Try this out. What skills does someone need to know to actually start using test containers?

Do you think so? I think if you're experienced with Java testing, like you know, Java programming and writing Java unit test, then you, I don't know, like after half an hour you have your first test. I'm sure it's that simple. And I would say having general Docker knowledge and experience is also helpful because if you never used Docker before, then you are maybe not maybe a little bit confused about some of the semantics and so on. Because we are on top of Docker but Docker or we are a leaky abstraction on top of Docker. So it's still clear that underneath Docker running.

So I know a lot of people are using Kubernetes nowadays. Does this matter if someone's using Kubernetes? This technology uses Docker the interchangeable.

It's not directly interchangeable because it's kind of a different layer or whatever. But so we have. So it depends a bit what you're doing with Kubernetes. We have now support for a technology called K3S which is from. Which used to be from Rancher and now it's from suse, I think. And it's a way to run Kubernetes inside Docker. Okay. And if you use this K3S container and test containers, then you can use test containers to spin up Kubernetes as Kubernetes test cluster, basically. And then you can do with this, whatever what you want to do. So if you're developing a Kubernetes operator, you can in a test driven way develop a Kubernetes operator with this local Kubernetes running in test containers. There's another project from a colleague from mine from atomicjar, it's called KIN container, which is very similar. It's Kubernetes in Docker. So it's a similar technology for this. So you can do this. Another thing is if you have your CI running on Kubernetes with also many organizations have nowadays and then sometimes they have CI set up in a way that CI can't run a Docker daemon. So there you have to be creative. So test containers needs to have a Docker daemon running somewhere. It doesn't have to be on the same node on which the tests are executed, but it has to be accessible. And then you can configure it with environment variables or whatever. But this has to be given. Yeah.

So I know a lot of times when people start using different technology, sometimes they use it in A way that it shouldn't be used. Or you're like, why? Why are you doing this? Have you seen any use cases of that? Like people using test containers? And you shake your head like, maybe that's not the best use of test containers.

So there is one very specific usage pattern we've seen a lot, especially with inexperienced test containers users, and that is using fixed host ports. So when you first start experimenting with Docker, you will very often see in the examples that you can just map a port to a fixed host port. Like you can map 8080 on Host, or 8080-80 doesn't matter, but something fixed. And this is also the syntax. And then when they start with test containers, they think, okay, now I start my database and I need to know on which port it's running, because I have to hard code the port somewhere in the config file. So I have to somehow make it happen that the test containers will map to this fixed host port. And we have a fixed host port class, but it's kind of deprecated and it's more there for historic reason. There's no good reason to use a fixed host port in the test containers programming model, so to say, because you can get the dynamically mapped port very easily from the Java object after the container has started. So you have to think a little bit, okay, how has to be the life cycle of my test suite and my application to inject this URL or this port at which this database or whatever is accessible at test runtime. But all technologies, so at least in the Java space and with the one I have used, have a way to achieve this. Like sometimes it's a bit easier, sometimes it requires a little bit of a workaround, but you can always do this. So with spring boot, have first class support for this nowadays, for example. And yeah, that's something people have to understand and that is very beneficial because if you would always map onto the fixed port, your tests would fail if the port is in use. And why would it be in use? Because maybe you start parallelizing or whatever, and if you always map onto a dynamic port, your whole suite becomes much more robust. And that's really something we want to have as test containers because we want to solve this bad reputations of integration tests that they are flaky or whatnot. We want to make them easy to maintain, easy to run and robust.

Absolutely. So are there any use cases where you think people shouldn't use test containers? Like, I know sometimes people may use test containers. Maybe we're probably not going to give them the benefit that they think they're getting.

I would like to answer with the opposite. Something where I think people can also use test containers, although you would think it's not the intention of the library.

Okay, even better. Yeah, yeah, yes.

So the library is called test containers, but there are non testing use cases where it's also very useful non automated testing use case, let's say like this. So one thing, and Sergey has a very nice blog post about this. Maybe we can put it in the show notes somewhere when I send you the link.

Yes, yeah, absolutely. Yep.

So it's basically using it as a replacement for Docker Compose for running your application locally in a development mode, so to say. So very often when people want to run and then interactively do something with the tested. Yeah, so very often people before run Docker Compose and they start the application both they do by hand and then the application connects with a local database or whatever. But what you can also do is basically write an alternative entry point in your application, an alternative main method or whatever that will start the containers as part of the application startup. And then you have a single process you execute and everything spins up, your application spins up the dependencies and yeah, that's very nice. And it will clean up everything automatically when the JVM process exits. Yes, that's a word because that's built into Test Containers, this cleanup feature. And yeah, so in a way here it's then a replacement for Docker Compose for this local development.

Very cool. So are there any common questions you get asked about test containers that you hear all the time? Maybe misconceptions or questions I didn't ask that you think people should have asked or should know about when comes to test containers.

So there's always a question about Kubernetes, which. Which we had already. Yes, yeah. And then often there are questions about using alternative container engines. So alternatives to Docker and yeah, we have of course Podman, we have there. And now for some time we have a project like Lima and Colima which I haven't used myself, but there are some alternatives coming up and the answer is that Test Containers is integrating against the Docker Rest API. So as long as an alternative container engine provides a Docker Rest API compatible Rest API, test containers will work with it. And yeah, we have seen such setups work. So with Kollyma especially, I know I supported some users in getting this working and with Podman there are some challenges yet. So I know the Podman team is working hard to make Podman really Docker compatible in a Docker drop in replacement, but it is not yet to my knowledge. And then there's nothing test containers can really do because test containers assumes a container engine that behaves like Docker regarding semantics, also regarding networks and so on.

So I know you're one of the co maintainers. Is there anything on the roadmap that you're working on that you think in the future releases people are really going to want or it's too early to delve anything about you may be working on that's in the pipeline?

So this reusable mode, it's something we have as an alpha or experimental feature for a long time now, like one year or maybe even two years. And bringing this to a real production quality where it's nicely documented and everyone can use it, I think that will be one of the best features and people really appreciate it. And it will again maybe change some testing paradigms because suddenly this integration testing TDD style becomes much easier because you have this overhead of starting the containers. And yes, although it's experimental and a little bit hard to find, you have to check the pull request to find the actual documentation. I know many of our users already use it very happily, so that's cool. And if this comes out, that will be a very nice, very nice thing. And the other change that's probably a bit overdue by now are some changes to our API and some general housekeeping. Like for example, we still have a Junit 4 as a dependency in the core of test containers and that's just there for historic reasons. So we also have support for other test frameworks or you can also use test containers outside of a test framework. So there is no good reason that we have this hard dependency on Junit 4. So removing this and as part of this, maybe having some cleanups for the API, some other patterns to make a construction of container definitions versus then a started container. A little bit more reusable or a little bit more flexible is something else we are exploring. Yes.

I don't know if I made this as clear as possible. This is all open source, correct?

Yes, 100%.

Everything we talked about up to this point, open source, free to use, no strings attached.

Definitely. MIT license, 100% open source.

Very nice. Cool, Cool. All right, Kevin, before we go, is there one piece of actionable advice you can give to someone to help them with their integration testing efforts and what's the best way to find or contact you?

So if you are maybe currently testing using something like an H2 database and you're just using this special JDBC URL that then will start the H2 database. If you go into the Test Containers documentation under jdbc you will find we have a similar special URL for test containers. So if you want to just try out Test containers, replace your current URL with this URL. That's all you have to do. That's even less than the half hours I said before. Like it's kind of five minutes or less and then you have your first go at Test containers and then maybe you enjoy it and you want to use it more. Also in a programmatic way. The best way to contact me is on the one hand probably Twitter, I'm there at as key view and also I'm reachable in the Test Container Slack. So you can find the Test Container Slack in our Test Containers documentation. And it's not only the maintainers that are there, also the community is very helpful and if you have any questions just pop them in there and we try to help users and collaborate on those things.

Thanks again for your automation awesomeness. The links to everything of value we covered in this episode. Head on over to testguild.com a392 and while you're there, make sure to click on the Try for Free Today link under the exclusive Sponsor section to learn all about Sauce Lab's awesome products and services. And if the show has helped you in any way, why not rate it and review it in itunes? Reviews really help in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack Automation awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Automation Podcast. Head on over to testguild.com for full show notes, amazing blog articles and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.