Test Before Merge. Not After with Sorin Dumitrescu and Shani Shoham

Welcome to the Test Guild Automation Podcast where we all get together to learn more about automation and software testing with your host, Joe Calantonio.

Hey, it's Joe. Welcome to another episode of the Test Guild Automation Podcast. Today we'll be talking with Shani and Soren all about how to test before you merge your code, why it's so important. Fionno Soren is the VP of Technology at bunnyshell and Shani is a frequent guest on the show who has scaled multiple businesses in the automation and testing space to the next level. Really excited to have him on here. So if you really want to learn how to increase releasing your code with less defects, you really don't want to miss this episode. Check it out.

The Test Guild Automation Podcast is sponsored by the fantastic folks at Source Labs, the cloud based test platform that helps ensure you can develop with confidence at step from code to deployment. For every framework, browser, OS, mobile device and API, get a free trial. Visit testguild.comsourcelabs and click on the exclusive sponsor section to try it for free today. Check it out.

Hey guys. Welcome to the Guild.

Hey Joe. Nice seeing you again.

Hey Joe, thanks for having us.

Thank you for joining us. Really excited about this topic. Before I get into it though, I think I really butchered your bios. So is anything you want me to include in your BIOS that you think you want the Guild to know more about Soren? Anything more than VP of technology?

So I'm going to try to keep it real short. As the VP of Engineering, I'm responsible for everything that's related to product realization. So starting with processes and ending up with what we ship and how we ship it, obviously. So everything in between is my responsibility here. I have a background in web development mostly. I worked as a web developer in various positions for over 10 years and I've been in management since like five years or so.

Awesome. And Shani, I know you've been on before, but anything I missed?

No, I think you covered that. I've been president and CEO of multiple testing companies. Last 121 Labs got acquired by Perfecto and Perforce and.

Yeah.

And you know, really looking to solve the next set of problems in testing and development.

Absolutely. Cool. All right. So Bunny Shell. What is Bunny Shell? I was doing some research on different technologies and Bunny Shell kept coming up. So I thought it'd be great to get you on the show to talk more about it. But for the folks that don't know how would you explain what Bunny Shell is?

I would say bunnyshell is an engineering productivity platform that essentially cuts the lead time to release. So we solved the problem of. But it worked on my local environment.

Full stack production replicas in the cloud. So, you know, typically the way that most engineering organizations work is you have developers kind of running a local docker. Then they merge that into the main branch. It doesn't work because the main branch is a little bit different. And many times they actually, you know, the version that they forked is already an old version and they're drifting. They spend a lot of time reworking the code, you know, merge that into a staging environment. Staging environment is different again. They spend a lot of time reworking the code, then it goes into testing. And typically most organizations only have a single shared staging testing environment. They run the test there, they find bugs, they goes back to the developer again, he reworks the code, he rebuilds the application. It's a very, very lengthy and unproductive process. And so what we do is essentially we trigger a full stack environment with every pull request that's in the cloud. So you can immediately share that environment with like product for uat, with QA for testing purposes. You know, QA does the testing on an isolated environment. You can do as many testing in parallel branches as you have. Essentially, you know, if you detect a bug, it's very easy to isolate that bug to that specific branch. The developer can literally connect using remote development, can connect to that environment, make changes to the cloud or to the environment in the cloud. QA will immediately see those changes, no need to rebuild and you know, wait again and again and again and. Yeah, and then once everybody's happy, you know, you just merge the code, we tear down the environment, and that's kind of about it. And other than that, we keep all the environments always in sync. So kind of that rift challenge that I explain, you know, no longer occurs. But the idea is with full stack environments, the code quality that the developer can produce is much higher. The feedback loop between dev and QA is much, much faster. And kind of the button X of, you know, having one or two shared staging environment and everybody's like merging into that and testing. That doesn't, you know, doesn't happen with a banisher like solution.

Nice. So it almost sounds like it solves the issue I used to have back in the day where it works on my machine. When you talk to a developer, it's like the ultimate environment to test in, almost, because everything's reproducible, everything's the same, ideally.

Yeah, exactly. By having a full stack environment as opposed to like a deprecated environment that only includes a set of microservices. It's a much better way of developing. Now your local machine is, you know, it's not an eight core server and so there's a limit to how many, you know, services you can run locally. And you know, your ability to mimic all kinds of databases, all kinds of cloud native services. And so that's, you know, when it's in the cloud, it's immediately shareable. It's a standard way, you know, you're in compliance and you're not relying on again, local machines which aren't meant, you know, to run lambdas and databases and stuff like that.

So Shani, I know we spoke a lot of times, you always seem to be a little ahead of the curve. I know your latest company that you do recently got acquired was AI based. So why Bunny Shell? Why this type of technology? What makes you excited about this domain? Why does this seem like something you think is really important nowadays?

So first of all, as an engineer by background, I resonate with the problem of you develop on one environment that's like a local environment, you merge to a different environment and therefore tons of rework. And you know, for tech companies, developers are the most expensive resources and so you really want to keep them productive. The other thing that I've seen, you know, with most of my customers throughout the years at different companies is, you know, we will sell a test automation solution. But, you know, spinning up environment was, you know, would always take forever. I just yesterday I met a customer of mine, VP of engineering. And I, I remember for them they implemented our solution at 21 and then it took them like six months to be able to create another environment to test on. And so yeah, just ability to spin up multiple environments, especially when we talk about test automation, we talk about running things in parallel, concurrent testing, et cetera. But the reality is that most organizations, at the end of the day, the bottleneck is the number of testing environments that they have. And so that's why I'm excited about environment as a service. I think it's kind of most of what we see out there is companies trying to build something similar themselves. And so now at the end of the day, you want your developers and your DevOps to really focus on managing productions and shipping code and features and not building all kinds of internal tools. Then bunnyshell, I think those, first of all, very sleek experience, nice blend between developer experience and kind of using your local ID, using APIs and then a SaaS solution that QA and product can use to spin up new environments themselves. And so yes, that's kind of why Bunishel plus, it's a great team and I really enjoy working with everybody.

Awesome. So I used to work for a large enterprise and you wouldn't believe in the amount of time we wasted on environments. It was just crazy. So, just curious to know how this all works. We had different ways of doing so, soaring behind the covers or under the hood. Now, how does a solution like this normally work, what's the technology involved and things like that.

So I'm going to start from the output, from the outcome that you get. You get to have your environments deployed on Kubernetes clusters. They run in total isolation from one another. So they're really sandboxed from this point of view. No, absolutely no interference, no shared resources, which obviously eliminates a lot of inconsistencies or false positives and eliminates wasted time. At the end of the day, you get that or those resources which are related to your environment. So besides your microservices or your main apps, you get to have your databases, cloud databases like RDS from AWS, for example, you get to have S3 buckets provisioned, or any other kind of cloud native resource that you need to have a full replica of a production environment. Obviously, for the Kubernetes part, we handle all the complexities, so we have ways of generating all the needed resources. We handle the ingresses, the services, for exposing the actual services, the microservices we handle volumes and we handle deployments, pods and so on. So everything is abstracted by BunnyShell. But obviously if you need to intervene and handle some very custom things or specifics related to Kubernetes, you can do that as well. So we try to keep it simple, but you can also leverage the full power of Kubernetes if you really need to. How we achieve all this is in a very simple manner for the user. Basically, almost any development team which runs containerized loads has a docker compose which describes their local environment. And you can start from that. So you can input your docker compose file or files within an environment and then we'll generate dev friendly syntax. We call it environment YAML, which our representation of the environment. Starting from that point on, you can add new services, edit, add volumes, ports and so on. Everything that you would need to do to represent your environment correctly. It's all linked with the resources which Terraform creates. We also support helm charts in case you need to deploy more complex loads on Kubernetes. So I think we got it pretty much covered when it comes to environment complexity.

All right, so how does this work with CI CD though? Isn't this like, shouldn't this already be part of cicd? How does it interact with cicd? Do you see it going hand in hand with cicd? Is someone checking code to their systems and then it spins up the environment? How does this all work?

Great question. I do want to just, you know, add two other things to what Soin mentioned. You know, it's being very modest, but what we do on the road is very, very complex. It works across multiple repos. And so if you have your front end working on one repo, your back end team working on another repo, you know, even if it's like something crazy like, you know, part of the team is on GitHub, part of the team is on BitBucket. Again, it's all seamless and you know, we do GitOps under the hood across all of these repos. And then it's also multi cloud. So you can again choose to deploy your development environment on aws, your staging environment on gcp, your production environment on Azure, and so doing that, it's pretty complex. Great question about the CI cd. And so first of all, we integrate into any development pipeline, TeamCity, Jenkins, Jaw, DevOps, BitBucket, pipelines, GitHub, Actions, etc. We have a simple API that you can trigger to spin up an environment and or for example, you know, execute a set of automated tests once the environment is up. And we know how to pass the environment, you know, as a variable to, for example, the base URL into your automated UI testing. And, but, you know, kind of what we do, I typically get the question of how you different from cicd. I guess, you know, the main differences is, you know, we spin up full stack environments where we constantly manage the state between the code, the infrastructure and the database. And so, for example, if we sense a change in, you know, the infrastructure configuration or the code configuration, we immediately trigger an update to all the existing environments. You know, if you tear down your code, you know, for some reason it crashes. We know how to destroy, you know, the infrastructure services, the database, and so kind of all this orchestration that's happening under the hood. Again, CI typically senses an event, triggers a set of actions based on that event, and kind of sends the results, you know, back to the next, potentially the next trigger. So again, we work very well with the ICD pipeline. Organizations that don't have CI CD pipeline, since we have webhooks into your repository. Then again, we don't necessarily need the development pipeline. We can sense, pull, request, trigger an environment. You know, if you don't have automated testing, you can do manual testing on that environment. Again, that environment is in the cloud with services that are already exposed, an endpoint that you can use. And then once we have a merge close, we'll tear down the environment. With CI CD again, you can use that environment to trigger automated testing and all sorts of other automated activities.

So, you know, Kirsten, I'm just playing not the devil advocate, but someone out there is like, so what is, Can I just do this with Docker and Kubernetes on my own? Why do I need a service? Or I mean, how difficult is this to do if someone's trying to implement themselves? Soren, I mean, obviously you, this is all you work on. So what makes this like, what's the secret sauce maybe makes this difficult that someone can't just spin up Docker and Kubernetes without maybe a lot of effort or a lot of, you know, infrastructure in place.

Yeah, so ultimately, Joe, it's code. So it's an app. Every team, or almost every team could probably build it. But the real question is, are you committed to doing so? Because you'll end up committing a ton of resources over a very long period of time and then you have maintenance. This is the kind of project that most companies tend to gravely underestimate. They tend to say, yeah, it's like a two or three month project in two or three people and 12 months later they still have five team full stack members committed to the project and they realize that they'll have to do maintenance like forever on the product. Right. Because technologies evolve. Maybe now you're using Terraform, but maybe you'll switch to Pulumi for newer apps or something like that. So it's the kind of product that needs to be in check with the latest tech at any times. Besides that, there's also the thing that Shani mentioned, because having multiple cloud providers, having multiple git providers integrated means obviously work to initially integrated and then maintain the APIs they get deprecated, updates need to be performed, and so on. But I think the trickiest part is the state management that also Shani mentioned, because you really have to keep track of all the resources. And I'm not meaning only Kubernetes here, because this is the simplest part. You need to keep track of every resource that Terraform has created. And this can become pretty tricky, especially if you have some failures along the way. So if you don't manage these properly, this could turn into very, very expensive mistake and have thousands of dollars wasted on hung up resources over the course of a year, let's say for mid size or large size company. And these are resources who maybe will get discovered during an audit or maybe not and they'll keep consuming money in the cloud or wherever they're deployed. I personally have knowledge of some situations like these in ultimately, for example, one company realized that they had performance issues while they were having pretty massive setup for kubernetes. And when they got to look into the issue, they realized that almost 40% of their resources were in fact not used. They were created and created and created, but nobody deleted them for testing purposes. So yeah, it can become really messy and pretty quickly as well. There's also the DNS management issue here because again, you don't just need to create some records, you need to know which of them are actually being used and perform cleanups and so on. And not having this mission of serving developers as your company's focus means that probably the developer experience will be subpar. So we're really geared toward how developers are working. And when I say developers, I mean development teams, right? I mean software engineers, both back end and front end, and I mean QA engineers, product manager and stakeholders ultimately. So everybody can use these kind of environments to ensure smooth collaboration and near instant feedback. And we all know how projects get dragged on because you need feedback from somebody, but you can't show them what you've been working on because you have no staging to put it on or UAT or whatever. So there are a bunch of things under the hood. And as I said, ultimately it's an app. It's a platform actually, and it can be built because it's code, but it will take a tremendous amount of focus and energy away from your core business. And it's not about the money ultimately, because no matter how much money you get, probably you won't be able to find talent to fill in your whole development budget.

Great point. Once again, I'm familiar trying to hire people for a team with different skills. Very difficult to do for sure. So this is an automation testing podcast. Just curious to know for people that are listening, maybe hard to visualize, can you maybe walk them through what a workflow looks like or how testing is done when you're using an environment as a service type of situation?

Yeah, let's talk about how the workflow works today. So today developer forks the branch, does his code check ins, merges that into the main branch. Again, there's a lot of free work that happening under the hood. Then you rebuild the application and then you start testing now while you rebuild the application. Also ideally while you're testing you typically other developers can, you know, merge their code because it might conflict and you know, would make it difficult to trace back defects to that specific developer. And so and again, typically the testing process doesn't kick in as soon as the build is ready. But kind of those, you know, the bottleneck is once you merge the code into a shared environment with bunny shell. Essentially as soon as a developer, you know, forks his branch and start doing his code check ins, you know, makes his first pull request and environment is spin up with a URL with the relevant endpoints. At that point again you can, you know, start triggering a continuous testing process or even do manual testing in parallel. And essentially again you're doing the test on an isolated environment. And if you think about multiple developers making multiple requests, each one of them has a separate isolated environment. Then again you can run multiple continuous testing pipelines on each of these branches separately. You know, each of them can have its own database, its own, you know, cloud services, et cetera. So they don't conflict with, with one another. Which is another challenge that I always had with automated testing is, you know, you run two sessions in parl and then they just conflict with one another. That's it. And then, you know, assuming that there is a bug again, the developer using remote development can, can essentially we create proxy between his, the cloud environment and his local machine. He uses his local ID to fix those changes. It doesn't need to rebuild the application again and again. Every time you make those changes the application, the changes are immediately applied to the cloud environment. You can run another set of regression testing, functional testing, etc. And then only once, you know, the test pass and everything is fine, you actually merge the environment and the code and that's pretty much the only time that you'll rebuild the application. So we save tons of time on rebuilding the application. Again, you can run your testing in parallel. You test on isolated environments, which makes it much easier to trace back a defect to a specific branch and just the feedback loop between dev and QA and even generally between dev and other constituencies. Like product is much, much faster.

So you know, once again people just heard the workflow. So you already touched a little bit on the benefits of using this. But what are some benefits from testing a testing perspective that you've seen maybe from customers or your experience, how this could save them people, time and money.

Yes, I kind of touched upon that. So essentially, first of all, you can run as many tests in parallel as you'd like, or as many test as feature branches as you have. And again, that's something that today, I would say majority of the organizations that I've worked with are not doing today. So today they have a single staging environment for testing purposes. So they either merge multiple branches and test it. And then again, it's very hard to trace back defects into that specific developer. Plus it delays the release process significantly. So, you know, with unlimited, essentially number of environments you can run unlimited regression testing, functional testing, UI testing again, you don't have to wait for these rebuilds again and again and again. And so you find the bug, the developer fixes that, you immediately run another set of tests. Cloning environments is super easy. So again, if I do manual testing in parallel to automated testing, I can either do that on that same environment or again, I can just clone that environment for manual testing purposes. And so, yeah, so just the time that it takes to, you know, identify and fix the defect, the feedback loop between dev and qa, the level of parallelism that you can run on is much, much faster. And the nice thing is many times I hear QA actually kind of complaining that they're dependent on DevOps to create additional environments for them. And so with Bunny Shell, it kind of liberates them to create as many environments as they like. They can take any template and just spin that up. And one of kind of the nice features which we didn't really cover, is the availability rule. And so, you know, kind of the concern, typically when you spin up a lot of environments in the cloud, is the cost of cloud. So first of all, these are thermal environments. They're temporary. You spin them up, you automatically tear them down. But also, you know, we can essentially decide per each environment. You know, I want this environment up until 8:00pm no developer works after 8:00pm I, you know, I cache the environment, spin it down, and then I keep the stake persistent. And then in the morning I spin it back up. And so that way we also save on cost of cloud.

So I'll be honest, we had somewhat of a solution like this when I was working. And the problem was people would just leave the environments up forever. They'd forget to tear them down. You know, you have to do a cleanup routine, someone have a spreadsheet and say, are you using this? Why is it running? Or you'd have to get permission to keep it up during a Certain time. Is any workaround around that or is it like a scheduling that makes it easier to handle that type of management? Actually of all the different environments now that could be up and running, costing lots of money because it's like you said, taking up resources.

So again we connect into your repository. So first of all you don't need to manually spin up the environments and tear them down. This is all automated developers. Once they're done with their branch they typically do a merge close. And so again it's something that they do as part of their workflow. And then again with availability rules, the whistles are just aren't running, you know, always, you know, when they're not needed. One of the other things that we've seen companies do as kind of a workaround is they just set up permanent environment that are again always up and running. They, you know, they have set up six environment, you know, each developer, you know, kind of picks an environment based on availability. The challenges there is one, they, you know, it's not parameterized, they need to reconfigure their ID and their testing, you know, every time to that environment that they picked up. And then again these are permanent environment so it's not really cost effective. So again it's, it's, you can script all of that, you can spend months kind of doing that. You know, we kind of do that out of the box.

So Soren, in the pre show we talked about something, you mentioned something about GitOps. I haven't had anyone actually talk about GitOps on the show before. Just really high level, you know, what is GitOps and how does this help with that type of a methodology?

So GitOps essentially means reacting to changes which are stored in git, right? That's the naming git operations. The main benefit here is that you have a single channel through which you get to push your changes, right? So everything passes through single queue. Let's say for changes which are file storing git, you also get all the benefits for governance, right? You get only some part of the team privileges to approve or requests to go into the main branch which are using and that main branch becomes the source of truth. Additionally, when using git and using pull requests, obviously you get to comment on these. So any, I don't know, any questions or any concerns are documented there. So it serves both for documentation purposes, but it also offers an audit trail. You get to clearly see when a change has been applied and by whom. Right? And what was the context that time? So if you need to get back in time like A time machine. Right. And figure out why something was done. You get to easily do that. And git is really good for this kind of changes. And I said again, because it's really important, you have a streamlined process of making the changes, applying the changes right for your environment. So only certain people can review, approve and then merge the pull requests. So nothing unsanctioned goes into an environment configuration.

All right, so we covered a lot of the benefits towards testing qa. Obviously that's the bread and butter of the test guild. So thank you all. But obviously you have to work with other people in the team, other parts of the organization. How does this type of approach help maybe developers or productivity or. Or even your CTO if you're trying to sell this solution to them?

Yeah, great question. So I would say kind of the two main metrics that we kind of see an impact in customers we work with are the lead time to deploy and then the developer productivity and velocity. So naturally, given the fact that the feedback loop is shorter, there's much less reworks because you, a developer works on a full stack environment and is, you know, kind of dev staging. Pre production. Production environments are identical, you know, means that he's just spending more time writing the code properly to begin with, which translates into velocity, which translates to the number of story points that you get done in a sprint, which translates into sprint predictability. Typically, many organizations, many engineering organizations starts the sprint with this amount of things and then very quickly, by the end of the sprint, they end up with that amount of things, you know, being pushed into production. And so higher velocity, you know, means you get more done and kind of the percentage between planned and actual is, is much, much higher. Other benefits, kind of the onboarding process of new developers. You know, typically a developer comes on board. I remember that from my days as a C of 21. We would bring a new developer and would take like, you know, after two weeks, I would ask what has this guy been doing? And he's, you know, he's still trying to set up his environment. And so when it's in the cloud again, there's no setup of environment. It's a standard, you know, environment in the cloud up and running very, very quickly. You're also more compliant because, you know, everybody work in a standard way. The environments are in the cloud and there's nothing running locally. So if you need to offboard someone, you know, it's literally a click of a button. Developers are just much more happy. Again, developers don't like the switching context of, you know, I worked on something and now I'm waiting for, you know, the build process to finish and for QA to under test. And so in the meantime I'm going to start working on another feature and then they have to switch back and forth between, you know, different branches. They definitely don't like the rework. They like to focus more on, you know, kind of building new features as opposed to, you know, reworking existing features. Much less merge conflicts because they develop on a full stack environment and because we're constantly keeping the environments in sync and so, you know, they don't experience any conflicts that relate to drift. And so yeah, I mean, essentially it's a much, much, much more rich and developer friendly experience.

Once again, I'm just thinking back to my experience for this big organization we had a, they used to have to give demos. In the demos they'd seize one of the environments. So you cannot be on this environment for this demo. And obviously someone would muck up the environment. It was very hard to scale these environments up because someone had like a 50 page document to get all the integrations in place. It sounds like they could set it up once here, take a snapshot. Anyone could use it. And if someone was doing a demo, it would be completely isolated so you'd have no chance of impacting, you know, any demos or sales things that are going on. Because someone needed a fully functioning environment.

This is exactly how we do demos. So literally at the beginning of the demo what we do is we spin up a demo environment, live in front of prospects and, and then we run the rest of the demo there. You can snapshot the database, you can again spin up the cloud, native services, the code and, and so you don't conflict with, you know, we override changes that someone else made to the database. So very, you know, you, you just articulated a great use case for us.

I just thought of another use case. Don't know if that's how it's used. Once again, we'd run like thousands of tests and sometimes tests would add data that would impact another test that would break it. So could someone actually use this to scale up and scale down a database to know that this test or this test suite always will have this clean data without it being corrupted by these other types of tests and activities going on.

Exactly. This is what I've seen in working with customers is they would run two automated, two concurrent sessions for automated testing and one would make a change to the database and then the other other one, you know, would trigger an action and the result would be, you know, wouldn't be consistent with what the test was expecting just because the two tests conflicted with one another. So with Bunny share, as I mentioned, the, you know, an environment is, includes all the dependencies, including databases. So you can either choose to use a shared database and you know, for example, I mean, RDS is probably something that you don't necessarily want to spin up an RDS for each environment. But then you, you can also decide that for certain environment, you are taking either the latest snapshot and, you know, going for an anonymization pipeline, or you're just taking a snapshot with certain data and you initiate the database that way and you write and then there's no conflicts. By the way, another use case that we see around database is envelopes. You know, if you're a data scientist and you know, you run an epoch and you need the data, you know, you certain, you need a certain piece of data, you want to be consistent with the data that you use to run your epochs. And so, you know, again, we can, with a single click, we can let data scientists, you know, create environments with proper database seeding.

Very cool. Okay guys, before we go, is there one piece of actual advice you can give to someone to help them with their environment automation efforts? And what's the best way to find Contact your learn more about BunnyShell so.

Obviously you can go to our website, BunnyShell.com and find out more or reach out to us on social networks, LinkedIn or whatever channel you find fit.

Yeah, I think with, you know, with software being very, very complex, you know, again, cloud, native services, gateways, you know, endpoints, databases, multiple repos. You know, you really need to think about your environment as more than just microservices and you really need to think about kind of local development. The idea is always going to be locally, but the code, the environment, that's with kind of the evolution of cloud. I think it's about time we move part of the development process to the cloud. So yeah, I just urge engineering leaders to kind of measure what the team is doing and how productive they are and then think about environments more holistically.

Thanks again for your automation awesomeness. The links to everything of value we covered in this episode, head on over to testguild.com a40e and while you're there, make sure to click on the Try for free today link under the exclusive sponsor section to learn all about Sauce Labs awesome products and services. And if the show has helped you in any way, why not rate it and review it in itunes. Reviews really help in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack automation awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Automation Podcast. Head on over to testguild.com for full show notes, amazing blog articles, and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.