Automation in the Modern Age using BigTest with Taras Mankovski

Welcome to the Test Guild Automation Podcast where we all get together to learn more about automation and software testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Automation Podcast. Today we'll be talking with Taris all about how they eliminate flakiness and testing at BigTest. Taras is the CEO of Frontside and product manager for BigTest, a new open source testing platform for the cloud native world. Looks really interesting. You don't want to miss this episode. If you want to get your hands on and know more about the latest and greatest in technology for automation testing, check it out.

The Test Guild Automation Podcast is sponsored by the fantastic folks at Sauce Labs. Their cloud based test platform helps ensure your favorite mobile apps and websites work flawlessly on every browser, operating system and device. Get a free trial. Visit tuskguild.comsourcelabs and click on the exclusive sponsor section to try it for free for 14 days. Check it out.

Hey Taris, welcome to the Guild.

Hey Joe, it's a pleasure to be here.

Awesome. I guess before we get into it, is there anything I missed in your bio that you want the Guild to know more about?

Well, I mean I'm an engineer first and foremost and I've been kind of swimming in the business world for a long time and so it's been a really fun journey. And this project, BigTest has been really a passion project and a love project, I guess.

Nice. So I just heard about BigTest recently. For the folks that don't know, how would you explain big tests at a really high level?

I would say it's a test suite that developers and engineers can that developers and QA can share. The goal of BigTest is to create a tool that developers want to use. So something that is fast and reliable but it tests the application the way the QA QA test application. So you know the way it tests as users use it so that you can have kind of shared confidence of in your whole organization. So QA QA developers looking at the same tests and they know what was being tested. You don't have to make compromise of reliability and speed for developer experience.

Nice now I guess why was this developed then? Is this an algorithm? Is it a tool? Is it something that like an add on to other tools?

I would say it's a combination. So ultimately want to get to is a kind of cohesive package that you could use to do testing right off the shelf. But we are small teams so bringing that to the world, you know, it requires a Lot of kind of iterative development. And the other part is, you know, it requires a lot of research because we have to go back and, you know, to improve certain areas of testing, we need to actually rethink the fundamental practices around testing. Because a lot of stuff that we. A lot of assumptions that we base our technologies on these days are based on ideas that have been around for a very long time, but they haven't really been updated to match the evolution that's happening in a kind of cloud native world. And so it requires that because everything is changing, we. We are shipping faster, we're free shipping more frequently. Our applications are being shipped as microservices as opposed to monoliths. So all of these things require that we apply the same kind of rigor.

That we see in architecture. We need to apply the same rigor to automated testing and make sure that we have solutions that match the modern world. I would say that the approach that we're taking is to first develop things that we know, because a lot of the ideas that we have in BigTest have been. We've been experimenting or using them on projects, you know, over three years. So, like, we have. One of the aspects of BigTest right now that's coming out is BigTest Interactors, which is essentially for web applications. It's a way for QA and developers to test, to share composable page objects that are essentially part of the component library. So you've got. You've got essentially kind of an API that. That you're providing as part of your component library. And this API is used for testing these components in tests. And so the benefit of that is that you have very reliable tests, because if the component library changes, your tests don't break, because the tests use interactors, which, you know, are kind of. They're in sync with the components. And so you've got this kind of connection that happens between Test Suite and the component library. And so this tool is available. It works with Jest and Cypress and also with BigTest Runner, which is an alpha. Right now, it's very much like a work in progress. So our goal is to make these tools, make aspects of the framework available for use with other tools, but then eventually offer kind of cohesive package that you could use off the shelf.

Nice. So you did mention there's some assumptions that people use that are kind of old, that maybe hold back their testing efforts. What are some of those assumptions, you think?

I think one of the biggest things. So we had this previous version of BigTest that was called Frankenstein it was a collection of tools. So, you know, like part of the test way to be able to test the web application. You had, you had Mocha Expect, which was used for, you know, writing assertions. You had karma for the actual runner. You've got the, there's a webpack in there somewhere to build application. You've got this, you've got this like five things that are supposed to work together to give you the ability to run your tests in a browser, but you can't. What we found is that you actually can't build a good developer experience with the system being composed that way. Because what happens is that in every aspect of the architecture you have a silo where data is siloed inside of that piece in the runner. You have all the information about being able to start a browser and then execute, inject the test and then run the, you know, open the web page like the runner has owns all of that. Then you've got the test suite. You know, all of the definition of the actual test that you want to run a part of Mocha and that information is not available to the runner. It's actually not available until you actually spin up the browser and then you run the execute the script. And then the assertions are actually not a part of the data structure at all because you only really know about what the assertion is when you're actually executing the test. So you've got all of these different aspects of the system. I mean, the other piece to that is that you've got like, if you have something like react testing library where you have a mechanism for asserting something in the dom, that thing knows about what you're touching in the dom. But each of these things, they work separately. And you don't actually have a cohesive data structure that represents each of those things together. And then what that happens essentially is that you don't have a way, if you want to create a really precise and comfortable and very responsive developer experience, or test writing experience, or test running experience. Doing that when you have these five silos is actually really difficult because each one of those brings its kind of quirks. And then you have to you basically mitigating those quirks in the glue. And it's really hard to make, make a cohesive experience where you've got this like four things, four or five things that are glued together and you've got this kind of glue that is just really hard to work with. And so we had to essentially like when we realized that it wasn't possible to create a good developer Experience, we basically had to start from scratch. And so at that point, when we realized that this wasn't that that's what we need to do at that point, we're basically, well, if we're doing that from scratch, then I mean, what other assumptions can we rethink? So, for example, like, one of the things that we've done with the test definition syntax is so like, fundamentally, one of the things that's unique about BigTest, I would say, is that the tests are not scripts. They're essentially a data structure. So the entire state of the runner, the builder, the text execution, the down to interactors, like every single aspect of the of the test suite is available as part of a data structure. So we can actually query that state using GraphQL. That's mechanism that's built into the actual runner itself. And so we can actually use that to build a CLI tool. The way of interacting with the big test system is using GraphQL. So when you want to run a test, you just like you run a mutation and that mutation executes a test on a specific browser, it'll be responsible using WebDriver to spin up the actual browser. Like once the browser spun up, it actually just. There's a mechanism for injecting the script while injecting the test mechanism, and then it executes that test. There's kind of like an interpreter inside of the testing system. So all of that, the entire kind of architecture has been rethought. What that allows us to do is it allows us to have a really granular control about how we execute the actual tests. And then having, you know, having this architecture being rethought, it then brings us to a point where we can actually design our tests in such a way that it makes the tests very, very reliable.

Yeah, I was curious to know. With this approach, I think you've actually addressed one of the biggest concerns for a lot of people with automation, especially developers, they get really discouraged quickly is flakiness. So I guess could you talk about how this approach can help with flakiness? I think you call it the convergence strategy. I believe in your blog post you talked about how you retain references and things like that to help you interact with the web ui.

Yeah, yeah, absolutely. This approach, actually, when we start working with the Frankenstein big test, like when we start working on that, we were working, we were inspired by the work that happened in Capybara community, because there's a lot of. In Capybara, they have a mechanism for convergence which is essentially like kind of rechecking if something is available on screen, or like, if you're asserting that, for example, that the button is visible, you can attempt to check that several times until you reach a point where you're like, okay, well, I found it. Or it's been like two seconds. Like, I'm not going to find it. Like, there's something, there's a problem there. So that's fundamentally the convergence mechanism. And this was inspired by Capybara. What was kind of really surprising, but also kind of serendipitous in some ways is that because we were inspired by Capybara, the creator of Capybara actually joined our effort and he's working with us on building big tests. So he was able to. So Jonas Nicholas, who is the original creator of Capybara, he's working with us to build the browser agent. What he brought was the understanding of how that mechanism and the limitations that they had in Capybara. And so what we were able to do in a second iteration of like, you know, using this approach is we, like, in big tests, we're able to have very tight control of the run loop. So there are two things that are kind of important to understand about how the runner mechanism works, which is the execution of the tests. There are two specific, two kind of separate, distinct things. There is the asynchronous actions that you can perform, and then there is assertions. You know, actions like steps, right? Like you want to click on a specific thing, but before you can click on that thing, you have to make sure that the element is visible on the screen, so you can attempt to check for that element's presence. And so mechanism like interactors actually do that. So if you use interactors with Jest, for example, they have this convergent mechanism without having all the big test runner. So you can actually get the benefits of that stability, because the way that the architecture of an interactor is such that they're running within the same JavaScript run loop as the application itself. So there's no way to let the time slip. So you can't. There isn't like an asynchrony where you can. Where between the moment that you checked for precondition and the moment that you actually invoked an action or the moment that you satisfy a condition, there is no way for the JavaScript application to kind of escape that. And that's the really, I would say, big difference in even. There's some really excellent tools that become available recently. Like Playwright is really powerful for spinning up browsers and API for controlling automation. It's a really awesome tool. But because you're running the script, your execution script is running in a different environment than your browser. There's always a moment when there could be a mismatch where the precondition that you've checked is no longer true. And that's where flake comes in. It's in that moment where you lost control of the run loop that you don't know if it's actually still the case. And when you continue, the application is not quite the same, but you don't really know that. That's where you get these kind of weird problems where if you run the application on your computer, the test works perfectly because you have a really fast cpu. But then when you run it on CI because CI is much slower, your tests are now failing. And we've got this weird problem where you can't really test on CI. And so, and we're all familiar with that with these kind of problems. And it really comes down to, I think it's the proximity to which the test runs relative to the application. So the closer you can run those, if you can run in the same run loop, that's when you get that you get the ability to have really tight control and as a result, very, very reliable tests.

Nice. And I think this helps with. I think a lot of people that use Selenium are familiar with the stale reference error. So this would help with those type of issues.

It sounds like, yeah, it's very similar. We actually thought about how we could make this work with Selenium, but unfortunately the architecture of Selenium is not really compatible with this approach. That's where I think that the biggest value I think that we can bring to the QA world is that it feels to me like unfortunately there hasn't been like in a developer testing, even if you look at how much work has gone into perfecting the developer experience of using Jest, it's a really, really comfortable test runner, but the same kind of expectations. The reason why developers love Jest is because it just fits like a glove. But it hasn't really translated to QA world. And I think it's a real shame because those two groups, like as an industry, you're kind of missing out on the opportunity to bridge that gap. And so that's what I'm hoping that we'll be able to address by making the tool that matches kind of developers needs. But at the same time it provides the kind of confidence that QA expects.

Absolutely. Now I know originally Selenium was based like it was tied to the browser, like it ran inside the browser. And I think they went away now and they created WebDriver. And now Cypress came out, I think Selenium originally did. So it sounds like when you run the browser in that same instance, it gives you more control in that BigTest ad, then on top of that other functionality to help you control that the DOM at runtime that you lose out. Using a solution like Selenium.

Yeah, yeah. I would say that BigTest is kind of like a hybrid. It's kind of in the middle between Cypress and WebDriver in the sense that it's. So we're using a WebDriver and Victes has a, what we call agent architecture, which is actually something similar to Selenium. So the, the, we have the web agents. The web agent essentially runs. Right now we're using WebDriver to, to open web browsers. It's kind of like a level up of the way the Cypress works, but inside of the web driver. So we're using the webdriver to, to control the browsers. And what's good about Cypress is that it is a script that you load, it runs inside of the browser environment that I think is really valuable. So what we do, so we have an interpreter for tests. So when you write a test, that test actually right now you write JavaScript. But that can change because at the end of the day the result is a data structure. So we take the data structure which we load it into the browser and inside the browser we have an interpreter that takes the data structure and essentially runs through that data structure. We're executing every step inside of the environment. But one of the reasons why Selenium I think is really powerful is because this architecture is really powerful which allows you to be able to use Selenium with mobile applications, for example. Right. It's a similar idea. And I think with big tests we ultimately are going in the same direction where we will going to have agents that are native agents that are going to be essentially interpreters for test data structure that is going to execute natively inside of the mobile environment. And so it's going to give you the benefits that you get from the proximity of execution to the application while having the same kind of ergonomics and speed that you get when you're using the browser.

Nice. Now you talked a little bit about how it helps with waiting. So you know, waiting a lot of times cause flakiness, but there's other issues that also cause flakiness. So is there any other functionality that helps Eliminate the flakiness a lot of people see with automation.

Yeah, I think one of the challenges, I think at front side we talk a lot about having different knobs to turn that would allow you to adjust the size of the test. So when we, when we talk about the size of the test, so when we say like big test, which is the name of the framework, it refers to being able to run, to be able to increase the size of the test without increasing the cost of the test. Right. So for example, being able to render the application and be able to invoke the application, it increases. So if you're testing a single component, that is a smaller test. If you're able to render that component as part of the application, that is a bigger test. Now when there's another element to this, which is that makes it really flaky, which is what happens when you hit the HTTP layer. Right. So when you, when you start to handle with. So you know, we talk about like cross boundary communication when you go over to start, when you start talking to the server, that is another kind of aspect that tends to increase flakiness. But there are ways to manage that. The challenge is that a lot of those things are not really separable from the way that the application is implemented. And so this is where I think where it starts to kind of enter into the cloud native world. Because in the cloud native world you really, you can't. The separations that we had before, we have a developer team and a QA team and ops team, like all working separately. That separation doesn't really work anymore because to be able to go faster, you need to be able to leverage each other's work much more effectively. And so the way this starts to overlap is that there's certain things you can use, like you can actually architect your application differently to make the tests more reliable. And this is the thing that's usually not available to. When you think of QA and development being separate things, that knob is definitely not available to you. But one of the things we've been using a lot to mitigate that. So regardless, if you can't adjust your application, one thing you can adjust is you could actually simulate HTTP layer. When I say simulation, I mean having a high fidelity simulation of your server. There are tools like Mirage that allow you to essentially create a in browser server that simulates how your server will behave. Now you can make that test even bigger if you were to run that same simulation on the server. But now if you can control both your browser test and your simulation environment, the simulation of your server, then you have a lot of control over how your application is going to behave in very specific test scenarios. And you have a lot of power there. So being able to kind of adjust the knobs on these things and having ways to do this without being able to kind of introduce these tools and introduce these techniques without having to completely change all of your tooling is really what BigTest wants to explore and be able to, like, make it super accessible so that there's a simple path. Our team can say, like, I don't need right now, like, we don't need HTTP simulation right now. But it's something that I know if I wanted to plug it in, like, it's really easy for me to plug in, you know, so that's one of the really, like, you know, this is what kind of speaks to this idea of making it easy to expand the size of the test and adjust the size of the test without increasing the costs.

Now, besides that, does it also help with speed? Because another thing that developers tend to say is it's slowing down by checking code into CI. It runs these tests and the tests take too long. Does big test help with that issue as well?

Well, it's. Yeah, absolutely. I mean, so running tests in the browser itself, like, so running the test interpreter alongside the application, like, that's super fast, because that's the fastest way you could possibly run the application, because you're really essentially running it at the same speed your application runs. Unfortunately, most of the real slowdowns of applications, they're really outside of the test. So if your application is using, like, if you have a single page application that's using microservices and you have like 20 or 30 requests to render single screen, like, no matter how fast your automated testing framework, your tests are going to be slow. So being able to simulate those HTTP requests so that those HTTP requests are handled in a browser can be a way to control the speed of the test. Like, Viktest does offer ways to increase the tests, but we do need to really, if you want to think about increasing performance of tests, we need to think about the application and testing a lot more holistically.

So if someone's listening, they're like, this sounds great, but it sounds complicated. Or, I have a bunch of tests already. Like, do I need to start over again? Like, if you already have a lot of Jest test or Cypress tests, are you able to incorporate big tests fairly easily?

You can start using aspects of it. I mean, that's our goal is to be able to if you have a Jest test suite and if you're using something like we have one of the organizations called Folio which is a. It's an open source platform for managing libraries. They are using BigTest for, with jest and React testing library and they're essentially rendering the components React testing library and they're using interactors to interact with the component elements. Because interactors shipped together with their component library, you can use interactors separately. We are also working right now, we have a project happening right now to create a simulation platform that's going to work alongside with existing Cypress and Jest tooling. We're working on making pieces of this available, but the reality is that it is complicated and I think that that's something that it's just inevitable. Having a good test suite is fundamentally complicated and that's why we are working hard to find ways to make these things more accessible is because there's just built in complexity that needs to be understood well and it's really hard to understand because there's so many pieces involved.

So now this is a random question. I was in a conference and I kept hearing about Flutter, Flutter, Flutter and how hard it was to test. When I was looking over BigTest, I could have sworn and mentioned something about Flutter. Do you know anything about. Does BigTest help you test Flutter native apps?

Not at the moment, but we have. One of the nice things about being consulting company is that we work with companies who have these kind of big problems. And one the problems they have today is, is being able to test the Flutter application. It's actually on our roadmap. We haven't started the project yet, but we're essentially, essentially our goal is to create a Flutter agent. So the nice thing about Flutter is that because it has like, it has a lot of mechanisms available for us to do this. We just need to do the actual implementation. So. And we know what a Flutter test suite looks like without BigTest. So it's just a matter of writing the agent that will be able to interpret the test data and execute the test so we have a clear path of how we're going to get there. We just haven't started the actual work of writing that agent, but it's like itching to start on it now.

This is all open source and eventually I guess it's pretty early. I think you're still in alpha, is that correct? And eventually more users start using it. Given feedback, it's probably going to get even tighter.

Yeah, I mean it's an open source project. It is A I would say design of this thing has been ongoing for five years. The latest iteration of BigTest has been under development for the last two years. And it's both an ongoing research project and also ongoing development and also ongoing implementation, putting these things into practice with our clients and others. And so it's available open source and we want people to be able to use it, but we're thinking, try to be really responsible about how we communicate what's ready. Because I know, for example, like interactors, we have customers that are using it. It's a solution that works, it's in beta. API is fairly stable. We don't expect there to be very many changes, but it hasn't been like, it's not 1.0 yet, but we're getting there. The runner is a completely different state in that we know what we want it to be, but we haven't gotten to a point where it matches our expectations. And so it's an alpha right now. And for companies that want to do that, are interested in finding better ways to do testing, it's a good potential conversation to have about whether you can use BigTest. But there are pieces that are in different stages and I would encourage people to look at each aspect separately and see what value can provide to their particular application in isolation.

Yeah, so that was one of my tests. Like, if someone's listening, who do you think would be the perfect audience to start utilizing right away? Or so they. Is it just for research right now with potential to actually help expand their framework using BigTest?

Yeah, I think interactors are ready to use. I think there are some very interesting possibilities around accessibility. And we. One of the things that we're looking at with interactors right now is improving keyboard navigation. We want to make keyboard more similar to how people think about working with keyboard. Like when you're, you know, there is something strange about the way that we write tests because the people say, you know, like, you select the input field and you type, you know, you input into that field. But quite often, like people navigate with their keyboard. So we want to make a keyboard interactor that would allow people to have like, really granular control over how they input information into forms, and that will allow to do some really detailed accessibility testing. That's what we're thinking about right now. But working with interactors for just regular writing, regular tests, and interacting with existing elements of the application that's available, people should start using that today. And you could use that with Cypress Ingest. That's just like right there. And if it doesn't work for whatever reason. We have folks who are ready to take feedback and start to make it better. So it's a really good time to start using interactors in particular.

Very cool. Ricky Tars, before we go, is there one piece of actual advice you can give to someone to help them with their automation testing efforts? What's the best way to find or contact you?

Yeah, one thing that I think could be really helpful and that can apply to any test suite if you have to wait for something like if you have explicit wait, find some way to introduce convergence. So just some way of saying have some piece of code that says I'm going to check every 10 milliseconds to see if that thing showed up. Because that fixed wait period, that is a fixed cost. It's like going to pay for parking where you know you're only going to need to potentially only need five minutes where you have to pay for an hour. That's how that fixed wait works. So if you can introduce convergence to replace that wait for with a convergent behavior and there are some testing react library offers this where you could say like wait for this element to become visible. That is way better than wait for 300 milliseconds or 500 milliseconds. If you can do that, that in itself is going to bring value to your test weight just by adjusting that one thing. And to find me, I'm Taras on Twitter and Taras on GitHub and tarasrontsite.com on email. I talk about testing all the time and everyone on my team loves talking about testing. So anytime we could talk about testing, we're happy to.

Thanks again for your automation awesomeness. The links to evident value we covered in this episode. Head on over to testguild.com a343 and while you're there, make sure to click on the Try for Free Today link under the exclusive Sponsor section to learn all about Sauce Labs awesome products and services. And if the show has helped you in any way, why not rate it and review it in itunes? Reviews really help in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack automation awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Automation Podcast. Head on over to testguild.com for full show notes, amazing blog articles and online testing conferences. Don't forget to subscribe to the guild to continue your testing journey.