Scaling Mobile Testing Pipelines with Anton Malinski

In a land of testers far and widely journey seeking answers, seeking skills, seeking a better way O the Test Illumination Testing podcast Guiding testers with our nation awesomeness from ancient realms to modern days, they lead the way. Oh, the testicle Ocean Testing podcast.

Hey, is mobile CI CD pipeline slowing you down? Well, if so, you're in for a special treat because today's guest, Anton Molinski, co founder of Marathon Labs, shares harder and lessons from scaling, testing, infrastructure and why healthy CI growth isn't just about speed. It covers a lot in this episode. You want to miss it? Check it out. Hey, Anton, welcome to the Guild.

Thank you, Joe. Happy to be here.

Great, Great to have you. I've been speaking to a lot of people in the industry, are hearing a lot about what you all are doing there. I guess before we get into it, Anton, you know in your bio, your bio you sent me, you wore a lot of hats over your career. So I'm just wondering what led you to where you are now.

The initial vector, I guess, is I started as a mathematician, specifically the cryptography field. So, you know, very scientific, very hardcore and very deep. And I never expected to go into software engineering, but I started working on security solutions in general and started working around the software and was like, I'm going to poke around a little bit, I'm going to take a stab at it. And working on it, I started to feel like maybe it's actually interesting tinkering with this puzzle of how to build something. And it led me to kind of go into the mobile area where I started building prototypes for mobile phones that were getting very popular. So everything had to be on the mobile and spend a lot of time there. And you quickly realize as soon as you get to scale, like certain scale, that you can't just verify everything that you've already built before. So you inevitably end up in this space where you need to verify stuff. And I started building this thing for a growing team where we were having growing number of use cases, features, bug fixes. All of this needs to be balanced. And I was building the CI CD story like we need more hardware and then we need to write better tests and like trying to balance this all at the same time. So, and this is how I kind of ended it, I think. Previously it was DevOps. Now it's platform engineering, whatever you want to call this field. But it's basically everything around the code that ends up on the production that users are actually using. So everything around it.

Love it, love it. So why mobile testing? Why does it seem Like CI for mobile apps is there's a lot of friction compared to maybe a web app.

Yeah, well first I guess where the pain starts is usually development cycle for web is very different from mobile. On web you can deploy it in several minutes, it's going to be deployed on your cdn, whatever you're using, Akamai, Cloudflare, whatever. But for mobile every release really matters. It's like hours and days of trying to see if your release is going to be like canceled by Google, by Apple, something is wrong. And so you really need to make sure everything is really working perfectly. And ideally you also don't want the team to stop working. You know, you might have 10 people in the mobile, you might have 50 and all of a sudden if all of them stop working for release, it's not really a good thing for the business. You want to still continue working on the next features and bug fixes. So making sure that whatever you're like doing in your code base is up to kind of, I guess a certain level of quality is. It's a growing pain. Like you either like don't know about it yet or you already know about it and you're on top of this game.

So once again in the pre show you mentioned something about Agoda I think is the company.

Yes. So this is the company where I got most of the experience I guess with the ACI and the testing and specifically UI tests for. For mobile and building mobile farms. So I joined this company end of 2015 and I stopped working there around middle of 2019 so like three and a half years experience journey. I started working as a regular like software engineer and transitioned into this infrastructure person and the manager for the core operations team. And this is where I've built like mobile farms with like hundreds of devices like for Android kind of roughly I think around 80, 90 Mac minis that are running all of our workload and lots and lots of different just regular Linux servers running everything from like emulated space for example for emulated Android devices and all the microservices that are needed there.

Cool. Yeah. And I think you mentioned something about you had some sort of loop going on. Faster test runs, more tests written, more infra needed optimization. Can you walk us through maybe what that loop is?

Yeah, well I guess, God, the whole story you have to understand when I joined the verification story looked like you stop working as an engineer, everyone stops working. There's an Excel spreadsheet with a list of scenarios where everyone was sharing and you have a kind of pass or fail and for several days the whole work stops and release is cut out for this several days. No feature work, no bug fixes. Everyone just goes through these scenarios basically like monkeys. You want to optimize this. So this is where we started. And I remember we had like maybe a couple of hundred appium tests that were run kind of once a month, but if you run them once a month, they're all always like half read. So like you don't have any confidence or anything. And where we kind of left off, like there's a whole journey in between. Like we ended up with like every commit being tested in under 15 minutes, everything including regression for all the different configurations for Android and fior. So the loop of kind of development started as, you know, we have this pain. How do we solve this is an iterative process. This didn't happen in a month, it didn't happen in one year. It took us like three years realistically to go from A to B. And at first we tried to address the like the first kind of pains with, you know, trying to get dynamic CI runners. You know, we just need build machines to even start running tests. And if they're hard coded, like it doesn't really work out of the box. So you need like if you want 10 more, you have to get 10 more. And once you get this, you start running your test. Usually I think devices on those build agents, right, you spin up an emulator on the build on the build runner and that kind of works until probably like 150, 200 UI tests where it's growing to this time when like it just takes long enough and you don't have really good control over it so you want to run it faster. And we've found that adding more hardware to this, it just didn't help, right? Like you get more build agents, it doesn't really help. There is a nice trick around it that people usually use, you know, with sharding. So you have, for those who don't know, like sharding is basically a concept where you parallelize the load into like this bucket. So let's say you have 200 UI tests and you want to run them on like in parallel with four different runners. So you get like 50, 50, 50, 50, get like four different jobs, execute them in parallel. And that's kind of, you know, you get more hardware, you don't really get more hardware, you get more build agents. Like you really need more devices that you want to execute the tests on. And since you are using kind of build runners as your hardware, that's kind of how I guess like where this solution grows from. And yeah, so this point I think we're roughly at around like one hour mark, one hour and a half for running our tests. And we added more hardware and we hit this wall of like 55 minutes. Like we give it like twice more hardware. It's still like not faster than 55 minutes. And we figured out that this is not a problem at all in the horizontal scaling, it's something else is broken there. And we were kind of concurrently figuring out how to write better tests and everything. But the main bottleneck there was around kind of the orchestration of running this. And specifically this is like test runner space where uh, it's a fancy way of saying scheduling algorithm, you know, like scheduler that tries to schedule tasks on your CPU for the operating system. Same concept. Just like you're scheduling tests for execution on lots of different devices or emulators, simulators, real devices. And we try to like work at that time. I think we experiment with a composer from Juno, with Fork, with Spoon. We kind of modified them for several months and, and it just didn't work for us. Like it, it was still kind of hitting exactly the same limitation. And the nature of it is everything in UI testing space and in general mobile testing space is super flaky. Like cables break, you know, batteries die. There's just something, some new update that's coming from you know, manufacturer and it has a pop up screen I remember like in our farm and it just like stops everything. You can't click on it through like any kind of software method. You have to physically go there and disable it so that because the manufacturer obviously wants you to update the news firmware, but you're actually running a 24, 7 service for everyone. So competing priorities. Yeah. And then we kind of worked on this, like tried to sketch out a prototype for something that could solve our problems. And the main two things that we were trying to address there were around the better kind of load balancing and parallelization and then also kind of addressing real failures. So kind of prepare like not hoping for everything to succeed magically. Like we just prepared that like everything will fail, devices will fail, test execution will fail, their tests are going to be broken. We just put it in the design. We have to work around this. And we basically came up with like statistical modeling and put this into the software. And I remember our first test run that we started, it was already like running everything in 15 minutes from like 55 that we just couldn't break break with adding More hardware, It's just impossible. So we found this spot where like what the bottleneck was for optimization of this particular problem and kind of we released this then I remember it was 2018. I released this as an open source project called Marathon and we continued kind of adoption of it internally and we got into the state where developers were happy with the solution. So developers are like, oh, I can actually trust my test because like every commit that I'm running right now is executing all of those tests. So like all of them are green, so I can trust them. And that means I can write more tests. And then in the next year our tests skyrocketed from 200 to like 1200 for both platforms. So this is like both we used native frameworks, so like XE UI test for iOS and espresso with a little bit of Uiautomator for Android. And the number of tests skyrocketed, you obviously find more issues and you solve them. But this phenomenon that I'd never expected, that trying to optimize the execution only puts more load on. You're going to have more test cases after that. And that's kind of the barrier, I think in general the 200, 300 test cases where I think a lot of companies are struggling.

So you know, you mentioned different frameworks. Does the framework you use matter? Like if you use Appium as opposed to native, does that impact how you're going to scale or is it irrelevant? You still hit these bottlenecks?

I'll start the answer to this question from the technical side. Appium is nothing more than the same espresso and UiAutomator and XCE UI tests. So Appium is just an HTTP server that is built around exactly those same APIs. So whenever you're running Appium you just have more layers of interfaces, but to the exactly the same APIs as you would use in native testing. So as any system that is like, you know, simple system is usually more reliable, like the more complex the system is, the more failure points you're going to have. So Appium from that perspective is usually much harder to achieve stable execution. It's not unheard of. I've seen companies that have like successfully implemented like thousands of IPM tests, but it is definitely a much harder effort compared to native test. And even between native test platforms for Android and iOS, the resource you like, you know those like usually your test case, what you're going to find an element by some ID or by text, you're going to like do some assertion, like click on something, interact with it. So generally iOS and XE UI test is 50% slower than Android. So at Agoda, I remember we standardized the format for all the test cases. So we kind of first had a specification, we used YAML for it, but it doesn't really matter, so it can be anything. So we first specified whatever the test case should be and then engineers took it for the actual execution and implemented it in the actual native framework using page objects and everything. But it turned out from the hardware resource planning perspective that exactly same cases take like 50% more on iOS, which was a surprise to me. I expected both native frameworks to be exactly as performant. And then if you add Appium on top of all of this, it's just gonna be slower and less reliable. So that means you're gonna spend more hardware on it and effort as well.

Nice. So you do have a background in mathematics. Are there any real world metrics you can or you rely on to let people know your CI growth is healthy or it's going to hit maybe a bottleneck?

Yes. So I guess in terms of metrics you can kind of see. So from hardware perspective and in general infrastructure, I would call the first important metric situation from the four golden metrics. And in this specific case, like situation can be queue time and the duration of your verification or the build. You know, like how long does it take for you to execute your tests and how much are you waiting in queue until your tests even start executing? It doesn't mean that your queue time has to always be zero, it's just impossible by nature because any company, it doesn't have a normal distribution of everyone working around the clock. Usually you clock in, you start writing some code, you push something, you test, then it's lunchtime and then maybe at the end of the day you're going to push your last commit and you actually want to test the final things. So usually you have those spikes in the load and that's where the queue times that starts to be, you know, non zero. It's going to be like 15 minutes, 20 minutes. But unless you have lots of money or you don't want to set a goal of queue time being zero, it should be something realistic, you know, like 10 minutes. It's not the end of the world, but it shouldn't be, you know, throughout the whole day. Like you should have times where you really have resources for running something and for testing duration, at least in Agoda and like, and later on, like with just marathon as a project in general, like through customer interviews, What I found is 50 minutes to 30 minutes is generally like psychologically okay for engineers to spend on waiting until their code change is verified. So anything like you can go for a coffee, come back and you still be in the context of whatever you were doing. If this green or red check in the CI is taking longer than 30 minutes, you probably are going to pick up some other task. Right? Like realistically you need to work on something so you're going to switch context and whenever that you actually get the result, you already forgot what you were working on. So like being in this flow state for an engineer is really important. So this time to feedback, this is where it starts to kind of accumulate. And especially if your checks take, I've seen teams where it takes like two, three hours. Like you start to pick not one additional task, you start to work on three of them and juggle all of them. And it's a pretty hard way to work, in my opinion.

Oh yeah. Even with vibe coding, I can imagine now it's even worse. You know, 30 minutes, people are already writing three new features, you know.

Yes.

Cool. So, you know, a lot of times when we talk about development, we talk about shift left and I assume it might be easier with web than it is mobile. Is it realistic to shift left when you're doing mobile type of development?

Yeah. So shift left, like, I guess like the whole phenomenon here, like is shift left can be described from production to staging environments, from staging environments to CI. And I guess we're focusing more on this like CI to local development loop. Right. So for like when building mobile infrastructure or a mobile platform, ideally you will like whenever you say you run your test cases in CI and something failed, you don't really want to people like have to like run it on very different devices. You want to build this thing as a proper service where like something failed on a device with a configuration, let's say it was a 5 inch screen, this specific DPI, this setting, like this specific version of the operating system. And you have exactly the same API on your development machine. You can use exactly the same CLI tool, for example, or like, you know, web UI and request exactly the same device. If it's a real device, you can get exactly the same device where it failed. And this way you can kind of shift left the signal from CI to your local execution. I don't really like the pattern of trying to push for a signal. Like if you can run something from your local and there is no real difference between CI and your local execution at the end of the day. So like If I want to check something, if it's working end to end, I'm just like my ideal user experience, I have a button, it just runs everything as fast as possible without any flakiness, without any kind of something unstable and just gives me a signal, does it still work or not? And in my opinion it's definitely possible, but for 99% of cases. So there are those things where you really need a specific hardware. So what I mean by this is we obviously have to talk about, you know, emulated and simulated devices versus real ones. So for simulated emulated ones you can spin them as much as possible, you can scale them, you can buy more servers for spinning up Android emulators. You know, buy Apple McLinees for running your simulators, that's very cheap. But if you want like rare devices, let's say you want some Chinese device without Google Play services, you're not going to buy hundreds of them and you're not going to buy like them for like scaled execution so that everyone can run those, all of those tests in like 15 minutes. And what I mean by this, so in our case, for example, in Agoda, we're running around 24, 25 hours of tests in 15 minutes. And just simple math, like how many devices do you need at minimum to even achieve this? It's around 50, right? So like it's around 50 devices is the minimum you need. Like and that's if you have multiple people doing this like concurrently, 150 devices of the same model, it's unpractical and not from the perspective of it's a once off investment. It's the problem of your users are changing devices. Whatever was popular one year ago, you bought those like, let's say you bought those 150 devices. You check it now, like your top 15 devices that your users are using, they're different, you need to buy new ones. And like what are you going to with the old ones? Like are you going to recycle them? Like what's going. And in a year you're going to do it again. It's unpractical to use this at scale in my opinion. It's like when the proper process here I think is when you see an issue and usually you see issues on real devices first you try to really replicate it on a simulator and emulator. There are lots of APIs available now. There is like anything around network shaping, packet delays, throttling, you have all the access to the camera, including faking video streams, you can mock gps. There's so many Things that you can do with sensors, even like for an accelerometer and try to reproduce it on an emulator and simulator if it's at all not possible. And there are cases like this, of course shift it to the real devices and that's where kind of balancing between those like real devices and virtual devices comes in. So how many things can you realistically reproduce on emulated devices? If most of them you can do, if you. It really depends on your application. Some applications are just simple crud things around API. They don't really use any specific sensors, they don't need like low level access to the hardware. So most of them can be realistically done on an emulator. But if you're doing something very, very interesting. I remember I was working like on one app, it was called book reader, called BookMate. And we were partnering with this company which was building this very interesting phone with an E ink screen on the back of it. So kind of front face was lcd on the back you had E Ink screen. And you can't emulate this like whatever you do, like it's just not possible. You're going to have to test on an actual device and there's going to be some proprietary API. So there's only one way to develop this. Like you can try to abstract everything behind interfaces, but there are those cases where this really is just a requirement to use real devices. But again, you want to try to strive for using emulated devices because those ones are easier to maintain, they're cheaper to support, they are updatable. Like whenever the new version of the operating system comes in, you can just update the package and deploy the new version of the simulator and emulator. But with real devices, whenever someone Even says Samsung S25 or something, which specific version of this device does it mean? Because they are varied between different markets even they might have a very different silicon chip there's inside, they might have very different like modem chip, they might have very different firmware version. So which specific device are we talking about? And even in that case of updating firmware that I mentioned before, like do you want to update your firmware? Because some of your users will update the firmware and some of your users are going to say no to this firmware. So whenever you have a crash from a specific device, like do you want to store all the different combinations of firmware that you just want to have at hand to reproduce the issues? So it's a very hard challenge.

Yeah, I was gonna say it sounds almost impossible to plan even if you do as best you can. Once you're in the wild, all bets are off. But mathematically, there has to be some way you can kind of have confidence in knowing you at least have like.

You know, well, you try to do you the best with what you can. So you're always limited by the resources. In this case, it's so even just the basic budget, like, how much hardware can you buy? How much hardware can you afford? So when realistically the first constraint is this one and the second one is like, how many problems do you want to fix for the users? You're never gonna achieve perfect software. It's impossible. There is no such thing. If we had an algorithm to do this, like, everyone would already be using it. So you want to try to do as best as possible for the majority of your users. And this is where it's super important to collect what your users are actually using for, you know, for your application. Maybe it's a very niche application installed for some market, and only like a certain number of devices are popular there. And this is where kind of trying to find a vendor who is going to give you those devices, like rent them out, you know, like browser stack or anything like this, or if you want to buy them, comes into play because you really want specific devices. Otherwise you're testing your app on like basically minority of the users and you want to make sure that at least the majority of the users have a good experience.

Totally. So, you know, I wrote my notes, I forgot to follow up to you. You didn't mention Marathon Open Source. Maybe a little bit more about what is Marathon Open Source? Because I kind of skipped over it without following up.

Yeah, that's all right. So Marathon is. So when like started as this orchestration and test runner in Agoda, we were trying to kind of parallelize everything as efficiently as possible and try to handle flakiness as well. So Marathon Open Source project is basically this test runner. It doesn't have any good UI or anything. It's just this component is in this very large puzzle of trying to solve, testing at scale and basically trying to. Because the ideal user experience, you have a button and you just have a result in this very short amount of time. Right? So Marathon is just a test runner for Android and for iOS that works with basically like real devices and simulated ones. And you just give it your application, give it hardware, and there is lots of different configuration that you can do there in terms of the execution. But basically it tries to balance everything as much as possible. So if devices, you know, dive God forbid. Or maybe they disconnect. It just works around this. The test run never stops. And it works in a very different way to conventional test runners, so to say, because most of the conventional test runners, they preplan everything. Let's say the test runner sees five devices, it's going to split all the tests into five buckets and you know, just continue executing until it's done. Marathon does it very, very differently. It doesn't expect the devices to even be there. It's just like whenever the device is free, it figures out what is the best statistically correct thing to do right now in terms of probabilities and encoding the things around like tests failing. So two biggest things around test run that are usually unaccounted for a test duration. So usually there is an assumption that all tests take exactly the same amount of time. So like when you chart for example into four different buckets, you assume that 50, 50, 50, 50 is exactly the same duration. No, because some of your tests are going to take one second for like checking your one widget, maybe you created one widget, make a screenshot, screenshot comparison, next test. But some of your tests are going to be scenario tests that are going to be like very long. It's going to be the include oauth. It's going to go like, you know, you're going to make some, you know, orders, you're going to go into your like basket, you know, go order, check. It's going to take minutes sometimes. So you need to account for those things. And the test runner has to be has to understand that this test, for example, takes longer duration. So I'm going to schedule only one for this device. But these tests, for example, are very short and I'm going to schedule them on a completely different device and altogether batch them. So this is one thing that accounts for. And the second one is just how flaky are the tests in general. So if it sees, for example, a test that has been stable for the last, it's always configurable. Let's say one week, it's not going to schedule additional retries. But if it sees that the test has been always flaky like it still passes. And most teams are very comfortable with the fact that it's. If one of the retries passes, the team still considers that the test is passing. Right. So we're in this mode where we're happy to pay additional cost for still having the test passing. And yeah, it just calculates the probabilities and figures out, oh, I need like three Retries so that I can make this test like at least one of them being, you know, green. But there's also another mode, like, if you want to fix that flakiness, it still sells all the metrics to like a metrics database. And you can see, like, which tests are flaky. For example, in Agoda we implemented the process where every week, like the team lead or dev manager received a report for like your team has like this statistic for the flakiness of your tests. And you probably want to fix those because they are really affecting everyone. The problem with the flaky test, it's not that like only specific team is affected. If you're running tests at scale, one test can basically make a bad day for every single developer because everyone is running everything. And this is where kind of having that statistic is very useful. So like, you can schedule preventative retries rather than, you know, running the test. It fails, I'm going to run another attempt. It just figures out I need three of them so that like one of them can actually pass and schedules them concurrently. And then if you have enough devices, you can already get the same result by still paying kind of roughly the same cost, but much, much faster. And you can also implement runs for checking flakiness. So when you fix flakiness, for example, how do you really know that you fixed it? I run it locally and it's green. Did you actually fix it? You really need those numbers for the probabilities. For example, the probability of passing was 0.7 and I want to get it to 0.9. It means that I need to run this test a thousand times on different devices, hopefully concurrently, and then check the actual statistic for my fix. Like, did it improve the statistic or not? And this is what Marathon Open Source project also allows you to do. So it's kind of like this engine of scheduling test execution as efficiently as possible for at scale. So where, like real problems happen, there's always fakiness. It's always going to be there like it was there in the beginning of time. I believe it's going to be the end. So you can only manage it and try to do the best the best you can, you know, and as much resources as possible that you can put it in.

Nice. So I first heard about you all from, I think you know Igor from Ingenious IO. He raves about how fast this is. So what's the other aspect of Marathon? I know you have the open source, but I believe you probably have another business that also does Maybe more of paid option.

Yeah. So kind of context there is Marathon open source started 8 years, around 8 years ago and like for the past around 3 years I guess we've been building this cloud offering. So the whole idea is like Marathon is just the engine. The thing that I wanted to build originally was this user experience of I just send my application as many tests as possible and I want to receive the signal as fast as possible by signal being what's actually failing and I want the signal to be reliable. Unfortunately it's not possible to do this just with the open source offering. You really need this knowledge around infrastructure like and how to tie these pieces together and have a process around infrastructure. There's always this new updates coming from WWTC and they're always breaking stuff around like all of this stack unfortunately. And what we're building right now is basically this user experience where you just provide us the binaries and we give you the results in under 15 minutes. So it does involve much more than the open source offering, but I know it's pretty popular. The open source part is very popular in the community. So lots of teams are using this. I think the largest number of tests in one run I heard was around nine and a half thousand, which I was very surprised to hear. Like I didn't know the teams were actually building like this, such large test suits. But then it depends on like what you're building and what the business wants in terms of reliability, you know.

Awesome, awesome. I hate to bring it up because a lot of people are sick of it, but AI. I hear about AI a lot with web. Is there any features or benefits of using some sort of AI LLM to help in mobile testing at all?

So it's very controversial, I guess. My personal opinion, I don't see yet any good place to put AI in in the mobile or even development. So my reasoning here is like if you're just starting out, it's much better to read the resources that are prepared like for everyone for the last several decades. Like they are targeted for people by people, they are handcrafted. AI is useful to get something very quickly out the door, but you really need to have a certain kind of experience, maybe like at least 10 years doing exactly the same thing and then this thing can prototype something for you and it's going to be much better than starting from scratch for sure. But you're still going to spend time on it. So I guess for experienced people, yeah, I can see some value in it. But the first place that comes to mind is probably writing tests like instead of, you know, like getting all, for example, the setup for the test, you know, setting up expectations for the mock server, setting up expectations on like preferences in your app and et cetera. But at the end of the day we, the approach before AI was like, I'm going to write my tests in a way where like I can do this in one single line of code. You know, put a junit row here. I'm going to have like some specific functions to set up my database. So whatever you're putting into the AI as a query as you like, what you want to achieve was probably already done with some framework for testing realistically. So I haven't seen yet anything that looks really interesting to me at least.

Okay, Anton, before we go, is there one piece of actual advice you can give to someone today to help scale their mobile CI CD pipelines? What would it be? And the second part is what's the best way to find or learn more about Marathon Labs?

Yeah, so I guess advices here first really set up the expectations like verification and quality. Space is really problematic because I believe like the only way to really achieve this, and you've probably heard this from someone on one of the podcasts, is there's this Kaizen approach to quality and all it boils down to is like, everyone really needs to chip in on this to succeed. It's a long term game. It's never like it doesn't take weeks, it doesn't take months, it takes years to get to the next level. And quality for almost every team and you really need to buy in an alignment from qa, from engineers, from product, from senior management that they like the team is going to be slower because they're going to do work on the quality. So first thing first, setup expectations. This is going to take a long time and if everyone buys in, then proceed with it. If someone doesn't really want to do this, I don't believe it's worth it. Like just try to continue where you are. It's you're gonna do basically go uphill with a large stone. So then try the second, I guess thing that I want to share is don't try to reinvent the rule here. I've been this guy and like I, I created Marathon because of the necessity. Like I didn't create it because like some there was. I just wanted to create a new project. So try to reuse whatever is there on the market. And I don't mean Marathon here. Like whatever you see as a solution, like any product, you know Try to work with the vendor, try to work with people who actually have like financial alignment to the product for the product to succeed. Because usually internal platform teams, they like, they do not have any effect on the business. Their impact is transitive. They're only helping developers. So the business is usually never interested in improving the platform. For example, like AWS does, like AWS sells their platform and hence they're really motivated for the platform to be very, of very high quality. If you're going to do something internally at best it's going to be some hacky prototype unless you spend the years on it. But where are you going to get the budget? So try to reuse whatever is there already on the market. For example, if you're doing mobile farms, there is the Successor of the OpenSTF Open Source Project. Device farmers start with that. Try to use native tooling but God forbid, don't go into this rabbit hole. Yeah, and also use solutions that are appropriate for your team size and the project. So if you're just starting out, if you have two developers and you have a very simple web, don't use any farms. Just go and spin up an emulator simulator on your build runner. It's going to serve you for the next year and a half. You don't need complex projects in the beginning. Maybe you want to think about the process of how you rule and write tests at that point. But you don't need complex solutions in the beginning. You can think about them maybe when your team is like maybe 10 people, 15. So use an appropriate solution for your scale. And regarding Marathon Labs, I think our main website, you can go to our main website marathonlabs IO like there is, learn about our business, learn about our product and there's also a link there to the open source version which you can just try it out, give us feedback as always and I'm staying always active for the community. So go to my Twitter, maybe we'll get a post link somewhere. I don't even remember the link to be fair. So reach out to me.

We'll have links to all this awesomeness in the comments down below. Thanks again for your automation awesomeness. The links to everything of value we covered in this episode. Head on over to test guild.com forward/a560 and if the show has helped you in any way, why not rate it and review it in itunes? Reviews really help in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe My mission is help you succeed with creating end to end full stack automation awesomeness. As always, test everything and keep the good. Cheers. Hey thank you for tuning in. It's incredible to connect with close to 400,000 followers across all our platforms and over 40,000 email subscribers who are at the forefront of automation testing and DevOps. If you haven't yet, join our vibrant community@test guild.com where you become part of our elite circle driving innovation in software testing and automation. And if you're a tool provider or have a service looking to empower our guild with solutions that elevate skills and tackle real world challenges, we're excited to collaborate. Visit testguild.info to explore how we can create transformative experiences together. Let's push the boundaries of what we can achieve.

With lutes and lyres the bards began their song A tune of knowledge, a melody of code through the air it spread like wildfire through the land Guiding tester showing the secrets to behold.