AltUnity Tester with Ru Cindrea

Welcome to the Test Guild Automation Podcast where we all get together to learn more about automation and software testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Automation Podcast. Today we'll be talking with Ru all about Alt Unity Tester and mobile automation for games. Ru is a senior test Consultant and management partner at altum. With over 17 years of experience, she believes all test is exploratory and applies this mindset to many complex testing challenges. When building test automation frameworks. Her focus has mostly been on testing mobile applications and embedded systems as well as automating game testing where she is one of the main developers of the open source Alt Unity tester tool. I'm really excited to dive into this tool. I think it's going to help a lot of folks. She has been also a BBST instructor for over 10 years and she's been working on developing the BBST materials and practical assignments that draw from her past project experiences. And she's also an entreprene that I'd like to touch on as well. Really excited to have her on the show. You don't want to miss this episode. Check it out.

The Test Guild Automation Podcast is sponsored by the fantastic folks at Sauce Labs. Their cloud based test platform helps ensure your favorite mobile apps and websites work flawlessly on every browser, operating system and device. Get a free trial. Visit testguild.comSourceLabs and click on the exclusive sponsor section to try it for free for 14 days. Check it out.

Hey Ru, welcome to the Guild.

Hey Joe, thanks for having me.

Awesome. Awesome to have you. We first spoke at Future of Testing, the applitools event and your presentation really blew me away. It's something I get asked about a lot is how do I automate games and complex applications. I think you have a really cool solution that we're going to dive into. But before we do, is there anything I missed in your bio that you want the Guild to know more about?

No, I think you covered it pretty well. That's me.

Cool. Cool. All right, so I always like speaking to entrepreneurs, especially tester entrepreneurs. So I'm curious to know why, like how did you start Altum? What was the focus? How did that happen?

So we started Altum basically because me and my other two main partners from the company, we were all three of us working as testers and really good friends. So we ended up saying, okay, we should probably join forces and do something together. Even if in the beginning we were thinking we would continue being freelancers. But it turned out we worked well together and it's now 13 years, I think since we started Altom and it's been one of the best decisions of my life to start the company and I feel like every day I get a chance to work on something that I love doing.

Awesome. So cool. So Ru, I'm just curious to know how did you get into automation games and is that the specialty of Altum is that just happens to be something that you happen to specialize in within everything else you offer.

So Altum we are really specializing in anything testing related software testing. I myself have started my career testing mobile apps, so it kind of stuck with me for a long time. I've been doing mobile testing in general for quite a few years and of course alongside just exploratory testing, I also started doing automation. And the introduction to the games industry started from project in Helsinki, Finland where I'm based right now when I was working with a company. It was then called bitbar, now it's Smart Bear. Since then they've been acquired by SmartBear, but they had a service called, I think it's still called Bitbar Cloud, where you could access Android and iOS devices in the cloud and run automated tests on them. But the project that I joined specifically to help them with was one for a company developing mobile games. And this was kind of my introduction to the game world for a long time stayed on just like the mobile games side, there's also quite a big market in Finland and in Helsinki for that. There's big companies, there's quite a good environment for working in that space. But that was also my introduction to Unity basically. And afterwards I started working with Unity not just for games, also for just developing applications. And this was specifically the point when I realized that there's a pretty big gap in the market when it comes to automation. You know, coming from a world of mobile testing where we have Appium and of course there's always some web service and you. I've always taken webdriver for granted and have been using it for quite a long time. Coming to the game or specifically the Unity world, it was quite surprising to find out that there's nothing that would offer the same type of end to end automation that we are so used to in other environments.

Why is that? I know you said Unity usually is used for game development, but I've heard you just mentioned as well that it's not only games games, you can actually do other types of development. So why do you think there was such a lack of tooling around it.

I don't want to say that there's necessarily a lack of tooling. I think Unity is doing a good job generally for unit testing and integration testing. But this is of course speculating and my own experience with different companies that we've worked with. But I think a lot of the times the state of the gaming industry has been such that the testing has always been done manually. A lot of companies, they're very small, so they don't have actual resources to invest in just testing. It's usually the developers or the users that do the testing. And big companies, I think, very often have entire teams of testers that have been doing game testing for a very long time manually. And it's still a shift that needs to happen. I think it will happen, but it's just, I don't know, maybe kind of like there was never a need in some cases for automation because there was so much manual testing being done anyway.

Absolutely. I've spoken with developers and a lot of times of games and they say, well, why would I write an automated test? Usually it's the software is released and I don't even have to worry about it anymore. I don't know if that's still the case. Maybe because it's a mobile app, maybe they do updates or what's the use case for needing to have automation tests then? For games?

I think, yeah, of course there's also always these titles that you release them once and never update them. But I think especially in the mobile game space, we are so used to weekly updates. There's a lot of games that follow. For example, you know, famous TV shows, they come up with new content every week, they update their graphics. There's this constant development, it's just like any other software development project, but maybe even updates more often than some web service or something like that, because they do try to keep up with the weekly whatever releases that come on the topic of the game. So I think that's one of the things that is shifting as well, the fact that there's a lot more constant updates coming. I think also on the mobile side, something that drives automation more than maybe on the console side is the diversity of devices and screen resolutions and hardware capabilities and so on. So that's a big kind of sign that you should try to run at least some basic smoke tests on as many devices as possible and automate those when you get to the fragmentation problem on the mobile space. So yeah, I think it is changing a bit. At least that's my hope.

Nice. So you didn't mention fragmentation is actually one of my next questions. You actually brought up a comment, I don't know if you remember during applitools future of testing your session. You said, you said testing an app was a little bit even worse when testing a mobile app. So things that you had issues with with testing an app were kind of almost even worse with a gaming mobile app. I think I'm getting that right. I'm not sure. And you called out fragmentation. So I guess a little more, can you go a little bit more into what is fragmentation? Why is it a little more difficult with gaming apps?

So yeah, so the concept of fragmentation is the fact that you are trying to serve the same build or the same application, the same binary. Trying to serve a lot of devices with very different configurations, whether it's the screen size or the hardware, the memory and the general performance of the phone. If we think of low end Android phones and high end Android phones, they're like two different products altogether, but you're still trying to get the same game to run on both of them. And of course, yeah, this is a problem for mobile apps as well. I think it's worse for games for a few reasons. One of them is a lot of games are targeting. You can't say, okay, I'm going to focus on what's popular in North America. You typically try to focus on the entire world and there might be big differences in like some device models are not even available in the country where you're based compared to other continents. That's one thing. Also, I think very often apps are targeted to a smaller demographic. You are or, you know, many apps. Of course apps like Facebook try to get to everyone. But you know, with some of them you really target people that are tech savvy maybe, or people that you can assume have the latest phones and so on. But with games, a lot of your target audience is going to be kids that will most likely have, you know, not the latest model but the previous one that they get from their older brother or from their parent and then so on. That's again worse. And then also games are very resource consuming. They usually require a lot of memory, good processors and so on. And the screen real estate is also extremely important. So if you have the game needs be just as fair to play on an iPad as it is on a smaller screen Android phone and trying to deal with that with the same application is something that becomes very, very challenging.

It's an interesting point you made about a lot of times kids play it and then it's usually on an older device. I've actually seen this with my nephews and nieces. It's kind of funny. So I guess that brings up performance. As you said, some of these are lower end things. So is automation, do you consider performance part of the automation piece? Is that something separate?

I would consider it something that you can at least measure through automation, especially if you're trying to cover a lot of ground in terms of number of devices and different configurations. I would think that automation in this sense is more okay. I want to be able to run the games and collect information about the performance of the system while the game is running. Or maybe frames per second for the game. Not necessarily in the sense of I want to create a load on the server that's running the game in the cloud or if it's a multiplayer game, for example. So more like performance on the local device that the game is running on is something that's very useful to have because performance in terms of response times and so on from the server, I think that's better done from just APIs rather than by playing the game whether it's through automation or not.

So are there any APIs that are available? I know a lot of times people focus in on UI testing, but with games, is there ever like usually it's easier to automate sometimes because there is an API or is that not the case?

I mean, of course it depends from game to game, but if I'm thinking in terms of if it is a game that connects to a server and we're trying to figure out how does that server perform, I think it's typically easier to just call the APIs that the game calls as well instead of trying to play the game to call them. But it depends from game to game, I think.

And also I guess it depends from game to game. But in general it seems like. I think you mentioned this logic tends to be more complicated with game apps. It's so open ended. It's not like a medical device where you know you have to do steps one through five. I mean the game, you could do anything, right? So how do you handle that? I guess sometimes people look at that and say this is even automatable because who knows what the heck the player is going to do.

Again, depending on the game. But some of them are actually extremely complex. With some of them you need to build a lot of the game world and the character stats and so on before you can even reach certain aspects of the gameplay. So that's one thing I think a lot of Companies solve that by building in some hooks that allow you to just load a specific level or load the game that is in a specific state through kind of like debug functions, let's say, or testability functions, if we want to call them. But then on the other hand, there's also the. So the complexity is one thing, but then a lot of games have a lot of randomness, which is something that's very, very difficult to deal with in automation. That makes it, you know, sometimes you have to be prepared for anything and that makes the script so complicated that, that they might not be reliable enough sometimes. But at least what I found is that we are still at a point where we'd be much better off if we simply had scripts that can start the game, play the tutorial, do the first steps to see that the game functions and performs okay on a specific device. Even that is complicated for certain platforms and certain game engines and so on. So there's a big step from there to doing real kind of like scenario testing. But that's also a possibility. Yeah, nice.

I want to dive into the automation piece of it. I think in your future of testing applitools session, you talked about pretty much three approaches, or not kind of three approaches, but three type of things. People tend to focus in on you progress. So the first one is Unity has a test framework built into it, so I assume that's where most people start. What does that offer and maybe what are some drawbacks to that?

Yeah, so they have the Unity test framework is something that a lot of developers write their unit tests in and the integration tests with. I think it's already quite a well developed practice to have that. But it's very tight to the Unity editor environment and ecosystem. So you're not going to run the game on the target platform, you're not going to get the APK for Android. You're still going to be testing inside the Unity editor. That's a great tool, but it also means like integration testing in games or in the context of the Unity test framework means more like you'd be building. You would instantiate a game object that's your character and then you instantiate something that's a wall in the game and then you try to move the character programmatically from this coordinates to a coordinate that's behind the wall and you'd see that the player or the character actually produces a collision event with the wall to test the integration between the wall and the character. So it's something that's where there's A lot of setup where you're very in control of the game world, but you're not really maybe testing the game.

Kind of in the state that the user would have it when they're just playing the game. So for example, Unity has a game like a loop that's always running, that's playing the frames with a certain frequency. You're in control of that. When you're doing integration testing, you can stop it, you can say, okay, now continue, now stop. Let me set up something so that makes it a bit different from the end to end or the end user experience. And that's where I think the Unity test framework is lacking on what I would call testing on any of the platforms that you support. Like the user would experience the game after that.

You mentioned a lot of people then go, well, obviously let me try Appium then, because Appium is made for mobile. And then they throw in some open. I think you said image recognition. So talk a little bit about that approach and maybe how far it can get you and then what some downfalls might be to that.

Yeah, so this was one of the first things I tried in my first kind of encounter with mobile gaming world. Yeah, of course, when you try Appium, it's like in the old days. You try to test a Flash application in a browser using Selenium. You don't see anything, any of the elements that are within the canvas from Unity. It's just one painted black box, basically. So you don't have any insight with Appium in the Unity hierarchy of game objects and so on. So a lot of people tried just basically starting up the game as usual app with Appium and then clicking at coordinates, which is of course not great, especially if you're trying to cover many resolutions and different devices. So that becomes quite pointless. But adding image recognition to that approach makes it, you know, kind of already useful because you can. Well, we developed a framework at that time working with Bitbar, where you could use OpenCV to. We would take a screenshot of the device screen with Appium and then we'd define these elements from the game that we were interested in. And with OpenCV we would try to. To find them on the screenshot from the device and assert, okay, the player is here, the enemy is there, and so on. So you could assert that things are happening in the game and you could also, for example, click on the enemy to attack it with your character because you knew exactly the coordinates of it on the screen. That was quite a, in a way, very simple approach because it Works on any platform. You don't need anything special done with the game. The only thing you need to be able to do is take a screenshot from the device and input touch events at coordinates, which Appium does. But the problem with that was that it only works for certain games. Or like, for example, if it's a game where speed is very important, then image recognition is always going to fail because it always takes a long time, especially talking about screenshots coming from a big tablet. It takes a lot of time to process that and try to find something on the screenshot and then go back to the game and click. By then everything has changed so much that maybe the character is not even in the scene anymore. So it wasn't good for anything like that. Also, while image recognition always works up to a point, there's always things are overlapping, the player is there, but it's just behind the wall a little bit and you can't see it and so on. So that doesn't work for anything. It works for simple scenarios and it worked quite nicely. But I think, yeah, my next step after that was also because then I started working with a project where we used Unity for apps where the graphics were not so rich, there was a lot of text. Image recognition doesn't work well for that. So I was already trying to think of trying to do OCR and so on, but that becomes again, same problems. But what I was really looking for was the ability to kind of interrogate the game and interact with the game objects in the game the same way you do with Selenium and WebDriver in a web browser with the DOM where you can look for elements and once you've located it, you can interact with it or get its properties, set its properties and so on.

I was going to say, I guess that's when altunity Tester was born. Where did you have that epiphany like, hey, actually I can get that information. How did you know I could throw. How did that even work? How did that even come to your mind? Because it seems like a pretty cool hack, a workaround to do what you did.

There were two things that happened. First, I started working with Unity a lot more and understanding just how the dynamics of the game objects and the hierarchy and how that works. And I realized that once you have a game object in the scene or in the level that's currently running, that game object knows everything about all of the other ones in the hierarchy. Which meant that if I could have a game object that I can talk to from Outside then I could just do everything that I can do now with WebDriver. So I was the first just basically understanding the Unity world, that was one thing. And then at the same time I talked to someone I can remember it was a meetup that we had in Helsinki, like a testing meetup where we were talking about with someone who was also using Unity for testing Unity apps. I think it was as well. And they mentioned that they had seen on GitHub a project that was adding a game object to the scene that was basically just logging information about everything in the hierarchy to the device log. And then from there with Appium, you could get the coordinates and do whatever you wanted to the via the just input coordinates and normal Appium operations. So I don't remember if I tried that or just implemented my own stuff where I was just logging to the device log information about the elements that I was interested in. And then I realized, okay, it's very easy to just open a TCP socket within the game and then connect to that as a back channel to the game. And that also offered like two way communication. I could ask for information in real time and I would only get information about the things that I was interested in rather than try to log everything past, dump all the information to the log. So that was kind of how it started, basically.

So this sounds like the most elegant solution, even what the guy was doing with putting into a log file or whatever. So it's really cool. And you actually open source it, which I think is even cooler. So are there any downsides to this? Because it sounds like, okay, now I can just do whatever I want. There's nothing holding me back.

So the downside, the biggest downside is of course that you need a special build. You cannot, you know, out of the box, you're not going to be able to automate anything. So you need your developer or someone to create a special build that adds this Alternity tester, we call it the prefab or the game object in the first scene of your game. And from there on you can just use it and, and use it for interrogating the game and writing your automated tests. But this is, I think, the biggest downside, let's say the fact that you need basically access to the code or a special build created for you to work with it. Also we kind of open source it to see how like, do people use it? Is this a real problem? Do people want to use it? So, yeah, we did get a lot of good feedback, but we still get a lot of questions. Because the setup part is not very straightforward. So this is still something that I would consider it a downside because of this, the fact that for example, if you use altunity tester to create an instrumented build for Android, that means you get a special build that has the altunity tester prefab in it. When you start the game on the device, that starts listening for a connection from the test. That also means that you need to create a bridge between the device and the laptop and your test to kind of be able to communicate. So the moment you go on the device, you need to do a little bit of setup and port forwarding from the device to the laptop to the test. So that kind of adds a bit of complexity to the just getting started with the automated test.

I could see that being an issue. But how hard for instigated and instrumented though? How many teams do you see that actually don't have a developer that can do it for them? Are there a lot of third party companies that are like, well, this doesn't help us. Is it very common that testers work with developers and it's easy to get instrumented for what they need?

I think. Well, I mean for a lot of companies this is not a problem. Of course the developers can create a build, you can create it from the command line in Unity cloud. It's pretty straightforward and you can automate all of that very easily. There are though kind of companies that specialize on just game testing that might not have access to the code even they are just delivered. Here's a game for Android, iOS, you know, PC, Xbox and so on. So they get just the binaries and they test them manually whenever there's a new release, let's say. So they cannot use automated approaches because of this because they don't have access to or ways to instrument the game.

So this sounds awesome. Everyone that's listening, I'd highly recommend getting Alt Unity tester. So how much development's going into this? Is it an active community? Where do you see a heading?

So I think we started in early 2018. We have like, I think we are now maybe four, five people on our side working on it, like on developing new things. And for example, now we try to improve the communication. We're going to release very soon, I hope a totally new version that has, that uses hopefully WebSockets instead of just a TCP socket for better communication. So we really put a lot of effort into this. We also have the open source, the kind of like let's say the Client and server side of it are both open source, but we also have a paid side, let's say to the alternative test tool, the group of tools and we are working on developing that. So basically if you have an instrumented game, if you don't have access to the code as well, so you can run the game in Unity Editor, then it's quite difficult to inspect and understand what elements are available in the hierarchy. So what we develop as a paid product is a so called inspector. Unity Editor is very kind of, let's say, intimidating because it's an IDE with a lot of features and you need to spend a lot of time in it to be comfortable with all the elements from there. So as a tester that might not be your comfort zone. So then the inspector can you can use that like you use the Appium Inspector or the Inspector, the developer tools where you inspect the DOM in the browser to see the elements, to look for locators, to try out your locators in real time and just interact with the game like you would from the tests. So it's just like a helper tool for that. But we do kind of actively develop it. We have quite a lot of already contributors from the outside for the open source side. We get PRs and new features added also from people from outside and at least we see it as growing and it seems like people use it and are interested in this and hopefully it will also change the overall kind of state of the industry towards more automation.

Nice. So like I said, we did meet at an applitools event, but I don't remember you actually talking about visual validation. How does visual validation tie into this at all? Or how do you use a visual validation tool like applitools? Is it something separate or we don't.

Have anything built in? I think you in theory could use something like that, but from the altunity tester perspective, that's something that can be added. We don't have that built in and I haven't yet tried to create an example with that, but I'm pretty sure it could be integrated in that way.

Someone could then have to worry about downloading this saying oh no, I need to have a paid service. It sounds like it's completely self contained.

Yeah, yeah, yes, yes, I love it.

All right. Okay, Roop, before we go, is there one piece of actual advice you can give to someone to help them with their game automation testing and what's the best way to find, contact you or actually work? Learn more about Altum.

I'm On Twitter you can find me there. I try to be responsive. We also have a Discord server for altunity tester. Like all our developers are there, I'm there almost all the time. So that's at least for the gaming world. That's a very easy way to contact us. We try to kind of do a lot of support and help people get started there. We also have a Google group, you know, all the usual channels let's say. But for myself personally, I think Twitter is the easiest to find me.

Awesome. And one piece of actual advice you can give someone with their game development if they go away now, how can I test it? Anything they could put into place right away you found really helpful.

I guess it's start with something very simple and grow it. Once you know that that very simple thing is, you know, working for you reliably and just build slowly, focus on reliability and then, you know, work from there. Because I think once you get started it can be overwhelming, especially in games like how much you could test. But if you start small and build on top of that, I think that would be my advice.

Thanks again for your automation awesomeness. The links to everything of value we covered in this episode. Head on over to testguild.com a365. And while you're there, make sure to click on the Try for free today link under the exclusive sponsor section to learn all about Sauce Lab's awesome products and services. And if the show has helped you in any way, why not rate it and review it in itunes? Reviews really help in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack Automation awesomeness. Us as always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Automation Podcast. Head on over to testguild.com for full show notes, amazing blog articles and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.