Test Automation in Gaming with Shane Evans

Welcome to the Test Guild Automation Podcast where we all get together to learn more about automation and software testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Automation Podcast. Today we'll be talking with Shane Evans, all about Game Driver. Shane is the founder of Game Driver, which enables video game developers, testers and producers to automate play testing, improved time to market, test coverage and overall quality and revenue. Shane has a lot of experience in the test tool space and has worked for some of the biggest companies in the world. You definitely want to hear about this awesome new tool I think is going to help a lot of people that are involved with game development. So check it out. The Test Guild Automation Podcast is sponsored by the fantastic folks at Sauce Labs. Their cloud based test platform helps ensure your favorite mobile apps and websites work flawlessly on every browser, operating system and device. Get a free trial, visit testguild.com sourcelabs and click on the exclusive sponsor section to try it for free for 14 days. Check it out.

Hey Shane, welcome to the Guild.

Thanks for having me.

So before we get into it, Shane, is there anything I missed in your bio that you want to tell us a little bit more about yourself?

No, I think that just about covers it. You know, I spent some time with Hewlett Packard with which became Hewlett Packard Enterprise and eventually Micro Focus, where I managed some very well known tools in Test Automation. Loadrunner for one, which was my introduction to the sort of software high tech world coming out of the private sector prior to that. And then I moved on to unified functional testing, which I think they're calling U of T1 now or something.

They're always changing the name. I have no idea what they're doing half the time with these names.

So I've been in the space for some time and I started Game Driver because I was actually doing game development on the side when I was working at Hewlett Packard and I noticed that there was a bit of a gap in the industry. I'd done some research, some education around the actual development process and when it came to testing, it just seemed like everything was very manual. So I started talking to people, I was asking friends in the industry and I, I attended some events. This is back in 2016 timeframe.

And as it turns out, for the most part, beyond unit testing, everything's just done manually and this is the way it's been for like 35 years.

So I set out to try and solve this problem and with my co founders, Rob Gutierrez, And Philip Mayhew. We came up with a solution and.

Did about a year of beta and now we're in production and have customers and everything seems to be going great.

Awesome. Yeah. I've spoke with a lot of game developers and what they do is they have their kids usually run through it. If it's a VR game, run through it and do the test manually for them. But when I asked if they had any tools to help them automate, a lot of times they said because it's usually one and done once it's released, they usually don't necessarily care about investing that much time in testing. What was the driving reason when you talked to developers, why they thought this would be a good idea?

So that was definitely the model for a lot of games. And back 10, 15 years ago that was the only model. But as things moved towards the online.

Continuous development, continuous integration model that we're used to in enterprise software, games have a much longer lifespan these days. So I used to play a very popular game called World of Warcraft and that's actually where I met one of my co founders more than 15 years ago. We were part of a guild together and that game has gone on to last over 15 years in production, has had release after release, both monthly patch type of releases and then these big bang add on releases called or dlc, sometimes they're called or expansions. But essentially it's the same game that's been in place for 15 years and there's a lot at stake with these games. So you know a game like Grand Theft Auto, which is very popular, I don't play it, but I'm sure you're familiar with it and your readers as well, or viewers.

When they launched Grand Theft Auto they made a billion dollars on opening weekend, which you know, relating back to enterprise software again is like a year's revenue for a lot of companies. Well, Grand Theft Auto has gone on to make over $9 billion. So is world of Warcraft. And when we talked to like the head of QA at Blizzard which makes World of Warcraft, we were shocked to learn that they too don't have a lot in terms of test automation. You know, the developers do their unit testing, but when it comes to integrated testing and play testing, 100% manual. And these games are becoming more and more prevalent, these kind of long lived titles. I play games like Destiny 2, Destiny franchise, which has been in production for over I think 10 years now.

And so there's a much longer life cycle and a lot more to be gained by automating some of the simple things that you do each and every release, every build, right, you give it to your testers and they're going to run through the same process every single time. Create a character, start the game, do some basic things to make sure that the sanity is there before they get into the meat of things.

Awesome. So how does it work? I would assume, I haven't tried it yet, that if someone had an API that you could, it would be easier to automate rather than going through a UI to automate the UI itself. So how does game driver actually drive the gameplay of an application?

Yeah, that's exactly right. So we have a framework that we use, an API that we provide and an agent that's embedded in the game itself. So when the developers are creating the build, they embed our game object within the game and that allows us to connect into the game, to control it, to interrogate the game while it's running, both in the development stage and when you've put together a standalone build so that you can send in commands like movements or query objects, say what is the score, what's happening on the screen right now, what's the position of this player object, etc.

How much overhead does that put? I know a lot of times when you add an agent into someone's code, sometimes it slows it down or causes unintended consequences.

Any feedback on that so far?

Nominal. I mean.

Our customers, our reference customers, are not reporting any sort of overhead, like impactful overhead in terms of running the agent. It's very lightweight. All it does is allow us to communicate into the game. So it provides a back channel to our API that says I'm waiting for commands. When we send in commands, it queues them up and performs those actions asynchronously or synchronously, however you want to run it.

Then.

Those commands end up providing the same type of input that a user would. It's just done through a different mechanism.

So I know there's some bigger game engines out there, like Unity. Does this integrate with these type of engines?

Yeah, so we started with Unity. They've got a huge market share right now and are growing pretty quickly. Even amongst the AAA studios that we talked to that have their own engine, so I won't mention their names because we're not quite at that stage yet. But some well known.

Shops that have their own internal engine that they either built or acquired, they're still looking at Unity for some of their smaller titles and that may grow into some of the larger titles as well, so it's growing in popularity and we felt like starting there would make a great first engine to support. But we did architect Game Driver so that it's portable. And the idea is once we've satisfied the Unity market, as we have all the features that they're looking for to do some level of automated testing, then we'll carry that forward to most likely Unreal Engine, which has been around for a long time and is really widely used in the larger studios. And then AWS jumped in with Lumberyard not too long ago and it's always been an interesting target for us as well. I'm not sure quite the adoption rate there yet for Lumberyard, but they're spending a lot of money and they're throwing in a lot of features, so I expect that it'll grow pretty quickly as well.

Nice. So if a developer wants to try this is it just like you said, it's pulling in a library and they're just coding it like they would normally code their application. Is that how it would work?

Yeah, I could show you if we're able to do a screen share, but essentially they would code API calls into the game. We do have a recorder in alpha right now. We're just going through the early stages of getting that productized. But the idea would be, even with the recorder, record the basic actions. It's not meant to satisfy the full comprehensive test suite, but to give you that starting point in terms of the actions that were performed by the user. And then you would go about inserting your control statements, data driving it, adding non user type of commands like querying or adding assertions to the test. But it would at least give you that initial.

Kick out the door in terms of starting your testing. But we found that the coding approach.

Doesn'T have a lot of overhead in itself. It's just that this industry is not, I would say, very mature in terms of test automation practices. We've worked with a few companies that are very mature, but we found that they actually brought their QA folk in from the enterprise space. So they're used to working with webdriver and other commercial tools, so that so that they're able to make up.

A test suite out of the gate. Whereas some of the more traditional gaming studios aren't quite mature in that sense yet. So there's going to be a bit of a market education, I think, in terms of getting people used to the idea of test automation. And we hope that that's actually a positive thing because there's armies of manual testers out there, thousands of them working in like shifts in some of the larger studios and they don't have a whole lot in terms of career growth opportunities. So it's really just manual testing. And either you do that until you become a manager or you get burnt out and switch roles because it can be quite arduous.

So who normally would create these tests then? Is it actually developers or would you foresee like in other places where you have someone that specialized in automation that would be as part of a. I don't even know if they're doing sprint teams in this type of development.

We do, yeah. Yeah. And I think you'll see manual testers who are technically capable of doing the test automation or maybe interested in doing that, who would take on this task from developers. I think initially it's kind of a mix where some of the teams that we work with are definitely following the kind of enterprise agile approach.

And they already have a test automation engineer sitting side by side with the developers. But we're trying to expose more of the manual testers to this capability because we think that there's a huge opportunity there to grow and learn within their careers.

So are there any quick wins you usually try to aim for when you're trying to implement this? Say, look, if you at least do this minimum, here's what you're going to save, or any scenarios like that that really makes people open their eyes and go, oh my gosh, we have to have this.

Absolutely. One of our earliest customers had a great example. It turns out they're actually not in the gaming space. They use Unity to develop a commercial solution the company's called In Context Solutions out of Chicago.

They were able to very quickly automate the top 200ish tests that are run for each and every build. So the developers would send them a build and they got to go through this checklist of functionality because their software, which is VR for retailers, to kind of map out how a store will look in VR before sending it out to the field.

So it needs to be something that a non technical person can just put on layout the store, do some testing with their target audience and then ship it out. And so they had a very comprehensive list of actions and features that needed to be tested for each and every build. They were quickly able to automate that and save something like 85% of their time spent manual testing just from that. And once they get that out of the way now you can focus on other areas that you might not have had time to test before and explore areas of the application that maybe saw a little less attention, automate those and continue. We actually think that from what we've seen, there's an opportunity here to automate a much greater percentage of work than what we've seen in traditional enterprise software test automation, which over the years kind of hovered at 20, 30%. There's some very mature companies that do a lot of automation, but as a whole it hasn't grown all that much. But because this is a very technical audience.

The game development community has some of the best engineering in the world, very talented engineers, and they're working in an environment where everything is basically accessible via the engine. Right. Whereas enterprise software is a comparison. You put on a website and then your end user has to fill out the website forms and go through the actions. But that's their interface. What we're providing here is a way to communicate with the game that allows a very high level of automation. That would not be possible just.

If we were interfacing with the game as the user would from the outside.

Another concern people might have is they need. You can't make a game, you can't build quality into a software. You need to build testability from the beginning. So if you're building a game, are there any things that need to be in place in order for Game Driver to work correctly? Do they have to have hooks or anything like that?

No, I think it comes down to best practices for developers in general, naming objects in a way that makes sense, that are easily identifiable. Again, this is a pretty mature space for development in video games. So the test automation is in its infancy, but development is not. And that gives us a really good opportunity to automate things in a much better way.

Going back into the 90s, when the wild, wild west of Web 2.0 and the enterprise really adopting test automation.

There was ever changing frameworks, there was no standards, so there was just a whole lot to deal with and the tools had a hard time keeping up. What we see in gaming is there's a handful of engines that everyone's converging on, and those engines provide everything from your animation to your lighting to your audio. They provide the whole package and they're quite mature. And that gives us a pretty unique opportunity, I think, not just in Game Driver, but in automation in general.

So you talked about an MMO game that you were playing, which requires like a load, and you do have a background on load testing. Is Game driver going to eventually, I guess it's far in the future, but are there any functionality now that exists in it or you could see helping people with performance testing, I would think that would be a bigger issue almost to help perform it is their application under load.

Yeah. So one of the challenges that we've heard from customers is definitely on the performance side, multi user kind of parallel testing. So we again developed Game Driver in a way very reminiscent of.

The.

Webdriver or Selenium Grid type of approach where you can control things remotely and therefore we can have multiple game driver agents running in parallel. That same company that I mentioned that's doing their build verification testing builds all of their software in the cloud. So they build in Azure and they run a bunch of tests in parallel using Game Driver. What it would be easily possible from that is monitoring if you have a central environment. They don't, by the way, but if you had like a central set of services that all of those game agents connected into would be, you know, monitor same sort of best practices you have for performance testing, monitor those for load, for concurrency, all of the resource utilization. And on the client side we're able to drive those workloads. One of the conversations we're having right now is with intel, who has a.

Graphics optimization tool for video games. I believe they're starting with Unity. I don't know if they support the other engines that I have to check, but it allows developers to optimize the performance of the game itself, not even on a large scale for multiplayer games, but on a single user scale. The challenge is we need something to drive the workload so that we can monitor the game and then optimize it. So without something like Game Driver, you're still playing the game manually to get the profiling information that you need in order to do the optimization. Insert Game Driver. Now, you don't need to test things manually. You can run it as part of your build process. You can drive the whole thing via APIs, no manual intervention involved, and then after each build, get not only the functionality report, but also the optimization report from, from a tool like Intel's.

So I'm not a gamer, so I'm probably not going to use correct terms here, but I would think it would also help you do more exploratory testing. I think you hit on this a little bit where you can, can you use Game Driver to populate, say your character so you get to a certain level and then you could start manually testing from there? I would, I would assume maybe that would be a user case.

You could, I think in, in most environments they probably have some, some sort of backend hooks to be able to do that. So instead of going through it manually, just say, load up a level 100 and off we go. I've seen that sort of thing in action in some of the larger games when the developers have access and can play around. What's important though, is that we don't leave behind that entry level stuff and that you're still performing those actions, whether or not you can have the ability to jump to the end and test something that you couldn't otherwise. There was a good example of this that we blogged about a few months ago with the World of Warcraft, which it's been in production for over 15 years. So you would think they've tested everything. Making a small change to the math, for example, that doesn't, you know, it doesn't have a huge impact or it shouldn't. Well, it turns out changing a small coefficient in the way that players did damage at lower levels broke the game. And without automation, how do you test that the game's just too big? You know, it takes years to play. It would take quite a long time to test unless you had some, some piece of automation in there to say, okay, you know, start up a new character and do some basic things at the low level. I think these things tend to get forgotten over time.

So we talked about the coding aspect of it, but I assume like any new framework out there, they have a record and playback type feature. Is this part of Game Driver as well?

Yeah, so like I said, we have it in alpha phase right now. We are hoping to have that launched in the near future. We're getting some feedback on that feature and we're always in entertaining beta requests. So for some companies that are testing on platforms that we don't support yet, we'll work with them. Recently we launched our iOS beta and we just make that freely available. So I expect the same would happen with the recording piece. Some companies have said it's not that big a deal, they just do all their coding right out of the gate. Others are looking for something to help them get started. And that's where a recording tool would definitely help.

What do the results look like? Do you get a report or is it in real time? You see when things are passing and failing, what's the end result of when a test.

You can watch it. Yeah, you can absolutely watch your tests run. Or if you have them running through Jenkins or some other.

Build automation tool, you would just get the report of the past or failed tests like you would any other.

So is there any functionality that Game Driver has that we haven't covered or you think people should know about one thing?

I think that's pretty cool. And it's a little feature, but we're able to actually control the speed of a game as well. So if you think about very complex scenarios, we can run the game at 2x5x speed, which is not something you can do manually. And that allows us to test much faster, not only in parallel, but let's run it at five times the speed, get things done.

That sounds really cool. For sure. So how's the forum? I noticed there's a forum here. Is the community growing? Are people helping each other?

It's pretty new at this stage. We had a lot of interaction with customers in Slack during our beta, so we would connect with them directly and we're still doing that with some of the trial customers. Others prefer to be more. Let me just do it myself. I don't want to get on your slide channel. So the form is quite new, but we do encourage people to come and ask questions. Of course.

Awesome. So if someone has, like you said, a normal background in automation, it sounds like this would be a seamless type of transition if they had to interact with game driver. Am I correct?

Yeah, absolutely. Phil, myself, Rob, we spent over a decade in this space. So we took a lot of what we learned out of enterprise test automation and tried to apply it here.

For example, the object identification piece. Maybe you don't know the exact name of the objects in game and the hierarchies can get quite complicated, much more so than the DOM in a web browser. For example.

We provide.

An XPath like interface to be able to search for those objects. So as long as you're a little bit proficient in regex, and I'm not a regex master, by the way, I still use tools to help me kind of form these things, but we provide the XPath type structure so that you can easily, you know, search for objects rather than knowing exactly where it sits, what the parent objects are and so on.

I forgot to mention, I don't think we talked about it. Most people think open source solutions all the time. Is this open source? Is it a paid solution? What's the roadmap for that?

It's a good question. Right now the framework is closed source, although we do have plans. I don't want to get, you know, too specific as to when, but.

I do believe very strongly in open source. I think for us to get there though, we need to mature the solution a little better. We need to get to a stage where okay, this works. It satisfies the majority of use cases. Now we can de emphasize the kind of the framework itself and focus on the bigger problems. And that's where I could see us either working with or entertaining some sort of open source strategy.

So what are the bigger problems?

Bigger problems, larger scenarios. Multiplayer testing. Like you mentioned.

There'S a desire for sort of an AI driven approach. And if you look at AI in testing today, it's very superficial. So you look at like what OpenAI does or what Google DeepMind has done. When they show demos of their AI in action, it's always using video games, but they're testing from the outside, they're playing it from the outside. So they don't have kind of a deep understanding what's happening in the game, they just see what's on the screen, which is great. But from a test automation perspective doesn't gain us anything. Right? If the AI can run right and avoid all the enemies and get to the end, it doesn't tell us if there's anything fundamentally broken in the game. So Game Driver, we expect, will provide the interface between These general purpose AIs and the game itself to be able to test at a more comprehensive, a deeper understanding level. And that to me is kind of the goal.

Shane, before we go, is there one piece of actual advice you can give someone to help them with their game testing efforts and let us know the best way to find or contact you.

So the best place to find and contact us first would be at gamedriver IO. We've got the trial for anyone who's familiar with testing and wants to give test automation a try. There's some tutorials there, the forum, as you mentioned, where we're available to answer any questions and we do have a community Slack channel that we invite our trial customers to. And in terms of getting started, I think as the tutorial will show for those who are maybe interested in test automation but haven't tried it because there isn't a whole lot of options available in video games today. Give the tutorial a try. We'll show you just how easy it can be to automate your testing. And once you get started with that, the opportunities or the possibilities are pretty much endless.

Thank you Shane for your automation awesomeness. The links to everything of value we covered in this episode, head on over to testguild.com a.

And while you're there, make sure to click on the Try for free today link under the exclusive sponsor section to learn all about Sauce Labs awesome products and services. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe and my mission is to help you succeed with creating end to end full stack testing awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Automation Podcast. Head on over to testguild.com for full show notes, amazing blog articles, and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.