223: Testing Dream Journaling (Smashing Sand Castles) with Noemi Ferrera

You can really see who is going to be developer or tester in the beach. And when you're small kids, the developers will be making the sandcastles and the testers will be these kids that enjoy breaking the sandcastle. And I was a strange kid that would like to do both.

Welcome to the Test Talks podcast, the place to go to geek out on software testing. And now your host whose mission is to help you succeed with test automation, Joe Colantonio.

Hey, it's Joe. And welcome to another episode of Test Talks. Today we'll be test talking with Nomi Ferreira. All about test automation, AI, virtual reality, machine learning and how to test games. Nomi has a bunch of experience. She's a master tester, automation engineer and developer. She's really cool, really knows her stuff, really enjoy talking to her. You don't want to miss this episode. She has some great insight around all things test automation related and more. Check it out. Test Talks is sponsored by the fantastic folks at Sauce Labs, the cloud based automated testing platform that eliminates the need to maintain your own Selenium grid and test infrastructure. Try it for free today. Visit testalks.com and click on the sign up now link under the homepage sponsor section.

Hey Nomi. Welcome to Test Talks.

Hello. I'm delighted to be here.

It's awesome to finally have you on the show. You joined us for the AI Summit Guild a few months ago and you really gave a lot of insight around AI and machine learning. So that's one of the topics I'd like to talk about today. Automation, testing around virtual reality, AI and especially games. Because you do work in the gaming industry. But before we get into it, can you just tell us a little bit more about yourself?

Okay, so my name is Noemi Ferreira. I'm originally from Spain, but I've been a little bit all over the world. I'm currently based in China and I work for a company called NetEase, which may not be so famous out of here, but it's actually one of the biggest in mobile games and in game setting in general.

And I started.

Programming when I was nine and I've been in and out of development and testing jobs, so kind of switching a little bit between them.

Awesome. So you just mentioned you've been programming since you were nine. How did that happen?

So my dad really liked Chess Masters, so he bought a computer. Just. Well, my, my feeling is that it was just because he has a Chess master game and he bought. It wasn't a Commodore, it was kind of a Sony, I think brand but it was one of these really, really old computers and that computers happened to come with a book in Visual. Well, not Visual Basic, Normal Basic, before Visual Basic. And I happened to be the only one in the family who found it interested and started looking into it. And the big problem that I had back at the time was that.

Every time we loaded a game, we will load it before lunchtime, my sister and I, and then we will play in siesta time in Spain, in the south of Spain. My parents will sleep for a couple of hours after lunch and we would use that time to play freely without them being like, oh, you've been playing for so long. So sometimes these games wouldn't load and that was the main issue for us. So I learned actually to start debugging on them. And I actually managed to fix a couple of issues and get the game running again. So we didn't have to wait another two hours for the game to load or whatever long it took. So that, that's kind of what got me started. Also, I really liked the games themselves, so I was really curious in knowing how it was meant to be created.

So, Nomi, it's interesting that you actually started off programming with games and now you're working for a game company. So it was that intentional that you're actually working for a game company now? Is that like the direction you always wanted to get get into when you were from nine on?

So first.

The book that I have talk a lot about Bill Gates. So first I was like, oh, Microsoft is my dream job.

So I did get that. I started working in Microsoft. It was my second company a while back in Ireland. And then I felt really empty because it turns out that when you manage to make a dream come true, you actually feel released for maybe a little while, but then you feel quite empty because you don't have any other goals anymore. So you have to find the next goal. And yeah, game was so another dream of mine. And funnily enough, okay, this is a bit embarrassing, but I'm going to tell you anyways. So I keep a dream journal. It's more kind of for goals, right? But I write everything there, even if the goal is unachievable or I think it's going to be unachievable, I still write it there and I had a few things written down and I kind of forgot about the journal for a while. And then this job in China came up and it was kind of perfect. It was good timing. I really wanted to come to China because I was learning Chinese for a While and I was curious about the culture and the places in here and also I was very curious about working a game company for a while as well, so. And there are not so many in Ireland. So it was kind of perfect timing. But it turns out that when I was packing to come here, I came here in 2017, at the end of 2017 and I found my dream journal and I had written down 2017, go to China, work for a game company. And then once I was packing, I found my dream journal and I have written down 2017, go to China, work for a game company.

So it's amazing how subconscious can work, right?

That's so true. That's pretty wild. So you also have I think master degrees in computer science and hardware and software. So you have the technical chops. But what got you into testing?

Well, so as you say, as you mentioned, I do have three degrees, one in Harvard, one in software and then a master degree in computer science. So I'm pretty much prepared for doing development. But.

It kind of makes me a bit bored to be in the development section in which you are part of a feature and then you won't see the whole complete thing until many months later. Maybe you don't even get to see it. So that's something that I really like about testing, that I get to see the complete product. And also I'm very.

Attention detail oriented. So I really like quality because it has that aspect of really trying to make sure that everything is fine and I like the organizing bits that comes with it. And I do have the quality thinking hat I think that. Well, I say many times that for me testing is like watching a movie after you read the book so someone else made the movie and then you get to point at all the mistakes and all the things that you think should be different as you imagine while you were reading the book.

And also I have another comparison. I think you can really see who is going to be developer or tester in the beach. And when you're small kids, the developers will be making the sandcastles and. And the testers will be this kid that enjoy breaking the sand castle. And I was a strange kid that would like to do both create the sandcastle and then break it myself.

So yeah, that's kind of the idea that I have to explain it in short words about testing versus development.

So I think of myself as.

For a very long time I had the dichotomy what should I do? If I should use my skills and develop or if I should use my heart and test, then I come up with the solution which is create new technologies but oriented to test it. Many developers will be doing front end or backend or mobile applications and I'm a sort of a developer that create tools for testing. That's the way I see myself.

How does that affect your testing? When you interface with the developer, are you able to say not only is there a bug, but maybe you should try this algorithm or this approach to make it better?

Correct. I like being in the code review with the developers and I happily will both point mistakes and also help them to think in a testing way so they can also create their own unit, test correctly and all that. But currently my current position, I don't deal with developers. We are actually developing tools ourselves, so we kind of both things into one.

So what kind of tools are you developing then?

So right now the tool I'm working on is called airtest and it's a solution for automate.

Applications and games and it's actually pretty cool to be honest. Don't want to be doing advertising or anything like that. I just really believe in the product.

So it's a tool that actually helps you test games. Because that's one thing I've seen when I've talked to developers. It doesn't seem to be an industry tool that they go to.

Yeah, so it's kind of, you know the way that we have some tools that have visual testing and then we have some other tools that have kind of.

DOM or object model extraction and then you can use that into creating your automation. It's sort of an IDE that will allow you to use either for many different platforms. So there's for Windows and there's for Cocos and Unity and there's for Android. So there's many different platforms that it can be used the exact same.

IDE.

And it's actually pretty accurate. And it's open source, which is really good as well.

Partnered with Google to do this. So it's a very nice app. It's just not famous enough in Europe and in the States. So hopefully we'll get to change that.

Awesome. So Nomi, could you just tell us a little bit more about the AEHR test project?

Sure. So there's two things about it. One is I am going to release a post soon in my blog to show how to automate Scratch. You know Scratch is for kids.

Yep.

So I want to show how to automate this using this tool because Scratch is based in Flash so it's really hard to automate. So it's a good approach. For teaching also kids how to test, which is one of my goals as well. I want to be able to bring testing to everyone, including kids. I think nowadays there's a lot of courses and a lot of tools to teach kids how to program and to teach people with no kind of not a degree in computer science in how to program. But there's nothing about testing. Right. And that's going to bring a problem at some point because I think they should come hand by hand and all developers should have some idea about testing in order to get very.

Quality application.

I love that vision and it's interesting. Like you said, I thought it's because I'm old, the developers I work with don't know how to test. But it's also new people that we hire straight out of college don't know how to test. So it's curious why testing is not part of any curriculum that I'm aware of.

Yeah, that's actually the thing that got me into test first, because I was working back at the time in IBM and I joined as a developer. I was an intern and when the internships was going to an end, actually I graduated when it was going to an end. I needed to find another job because my team didn't have budget to hire full time. So I look around and I heard about this job in quality and I didn't learn a lot of quality when I was studying. However, it sounded so good. To me. The word quality means the world. It sounds like a high level thing and many people will see it as a lower level thing. So hopefully we'll get to change that vision as well and get people to, you know, be proud of having quality of their programs and apps.

Awesome. So you also seem very progressive on trying new technologies, one of them being virtual reality. And I was speaking with a previous guest on Test Talks, BJ Aberly, about how it seems to still be in its infancy. And he gave like a three step approach of how he handles VR. So how do you handle VR testing? Do you have a process that you use or is it the same skills that you use to test any other piece of software?

Okay, so actually I heard this podcast today, so I'm very happy that I happened to hear it. He was very, very interested. I follow him on Twitter right after. Yeah, I agree with many things he said. And in regards to these, yeah, the steps will be pretty similar to any other application, but there are certain things that are very specific of both AI and VR. Although I've been working more with VR.

Which I feel a Little bit ashamed of that because I know a perfect guy that is working on AI open source project, but I just haven't had the time to help him out because I moved to China and all. So anyways, yeah, I think the steps are very similar in general. The tools will be also pretty similar, some of them will be different.

And then you have a couple of things that you have to take into account that are separate or special to VR rather than other applications. And I explained a little bit on my talk, my Selenium talk.

This year in India. Basically I call it safety tests, which is when you kind of have objects around you and your VR experience require you to move. You can get to hit these objects or even, you know, you can lose balance and fell off. So you have to be aware of your surroundings. Also many of the headsets will come with the audio and these headsets will, you know, will immerse you into the experience not only by your vision, but also by your audio, by your hearing. And this means that if there's anything happening outside, you may not hear it. And it may be as important as an alarm, fire alarm, carbon dioxide alarm, Is it dioxide or monoxide? Anyway, it could be an alarm happening around you and you may not realize, or it could be someone entering your house and you're immersed in the experience. Because VR is about creating a virtual world and it's about creating it.

As close to the reality as possible. This brings this issue in safety. And then the second issue with safety is when you have programs or application or games that requires multi users and that means that two users may be in the same space. And this may make some of the users uncomfortable if the other user is too close to them. So there's been some people, especially woman, that has felt abused over VR even though nothing physically was happening to her. And you know, not only that, I'm also quite of a geek, so. So I like watching playthroughs on YouTube. So some of these YouTubers are really famous and when people in the, for example, in VRChat realize that it's them, they straightaway go towards them. The first thing they want to do is touch them, play around and all of that. And there's no established security around this. So I think it's something else that we have to take into account besides safety. Another thing that is very specific of VR is when you feel dizzy because of the movement. So there are different ways of moving in VR and you can get to feel that dizziness there.

That's actually one of the funniest things of your presentation I thought was you actually use your mother to test motion sickness. And I thought that was hilarious.

Exactly. So this dizziness is called motion sickness. As you said, it's caused because your eyes are seeing the movement, but your body is not feeling it. So you physically are not moving and your eyes are seeing it. And it's exactly the same effect that happened to you when you're in a boat. So not everybody has the same degree of motion sickness. In my case, I'm pretty used to this because I play many games and also kind of used to my eyes not following my body feeling. But my mom, she obviously is not as used to this as I am. So, yeah, I use the poor woman as a tester in this case because she has more sensitivity around this. Also for scale and for feeling of how a room feel. Maybe you develop an application or a game and you think you want to make the scale feel like it's very big or very small. And then you, you give to someone else and they don't feel the same. So all these sort of things, I think is important to test with different people. Which means, you know how everybody's like, oh, the testers are going to be obsolete, whatever, but actually it's not. You actually need many different people to get an actual feeling of this, to have a proper solution for that. And while sometimes maybe you can use different people on your team, it doesn't need to be a tester. But you know, you're always going to need tester around to do this sort of stuff. Then after that, after you got the testing, you can set up your game or application to be configurable so they can change the size of the distance of the object depending on the size of the person that is using that game. And according to motion sickness coming back to it, there's another. There's a few tricks that you can do so people don't feel this. One of them is to make them feel like they're inside like a car or a cabin. So when everything moves around you, you feel like you're in the car. So it's okay not to feel the movement. Which makes me think that when cars start coming up and, you know, trains and all that people may feel this motion sickness in there as well at the beginning. But then people got used to that as being a possibility that you may be inside something and feel other things moving. So we don't get sick with this. Usually not many people will get sick with this anymore. So I will assume at some point the motion signals threshold is going to be way higher for everyone.

Very cool. So, Nomi, I think another misunderstanding with technologies like VR is that it's only a manual testing type process. And once again, when I spoke to bj, he gave a few examples of using tools that helps him, like using Chromecast to record and play it back. But I've been reading up and I noticed someone was talking about there's a way to actually create heavy developers to couple your application objects from the input device, which then allows you to simulate test input. So I guess my question is how do you automate some of these things and what tools did you use to help you with that?

So one of the things I realized when I was listening to this other podcast is that it really depends on the device you're using. It depends on the technologies you're using. In my case, I'm mostly using Unity and Unity does have some unit test possibility to check objects and the relation with them. It's not really end to end thought it's more for unit testing, but it's the start. Then another thing that I was working on actually, hopefully it'll be released soon with the Aertesk project that we were talking before is adding VR capabilities for Cardboard. So for Google Cardboard, so far we haven't added any device for gesturing. But basically the big difference between VR and any other application is that in VR you don't. You have objects that are maybe behind you. So in order to access to this object and in order to do certain actions, you need to be moving the camera. Now this is kind of issue because at least for Android Cardboard, Google wasn't very happy with people moving the camera. They think that users should be the only one responsible of moving the camera. So it was slightly challenging to do that. So basically what we managed to do is to rotate the camera on demand. So you could rotate the camera either toward an object or with the degrees you want, and then instead of clicking on the object itself, you can click on the screen and that will effectively click on that object. So you can click anywhere and, and other things that you can do is like check your location, check your position, check the object, also location, position, color, to verify all this information. But it's really very similar to any other application.

Awesome. So another technology I think you've been researching and using is artificial intelligence and machine learning. So can you give us an example of how you've applied some of your research to real world testing? Or have you done anything recently with AI or machine learning that you think people would find helpful.

So I did some kind of side project just for my own, just because I happen to have been in companies before in which you arrive to the company and you have a lot of test cases there and the tester that was working on them is gone, you have no one to ask, and you have to verify the next release as fast as possible. So my question was, okay, how do I decide which test cases to run first and back at the time I just went through everything and tried to remove the ones that were not valid anymore and try to understand the ones that were valid and try to maybe execute first the ones that will have been touched by the developers in the particular feature that we were doing. But maybe then I can skip some bug related to something that failed because of change that affected it somehow. So it was pretty tough. So I come up on my free time with a solution that use tool based system. So basically I can decide on certain rules of certain properties of the test case, such as the risk or the novelty or the priority of the test case or of the feature. And then given these rules, I can say, okay, if the priority is high and the test case is new, I'm definitely going to execute it. Maybe if the test case is old and priority is also very low, maybe I won't execute it. And then give all of these together to the machine and then the machine will tell me, you should execute this test case, this test case and this test case. If you want to get your best in 10 minutes, if you only have 10 minutes to test, you should execute these three test cases. Maybe if you have an hour, maybe you can execute these six ones. And.

It'S kind of handy in a way, even though you have to prepare a lot of things. So it really makes sense if you do have a lot of really old test cases and you don't really know about them. So that was the first solution and the second solution I worked for, even though it's still not in the any I haven't done any talks or any post about this is using machine learning to do the same. So basically using machine learning, instead of me giving the rules, the machine will tell me as well. Okay, it turns out when the risk is high and the test case is really new, it turns out that you won't find any books. It's really very unlikely that you will find books when the risk is high and the test case is new. And maybe you wouldn't have thought about that, but maybe it turns out it's true because maybe the developers put more attention into it. So the machine can bring up things that you would not have imagined even as an expert. And then it will also give you well with this consideration then these are the test cases you should run. However, the problem that machine learning has is that it needs a lot of training to make a good system. So you need to train the system very well with a lot of cases. So you do really need to have lot of really old test cases. You should have kept the results and the bugs that are being found by them so you can kind of train the system this way. So it's not very useful for most companies because they will keep the test cases great to a load one and you know, update them and all of that and try to run in parallel so you manage to run mostly all of them. You don't need to decide which ones. And that is why I haven't finished it implemented because I never got actual data that I could use for this. So I'm just have mock data.

Awesome. I love that concept of using it to let you know what are the critical tests you should be running on a given time frame. So you brought up a point about training your test data and that's a common term I hear, but I'm not sure people necessarily know what that means. Can you just give us an example? How do you train your AI test data?

Basically.

Usually the train is related to neural network or deep learning system in which basically the machine is going to adjust itself after every right answer or every wrong answer. So you basically have to imagine you're training your machine to differentiate between cats and dogs. So you send a picture of a cat and then the machine say cats. You say okay, that's fine. And now you find a picture of a different sort of cat, maybe a Persian one. You send it to the machine and the machine may say dog and then you have to say no, it's not right. You know, that is in a way how the machine will learn. So so then the next picture you send about another Persian, maybe the previous one was white and this one is a black looking Persian cat. And the machine will say cut and then you say it's correct. So you send a third person cut over maybe a different color, maybe a different position, and the machine may say yes or not. And this is the way that the machine is starting to learn. So it kind of correct itself. So when you kind of give rewards or punish the machine when it does well or wrong, it's called like that, but doesn't really mean that you're going to Bring a whip. Whip your computer. It's not about that. You basically will send a weighted answer to the machine so the machine can use this into their computation. And then after a while of seeing a lot of different cats is kind of similar to how we learn, right? You know, after a while of seeing a lot of different cats and a lot of different dogs, we are able to differentiate and say which one is a cat and which one is a dog. Maybe when we are really small, we just think that everything is a cat. In fact, funny story, I heard that the Chihuahuas were thought to be rats for many years. I don't know if it's true or not. You know, it's kind of similar. So this is the way that the machine learns. And then once the machine has learned, you stop the learning phase. So you're not going to reward and punish anymore, and you just pass whichever image and you expect that the result is going to be accurate. It's like the exam for the machine. Do you know now or not? That's kind of how it works.

Very cool. So is there ever a point where, you know, sometimes we get locked into, I'm doing VR, I'm doing AI, I'm doing machine learning. Is there ever a point where you see AI and virtual reality merging? And, like, how do you have to account for that?

Yeah, definitely, I imagine, for example, it's. The thing is that I don't see them merging as one technology. It's more kind of being used at the same time. In the world of video games, there's this video game that is called hello Neighbor, to give you an example. And this video game, the goal is that you've seen your neighbor doing something nasty in his house, and you kind of sneak into the house and try to get into a particular room. And every time the neighbor cuts you, he will set up. You will be sent back home, and he will set up different sort of traps for you not to get the same path. And this is done through artificial intelligence. I can perfectly see how this game can be the same exact game, can be implemented into VR. And then you say artificial intelligence in order to manipulate the agents against you play that you play against. So, yeah, I definitely think that they both can easily be used together.

I thought you said you were working on a project where you're using VR to teach testing. Is that the ear test that you were talking about? Is it something different?

It's called VR museum. It's not really like, it's kind of you're in a VR museum and you look around and you find mops, I think it's called, right, the flying bug in the houses. So you, you find these points and then each of them will tell you about the type of testing. So a little bit about security testing, a little bit about maybe functional testing and tell you what it is. So you kind of playing because you're in a room and you have to find this box around.

But you also learning and I think VR is very useful for teaching people things. There are many VR applications to teach doctors how to operate without needing to be in an actual operation. Or, or I have a friend of mine that was working in an application to teach people how to solder, how to use the soldiers to, you know, to build structures without using actual physical things that will be very expensive. So that, that's really interesting application of VR. And why not? We could use it for testing. It's probably not the, the most needed field, but we could definitely also use it for testing. Why not?

Very cool. Okay Nomi, before we go, is there one piece of actionable advice you can give someone to Prove either their VR, AR AI based test automation or testing efforts? And let us know the best way to find or contact you.

So you can contact me on Twitter, I'm also on LinkedIn. I have a blog, so you can also leave a comment if you find something interesting. I would really appreciate that. Or if you want to give me some feedback. And in terms of what can you do, it's just like with everything, play around, attend events, meetings, meetups. If there's none close to you, create one and start checking into things and be curious. I think that's one of the reasons that I think that testers are similar to links. And that's why I call myself the Test Links in many places. And it's because links are very, very curious. So be a tester via links, be curious and go and do stuff.

Thank you Nomi for your automation awesomeness. For links to everything of value we covered in this episode, head on over to testtalks.com223 and while you're there, make sure to click on the sign up for a free trial link under the exclusive sponsor section to learn all about Sauce Labs awesome products and services. And if you head over to automationguild.com you can start to see some of the speakers I have lined UP for the 2019 online conference dedicated 100% to creating automation awesomeness. It's the biggest one in the world. So if you want to get in early, head on over to automationguild.com put in your email and you'll get info as soon as I open up the guild for registration. So that's it for this episode of Test Talks. I'm Joe and my mission is to help you succeed with creating Automation Awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Talks podcast. Head on over to www.testtalks.com.com for full show notes, amazing blog articles, and other automation awesomeness.