Node.js Desktop Automation Testing Framework with Simon Hofmann

Foreign.

Get ready to discover the most actionable end to end automation advice for some of the smartest testers on the planet. Hey, I'm Joe Calantonio, host of the Test Guild Automation Podcast and my goal is to help you succeed with creating automation awesomeness.

Hey, it's Joe and welcome to another episode of the Test Guild Automation Podcast and today we'll be talking with Simon all about desktop automation testing frameworks for Node js. Simon is a self taught, half trained software engineer who likes to dive deep into technical topics and cares about semantic versioning, testing and API design. Has a really interesting history behind what he's done and a really cool solution I think you're really going to enjoy. He's been working on Nut Tree Nut js. It's an ecosystem to make JS desktop automation more enjoyable. So really exciting to have on the show. You don't want to miss this. Check it out.

This episode of the Test Guild Automation Podcast is sponsored by the Test Guild. Test Guild offers amazing partnership plans that cater to your brand awareness, lead generation and thought leadership goals. Get your products and services in front of your ideal target audience. Our satisfied clients rave about the results they've seen from partnering with us, from boosted event attendance to impressive roi. Visit our website and let's talk about how Test Guild can take your brand to the next level. Head over to testguild.info and let's talk.

Hey Simon, welcome to the Guild.

Hey Joe, thanks for having me.

Great to have you. So is there anything else you want the Guild to know more about? I looked a little bit in your background and seems like you've done a lot of different things. So just curious if you can maybe expand on the bio that I gave.

So basically the bio sums it up pretty nicely. So before I dove into the topic of computer science, I used to work as a mechanical engineer. But I've been interested in computers throughout my whole life. And once I finished my basic training as a mechanical engineer, I decided that I should continue to go to school and that ultimately led me to end up with a CS degree.

Nice. How did you get involved in testing though? Because it seems like you have more of a background in development.

First and foremost, I'm a software developer. Yes. But in one of my previous gigs, one of my previous companies, I've been on a development team that was specifically tasked with developing an end to end testing framework. So it somehow relates. And also development is also pretty close related to testing. So whatever you develop, you want to have tested.

Absolutely. I just Find sometimes developers don't into testing. So do you have any recommendations of maybe for developers to get them the tools that actually work better for a development type of environment to get them more involved in testing their code more frequently?

Once you get started developing a greenfield project, a proper testing setup should be the first thing you should consider when you get going. So I wouldn't start a project without having a proper test set up at hand. And I mean essentially it only gives you benefits. So let's say you're developing an application, let's say a backend, and your backend has a whole suit of reliable and meaningful integration tests, for example, your future colleagues will be really grateful, I can tell you, because having a meaningful set of integration tests allows you to get started developing and evolving your application without worrying of breaking something. Because if you have your business logic and the stuff that provides business value to your company properly tested, you will definitely be sure that you are not breaking something in lose money.

Absolutely. I think when people think of automation though for developer, they think of unit testing. So do more than unit testing. Seems like you're also more involved in integration and end to end testing.

I don't know who coined that sentence, but there's a saying that you should write integration tests, quite a few of them, because integration tests give you, in my opinion, more confidence than a unit test. And they also allow you to move quicker when you are refactoring something. I mean unit tests, there are use cases where a unit test is definitely the right solution. But when thinking about complex systems that interact with other distributed systems, you should definitely keep an eye out for unit tests. And let's say if you add a front end to your whole application stack, then you should definitely also dive into end to end testing. Node js, for example, basically combines all three of them because there are unit tests on code level, but there are also integration tests when it comes to, let's say, loading third party plugins. They are connected via defined interfaces. And there's also a whole suite of full end to end tests that basically evaluate the whole function of the framework.

Absolutely. So I want to dive a little bit more into NUT js since you mentioned it, I guess at a high level, what is NUT js? For people that don't know that's never heard of it.

So Node JS is a Node JS framework that is written in TypeScript and it's a framework that allows you to automate your desktop. And by automating your desktop, I am talking about simulating keyboard input. Simulating mouse inputs, accessing clipboard content. And you also have the possibility to add plugins to the core framework which will allow you to perform on screen image search for for example, or ocr. So you can essentially work with whatever you see on your screen.

Cool. So I think when people think of front end automation, especially in this day and age, they think of a browser application. But it seems like this was made specifically for desktop. So why did you create a tool for the desktop? Did you find that some tests required more than a browser?

Not everything happens inside a browser these days, and especially if you think about an end to end testing scenario where you have, let's say a web application, let's say a webshop, and you want to create an end to end test that verifies the full ordering process. So let's say a customer accesses your web shop, puts something into the basket, performs a checkout and then let's say he receives an invoice. But the invoice is a PDF file and you want to validate that what ends up at the customer site is actually a valid PDF with the proper content. Then that would happen outside of a browser.

Absolutely. So I know you've been developing this for a while, so. And also I think it has like a lot of stars on GitHub, like 1.5 or more. So what use cases do people usually use this for? I know there's a lot of choices out there, not that many for desktop though. So that's why I was really excited to see it. Like have you seen people use this and how long have you been developing it?

So I'm developing it since August of 2018. So it's not a 50 year, well use cases where it's been used. There are quite a few. So actually a lot of people are using it for game automation. But what made me really happy is that I noticed that there's a free online school that teaches JavaScript or Node JS development and they are using it to build a WebSocket server with their students. So that's essentially that's a remote control. So you can remote control your desktop via WebSocket connection. But there are quite a few of other topics that Node JS is used in. I know of one example that uses it in a video conferencing tool where this can share or where you can access your peer's mouse, for example. And the underlying technology that is used to perform the remote code execution or to the remote mouse movement is not JS and maybe you know, scriptkit and the underlying technology that performs keyboard actions in Script Kit is not JS nice.

So yeah, what I love about this, it seems like you set it up also to be expandable like you mentioned. So there's different kinds of plugins and one of them that caught my attention was visual automation and testing. So who creates the plugins? Are you creating all these plugins on top of it? Like who maintains it, who's creating the plugins? And maybe talk a little bit more about the different plugins and the different types of things it can do.

So when I started out developing nutjs, there was no plugin system back in the days, it consisted of essentially what is nowadays the core of the framework. And that core also contained all the code that was used to perform on screen image search. And after a while I realized that this was bit of a problem because most users were not that interested in on screen image search in the first place. So when they started out they were mostly looking for mouse and keyboard automation. And back in the days.

All the code that was used for on screen image search was limited in which node versions it supported. So naturally it was always a problem because I was always running behind updates, I had to recompile binaries and all that stuff. So people continuously asked whether it would be possible that they use Node JS without all the image matching stuff. Because without the image matching it was usable with all the recent node versions. And that led me to think about the whole architecture of the project and I started to think about that. It would be cool to have a plugin system that allows you to use the core on its own, but if you would for example want to use on screen image search, then you would just have to install a plugin, it would self register and you would have all the functionality in place. So that led me into refactoring the old framework, moving the image matching code into its own plugin. And at the moment I think I am the only person that's actively publishing new plugins. But there is also forks of existing plugins from myself that for example say updated to recent versions or add support for additional node versions and stuff like that. All of the plugins that are listed on the NodeJS website are from myself.

Nice, but I also like it's cross platform. So if I'm correct you can run against Windows, Mac, Linux, all the platforms, which is pretty cool.

That was also one of the main goals I tried to achieve when I started developing it, because one of the key motivations why I actually started developing my own tool was that all of the existing frameworks were not that actively maintained or more likely unmaintained at that time. I also wasn't happy about the current state of these frameworks because all these desktop automation frameworks are essentially relying on binaries. So you have a lot of C code that accesses operating system level APIs to perform all the automation for, let's say mouse movement. And all of the existing frameworks did not provide proper way to easily install all these tools. So you would have to recompile all the native code on your machine once you install it. And that would be a big problem if you would want to integrate the package in any kind of application you want to distribute to users, because you would be on your own when it comes to how do I manage package distribution, how do I cross compile stuff for different platforms? So that led me to think about how could I manage.

To have just a single package I can install via npm install like the default way in the Node JS world and it just works out of the box.

So yeah, that's another problem. Sometimes some of these tools require a lot of step and this seems like it's just from the command line, one install command and there it is. Is there anything anyone needs to know before they use it? Do they need to have CERN anything else installed on the machine in order for it to work? But like you said, is it truly self contained?

On Linux There are two requirements for X11 libraries to be installed, but on both Mac and Windows it's really just an NPM install and everything runs out of the box. And it's also the same for the image matching and the OCR plugins, because these plugins are essentially also binaries. And in most frameworks that also do on screen image search, most of these frameworks are using OpenCV in the background. So NodeJS does it as well. But for the most part what I have seen on other frameworks, they are relying on users to install OpenCV binaries on their system. And I can tell from experience that installing OpenCV on Windows is different story than installing it on a Linux box, for example. And then you would also have to make sure that you have installed the right version. And that was also something I really didn't want my users to care about. So the image matching plugins I provide are fully self contained. They ship all the required libraries and you can just install them, wire them up in their code and they are ready to go.

So why did you use Node JS or JavaScript rather than Python? Because when I look at your history, it seemed like you started off using Python. Is there a benefit or is there a reason why you chose this tech stack to write this in?

I used to write a lot of Python when I was still in university, but over time that focus really shifted towards the whole Node JS and JavaScript ecosystem. And nowadays I hardly write Python anymore. So you could say I'm all in on the JavaScript and TypeScript ecosystem.

Nice. So talking about that, I think this also integrates with popular libraries like Jest. So how does it plug into Jest or how would you use this with Jest?

That was actually an experiment that I started out of curiosity because I started thinking about, okay, I'm now able to search for something on my screen, but how would I use that in case of actually writing a test? And then I well, so all the tests I have in Node JS are written with Jest. And so I started thinking about maybe I can integrate it with the existing framework I'm already using. And I started reading up on the Jest documentation and I noticed that it's pretty straightforward or pretty easy to write your own extensions for the framework. So I gave it a try and added my own matchers. So now let's say you want to, for example, verify that your screen is showing an image. Then you could write a typical Jest expectation in the form of Expect screen. So you pass the NodeJS screen object and then you can formulate your expectation in a form of to show and the path to an image file you have on your system. And so the integration with Jest basically came out of the same motivation I had with the overall API design, because I think it's a really nice way to structure your test expectations the way it is done in Jest or in Mocha, by writing expect whatever test object you have. In my example, to show an image on my screen or not to show it makes it really easy to read and understand. What's the actual expectation of your test cases?

I think I also saw something about Sakooli and Selenium. So does this integrate at all with Sakuli or Selenium? Or placement for solutions like Sakuli or Selenium? Or is there any use cases where you would use this with Selenium? Anything like that.

So what is the connection between Nudge and Sakooli? Well, I started developing Nude JS in my free time, but, well, I was one of the lead developers of Sakuli.

So back in the days when I took over the development and maintenance of Zakoli. It was still in version V1 and was written in Java. So it was a mixture of Java code in combination with a Nashorn JavaScript engine that was integrated in the JDK or the JVM. And once I had the prototype of nude JS in place that validated that it is possible to perform a desktop automation with JavaScript, I pitched a rewrite of Zakooli v1 with Node JS.

So is Saculy is this still an open maintained project?

So I'm no longer with my previous employer and I don't think that it's currently maintained.

Gotcha. So you also mentioned, you know, in your bio, you know modern software testing relies on containerization. Kubernetes Docker Stupid question is any way you would use NUTS JS to help with maybe maintenance of environments or, I don't know, setting up environments for you automatically.

So I wouldn't recommend it if there is a way to perform it via, let's say an automation tool like Ansible or Terraform or something like that. But if you have a setup that requires user interface interactions, then you could definitely do that. And one thing I'm considering for the future of Node JS is also a remote plugin that allows you to basically write code on one machine, execute it on one machine, but the actual execution or the actual interaction with your mouse and keyboard happens on a remote system.

Oh, that's neat. Now why would you do that? Just for remote provisioning of things or setting things up remotely, obviously.

Exactly. That could be one use case. Yeah, if you distribute it, or if you distribute the remote calls over a bunch of machines, you can automate setting up whatever applications.

Awesome. So what's. I don't know if you just released it, but on the website it says release of version three. So what's new in version three that added to your already existing code base?

So with version 3 I restructured the plugin interface so that with version 3 it is now possible to provide arbitrary plugin data. So if you would write your own plugin for Node js, you could define an interface of your optional data your plugin needs or your plugin is able to process. And that would give. Or that gives plugin developers a lot more flexibility on how they can add additional functionality. So let's say your plugin should allow users to pass in. So one example is from my second image matching plugin I created. And that plugin allows users to specify the amount of scaling to apply when searching for an image. And this is something that is not contained in the default interface for plugins. But with this new arbitrary data you can pass along to a plugin or you can pass down to it. It is now possible to configure that anyways. And along with release of V3, I also released that OCR plugin I mentioned earlier. So that gives you the possibility to search for text on your screen. So you could say screen dot, find, single word hello and it would locate the text on your screen. So you could use your or you could move your mouse towards it, which gives you a really nice way of interacting with your ui. Because you can enter, you can restrict the search area to look for the text to let's say the area of the currently active window. And in case of let's say a pop up you have to confirm. You could limit the search area to the area of that window and still say screen dot, find single word, confirm. And this way you could move your mouse cursor to the button that says confirm. Click it, close the pop up. Stuff like that.

Nice. How accurate is that? Do you have to worry about pixels differences on different machines? If you've developed on one and put on another? Is that just old school thinking that I have?

So for all the image based stuff that's currently still a bit brittle, so if you switch it from one machine to the other with a different screen resolution, it probably will break because the image locate so that the algorithm that performs the image search is not that flexible when it comes to different resolutions. That's something I still want to improve, but I don't find the time to do so.

So for the OCR run I would consider it pretty stable because most of modern desktops do have high enough resolution for accurate OCR results, my guess would be that OCR should be stable between different machines.

Just a random question. I was just looking through the release file and I saw something about Jimp Security Advisory. Is that something? What is Jimp?

Jimp is an npm package that NodeJS uses internally and a while ago it had a CVE that was not yet addressed in an official package release. So what this advisory does is that it basically gave nudges users an instruction on how to force transitive packages to be resolved in a way that the CVE did not affect them. Gotcha.

Cool. I guess if someone's listening to this and like, wow, I need to learn more, I'd love to get this implemented, but I need a little help. Do you offer consulting services?

If you ask me, I'm happy to help.

Awesome. All right Simon, before we go, is there one piece of actual advice you can give to someone to help them with their automation testing efforts and what's the best way to find or contact you?

The easiest way to contact me is probably via email, so you can either reach out via the contact that is listed on the nudge JS website or hit one of my websites like s1h.org and you can also find me on Twitter GitHub. I'm running a Nudge JS Discord server. Happy to onboard, no users there.

And well I actually started the Discord in the hope of catering some kind of community around the framework, so I'm always happy if new users or new people join.

Thanks again for your automation awesomeness. The links to evident value we covered in this episode. Head on over to testguild.com a442 and if the show has helped you in any way, why not rate it and review it in itunes? Reviews really help in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack automation awesomeness. As always, test everything and keep the good. Cheers.

Hey, thanks again for listening. If you're not already part of our awesome community of 27,000 of the small artists, testers, DevOps and automation professionals in the world, we'd love to have you join the fam@testguild.com and if you're in the DevOps automation software testing space or you're a test tool provider and want to offer real world value that can improve the skills or solve a problem for the Guild community, I'd love to hear from you. Head on over to test guild.info and let's make it happen.

Sam.