235: OpenTest Automation Framework with Adrian Theodorescu

Hit it.

Welcome to the Test Talks podcast, the place to go to geek out on software testing. And now your host whose mission is to help you succeed with test automation, Joe Colantonio.

Hey, it's Joe. And welcome to another episode of Test Talks. Are you hungry to learn more about a new open source framework this year? Because that's what we'll be talking all about today in this episode with Adrian Theodore Escu, a developer turned tester with over 17 years of experience. Adrian shares his involvement with creating Open Test, which is a free and open source functional test automation tool for web applications, mobile apps and APIs. And based on Adrian's description of it, it could have you soon saying, mm, mm, mm, mm, mm. I'm loving it. Check it out.

Test Talks is sponsored by the fantastic folks at Sauce Labs, the cloud based automated testing platform that eliminates the need to maintain your own selenium grid and test infrastructure. Try it for free today. Visit testtalks.com and click on the Sign up now link under the homepage Sponsor section.

Hey, Adrian, welcome to Test Talks.

Hey Joe. Thank you for inviting me. It's an honor and a privilege to be here and thank you for everything you do for the test automation community.

Well, awesome. Thank you for that. It's an honor to have you. We've been talking back and forth through email about a new. Well, I think it's new. It's an open source automation framework called Open Test. And, and I know you have a lot of experience with automation as well, so I thought we'd talk a little bit about automation frameworks as well as Open Test. But before we get into it, could you just tell us a little bit more about yourself?

Sure. So I'm a Software Engineer with 17 years of experience professionally and at least 10 years of writing code as a hobby before that. So I've been doing this a while now. I started my software development career in Bucharest, Romania and I moved to the US in 2011 with my family. I'm now working for a large multinational company as a software architect. We've been happily living in the very windy area of Chicago for about seven years now. My journey with test automation had only started until three years ago, but you could argue the software development is automation. Right? That's what software developers do. They try to make machines to do what they want. So basically that's about it for me.

I know you have three years of automation, but you've been doing this for 17 years as a developer. What got you into automation?

That's actually an interesting story. Like all software engineers, I didn't really think that test automation or the QA practice is necessarily very interesting. However, one day my boss called me into his office and he asked me to help our QA department with, you know, getting off the ground with test automation. And I have to admit, initially I was not very thrilled. But when someone asks you nicely for help, I guess you should help if you can, especially if that someone is your boss, right? So that's how it, how it started. I didn't exactly know what was I supposed to help with. Writing code, helping people, you know, guiding them to what tools to use or how to approach this. So we first started looking at doing this. Using a commercial tool, I guess, would have been the easy way out, but for various reasons, we came to the conclusion that this wouldn't work. We have tried this quite a lot of times with a lot of commercial tools. I think we probably tried them all. We have a lot of teams in a lot of places, a lot of corners of the world, and I think it's fair to say that we haven't had a great deal of success with commercial tools.

Why do you think commercial tools for your particular situation weren't working for you?

So if you think about this, it kind of makes sense. So when you use a commercial tool, let's say you have a team of 10 test automation engineers, and we have smaller teams, like three people, and we have very large teams, like 20 or more people. So, but let's take 10 as an example. When you start your test automation project, one of the first decisions you have to make is how many licenses do we get for the tool, right? And someone comes up with an idea of, I know, let's get two. Someone says, two, that's not enough, let's get five. At some point, let's say you decide to get three or four licenses. I can tell you initially that maybe might work, but there's no way you can use those people to their full capacity if they cannot all work in the tool at the same time. Sure, someone might be off to lunch. Somebody else might be using that floating license while these guys is having lunch. But this is going to cause a great deal of stress and pressure on those people. First of all, you can only start doing anything after you get some approval from someone. And then if you use a floating license, probably have some licensing servers somewhere, and you have to connect to that server using some custom port. So you have to talk to your networking team to open the port. And you probably know some of the pain that this Causes. And let's assume that money is no issue, right? You have the money to pay for it. Maybe networking is no issue because maybe in a smaller company things happen faster. How about later on if a different team in your company wants to leverage the test automation suite that you have created, Right. What's the point in writing tests if no one is going to use them? When these guys are asking you about your test suite and how to use it, first thing that they're going to hear is that it's going to take at least $3,000 for a license and it's going to take probably two weeks for custom port to be opened and who knows what other bureaucratic hurdles, right? And I think this is actually a very, very down to earth explanation about why test automation doesn't work or it's very hard to make it work with commercial tools. And one of the main reasons why we decided to go the other route, the open source route. So at the very beginning of this project that I got involved with, we decided that we are only going to use libraries that are open source, not only open source, but that are using a liberal license like MIT or bsd and no GPL stuff or things that might put us in danger of getting sued. Someone makes a mistake, makes a change to some of those tools, you can easily get in trouble. So we wanted people to be able to get started with the tool without having to think twice, without having to ask approval from anyone or pay anyone, any, anything. That's basically the, the reason why we, we decided to create Open Test.

Awesome. So I guess for the folks that don't know what is Open Test and what, why did you create it? There seems to be, you know, you go to GitHub, there's thousands, not thousands, hundreds of automated automation frameworks already out there. What made you think, okay, there's nothing out there that we can modify, let's just go and create our own type of solution?

Yeah, that's, I guess there are many, many reasons. The short answer to the question is we just couldn't find an open source tool with the features we needed. We can talk about what those features were. So first of all, we needed the tool to be open source, right? We had no doubt in our minds that this is the right decision to make. But then we looked at various tools and one, I guess, big challenge that we have in our company is that we have to be able to work with people of all skill levels. So we have people that are very skilled, that are coding gurus and we also have people that dabble with coding a little bit, but are not very experienced. And then again, we have people that are not coding coders and don't want to be, or maybe, I mean, don't have any inclinations for that. And we still have to make sure that we are leveraging those people to their full capacity. Right? So we realized that whatever tool we're going to use is going to have to be able to adapt to all of those people. And that right off the bat excluded any options like writing Selenium tests in Java or Python or JavaScript or you name it, right? That was clearly not an option for us. The other reason, which might not be very applicable to other teams, is we needed a tool that can do seamless distributed testing. This is again a more exotic design goal, but for us it was very important because our first test automation project required or involve a mobile application and a point of sale application. And of course, those two applications run on very different types of devices and run on different devices, first of all. So you have to make sure that your test framework or tool supports this scenario. So this is actually one of the scenarios that we have best support for. When you write a test with open test, you basically almost, I mean, you don't have to do any complex setup and the test reads nice and it's easy to write. So distributed testing is one of the features that we are most proud of.

Sorry, don't mean to interrupt, but could you just tell us a little bit more about the distributing testing feature? Because it seems like it's one of the really unique features of this particular open source framework.

What I mean by distributed testing is the ability to execute test steps on more than one machine in one single test. So you have a test. Let me give you an example. You have a test that starts in a mobile application. In our case, we want to be able to display an offer for a customer. We click on that offer, we we get an offer code, and then with that code we want to go to the point of sale application and we create a new order. We put the offer code into the order, we finalize the order to redeem that offer. So at which point we are done with the point of sale and we go back to the mobile application to make sure that the offer was removed to the customer's offer page. Offers page.

That's what I mean by distributed testing.

Awesome. So before I interrupted you, what were some of the other features you were just about to mention?

Other features that we looked for is a tool that plays well with source control. So we excluded any tool that used a proprietary format for storing their data or their test logic. We didn't want to hear about using Excel for either storing test data or test logic. And for the life of me, I don't understand why so many people are so in love with Excel. I mean, Excel is a wonderful tool if you use it the way it was designed. But when you store your test data in Excel, that file is a binary file. So if you and me both make a change to that file at the same time, then one of us is going to lose his data. So that was one thing we didn't want to have to deal with. For us, it was really important to be able to work with large teams and geographically dispersed teams. So source control was one of the very important points on our agenda. And then we just wanted something that has a solid, elegant design, you know, an extensible design. Something that can adapt to problems that we don't know about yet. And we all know. I mean, if you've been in software development for a while, you know, first of all, test automation is software development. This is why I kind of put an equal sign there, a different type of software development, but nevertheless. And if you've been doing software development for a while, you know that there is always something you haven't thought about. And this thing will occur maybe one year or two years down the line, but you have to be prepared to handle that in some way. This is what Open Test was designed to do. Extending Open Test is very easy without changing anything from the core test framework or test tool. It's simply a matter of creating a Java class that derives from some base class that Open Test provides. And you override one method, whatever you put in that method is what's going to execute when you run that keyword that you're implementing. So suffice to say, it's very easy to extend. And we already proved that we have a desktop application, our point of Sale application, which is not a very easy to test application. It's a Windows desktop application, but it doesn't use any Windows native controls, so you cannot identify objects using the usual type of techniques. It's basically one big image that you need to be able to click and interact with. And we wrote a number of custom keywords in order to be able to test this application. They are of course not open sourced, but with that we proved that open source is extensible and everything works the way we designed it initially.

Awesome. Now it looks like it supports Selenium Appium. You have Support for rest. So is the framework basically a wrapper on top of these that I think you mentioned? Keywords like give you keywords that you can use rather than starting from scratch. It's almost like scaffolding that makes it easier for a newcomer to come in and just start working, not having to build out all these other reporting and other tools like that.

Yeah, yeah. So one of the things that I try to avoid, and I think most software engineers, if they have.

Some experience in the field they try to avoid, is reinventing the wheel. Right? So you definitely don't want to solve any hard problems that someone already solved for you. So from the get go, we knew that web testing and mobile testing, those are hard problems. And we looked to the market for libraries and in particular open source libraries that can do that and do that well. And I think we can all agree that Selenium and Appium are two of the libraries that are basically proven and solid and are the undisputed leaders in their segments. Right. I know there's a lot of talk in the test automation community about new and revolutionary ways to do web testing, but I'm having some. I have some reservations about the effectiveness of those tools, at least for the time being. So when it comes to AI or when it comes to designing custom tools based on Node JS and Electron and the Chrome engine and embedding them in desktop applications, you know, it's kind of a long shot for me. There are projects that are doing a lot of interesting things and I'm keeping an eye on those projects. But I believe that one rule of thumb we should follow with test automation and software development is stick with what works, stick with what's proven, don't assume risks unless you have to. For us, it was a very simple decision. Selenium and Appium are really nice libraries and well supported libraries. And for API testing, that's actually much easier because Java has already a lot of libraries that can do that very well. We chose to go with the Apache HTTP client library, and speaking about Java, one other important decision that we had to make at the beginning of the project was the programming language that we wanted to use. We chose to go with Java because it's a mature language and it's basically the language of test automation. I know different people have different opinions on that, but if you take a look on Stack overflow, I think you'll find it's much easier to find answers and help if you're using the Java language. So this is why we chose to go with Java and Of course, one of the important reason was that we did need this to be cross platform and we wanted to support as many devices and scenarios as possible. So if you look at the architecture of Open Test, you'll find that there are very few test automation scenarios that we cannot support. So as long as a device can run Java, you can use that device as part of your distributed test and that happens seamlessly. It's very straightforward to write a distributed test with open test. So that was another design goal. And I already mentioned we wanted the tool to be cross platforms. So this excluded some other very good test automation tools. I don't want to give any names because I wanted to sound like I'm bashing anybody, but there are definitely a lot of good tools out there which only support Windows and we could not afford that. So, yeah, for all of these reasons, we thought that our best choice is to go with writing a tool from scratch. I guess if you were to ask me, one of the tools that came closest to what we needed was probably the robot framework. But the robot framework also had a few things that were not right for us. So first of all, they use Python. Python is, I think, a very nice language, but as far as test automation, you have a better chance of success with with Java. And the other thing is the distributed testing aspect that we were using heavily is not very well implemented. And yet another thing is the syntax for writing tests. We didn't think that syntax was very nice to use, especially for beginners, and it had a few design decisions in there that we didn't like. We wanted a tool that does a very nice separation of concerns. So when you write a test, you want to only see in there the things that you need to see for that one test that you are writing with robot, the way you write the test. I mean, I'm not an expert in robot framework by any means, but suffice to say, there were a few syntax issues there that I didn't think were fitting well with our goals.

Awesome. So I guess another thing that comes up a lot, regardless of what framework you're using, especially Selenium and Appium, is flaky test. And a lot of times I see it's due to people aren't using the right weight mechanism. So because you built a framework on top of these tools, did you add any other functionality that helps? Maybe someone that's not really a developer or someone that doesn't understand automation to bake in some sort of way to make the test less flaky?

Yeah. Thank you for this question, this is actually one point that is really important. It's a pet peeve of mine actually. I want to do a little rant. I actually just wrote an article called Selenium is not Brittle. You're just not using it right. And what I explained in that article is that Selenium is a tool and it is a very good tool. However, if you think that you can click an element by doing element click, then you have a problem, right? Because there's a whole host of things you have to look at before you do that element click. So first of all you need to make sure that your element exists in the page and that's probably what every test automation developer out there knows already. So most elements nowadays are part of a single page app and they are not in the page from the very beginning. They are created dynamically from some asynchronous JavaScript that runs in the background. So that's one thing, just wait for that element to be present. But after that you still have to make sure the element is not covered by something else. So actually there's something even before that you have to make sure that the element is enabled and is visible, right? A lot of elements are created, but they are hidden initially and they are only shown based on some conditional logic. So you have to make sure that the element is visible and enabled. After that you have to make sure that the element is not covered by some, I know, lightbox element or progress indicator or who knows what. Right? And even after that there might still be reasons for the click to fail. So when you click, make sure you catch that exception and make sure you retry the click because it might not work the very first time. So when you put all those things together, you'll find that you end up with quite some complex, with quite complex logic. And that logic is obviously not going to easy to implement by a junior developer. And unfortunately most of the times we ask those junior developers to write test automation and that's I think one of the problems in, especially in large companies. So open test is trying to shield the test automation engineer from all of these problems. And frankly, even if you are an experienced developer, this is still the right way to go. Just build these pieces of higher level logic, encapsulate them nicely and then hide them away and use them from some library. So you don't want to have to deal with all this complexity every time you do a click. Right? So to answer your question, yeah, we have a lot of logic in there that was rather painful to to build and stabilize and get right. That makes sure that all those weighting mechanisms work right. And especially on the appium side, where you don't only have to deal with the problems I explained, but you also have to be able to swipe to get to the element you want to interact with. And that's much harder to solve problem than you would think. You have to think of all of these things, like do I swipe in the whole page or maybe do I swipe in some smaller scroll view? If my page has multiple scroll views, which one do I want to swipe in and how long do I want to swipe? Did I reach the end of the page? How do I know that? There are so many things to look at, I don't even know where to start. So I guess all this is to say is don't try to reinvent the wheel. Try to use the work of people that came before you. And that's what we try to accomplish with OpenTest. Give people a tool that they can rely upon.

So I work for a huge company and I'm not going to mention companies I know you work for. A large company may even be larger than the company I work for. So is this a tool that's used in house by everyone or is it just by like how many people actually are using this tool in house?

Yeah, so this is interesting. This project started, it was in the beginning.

I would call it an unsupervised project, meaning that we had all of the freedom in the world to make our tool selection and to make our design decisions. Nobody cared about the tool that we're using. Everybody was just looking at trivial indicators, like how many tools, tests did you write, what percent of the manual tests did you automate and stuff like that. Right. So I think that was one of the main reasons why we succeeded with open Test. And yeah, this started in one of the teams and we have countless teams, test automation teams across the enterprise. So after about one year, people started hearing about what we were doing and started asking questions and maybe some of the teams were toying around with the idea of starting to leverage OpenTest. It wasn't even called OpenTest at that point. We came up with a name later on when we decided we wanted to open source it. Eventually, I think after about almost two years, some teams started to leverage it and yeah, word got out, more and more teams adopted it and I think it's to a point now where it's kind of the de facto functional testing tool across the enterprise. A couple of very large vendors, very large Consulting companies that are leveraging it now and they started leveraging it for other accounts. Also, the tool is not mandated by the company. There's no written document that says if you do functional testing, you should use Open Test. But it just I guess emerged as the de facto standard.

Is any user case where you think someone should not use this framework?

Yeah, there are a couple of.

Use cases that don't make Open Test the ideal tool. First of all, we don't support desktop application testing yet. It would be a very easy thing to add, but we didn't have this need. I was telling you that the only desktop application we needed to support so far was this point of sale application which.

Wasn'T implemented using Windows native controls. So we had to write some custom logic to test that. Anyway, so we are not doing desktop application, but there's a really exciting project that was started by Microsoft called WinApp Driver, which is basically an Appium driver for Windows applications. So you can basically use the same approach that you use for mobile testing, using Appium to test Windows native applications. Right. So they, the whole webdriver thing and the whole paradigm that Selenium introduced, you can use that now to test Windows applications. And when I say Windows applications, I mean Windows Forms, I mean native controls for sure, Windows Forms, wpf, you name it. And it's supported by Microsoft, which I think it's a great.

Thing. So desktop applications we don't do yet. Then of course, if you talk about performance testing, Open Test is probably not the best to do that. Security for sure, not the best tool to use. So Open Test is geared towards functional testing and in particular to Web, mobile and APIs for the time being.

Okay, Adrian, before we go, is there one piece of actionable advice you can give someone to improve their automation efforts or to improve their open test automation efforts? And let us know the best way to find, contact you or learn more about Open Test.

Sure. So one advice that I would give to test engineers is to keep in mind when you implement your tests that you can add levels of abstraction to your test as necessary. And I'll translate that in a minute. This will basically future proof your tests and make it easier for you to write tests. So what, what do I mean by adding levels? Levels of abstraction? If you want to click a button or if you want to put some text in a text box or whatever it is that you want to do, you should stop for a second and think about what is it really that you, you're trying to do there. And most of the times you'll find that what you're doing is you are clicking a button to add a product to the shopping cart. Or maybe you are filling a registration form or something like that. Well, if you're filling a registration form, then what you should do is you should take all that logic to fill that form and you should move it into a separate utility method that deals with that complete form. It interacts with all of the elements in the form and at the end it clicks the submit button, call that method something like Submit registration form or whatever, and there it is. Now you can use it over and over in hundreds of tests and when the application changes, maybe they add a new mandatory field into that form. All you have to do is go back to that utility method and change that one method and all of your tests will continue to be working. And yeah, don't fall into the trap of thinking it's easier if I don't create this abstraction layer, and I would probably be gone from the company by the time we need to change this anyway. This is a very slippery road and most of the times it comes back to haunt you. And even if you do leave the company before they change the app, it's still bad for your karma points. So not something you want to do. So that would be one actionable advice, I guess. And for the managers out there, my advice would be do everything you can to go with open source for test automation. Maybe open test is not the solution for you, but whatever that solution is, make sure you are using open source and only use commercial tools for things that really cannot be done with open source. Or if you have any other compelling reason to choose a commercial tool.

Thank you Andrew for your open source automation awesomeness. For links to everything of value we covered in this episode, head on over to testhalks.com235. And while you're there, make sure to click on that. Sign up for a free trial link under the Exclusive Sponsor section to learn all about Source Labs awesome products and services. And if you haven't already registered for the Automation Guild, why not? What are you waiting for? Register now and experience automation awesomeness in the new year and discover how to accelerate all your team's test automation efforts. Register Now. Head on over to automationguild.com so that's it for this episode of Test Talks. I'm Joe and my mission is to help you succeed with creating automation awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Talks podcast. Head on over to www.testtalks.com for full show notes, amazing blog articles and other automation awesomeness.