241: Automate The Planet with Anton Angelov

Hit it.

Welcome to the Test Talks podcast, the place to go to geek out on software testing. And now your host whose mission is to help you succeed with test automation, Joe Colantonio.

Hey, it's Joe. And welcome to another episode of Test Talks. Today we're going to be speaking with Anton Angelov, the author of one of my favorite blogs, Automate the Planet. Anton is the CTO and co founder of Automate the Planet and inventor of Belltrek's Test Automation Framework and the MESA Distributed Test Runner. So he has a ton of experience with automation testing. I've been wanting to get him on my show for a while. I finally reached out to him and he agreed to do the interview. So a lot of good stuff here. You don't want to miss it. Check it out. Also, I just realized I've been doing the podcast officially now for five years. I can't believe how fast time has gone. I've been so blessed to speak with the biggest names in the test automation field. So I'm so thankful for you for listening in and supporting us through all these years. Thank you. And if there's anything you want to hear about or know more about, or you want to be a guest on the show, why not reach out to me@joesttalks.com.

Test Talks is sponsored by the fantastic folks at Sweet Dos Labs, the Cloud based automated testing platform that eliminates the need to maintain your own selenium grid and test infrastructure. Try it for free today. Visit testalks.com and click on the Sign up now link under the homepage Sponsor section.

Hey Anton. Welcome to Test Talks.

Hey Joe. I'm happy to be on the show finally after so many talks over LinkedIn. So. So yeah, awesome.

Today I'd like to talk about all your experience around test automation, especially more about your site Automate the Planet, as well as some tools that you've created in open source. But before we get into it, can you just tell us a little bit more about yourself?

Yeah, sure. Right now I'm running two businesses. I own two companies. Also I'm working part time as a consultant. In my free time, I really enjoy reading books. Not only technical stuff, but about psychology, history, anthropology, sociology, etc, etc. Because I really believe that knowing all about this stuff can help you to understand the world better and help you to manage tough situations and choices. Besides that, I try to write articles almost each week. Recently I started to write science fiction novel.

Awesome. So you do speak around the world on automation and you seem to create a lot of different solutions around automation. I guess the first thing I noticed was in one of your sessions you gave some reasons why you decided to make your framework into a product. Can you just tell us a little bit more about that?

Well, part of my job in my past company as an architect was to write similar frameworks and test automation solutions. And because right now in Bulgaria, I'm like all people in the field know me, many companies wanted for me to hire me to build similar frameworks for them. However, on the third time when I visit such a company, I told them that if every time I need to create again and again the same thing, it will be really hard to maintain it. You know, because in my past companies, because you are in a rush always to test the new features and like management usually don't spend so much time allowing you to maintain and write new features to the framework. Usually the framework is not well documented, it's not well tested, etc. And because of that, I really decided to create this framework as a product so that we can concentrate to be high quality, having all the features to be well documented and well tested, so that the support of it is tensorwise, so that it if I go to consult company Hicks, I will just let them use things that we develop and handle the support. Not like supporting 10 solutions. That was the reason.

So I guess a lot of things that people get confused with is a framework. What is a framework? I think in one of your sessions you talked about the difference between a library, a framework and a test automation framework.

Actually the framework is. Many people think that they own a framework, but they don't actually own a framework, they own a library. Like they just implemented a few custom methods around Selenium webdriver and they think that they have a framework. But this is not a framework. The framework is a complete solution that have many things. For example, it implements high coding standards like using tools like StyleCop for, you know, these warnings in the IDEs mentioning the coding standards. Another thing is for example the integration with reporting portals like Zafira or allure and etc etc. The framework also gives you the possibility to extend and customize the solution because many people just hard code everything and you know, usually these libraries are not scalable. In my past company I had to write a framework that was used from more than 10 teams. And when you see such a scale, usually these teams had to test and write automated tests about different websites and they had different needs and usually the initial implementation was not working out of the box for them. So we had to think of ways how to allowing them not just to write just another method in the framework, but to be able to customize it for their needs. This is part of the framework to be extendable and customizable. This is also the same is valid for the different environments because usually these automated tests have to run production on, test on, continuous integration and so on and so on. This is my short answer.

Cool. So you mentioned it needs to be able to be customizable and extensible. So are there any way we can actually write test that test framework to make sure it's working properly?

Yeah, exactly. Actually this year I will talk at Appium Conf in India and this will be my thought there. How to test the framework because the part where the framework is a product is where it is tested well. And to make sure that it's working in each update of the underlying framework, such as Appium, WebDriver, etc etc tested well. And because of that for our framework we created I think more than 5,000 tests besides all the unit tests, for example type of tests, where I call them warning tests where for example, if the framework wraps some action of the WebDriver and you know how you use it in the framework, you create similar tests only with WebDriver so that when you update the WebDriver if some of these tests fail, then you know that it's not good to update the framework. This is especially true for Appium, because in the new version, at least in C, half of the old features are not working yet. This is how we decide whether to Upgrade or not. WebDriver IPM in the framework. This is true for the browser integrations because as you know, for the different browsers there are different.

Drivers. So we have a single test.

Which tests the different features of the framework executed against all of the browsers. And we do this upgrade tests. For example, we update the Chrome driver, then we execute the test and see if they work correctly or not. If they are failing, just try to make a fix for the browser or yeah, whatever. But the thing is that we test owes against all browsers and since our framework is cross platform, this means that you need to repeat all of these tests all operating systems, meaning Windows, Mac OS, Linux, etc. Because we noticed that if something is working from Chrome on Windows, it's not always working on Mac.

So besides that, I think another thing people struggle with is third party dependencies. Is there a way that people can test that as well before they start taking in third party dependencies?

Yeah, again because of that we have actually test the framework and we have continuous integration and test environment to test all of this. For example, we distribute our framework as nuget packages and usually we have a live version which is well tested and we know that it's working with the third party dependencies that are in it. However, since we use git we usually create for the new releases a second branch which means release higs branch and then we use I think it was called myget the service where we use it like test environment for deploying our new release packages and our continuous integration migration actually executes against the test environment. Meaning that if I see that there are updates for any unit, Ms. Test, etc. We update them and then we start an automated build checking whether all of our framework tests are passing or not. If they are not, I just observe whether why not fix them that it's the and if something that is in the third party dependency I see if I can fix it myself or just write a support ticket or submit an issue in GitHub to the to the owners of the third party dependency. This is the case for example with APM and WebDriver. I have submitted lots of issues there.

So as I was preparing for this interview I came across something you you may have said. I don't know where I found this, but something about the full stack automation frameworks are the fifth generation of tools. What do you mean by full stack test automation frameworks?

By fifth generation I mean usually right now there are lots of open source test frameworks which are great like Selenite. There are also a few others that are popular ggi. I think they had a solution in Java and C. There are more others. However, most of the frameworks and custom frameworks that people usually build they lack a few things that I think are important for the new generation of tools. For example, most of frameworks are dedicated only to testing web. Usually they lack of tooling for troubleshooting. Many of them already mitigated the issues with the flaky test. For example, they have features or automatic way of weighting of elements. But if the test fails usually they are not so good in finding out what is the problem. So I think one of the things that is important for this new generation of frameworks is is to allow you to troubleshoot better the results. For example easy way to create video recordings of the failed test. Way to create full page screenshots on test fail. Another thing is that as mentioned I think that the new generation of tools should allow you to automate everything like the WebDriver star approach, meaning that you need to be able to automate desktop, web, Mobile, Android and iOS API etc. However, since this is a single framework, the API like the methods and the way how you write this should be similar. Meaning that it's like really tough thing to do when you design a framework to write it in a way because you know these are very different technologies, it's a little tough to design it so that the way how you write this is almost identical. Because I think this type of frameworks should not be so tough to learn. Meaning that the learning curve should be. You don't need to read the whole documentation to learn to write a new test. I think also the era where this type of tooling is working only on a single operating system is over. Meaning that it should be cross platform and allow people to use different IDEs and different operating systems. Because of that most of the technologies are going like completely cross platform. And I think the most important thing about this full stack frameworks as we talked to be extendable, extensible and customizable. Meaning that if they're so generic they should allow people to write their own plugins to allow them to override the behavior of the existing behavior of the framework and actions, etc etc so that it can fit their needs and their team.

Awesome. So speaking about frameworks, I believe you created your own as well called Bellatrix. So is Bellatrix something you would consider a fifth generation framework? And if so like what are some of the baked in features in Bellatrix that gives people easy to get started right out of the box with all this functionality?

Yeah, Bellatrix is generation framework and we strive to build it that way. The biggest benefit that it uses the core frameworks like WebDriver, API, WinAppDriver, Rest, Sharp, etc etc under the hood however, to build this kind of features around them because they are not frameworks, you know, they are just libraries for automating browsers, phones, etc. The thing that we built on top of them is for example, besides the troubleshooting features that I already mentioned, one of the things that we invented is called getting started guide, meaning that you just download a complete solution, folders inside it, having completely working examples with comments, etc. So that you can check how each feature is working and on top of that there are some exercises for each feature so that you can write your own test etc. Or extend the solution. And I think this is like much better approach than just like the old approach where you teach people yourself or give them like 200 pages long guide how to write tests Another thing is that maybe the coolest thing of all is we have like over 10 things to allowing people to customize and extend the framework. Many of the features that actually we built like video recording contest failure, BDD logging, highlighting and highlighting elements during execution, etc. Are built on top of this built in extensibility that we give. Actually for the newest release we are working on a really cool feature called that will allow people to reuse their webdriver tests and use them for wall testing. Meaning that in the wall test actually you don't actually like start 10 browsers etc. Actually you first run the test in the browser. We have a feature which can capture complete HTTP traffic, so we use this feature to write the traffic of the last execution and then we have a separate module of Vertrix which you can create wall tests using this already recorded and actually so cool that even the assertions and modifications of each test and test driven are recorded and allow you.

To.

So another thing that people struggle with is running their test in parallel or basically even running their test. I believe you have another project you worked on as well called mysa. Is Mysa pot of Bellatrix or what does Mysa get you?

Mysa actually is an open source project. It's a distributed test runner which we created like a year ago during the need to execute all of this test that I mentioned that we have for the framework and it was a struggle because of the lack of tooling in this section of the test automation. There were a few existing tools like for example Microsoft test agents, but they are not cross platform. So we had to think of a way how to build such a tool so that we can distribute our tests across multiple platforms and of course support our. NET core native test runners.

I thought that it is going to be cool to give it to people for free and to help us to design it in a way that it can be further improved by allowing to build different plugins for the different languages. Right now we support. Net and JavaScript Protractor and we are working right now on Java plugins. Actually I don't know another tool that does the same for a couple of languages. Usually they are only programming language specific. MYSA actually is allowing you to execute tests across multiple machines and run them in parallel and give you lots of other cool features like retrying the failing test, passing some data, smartly balancing the test across multiple machines and this helped us to optimize the maximum our continuous integration execution of our framework tests.

Very cool. So I forgot to mention, is Bellcheck open source as well?

No, Bellatrix is not open source. Actually we sell the source code if someone wants it. But my belief is that on purpose we didn't open source it because I really wanted to keep it high quality because I was part of many open source projects, many people were writing, they don't follow always the guides. Even right now in the Webdriver project there are only a few people that participate. These are the core contributors and there are so many like pull requests etc. Then don't have the time to review them. And this is why we decided that it's better to be like the tool is free to use up to some extent, but it's not open source because even if I open source it, the code on so many places is so hard that nobody always is going to contribute. This is the case with me as well. It is open source, but usually I need to write most of the plugins because people don't understand how to write a complex multi threading code.

Okay, switching gears a little bit. You mentioned you do read a lot. Is there anything you're reading right now that you think has changed the way you thought about testing even though it's not necessarily related to testing or automation right now?

Actually from one famous publisher they invited me to write the second edition of Selenium Best practicing nine patterns and right now I actually reading the first edition since I hate it order but I started to read it and at first it was to.

Not so many, not so much into the design patterns, etc. Because it started explaining selenium IDE and like basics and I was why should I read it? But then right now I read it because you know in order to write the second edition I need to know what it's about. But then I read about a few things that I didn't know that it's called that way. For example black hole design pattern for example. This is something that actually we used in Bellatrix as well. But I didn't know that this is the name, this is approach where you block third party dependencies on the website like Google Analytics, Facebook integrations and so on so that your tests are running much faster in continuous integration. As mentioned, we have similar feature in Bellatrix but I didn't know that this is the name of the pattern. And I read today about it and I was really happy about knowing this. And yeah, there are lots of insights in this book actually that I didn't know that this is the name of.

The patterns and I'm not sure if we mentioned as well, you're also a frequent blogger on Automate the Planet. Why did you start Automate the Planet? I know I started to help me learn by teaching. Just so whatever I was reading I was able to put in practice. And then if I was able to teach it, it made me know that I understood it. Is that one of the reasons why you may have started Automate the Planet?

Actually, I started like five years ago. Back then in my past company, I really wanted to do test automation 100% on. But back then we were like depending on the position, you do what your managers told you to do. Like.

You'Re not always allowed to work on what you think it's interesting, etc. So I was reading lots of books and watched a lot of courses. So I wanted to practice all of these new insights and working on this project.

And I did that in my free time. And during this period I started to play with the different new technologies that I was learning and decided that especially in the C world, many of these technologies was not so popular. So I decided that, I don't know, this is my way of contributing and teaching people, especially the net world, how to apply these new things. And with the time I started not only doing tutorials and giving examples and getting started guides, but writing my own.

The more I got into the programming design patterns, frameworks, etc. I started Paint my own implementations. For example, I have like 20 articles about applying design patterns in digital automation.

Which was like, lots of people told me that they learned a lot from this kind of series.

So Anton, I know in the beginning you mentioned soft skills and a lot of times people might find that like, what does it have to do with automation or testing or development? So if you just expand on what you meant by why you think soft skills are important.

During my career, like I talked with a lot of guys that had really amazing hard skills, really good programmers, etc. It was really hard to communicate with them. And I think that soft skills helped me a lot, not only in conferences, networking and, you know, teaching, because you cannot with people, you cannot teach them when, if you develop the greatest framework and library, etc. You need to be able to explain to the people, to your team, why this new thing is better than the previous solution, teach them, etc. And one more thing about the soft skills, maybe this is why I started this science fiction novel, etc. Is that I think it's good people to be inspired, you know, to be happy with what they do. During the work because many people are just, I don't know, usually my colleagues are really happy with what they do, but they are people that just go programming and testing only for the money and I think people should be inspired and get creative. This is part of the why I go to conferences to talk to inspire people. But I really think that blogging, writing articles, sharing opinion on Facebook about different books that you read is something that.

Contributes to the overall inspiration of people to work and help with what they do.

Okay Anton, before we go, is there one piece of actionable advice you can give someone to help improve their automation testing efforts and let us know the best way to find or contact you.

My advice is to spare some time instead of checking your Facebook etc. At work when you need some time to rest, to have a few articles about automation and read them during work with your coffee. This is my advice so that you can be always up to date with the latest trends and improve all the time. This is something much easier than reading a whole book about something and how people can reach me. They can reach me through my blog, ultimatetheplanet.com or using LinkedIn. These are the best channels.

Thank you Anton for your automation awesomeness. The links to everything of value we covered in this episode. Head ON over to testtalks.com241 and while you're there, make sure to click on our sign up for a free trial link under the Exclusive Sponsor section to learn all about our awesome sponsor Sauce Labs and their great products and services. So that's it for this episode of Test Talks. I'm Joe and my mission is to help you succeed with creating automation awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Tasks Talk Talks podcast. Head on over to triple W.TestTalks.com for full show notes, amazing blog articles and other automation awesomeness.