Automation Tips and EndTest with Liviu Lupei

Five, four, three, two, one.

Drop that.

Welcome to the Test Guild Automation Podcast. We're all getting together to learn more about software testing and automation with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Automation Podcast. Today we'll be talking with Livou Lupe, a software engineer and co founder at ntest IO, all about some unexpected ways to improve your automated test, as well as probably AI and their cloud based testing solution and tests. You don't want to miss this episode. Check it out. And if you haven't already got the perfect gift for that, hard to shop for automation engineer in your life. Make sure to head over to automationguild.com and get them a ticket to the 4th annual Automation Guild, an online conference dedicated 100% to helping you succeed with creating automation awesomeness. Our goal is you learn a tip, tool, technique or best practice that's going help accelerate you and your team's efforts in 2020. You don't want to miss this event. Register now, head on over to automationguild.com and give the gift of automation awesomeness in 2020.

The Test Guild Automation Podcast is sponsored by the fantastic folks at Sauce Labs. Their cloud based test platform helps ensure your favorite mobile apps and websites work flawlessly on every browser, operating system and device. Get a free TR visit testguild.com sourcelabs and click on the exclusive sponsor section to try it for free for 14 days. Check it out.

Hey Levou, welcome to the Guild.

Hi Joe. Glad to be here.

Awesome. We have a lot to talk about. There's a lot going on in the industry, so I thought we'd go over some high level concepts about maybe newer technologies as well as ntest. But before we get into it, could you just tell us a little bit more about yourself?

Okay, so I'm mostly a software engineer who has worked a lot with automated testing and I've been working in this industry for almost seven years.

And you know, in those seven years, probably a lot has changed. And you know, you wrote an article a few years back that's still relevant. I thought we'd dive into a little bit. And the first point you brought up was that as a test engineer, you need to embrace ever changing technology trends. And I think that's something that some testers avoid. So is there any current technologies you see out there that testers may be avoiding that they need to embrace a little bit more?

Yes. So during my experience I noticed that testers tend to not be so much into innovation as developers. So they're a bit, let's say, hesitant to use maybe new technologies. And right now, I mean, just because the technology is new, it doesn't mean that it's better than the old one. So I'm actually a big fan of supporting Selenium and Appium and technologies which.

Are well established in this domain.

You do have a product on Test IO, does it actually use Appium and Selenium under the covers?

Yes, yes. The engine for finding the elements in the web tests, it does use Selenium and the engine from the mobile test, it uses Appium because we didn't want to, let's say, reinvent the wheel. These are technologies which are good. And despite, let's say all the trends from the previous years, which are, you know, let's say bashing Selenium, we think that it's a really good technology and it's supported by a huge community.

So I think one of those trends that testers tend to avoid, maybe when they're older like me, because they've had a bad experience, is Codeless. So anytime anyone hears codeless automation, usually they just block it out. Is that a newer technology that may have approved recently that was different than it may have been maybe 10, 20 years ago?

I actually remember from my first years in the domain that the feeling was the same, that when you hear about something codeless, you would think about, let's say, these classic tools that you would just install on your Windows computer and then you would try to record something and they would use, let's say mostly image based recognition and the tests would fail if you would change your screen resolution and stuff like that. So. So yeah, they do have a bad reputation. I think it's an effort from anyone who's working on a codeless solution to deal with this reputation and try to get the users to get past it.

So this is one reason why I like talking to vendors or people that create tools. I believe that your tool does talk about codeless scripting. How do you make it so yours? How do you make it so that codeless testing is more reliable?

Well, during my experience I noticed that most of the employers would hire me just to create an automation framework. And everyone everywhere was just talking about, yeah, we want to have automated tests. We have these Selenium tests, we need to work on them. And in general it was automated testing didn't get such a high priority in the company. It was always something like maybe nice to have, but not necessarily mandatory. And a lot of the automation testers would just end up doing manual testing. So yeah, I think the codeless part helps because you don't have to reinvent the wheel and go through all the trouble that we and other vendors went through just to, to make it stable, to make the weights automatic and, you know, just get the best out of Selenium because I think it's a fantastic library and you know, with some, I wouldn't call them improvements, but with some tweaks, you can make it like even more friendlier and easier to use.

Nice. Also, I guess with Codeless there's a few ways you can approach it. Just curious to know how you approach it. Someone could say cucumber is a codeless way to do automation or using like a dsl, some sort of some simplified English like syntax that a vendor creates that just does the automation code in the background. When you say maybe like login and it automatically knows, you know, go to a site, click on these fields and enter in the data. How does your solution work?

It's not, let's say, a keyword driven solution. This was taken into consideration. But I think, and this is my personal opinion, that the problem with keywords is that sometimes it's not so intuitive what keyword you're supposed to write. For example, maybe I want, let's say I want to write some text in an input field and I don't know, should I write right text, should I write send keys or should I write input dot value equals? So it might be a bit, it might make me feel a bit, I don't know, not so sure about what to write there. So the approach here is to use the user interface. A user interface where you define each step and each step is defined by an action, and you just select an action from the drop down and then you define, let's say, the rest of the parameters. For this case, let's say that you want to write text in an input. You select from the action, you select write text and then you're asked to provide the locator for that element and also the text which will be written inside that element. This is obviously the manual approach. You can just use a Chrome extension, like many other Chrome extensions out there, that you would use to record the test.

Nice. So another trend or another thing that a lot of folks get hung up on or really have issues with, is making their tests more stable or making stability a priority. So a lot of times you'll run in a continuous integration environment and no matter how hard you try, you still have like 3 or 4% that are failing. So any Tips on how to get around or how to make our tests more stable or make them a priority that when they do fail, we do take care of them.

Yeah, this is indeed an issue. And like you said, it's. You notice it the most when you integrate your test into a CI CD system, because in that moment, every failure or error, even if it's a false one, it will block your workflow. So I think a lot of people tend to, let's say, just blame Selenium for the flakiness or for the tests not being stable. But I think.

One way to do it is to try to make each step and each connection to every element as stable as possible. For example, this means maybe sometimes writing your own XPath and let's say you have a situation where you need to click on a button and that button doesn't have an id, but it has many other attributes. Here you have two approaches. If you just get, you know, if you take the lazy path and just get the XPath from Chrome, it will give you some long XPath which obviously won't be stable. And the next time you'll add a div element in that page, that XPath will no longer work. But instead you have the alternative approach to Write your own XPath based on certain attributes. Because each element has an attribute most of the time, which makes it unique, or a number of attributes, if combined, will make that element unique. So I think just doing a little bit of research and.

Trying to find the best way to make it work is the solution.

Very cool. And your answer just reminded me of another follow up for Codeless. You mentioned sometimes you need to change the XPath or modify it. A lot of codeless solutions require you to. It's like a black box, you don't really know what's going on. So with something like say ntest, can someone modify whatever is being created automatically for you behind the scenes with your solution?

Oh, yes, absolutely. When you record the test.

Each step is, like I said, defined by an action. And if that action needs an element, it also contains a locator, a locator type, and a locator for that element. This can be any of the standard Selenium locators, as we all know, ID class, name, name, XPath, CSS selector and the steps, they can be easily edited by the user and they can change whatever they want. So it's not, let's say completely black box. What is, let's say behind the black box is the actual methods.

For identifying the elements and let's say other procedures from the engine.

So another point you bring up in your article is visuals are important. So back in the day, basically what you're doing when you're doing functional automation is you're just driving a browser like a user would by interacting with it. So clicks and things. But you really didn't do a lot of visual validation. So why do you think visuals are important nowadays?

Well, I think they're important because the though applications are more visual than ever and it's something that is really a problem that you can easily identify. It's like super easy to find and a lot of people just skip them all together. And.

Most of the, let's say the testers who are building automated tests, they tend to verify the user workflows, see if the login works, see if that works just from action to action to action, and validating some, maybe the presence of an element or the text inside the element. And what's ironic about those tests is that if they would run them on a website without any css, just a website that just looks awful, they would work and they would say that everything is okay. And nowadays you have so many tutorials online on how to do a basic screenshot comparison.

Really complex. So we have all these libraries that can help you and it's something that literally you can implement in a day and it will give you a very good return on investment.

Nice. So first tool that comes to my mind is applitools for visual validation. Does ntest work with applitools or do you have your own type of visual validation functionality within your solution?

I'm familiar with applitools and I think they're doing a really awesome job. And test does have its own actions for what we call screenshot comparison, in the sense that you can perform a screenshot comparison for the entire page or for a particular element. And obviously you can tweak the tolerance. So you set a tolerance in percentages that works for you. And if the images have different sizes, because most of the time you may be running those tests in different resolutions or on different environments, the image which is bigger is automatically resized to match the image which is smaller. So the screenshot comparison will work even in this case.

And as I mentioned, you are the co founder of ntest. But in this article you write you can't automate everything, which is kind of funny because usually back in the day a test tool vendor would say you can automate everything. Here's our solution. So what would you say you would not automate or you don't think would be a good idea to automate using any type of automation tool?

Well, it depends. Let's say there is no let's say simple answer to this question. But actually the first thing that came to mind was realistically speaking, as much as we love the shift left movement and everything, there will always be a moment when you just, you're releasing a patch. It's like release day and you're maybe release this tomorrow and you're adding a last minute patch and you need to validate something. If in that moment you don't have automated tests, it would be a bad idea to start writing them, then you can just manually validate one. So we shouldn't shame people who, you know, from time to time do manual testing because we just have to give up on the idea that everything can be automated. This is important in the context that companies don't have infinite budgets for developing projects. So if you would have infinite budget for developing a software project, yes, you would probably automate everything.

And so automate everything. Another point you bring up is cross browsers. Just do it. I know you wrote this in 2018 and now you do have a tool out there and you have customers. Do your customers tend to want to test against multiple browsers still? Is that still an issue or something you think teams should be doing?

Yeah, and for them it's actually for a lot of them it's important to know that the browsers are on realistic environments. And by realistic environments I mean that you have a Chrome browser on a Windows machine or a Mac OS machine because it can happen and it has happened a number of times that you have a browser on a Linux machine and you have some automated tests and you're not detecting any issues or bugs. And then when you pass the site to the end user, they will open it in their Chrome browser on maybe Mac OS and they'll have bugs there. So this is something that I think is really important to test on accurate environments.

And I guess your next point is remember the little things. Could you just tell us a little bit more what you mean by remember the little things when we're creating automated tests?

Well, for example, you might, let's say, you might just have, let's say the instinct to only test the user flow, see that it works and just put an assertion there just to make a pass and to know that it works. But there are, so let's say there are a lot of details that we tend to forget about. For example, the meta tags and HTML which are like super important for any Business and you need to test those. Even if they're not visible, you dig through the page source and look for them manually or automatically. Depends on how much time you have and what your approach is. For example, the cookies. As you may know, you don't hear testers too often talking about testing the cookies because they make the assumption that okay, if the site works, it means that the cookies are working, but there are a lot of other cookies which are placed and again are relevant to that business. These are, let's say, some examples.

Now that's a great, great point because a lot of times people only focus maybe on the high level things and not really taken to thought their business domain. So like you said, maybe a certain business really relies on a certain cookie. And so that would be something you want to test where it's not like a standard, just everyone should be testing cookies.

Yeah, yeah. And I think in general it's healthy for let's say a test tag to be business oriented, to know the business. Because in general the tester is, let's say, the bridge between the product owner and the developers. So we as testers have this responsibility.

Your next point is sometimes mobile comes first. I know even having a small website, when I look at Google Analytics I can see the percentage of people that use a web browser versus mobile. And mine is mainly web browser. But I would think sometimes folks just automatically assume that it's always going to be a web browser and in fact maybe you're an E commerce site and people for some reason use your mobile app most. So is that something you see people sometimes neglecting is the mobile testing aspect of their site?

Yeah, and this actually depends on the generation and including myself, I mean I have, let's say that tendency to okay, I want to buy something online or I want to check something out, I open my laptop and access that URL and do what I have to do. But it's so surprising that people who are younger than me, they just do everything on mobile. And it's just mind blowing how much mobile traffic is increasing from year to year. It's amazing. And I think for anyone who is over maybe 25 or 26 years old, it still comes as a surprise that people rather use their phones to buy something online than than a website on a desktop where I was like, absolutely.

I'll probably edit this out. But my wife is older than me, but she uses her cell phone for everything. Even like filling out government forms and it always shows up. Like you could tell it wasn't optimized for Mobile and it just drives me nuts. So I think some people don't even think about that. A lot of people are going to try to do it on mobile sometimes and you may want to optimize for that as well.

What about yourself? How do you refer to mobile?

I cannot stand doing anything on the mobile device. I have to go on a desktop for everything.

Yeah, I understand that. I understand that perfectly.

Very cool. So another thing that you wrote about was that not just testers, but everyone should test. When you do talk to customers, is this a hard sell? Do you see organizations embracing this type of philosophy?

Well, in a, let's say ideal world, you would want, let's say the product owners to test, you would want the developers to test. But in the real world, from what I've seen, this doesn't tend to happen. And sometimes not even the tester has the time and the resources to test. But I think this is something that should be taught as a responsibility in software development. I think a lot of testers out there were in the situation that they had, let's say, a new functionality to test and it was delivered by the developer and maybe the environment wasn't even working or something like that. And for, let's say people such as maybe product owners, I think they usually tend to verify the software and their input is super valuable. But most of the time their input comes very late in the development process.

How old is ntest? How long has that been around?

Oh, I actually started this entest project back in 2015. It was released at, at the end of 2016.

So having a background in automation first, before creating a tool, what did you bake into the two that you thought would help automation engineers based on your experience of what you dealt with as a hands on automation engineer?

Well, there were some things that let's say didn't seem so intuitive at the beginning, such as having to add a weight for an element each time you want to interact with it. Because we tend to think that, okay, I want to interact with that element, of course I want it to be on the page, I don't want to try to interact with it before it's there. But the main problem that I was trying to solve was that discussing with a lot of people who work in testing, you would often meet, let's say, the type of person that is a test tug is a valuable asset to their company. But they're most into manual testing and a lot of times they would just say that, okay, I want to start automating, maybe I'll start Learning next week, maybe I'll start learning next week, maybe we'll do some automated tests. And this was always pushed back due to, let's say, higher priorities or simply the workflows. And I just wanted to offer a tool that would allow a tester to create automated tests without having to code because let's say this was mostly the main reason that stopped them from doing this.

I'm a big believer in there's not one tool that's right for everyone. That's why I like having a lot of different vendors in this space, because they all have unique differentiators that make them maybe a prime tool for one team that may not be the right fit for another. So I guess my question is what kind of team do you think would be a good fit for your solution?

Well, we're mostly focusing on, let's say.

SMBs, so businesses that are not quite enterprises because let's say we don't actually have a sales process to sell to enterprise customers. And because the price is, let's say, something which isn't expensive, it's targeted at SMBs. So companies who don't have budgets of tens and thousands of dollars to spend on automated testing.

No, I love that because like you said, every company is different and budgets are different. So a lot of hang ups sometimes with smaller teams is they may not have the knowledge to build a framework yet because they don't have a big budget. They tend to write it and they tend to write a really bad framework. So this sounds like it might be a good option for people to check out if they're in that type of situation. For sure. Is there anything different about your solution than say some of the other tools out there that maybe make sure is a little bit unique or different?

I have a good opinion about every tool and library out there, but obviously just like when you want to buy a car, maybe one type of car model is the right one for you and the other model is right for someone else. We tend to focus a lot on simplicity, on the fact that our idea is that you sign up and in the first minute you should be able to create an automated test that you shouldn't get the feeling that, oh, what is this huge application with so many options that I don't know even where to start. We've all been in that situation where you start using a solution and you're just overwhelmed from the first minute. We try to focus on simplicity.

But not trimming on features. So we have a lot of functionalities and features, a lot of things can be done, but it's focused on simplicity.

So you did talk about shift left and a big move is trying to get software testing introduced early and early in the development lifecycle. And when developers release code, it gets tested as soon as possible. How well does NTest integrate with, say, someone's CI CD systems or workflows?

So we have this NTest API which you can use to execute a test and you can also use it to fetch the results and to do, let's say, other actions in the platform. So mostly you would use this API to integrate with your CI CD system, such as Jenkins. We also offer webhook notifications, so you can just add a URL there, an endpoint, and we'll send the results automatically to that endpoint. And for, let's say for certain integrations, we built our own, let's say custom integration, such as Slack. We have a Slack app which you can just install and you will get the results in your Slack channel without having to use the API.

Okay, Li Vu, before we go, is there one piece of actual advice you can give someone to help them with their automation testing efforts? And what's the best way to find, contact you or learn more about end test?

I think the best advice I could give to someone to, let's say, improve their automated testing and overall their testing is to focus on the product, focus on the business, know the product and know what to expect and to put themselves in the shoes of the end user. Because that's where you tend to, let's say, find the things that are not so logical or the things that might annoy you as an end user. Because most of the time, let's say, when you're a tester, you start working with, you know, the alpha version of an app and then the beta version. And if it's badly designed from the start, you just get used to it and then.

You can no longer see that it's not intuitive or it's a really bad experience and for people to contact me. I'm not on Twitter or anything like that, but I am on LinkedIn, so I think that's the best place to contact me.

Thank you Livu for your automation awesomeness. The link to everything of value we covered in this episode. Head on over to testguild.com.

Alike automation284 and while you're there, make sure to click on the Try it for free Today link under the exclusive Sponsor section to learn all about Sauce Labs awesome products and services. And as I mentioned, you don't want to miss Automation Guild. Head on over to automationguild.com and accelerate your automation efforts in the new year. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe and my mission is to help in helping you succeed in creating full stack end to end automation awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Automation Podcast. Head on over to testguild.com for full show notes, amazing blog articles, and online testing conferences. Make sure to subscribe to join the Guild and continue your testing journey.

This has been a Joe Calantonio production.