When to choose a low-code automation tool with Diana Oks

Welcome to the Test Guild Automation Podcast where we all get together to learn more about automation and software testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Automation Podcast. Today we'll be talking with Diana Ochs all about when to choose low code automation tools as well as other automation topics. And we may even dive a little bit into security testing. We'll see. But Diana has over six years of experience as an automation engineer. She currently works as automation engineer at Vulcan Cyber, which is a startup based in Israel. She has a really cool background. She has a double degree I believe. One is in engineering and one is in physics. We'll dive into that as well, hopefully see if I got that right. But really excited to have her on the show. She joined us this year at Automation Guild and really was an awesome speaker. Really dropped a lot of knowledge, so I wanted to share it with you as well. You don't want to miss this episode. Check it out.

The Test Guild Automation Podcast is sponsored by the fantastic folks at Source Labs. Their cloud based test platform helps ensure your favorite mobile apps and websites work flawlessly on every browser, operating system and device. Get a free trial. Visit testguild.com sourcelabs and click on the exclusive sponsor section to try it for free for 14 days. Check it out.

Hey Diana, welcome to the Guild.

Hi Joe, great to be here. Thank you for having me.

Awesome, awesome. Great to have you. Like I said, you were a speaker Automation Guild. So I'm really excited to have you join us on the podcast. Get asked about this all the time is about boot camps. Do they really help? How do I choose one? What made you even think of like, I have a background in physics, material, engineering. I heard this thing about like, how did you get into automation or software development? And then how did you know? Like, here's a good boot camp that's actually going to give me the skills I need to to find a job. If I invest in this as well.

While looking for a job, kind of apply to all graduate people. Do you want to try this out? It was not an innovative idea, but it was like kind of. There were only two places that did it and they kind of applied. So I took both of them and tried and tested and one did not accept me, but the other actually took me. So this is how it started, but I didn't know if it was a good thing. But it kind of was a leap of faith because they really promised a job in the end. Unlike we'll try to help you, it will be more, we will find you a job. Of course you have to sign an agreement to work with them and for maybe a lower wage than it's used in the industry, but still they find you a job and you sign to work two years through them. So by the end of these two years. Sorry, two years. And by the end of the two years you're kind of free to choose whether to stay with this company or maybe sign another company. So you end up with the knowledge, with experience and you're a kind of free agent to go.

Nice. So the current company you're in now also, I believe you were a first time hire in testing or qa, is that correct?

It's the first time I'm using testing in a company. I wasn't aware actually of testing before I came there. I did more really testing, but my own framework, usually it was created inside the company. The company was responsible for that. Some testing depend on the company actually, but were different kind of testing. For example, while I work in Philips, which was medical company, not Philips, the Lightning, they had like a lot of teams doing manual testing and at some point they decided to outsource that and doing automation testing because it really was too much for a test. Some tests were only tested once in two years, it's just a week and understand the amount of tests. So there it was, more desktop application. So the whole infrastructure was built inside the company and it should work with their tools and not outside tools. So think of everything you know about web testing or mobile testing. So it's really not applying here. You need to kind of invent your own infrastructure and things that will work. On the other hand, when you work with company that are web based or stuff like that, so you take all you know, because essentially automation or testing for that matter, it's not either testing mobile or API. You test and test is a methodology. You can apply the same rules, the same knowledge, the same base rules on anything you test. So. So it's more of how do you take your knowledge and transform it to other places?

Absolutely. So I guess a lot of times when you join a new company, they already have a framework in place or you know, they have. These are the tools we use. What was it like at Vulcan when you joined? Did they already have something in place or were charged with finding a solution to help them with their unique testing needs?

Okay, so Vulcan was then kind of, we are friends and we have an idea. So we take few people we know and we create this real platform that does what we envisioned but at some point when your product became a real thing, you try to sell it. At some point, you know, every startup has this, we have this idea, we think it looks real and now we kind of want to start to sell it to customers. And at some point you start to be really, really responsible. You can't say it might work, it has to work. You have clients and now you're in charge with and you have this responsibility to make your product work. So about two years after the startup was founded, they kind of came to this conclusion that they don't have any qa. The QA was done by the product. They really don't do good job. They kind of sweep through the app, everybody's in charge on their area, so they kind of check this work works and it wasn't working. So before even considering me, they took Testim, I think for a tryout to see maybe they can make this work. So when I came they told me, look, we tried this but we don't have anyone who's in charge in that. So at this point you can either choose to use them or create your own framework. We will go either way. So I was actually interested. I called them, we scheduled a demo because apparently there is a real company. I didn't know that.

So I found a contact person on my WhatsApp company and I asked them, hi, I know you work there, can you tell me about it? Can you walk me through and say I can even do more? I can schedule the demo, so I'll tell you what we do and you can ask all questions. And we scheduled a demo and he answered all my questions. I do have to say that I really have bad experience using Selenium. Not because it's a bad tool, because a lot of time when developers develop things, we're not always able to test them. So you kind of start to look for all these hacks and works around and the Internet is full. How do I test this? Unique drop down with unique and it start to pile up into a lot of solutions to really cool features that the users see but are really hard to test. So I had this all small questions, how do you handle this and how do you handle that? I assume that it will have at least partial solution to all sorts of issues and if they can provide some answers to that, then I think I can use this. At that point I wasn't sure if we're going to do only UI or API. If I'm doing both, then testing wouldn't be a good answer. But if I'm Doing only ui, maybe testing is a good answer. So it was a matter of decision, how do we proceed? And since the company didn't have any coverage whatsoever and the few tests that was weren't good enough, it was a matter of, okay, so if we go with this UI and it will take me about two or three months to build a good framework. Because a good framework has to work not only with the testing itself, but also with what we have already in the company. So for example, we're using GitLab, so it has to be part of this git, or should I use Python or should I use. So it's all sorts of decision you need to make. And also if you use open sources like Selenium for example, then so it's not really free because your time is not free, you need to maintain it and you need ownership for this project and you need to make it really to fit your product. So it's not really free in that matter. But it takes a lot of other time, the workaround and you didn't even write a single test. So this was all part of the solution, the solutions that testing provided actually nice.

Now I know a lot of people, they're either hardcore open source or, you know, they go crazy. But I think you brought up some good points of some reasons why maybe you want to look into other options other than open source. I believe even Testim behind the scenes, I think might use Selenium as well behind the scenes, but it has extra functionality built in to handle all these scenarios like you mentioned that you don't have to create from scratch. So I guess one thing that might be concerning, as you said, you weren't sure if you're going to do API testing, what happens if you invested in a vendor based tool like Testim and then maybe a year down the road they're like, oh, by the way, we also need to do API testing. Does that make it more difficult now that you're tied into a vendor rather than an open source solution? Or is that something that even matters?

So we're here down the road about a year and actually it comes to, I think we have a good coverage, but now we have areas we're not handling and we can't handle them through the ui. So we do need to make those API tests and we really cannot use Testimon at this point. So we need to create our own framework. So does not eliminate all the work Testim has done because it really in place, we have about 200 tests that really test the UI and it probably will grow even more because UI is changing and we can do even more elaborated tests, but we cannot ignore the API. So we will have two automation projects. And I think everybody kind of think of you need to have one solution to a lot of things, but you may have more than one solution and this is okay. Think of it like the developers have unit test and I have my UI test, for example, and this is not the same. So what? So it shouldn't be always like I want something that to fit at all. Because you might not get this in tools. You need to understand that some tools will answer, will check boxes on one thing and other tools will check boxes on another thing and it's okay. I don't think it's supposed to scare us in this point. We should be able to use more than one tool or create more than one framework to match all sorts of stuff.

Nice. So do you have an approach now then when your team's running the test, how do they know when to use API testing or what tool to use for API testing and when to use testim? Do you have like a method that they know? Okay, this is a UI test, so I know I need to use testim. Well, there's an API test now I need to go to this open other solution to do API testing.

So actually I guess it depends on company policy maybe or maybe the company agenda. But for me, if I can do it on the UI and it will cover me more bases, then I'll probably do the UI test because I can do it easily using testim. However, if I see this test is prone to fail or maybe is not really accurate on the UI side, for example, if I need to check data, for example, I don't really need to open my browser and to see the whole data thing, it won't bring me added value. There is no added value in opening the browser and checking the data, unless I want to know it's in place or how it looks or maybe if it corresponds with something. But if I only want to see the data, like kind of JSON files, then API would do this quickly and easily and I don't have to work this all around. So it all depends on your definition of done in that matter. And how would you define your success testing the data or testing the data in some place or some location? Or does it, you know, it's in the boundaries, is it in the dropdown? So it all depends really on the matter of your definition.

Cool. So another thing with a vendor based tool in this example happens to be testim, but it could be any vendor based tool. I think testim though is one of those solutions that kind of markets themselves as a low code tool. And a lot of times automation engineers, not only are they turned off by vendor based tools, but when they hear low code that turned off even more. So can you just tell me a little bit more like what is a low code solution or automation tool and what benefits does it give you and what maybe are things that are myths that people believe about low code that really aren't true?

Absolutely. So low code does not necessarily mean no code at all because I can assure you have a lot of snippet of code inside. Because they really cannot check all the boxes Selenium has and they can't really cover all the validation that my app has or your app has. But it does provide you a really simple solution to create UI tests through clicking and recording and then just rerunning the records. And the added value is the screenshot. So every step has a screenshot. You can add a description. What it does, you can really add or define or redefine your selectors or css. So basically you have control on every step. Just your step in that essence are recorded instead of you writing them down. So if you are scared it's not personalized enough, then you can personalize it enough, you can determine what it does, you can determine the validation you have. You can add it to suites, you can add it to groups. So it depends really what you need to do. And you don't need to be scared of duplication because they have for example this cool feature of shared group which I use a lot because I have same step, for example about 10 or 15 tests which checks the same box because at the end it comes to this flow that check in a box. So I just use the same step and I don't need to do it multiple times. For example, or do you have a login for your application, don't need to do it hundred times, you just click record it once and reuse it in all your tests. So you don't have to be scared of that because on one hand it's easy to handle and on the other it's really easy to maintain. So both and it's kind of really understandable by people that are not really writing your text. So if it's simple enough, anybody can understand and it's kind of self explanatory.

That is also a big concern people have is we'll have a lot of duplication. So how does it know there's already a login. If you have like a big team and someone's like, oh, I'm going to do this script and they start recording and they do a login. But there already has a login exist, does it tell you, hey, by the way, this already exists? Or how does that work?

So they have this feature just clear. We're talking about testing, because this is what I use. But there are really other tools, for example, so I'm totally for that. I just speak from my own experience. So testing, for example, has. Let's talk about login, because I think it's kind of the most common step. So they kind of have this base URL which they open the browser and navigates to this basic URL. And then for example, our application has your landing page, which is the login page. And so it has those five steps. Click the username, setting your digits, for example, or your user, then click your password, set your password and click login. So kind of five steps and validate, for example, you were logged in, because if you weren't logged in, so it will fail. And then you save this as your shared group and then you can call it in every step you create, just so it will show you that you have X tested using this step, for example. But just you have to remember, if you change the user, then it will change your user, for example, all across this step. Don't change shared groups without understanding the repercussions.

Absolutely. So how do you create a test set? Is it always recording or is it if already. If you already had like the framework's been around for a year now, so it must have a bunch of functionality. Does UI have like a dropdown where it shows all the existing functions or methods you can use already or how does it work after you've already been.

Yeah, so two options. If you have like similar, similar, you can clone your test and add some. Just the few steps that are really different. Or if you need to create your own step, then you create new step and then you can add. They have like box like this that has a drop down and you have two options or three now they kind of change a little bit ui, but you either record your next step, you have testing methods like validate, or add something that was really incorporated in the application. Or you can go to your own saved shared steps and use them. So you have a dropdown with all your shared steps and you can choose them. So you can basically build your own test using your own shared steps.

Cool. So I also know a lot of People when they're using an API for automation like Selenium or whatever open source tool, a lot of times they struggle with things that are time consuming, like their weight approach. Like they don't have the right weight mechanisms in place to wait for something to be ready to be interacted with because it's not on the page and fully loaded. And so they end up having these weird issues with flakiness because they don't have the right weights in place, the right mechanism in place to handle that. Do you have to worry about that with a vendor based tool? Does the vendor based tool take care of waiting for you automatically without you having to think about, okay, now I need to add this wait or this wait for type of assertion.

They have two mechanism, I think and we can talk about them both. So the one is the classic wait and they have really inserted like wait for a certain element to be visible or for example wait for not to be, wait for a text, wait for a download and stuff like that which is incorporated in their application. On the other hand, they have this mechanism that tells you how long it will take to fail for a step so they can retry during this whole time. You said like for 10 seconds. It will try for 10 seconds and then will fail. Or you can say wait until this element appears and then try the next step. So both will give you the same. But I think we're talking about right method to work, then wait until there is an element and test. This element would be more correct flow rather than more means to achieve your success for that matter, rather than options. But both will work for that matter.

So how do you run your test with the vendor based tool? Does it integrate with cicd? Do you have that in place at your company? What's your workflow for your automation?

I actually really enjoy work with the ui. I know most people don't always like that, but it has its own benefits because UI means like everybody can open this URL and see it for themselves. Which is not bad because as I said, if you're a small team and you're the only one that does, then I'm the only one that knows what these tests are doing. And on the other hand I need to share it with my developers so they need to know what those tests are doing in case they fail. So it's kind of easy because here I send you my URL for the fail test and you can see the failure or where it failed or have them the screenshots. So it's kind of easy to understand and on the other hand, you have this, your CI CD in terms of we have integration doing GitLab. I think it's not really popular tools but, but we use it. They have the CLI options that you can actually call your projects and to run. And you have this suite name or the plan names. So essentially it kind of work both way. Either using the ui, I have scheduled suites and schedule time. So I actually have this from my end. And on the other hand, when my developer is doing deployment, they have tests for pre deployment. So we call this pre deployment suite, for example. But we can use it only from the cli. I think it's kind of was their purchase. But I really like the ui, I think.

Cool. Can we talk a little bit about what you're testing? Because I'm looking@VulcanIO now and it looks like it does some sort of security automation. So it's like you're automating something that's being automated. How does that work? What's that like?

So a little bit about kind of the cybersecurity, which is a fancy name for a lot of things. But essentially every company, every company that really has to, or at least big companies, has to have scanners. They have scanners in their companies, usually large company, they have scanners like tenable equalis, like Rapid7, that scan their whole assets. For that matter, every computer they have, every server they have, whether it's cloud based or a real server, it's all scanned through their network. And the scanners let them know if they have vulnerabilities and what assets are really vulnerable at that point. So what we do is we do not scan those systems. We only take the information from their scanners. And for that matter, we have part one team that does all the integration. And we have integration about 30 connectors. Some of them are scanners, others just brings you the data, other brings and fixes the data. So it really depends on the type. And then we have our system that does this whole prioritizing and gives credit for vulnerabilities. By our own algorithms. We can prioritize what is important to be fixed first or is not important to be fixed or can be postponed. So the most important thing to companies that has vulnerabilities is to know what can be fixed or what can be postponed or how it will affect. Because think about it, you have a large company and you need to do a reboot to some software or some browser or some, I don't know, operation system and it will affect your team and some data might be Lost you would consider am I willing to take this risk or am I not willing to take this risk? So sometimes it's kind of easy as like change your password, but other times it's kind of a little bit more complicated than that. So companies need to understand what is important. And using algorithms, they can actually insert their own favorite algorithms, for example by tagging or by prioritizing by weight, asset types, all sorts of things. And this will give you this list of what is in risk in critical high medium, for example. And the third part is after all this data is in the system, you can choose to take your action. And by taking action means you can either choose manually how to take an action or you can do like kind of a playbook we have in the system which you define a rule and it will automatically will run. And once and something is defined, for example a new vulnerability on your asset was defined, then it will take action. One of the actions you defined and action in that essence might be open a ticket if you have Jira or ServiceNow or send an email or whether you want to deliver a fix. So usually fixes are a little bit more complicated because it's different.

Some different company, but it's usually different people in different companies are responsible for tracking or tracing. So the whole cybersecurity, all the security companies is a little bit more complicated than that. So it's not always the same people that are responsible for scans and fixing. So it's kind of this platform allows them to interact to understand what is important and what can be fixed and kind of using the same system to understand where or how your system will vulnerable. And the second thing they talk about is that we can deploy fix, so we can deploy ansible scripts or chef scripts or SCCM scripts. So this is kind of the take action part in that manner.

So how do you know what to test or automate in that flow? Sounds very complicated because it sounds like you need to have something in a certain state. You need to have a bunch of data like if on this vulnerability and then this vulnerability can be fixed or this vulnerability can be delayed. How are you automating or testing? How long do your tests take to run? Because it sounds like a long flow that you need to actually automate.

It's really, it depends. It's really kind of as I said, we can divide into three part how do we bring the data, which is important to test, to bring all your data. Then once the data is in the system, you need to make sure it's really handled correctly for Example, it has fixes. How do you match fixes to vulnerabilities? How do you think about it? You have for example a Mac computer and then suddenly you have a Windows fix. It's kind of awkward and you want to avoid it. So we have a really big research chip, not big, but it's kind of. It has this task of bringing the right solutions, the right fixes, the right remedies for the vulnerabilities. And they actually built, if you look in our website, they have this remedy cloud which you can look for vulnerabilities by only clicking the CVEs of the vulnerabilities, which is a major project and it's free, so pretty much happy to use it if you want to. And it's really hard work and the research team really does a lot of work in that essence to make sure you bring the right remedies, right fixes to the vulnerabilities. So it's a lot of on our behalf and also the algorithms that really merged all the solutions.

So Diana, I've worked for a lot of enterprise companies, a lot of big, big, huge companies and I've always had better luck with vendor based solutions where I know a lot of times you have these in house like one or two people type companies and they have no problem using a open source tool like Selenium. So I'm just curious to know, to get your opinion on like when you're choosing a tool, does it matter if you're a big corporate enterprise company as opposed to like a 12 person type operation?

Absolute.

Really a different way of work and thinking. Big enterprises usually have bigger teams or many teams that each is tasked with different, for example chunk of information or testing on. Sometimes a lot of teams do not even test the same things in the application or in charge of different things. And then you have the whole framework team that really builds tools, for example, and I've had it. So. So it's usually not the same people, not the same group or not the same even areas. But if you are a one person or team of two, for example in small companies, then you are the answer. You the person that builds this whole framework, you design it, you use it, you make it work for you and you're in charge of it. So for example, you need to interpret your test for your developers, you need to work with them or along with them to plan the test to see what features they do. So you're really hands on all this process while it's being planned and executed. So once the feature is out or even before it's out you already know what to test and how to test and you can have the say or the feedback. Sometimes I really have to stop deploying because I think we have a critical bug or a showstopper for that matter. So you really have a stainless steel and it's really big in responsibility because essentially you are the one person and sometimes they can talk to you in the evening. Hey, we have this problem, can you please help us or rerun the test or check what is matter and it happens. So I think this is the biggest difference because you have less people to be in charge with all the testing in the company.

Okay Diana, before we go, is there one piece of actual advice you can give to someone to help them with their low code automation testing efforts and what's the best way to find or contact you?

Please reach out in LinkedIn. I think I answer everybody that ask question. Personally I used you to contact other people so I vote for the community. But also I cannot tell you to use distro or other but I think if you have the opportunity to use AI tools or tools that incorporate AI and makes our life easier totally, you should consider using them. I'm all for making our life easier and I think the main goal is not to create more code, but to make tests more reliable and more approachable. And so we should totally use tools that will help us doing that. So never. At least don't reject using tools unless you really have a good and valid reason for that.

Thanks again for your automation awesomeness. The links to everything of value we covered in this episode Head on over to testguild.com a.

And while you're there, make sure to click on the Try for Free Today link under the Exclusive Sponsor section to learn all about Sauce Lab's awesome products and services. And if the show has helped you in any way, why not rate it and review it in itunes? Reviews really help in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack Automation awesomeness. As always, test everything and keep keep the good Cheers.

Thanks for listening to the Test Guild Automation Podcast. Head on over to testguild.com for full show notes, amazing blog articles and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.