Building a Real-World Automation Robot Framework with Dave Martin

Welcome to the Test Guild Automation Podcast where we all get together to learn more about automation and software testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Automation Podcast. Today we'll be talking with Dave Martin, all about building a real world automation robot framework. Really excited about this episode. Dave Martin's an old colleague of mine. He does a lot of cool things around automation. So I'm really excited to hear what he's done with this. David's been in the QA profession for over 23 years, and from the very early days in QA, he's heavily leaned on test automation. Through his career, he has built test automation frameworks from the ground up using a variety of tools. Most recently, he built an automation framework using robot framework that regularly runs about 50,000 tests and now has about a dozen contributors. And that's what we'll be talking all about today. You don't want to miss this episode. Check it out.

The Test Guild Automation Podcast is sponsored by the fantastic folks at Source Labs. Their cloud based test platform helps ensure your favorite mobile apps and websites work flawlessly on every browser, operating system and device. Get a free trial. Visit testguild.com sourcelabs and click on the exclusive sponsor section to try it for free for 14 days. Check it out.

Hey, Dave. Welcome to the Guild.

Hey, thanks, Joe. Good to be here.

Awesome to have you on the show. Before we get into it, is there anything I missed in your bio that you want the Guild to know more about?

No, I think that was great. I mean, one other interesting tidbit about me is that I actually spent a couple years working in the Virgin Islands, which was quite the experience doing QA in paradise. So that was awesome.

What happened to that gig? That sounds like an awesome job.

Oh, what happened to it? Oh, Hurricane Irma happened to it. Oh, yeah. Quite the experience there. That kind of put a nail in that coffin.

So you've worked at a lot of different places. I think you also worked at a church of Latter Day Saints and I think to me that'd be like working at the Vatican.

So a lot of cool, right? Yeah, that was a good job. It's the church that I'm also a member of and did a fantastic project for them that was just so much fun. So, yeah, good stuff there.

Very cool. All right, Dave, so what industry are you in now?

I work for a security company. Online security network security, that kind of stuff.

Very cool. So I've been hearing a lot about robot framework. I actually Spoke to Pekka Clock who is the creator of Robot Framework. But I haven't had anyone on the show that actually has used Robot framework day to day. So I'm really excited to dive into it. Before we get into that though, Dave, at a high level, what is the robot framework? For folks that don't know, Robot framework.

Is a open source automation framework, primarily keyword driven. The thing I love about it is it's the Selenium library is just one of many libraries that come out of the box with Robot framework that are just built in. I can do web testing using the Selenium library. I can do operating system level operations, I can do terminal based operations all within the same automation framework using the same keyword driven syntax. So I think that's. And it's been around for I think since 2007. So it's pretty mature. It's got a great community. But in it just what it is.

Nice. So Dave, when we used to work together we did a lot of things with HP which is now Micro Focus's products. But the thing with that it was a paid vendor. So is there a reason why you chose Robot Framework because not only does it handle web, but it also handles other type of automation?

That was one of the benefits of the Micro Focus tool set or the HP tools at Mercury was that it had a lot of different plug ins, it could handle different technology types. I guess that is part of the reason why I leaned on Robot is because of its ability to handle different technology types within the same framework and it's free and open source and I'd been doing a lot with Selenium leading up to that and I just heard enough about Robot that I wanted to really jump in. So my current employer, I had the opportunity to build something and I thought why not? Why not grab Robot framework and give it a try. Never even touched it before that, you know. And within two weeks, maybe three, I had probably a thousand tests running. So I built something in two or three weeks that took me manual effort, took me about three weeks to run manually and I automated it in two or three weeks. So ROI was instant.

So yes. How are you able to scale up that quickly? I guess it uses Selenium under the COVID So it adds a keyword approach that makes it easier. Like what made it so easy to scale up like that for you?

Yeah, yeah, no, I think just the keyword driven nature of it. The I used, I mean, I guess page object pattern is probably not the right word because it's not really object oriented. In its approach, but I organize my keywords in such a way that they were easy to find to reuse. So I built kind of a granular set of keywords, then kind of a mid tier set of keywords that brings multiple together. And then I just kind of chunked it up like that. And then the reusability was pretty awesome and the maintenance was almost nothing. I mean, I had to tweak a few locators here and there just because they might not have been robust enough, but because of the way I was already organized in my page and things like that, it just, it made the maintenance super easy.

So once again, Dave, I'm not sure, maybe some people, I guess keyboard frame type of approaches are kind of older, but people that are new to automation may not be familiar with it. So you mentioned keyword approach a few times. What does that mean in the context of robot framework?

You know, I guess it's just there's no real coding involved. If you're using the available libraries that robot framework comes with, those have keywords and the keywords just execute internal type of syntax that is unique to robot. It's very easy to pick up, very easy to read, very easy to write. And so, you know, by its nature these built in libraries are have a bunch of public keywords that can be leveraged. And it's as simple as saying I want to use this library and this keyword and then whatever arguments need to be passed into it. There's no coding. Like if I want to build my own libraries, that's the only coding that's involved. Otherwise this keyword driven syntax is I want to run, I call the keyword by name and then I just pass it whatever data I guess that it needs that keyword requires. Maybe for example, there's a keyword called click and so I call click and I pass it the object I want to click the locator that I want to click on. Simple as that. You know, input. I want to input text into an input object. I call the input keyword, pass it the locator, pass it the text that I want to, I want to input. So I don't know if I'm describing the keyword driven syntax properly, but there is no code that I am writing to do automation essentially.

Gotcha. And I assume the scripts look a lot more readable than if it was straight up.

Absolutely. Very, very readable. Anybody can look at one of these tests and understand what it's doing. And again, that's also part of the design as well, right? If you build a keyword that is doing multiple things, you want to name it such that it's very clear to somebody calling that and reading that what it's doing.

Now, I know Selenium, they tried to make the API as readable as possible, but this sounds, it takes it to even a more readable approach where it's less like looking like code and more like English like syntax than you're creating your scripts in without you having to roll your own type of functionality. It sounds like.

Absolutely. Yep, exactly. Now, if you're looking at the code in the libraries that you know, that's actual Python or Java code, depending on which implementation of Robot you're using. I use the Python implementation. And so those libraries are written in Python and you can go and look at that layer and write your own libraries, which is you're going to be writing Python code at that point. And we have written a few libraries to help us out. I guess that's another thing I love about this is it's got something for everybody. Your hardcore automators, these coders, they can be building libraries that do very, very custom operations, while your, you know, maybe your lower level automators or even your manual testers can easily be trained to write, testing, to use those keywords that are being exposed by those libraries in their testing. So it brings automation to the masses is what I kind of like to think of it as.

Nice. So it sounds like you have a very successful framework and you were able to get 50,000 tests and have dozens of contributors. So how is it to get contributors on board? Because a lot of times when you start getting, adding them to a framework, they stop mucking things up. So is there a standard you use or does the framework make it so that it's just easier to contribute without causing a lot of overhead or bugs? Framework bugs?

Yeah. No, there are certainly a couple of parts of the framework that are pretty sensitive. You don't want to be messing with it. But for the most, the vast majority of this I don't require, you know, reviews on check in or anything like that. They just, I trust the people to, I want to enable them. And honestly, we've never really had a major issue where somebody's checked in something that broke a bunch of other people. Right? They most people are working in their areas of domain and there's not a lot of crossover where I'm going to break my coworker if I make a change. There is the possibility for sure. We encourage local execution, testing out your change, making sure it works before you push it up to the code repo. But it's, you know, I have a small coding etiquette document that I've created, but it's very open. Like there is another framework in play at my current company that's very closed and locked down, very difficult to add anything. I wanted to create something very different from that. And I think just the, just the nature of the usability of this thing, if somebody does make a mistake, it's very easy to fix because of the maintenance, the way we've organized the keywords and whatnot.

Yeah. I don't know if there were any gutches with Robot framework because you've dealt with a lot of different tools. Was there anything like robot framework? You're like, oh, that's kind of unique to robot framework. I wish I knew that ahead of time or.

All right, well, the biggest gotcha with Robot is everything is delimited by a double space. And more often than not, when somebody comes to me and says, hey, this isn't working, I look and I'm like, you're missing. You've only got one space between these two arguments. You need two. Or, you know, so it's thinking that you're passing in one too few arguments into your keyword or something like that. So everything is double spaced, limited. And that does tend to be a bit of a gotcha. But it's the first thing I look at now, right when somebody comes to me and says, I'm having trouble here, what's wrong? And it's inevitably, probably 50% of the time somebody's left out a double space. So that's very unique to robot.

Nice. I also think they have a lot of libraries that you can pull in. So any chance of libraries conflicting with one another? Is that something people need to be aware of?

So to date, I don't think I've had any conflicting libraries. I do keep a requirements. Txt file that I'm actually regularly updating. I'm not afraid to update to a newer version of a certain library. It's something that I'll. I'll give it a go and try it out myself first. But as far as libraries conflicting, I haven't seen that be a problem.

How about execution speed? Does that add any overhead you would get if you didn't? If you just use straight up Selenium?

I don't believe so, Joe. I haven't done that. I haven't done any speed comparisons between straight Selenium and a robot, but I've used Selenium enough that I haven't noticed anything. I mean there are areas where I have some implicit weights and so there's things that I'm waiting for that. But as far as just the order of execution it seems to go, you know, once those implicit weights are met, it seems to just move right on.

Cool. So are there any must use libraries in robot framework? I know they have like built in ones. They have ones for strings and date time. They have external ones for Android libraries. I don't know any ones that you use that seem kind of like out of the blue that people want to know about. Like, hey, wow, there's a library I can consume right out of the box and not have to worry about developing it myself.

Right. Gosh, the ones that you mentioned, there's a dialogs library I actually use. There's the strings library. I use quite a bit. Date time. I do use obviously the built in which is kind of the core Selenium library, obviously. Operating system library. There's an SSH library I think, if I'm remembering right, I think there might even be an SCP library or something. But I use quite a bit. Let me pull that up and I'll follow up on that here.

Yeah, that's something for everything. RPA framework, they have some for SOAP library ssh. Like you said, it's crazy.

Yes, yeah, yeah. And there's also a library that we use to support that supports Auto It. Oh and we had to use AutoIt for a few things and so we have a library that we use that helps us with to integrate with autoit.

Yeah, I'm learning on air does turn into this up for Kafka also. This is pretty cool.

Yeah, absolutely. No, the community is very active and I've even reached out to Pekka Clark myself and asked him a few questions and he's answered them like through a Slack channel. It's pretty crazy. Community is very active and you throw a question out, you're going to get an answer.

Nice. So Dave, how's software development work where you're working right now? Do you do all the automation? Is there an automation group? Are you embedded in Sprint teams? How does that work?

Our QA is our QA team. There's no separate automation QA versus regular qa. We're all qa. We're all expected to do some automation. We are embedded into or we try to follow an agile Sprint type, you know, release cycle, I guess. And just like everybody else it's like agile scrum fall. Right. But we do our best. But yeah, we're embedded in the product teams that follow an agile type release cycle. And I report into a QA director, but overall we report into our, into our product team managers.

Cool. So I think one thing people might be getting confused by is this. You're using the robot framework. You created a framework for a framework. So, so what do we mean by that?

Right. Robot framework is the name of the framework that I am using to build my internal framework. And I guess what I mean by that is my implementation of robot framework is my implementation of robot framework. So I call that my framework. But we actually have a name for it. But maybe framework is an overloaded term, I don't know. But I have my implementation of robot framework. The way that I use it I refer to as my internal framework. Right. The framework that I built. And it was one of the things that let me back up just a little bit on how this actually got started. Joe. I got hired on knowing that I had years and years of automation experience. And one of the first tasks my boss gave me was testing this very, very, very manual process that was incredibly painful. It took, like I said, like two or three weeks to execute this particular test. Had to verify tons of different data points. I why am I getting asked to do this? Right. I'm an automation guy, you know. And then it dawned on me, like, he doesn't want me doing this manually. He wants me to build something for this. He wouldn't have asked me to do this. Right. And so I thought, you know, I'm going to do it. After going through this process once, I committed to myself, never again. And so the very next time I picked up robot framework, I didn't even know it. And I found a course online, a Brian Lamb course on Udemy. I watched the first part of that course and then was off to the races. And so any managers out there, I'm telling you, that was very effective. You know, give your automator the least desirable task to run, the least desirable manual task to run, and I guarantee they're going to find a way to automate that thing.

You know, I would hope so. I guess I'm a pessimist. I've seen people that will just do that manually to death forever. So it's great how you took that initiative.

Sure.

So, yeah, I think it's a great technique. Maybe hire an automation engineer, give them a really manually intensive task, and eventually I guess a solution will come out that will help make that more streamlined, right?

Yeah, I hope so. Right. I mean, it worked for me. Maybe it might not work for everyone, but I thought that was pretty good. Pretty good management there.

Cool. So your tests seem to run regularly. So how does that work? Do you have it running in a CICD system? Is it easy to run in a. Like in Jenkins? How you run every. Do you run every night? Every week? How's that work?

Right. I run on every build. I also deploy the builds to our test environment and I've actually automated that with Robot as well. And that's entirely executing a series of terminal based commands on a Linux system. And I automated that with Robot aside from Selenium, like it's. I even automated our API testing with Robot to get back to your question on the cicd. Yes, Jenkins has been awesome. There are plugins that allow you to interact directly or publish results that are robot results. The build script is really easy to build that robot command to execute your tests. And then we actually even have a post build process in Jenkins that uploads the results to our test management system.

That was. I was really worried about how we're going to integrate with test management and all that. I mean it took like an hour to figure it out. It was pretty, pretty wild.

Nice. I know a lot of drawbacks to a lot of open source solutions is the reporting isn't that robust and you have to get other libraries to plug in. So how's the reporting with Robot framework?

Oh my goodness. One of the best things about robot framework is a built in reporting. It is right out of the box. You get awesome HTML style reports or XML or whatever you want really, but it's they. I'd say it's probably the best feature of Robot is built in reporting. Very easy to drill down and see what's going on. Every time there's a failure, it automatically captures a screenshot for you. You can capture a screenshot whenever you want, but every time there's a failure it's automatic. You can, you know, inject your own messages into the log as well. I've created a couple of keywords that we could that or they use built in keywords but I create an abstraction layer above it. That's one of the things I like to do, Joe. And maybe that's not good practice, I don't know. But I like to build an abstraction layer when I'm using a built in keyword, something that's coming straight from a robot framework, I try to create a wrapper around that keyword. So if I ever had a rip out robot framework, it wouldn't be as heavy of a Lift. As if I was using robot commands everywhere. Right. I just go to my single abstraction layer and I update my call to robot keyword to something else and off I'm going. But yeah, so I create a message. I wrap one of the robot keywords to trace or debug or info or warn or error level logging wherever I need to in the application. So I just call my keyword, hey, this is a trace level log. If I in Jenkins, if I want to run my test with trace level logging, I pass that in along the command line and the. And now I've got trace level logging throughout my application. So a lot of my keywords, I've built that in. Right. If I hey, I'm doing a trace level log here. I want to see exactly what's going on at every step. So some of my keywords may look a little daunting because there's so many log messages in them. I think it's important to have good logging and robot provides it. So use it.

Absolutely. I actually love that method of layers almost. That's how I created mine when I was at our other company. We use BDD on the front end. Would call one layer. That first layer is like a business level thing like patient login and that would call a technical layer. And then in theory, people then wouldn't have to know how the sausage is made. So you could pull up a selenium library and plug in like water and it would still work with minimum updating because you would just have to update those methods for the technical piece of it.

Exactly.

Yeah.

You know, I like to think of it as Legos. My really low level keywords. My. The ones that do one thing and that's my single Lego. That's one Lego. And then I've got these other ones that are. They might leverage several keywords from the same page or page object. You know that. And that's creating a very simple structure with Legos, maybe all the same color or all the same size. Right. And then you start getting into more things where you're. I'm bringing together keywords from multiple different pages to build a kind of a workflow type process that can be called. It just makes it easier when you're building tests. Anything you can do to make it easy when building a test? Hey, I'm creating a patient. Let's say in the healthcare, there's a lot of steps involved in adding a patient or creating a patient record. Right. I might have a keyword called create patient. It'll take quite a few different Arguments and it's going to call a whole bunch of different keywords downstream. But in my test, every time I need to create a patient, I'm calling one keyword and that does all the work. So I think that's the kind of approach I'm going for.

Nice. Now you did say you work for a security company. Is there any type of built in security functionality? Do you test any security using any plugins or anything with the robot framework?

Like with many companies, security testing is kind of its own specialized function. So I don't, and I don't know, I haven't really looked into it. Perhaps there are ways of doing it in robot. I haven't investigated that because we have very smart people that are in charge of kind of that level of testing that are specially trained for that and I'm not. So I do try to integrate a little some low level performance testing. So again, you need good performance testing tools for performance testing. And it's more that I can do a little bit of load. Right. I can do an operation a whole bunch of times across a whole bunch of different machines using a robot script to generate load. So I guess it might be more load than performance.

Yeah, so that brings up a good point. How does it run in parallel? A lot of times it's hard to run certain frameworks in parallel. If you're doing some sort of performance, then is it easy to scale up and run a test across multiple instances?

Oh sure. I mean I use a selenium grid and an in house selenium grid and so yes, it's a simple matter of I just create a Jenkins job and fire off a whole bunch of that same job. Or there's many ways to do it. I do it a few different ways, but I have a multi job in Jenkins where I can execute the same job many times but with different parameters each time, that kind of thing. So mostly Jenkins is helping me drive that though.

Okay, Dave, before we go, is there one piece of actual advice you can give someone to help them with their automation testing efforts and what's the best way to find or contact you?

Sure. My advice is to just jump in and learn, right? There are so many resources online for learning robot. Well, maybe not so many. There are Enough though. Udemy, LinkedIn, learning, pluralsight, I believe. I mean, you name it, there's going to be some courses out there and I think it's well worth. I mean I spent some of my own money on the Udemy course just because I wanted to learn this. And so there was that internal drive, I think that has to be there, right? Find a business problem, something that you can identify, a quick return on investment, do a little bit of side learning and then jump in and show some roi, automate some process that's painful and put on a little sales pitch to your boss and say, hey, this took so long to do, I've now automated it and boom, show it off. But every little step along the way when there was a little challenge just took a little bit of research something there's an answer out there and leverage the community. But yeah, you've got to be proactive. And I think QA does involve a little bit of salesmanship as well sometimes. So I guess that's my advice. And as far as contacting me, contact superdavemail.com is my personal email address and happy to answer any questions that people have about robot. I'm pretty passionate about it, so I want people to be successful.

Thank you Dave for your automation awesomeness. Links to everything in value we covered in this episode. Head on over to testguild.com a320 and while you're there, make sure to click on the Try it for free Today link under the Exclusive Sponsor section to learn all about Sauce Labs awesome products and services. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe and my mission is to help you succeed with creating end to end full stack automation awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Automation Podcast. Head on over to testguild.com for full show notes, amazing blog articles and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.