243: Creating Klassi Frameworks with Larry Goddard

Hit it.

Welcome to the Test Talks podcast, the place to go to geek out on software testing. And now your host whose mission is to help you succeed with test automation, Joe Colantonio.

Hey, it's Joe. And welcome to another episode of Test Talks. Today we're going to be test talking with Larry Goddard, all about creating a classy or automation framework. Larry will discuss some JavaScript related testing tools that have helped him to create tests for ui, API and visual validations, as well as generate reports and run tests in continuous integration. So listen up and discover how to improve your JavaScript based automation testing framework efforts. Check it out.

Test Talks is sponsored by the fantastic folks at Smart Sauce Labs, the cloud based automated testing platform that eliminates the need to maintain your own selenium grid and test infrastructure. Try it for free today. Visit testtalks.com and click on the sign up now link under the homepage sponsor section.

Hey Larry, welcome to Test Talks.

Yeah, thanks Drew. It's my pleasure to be here.

Awesome. So I've been following you for a while. You joined the guild and you have a lot of great insight with the automation Guild Slack community. And one of the things I noticed was you were talking about frameworks. So today that's why I'd like to talk about all your experience automation frameworks and your experience creating UI API BDD tests. But before we get into it, can you just tell us a little bit.

More about yourself, about me? I don't know, I'm a tech junkie as you would call it. Been playing around with tech from since you used to plug your cartridge into your keyboard to do stuff and went on and do systems engineering.

And then after a while, a couple years and that, and then through some faith and a lack of a friend, get into the, the whole testing thing. And I must admit after that point I didn't even realize it had something called testing, you know. So yeah, as I get into it and over the years, you know, you move from standard off the shelf products to open source things and, and outside of that, I'm a keen rugby player. Don't play anymore, stop. About five years now, but I played for like teens of years, I think like 25 years non stop. You know, I went through the whole, the whole ropes with it and if I, not if I wasn't playing sports and I wasn't think I just like to chill. So that's, that's about me really.

Very cool. So how, how long you been testing for then?

I've been testing for roughly Namaste 2000 so that will be this year, 18 years.

And how did you get into automation from testing?

Well, I got into automation like I say, from over the time. You know, it was all manual testing and then it had and it was all Mercury Interactive, Load Runner, windrunner test director and stuff like that. And that's how I started into it and you know, just progressively move into.

Open source and now. Yeah.

So you mentioned a lot of automation tools there, so you have a lot of experience. So I'm just curious, how do you choose an automation tool to use?

I make my choice based on one, based on what the. Not what I actually testing but based on what the tech stack is where I'm working at a point in time I try to keep my test tools in line with what the devs use and so it's easy to get only for the. Easy to get help like that, you know, like. So I'm doing something on your stock. You can readily, readily get help from one of the devs and them to whatever it is you're working with. So that's how I kind of think that's previously but now because I started getting into frameworks and things, I kind of went down the road of choosing a particular language and kind of sticking to that unless it's something specific for a client.

So what language do you stick with or did you decide to choose on.

JavaScript node JS and why is that? Because I did a look and I tell myself, you know all these, I look at it from the point of all these, the new startups and, and all these things, they all goes in, they all uses this technology stack, you know, React JS and all these weather wonderful stuff. And yet you find all the QA frameworks out there was, was mainly Java Java based and one of the problems was you either had to learn Java or the testing was completely isolated from everything else. So if the QA get hit by a truck tomorrow, all the testing stopped because none of the actual people like you know the devs and they know Java anyway because they're into the soldier and that's why I decided to go down that JavaScript route.

Cool. Yeah. For some reason I noticed everyone that's into JavaScript rather than using the Selenium JavaScript binding, they tend to use WebDriver IO. I believe that's what you decided too as well. I'm just curious to know why you chose Webdriver IO.

Oh yeah, I chose Webdriver IO because of the way it is. I used to use like just you know, Selenium webdriver like everybody else. And then I was doing a job for Cambridge Assessment, Cambridge University, and I was looking at how I could do this and get all of the help with less time as possible. And one of the things I realized, I started looking around and one of the things I realized is webdriver IO and I only know about it at that point when I was actually looking, it was that is the way the commands are. It's very super intelligent, for want of a better word. Where as a normal Selenium webdriver you'll have to say find element by CSS or ID or by name or partial text. With webdriver IO is just a simple driver, dot, click and you put the element and in the background it will decide based on the element you pass. It will be. It will, in the background will decide if that's a CSS or if it's a, a partial text or you know, one of those expat or whatever it is. It works it out, let's say for itself. So another one of the things, because like I tell people I'm lazy, so anything to help, anything to help with your progress is made. And knowing that it makes a test a little stronger as well, because you wouldn't like grab an ID and pass it as a name because your test will just automatically break anyway. So it avoid all them things. So that's one of the reasons why I went down the web IO route.

So is this something you would recommend to someone if they were looking for a way to automate their browsers?

Yeah, at the moment I recommend, I'm a staunch advocate of Webdriver IO. I recommend it to everybody that I talk to and, and I come in contact with where they're asking for, you know, some feedback on automation. And I also. One of the things is that they'll just recommend it because they, I tend to kind of like explain why they would use that over the normal Selenium web driver.

Very cool. So is there any functionality of webdriver IO you don't think a lot of people know about that? They should. They're like, wow, if they only knew it did this. I mean, what you already mentioned already is pretty awesome, but is there anything else that it gives you?

Yeah, one of the things, I haven't actually used it in any project as yet, but one of the things with Webdriver IO is to give you the ability to work with Shadow doms and that Shadow DOM is a big thing with Selenium and everybody else at the moment. I actually created Two little functions that would grab the main DOM and then find the shadow dominant and pull whatever idea need from in there. It was a kind of a workaround. But right now with this latest release of Web IO, they literally put it in there. So you can now literally call your shadow DOM directly.

So the folks that don't know though what is a shadow dom, because I've heard this a few times and it seems very complicated to automate.

Let me see, to use my words, is it's where you have your DOM where you find all your elements and stuff, but then inside that it is something like react. So it generates like a sub DOM of the main dom. It's not readily available for a point and click. I mean, that's about the best way I could actually put it without actually looking up the true meaning of it.

So you have Webdriver IO and if you look on GitHub, you have this all documented as your framework. What I thought was interesting also you used a tool called Resemble JS for visual regression. I've never heard of Resemble js. Can you just talk a little bit more about that?

Yeah, funny enough, I was looking at, I actually when was coming out, I was looking at, you know, you have API tools and, and all these commercial, commercially available things. And one of the things I was looking at is how could I have an open source application that will allow me to build in something and enter that later when you the user decide, oh, that's looking like a cool framework, let me use it, it will remain open source, meaning you wouldn't have to get an account anywhere and pay any money for anything. And I was just, and I was just doing a search really and trying to figure out what could do what and what can do where. And that's how I came up with, with our Resemble js. I looked at what it could do, I tried it out, you know, run a simple test on what it could do and then I kind of write a little wrapper around what I wanted to do. There you go. So like I said, I didn't know about it prior to that at all.

Very cool. So give an example of one of those wrappers would look like, what do you use Resemble JS for? Like an example on a real application?

It's Resemble js. It's a little tool.

I would say a lot of tools. One of those little things that what it does is do image comparison. It allows you to take an image and compare it and then take another image of that same page at a later date and compare it with the previous one that they took. That's why it does in a nutshell, I use it for, like I said, visual regression and you still have it allows you to like block out. You could block out a part of the, let's say like let's for argument's sake, a clock, a live clock, which means every time you take a picture of the page the time will be different. So that means that image will fail. So you can literally block that clock out by passing in the XY coordinates of where the clock is on the page and will grab an image of the page and put a black spot over the clocks. It will ignore that. So it's just an image comparison little tool.

So how reliable is it? Have you noticed if it's flaky or is it pretty good, pretty solid?

Well, so far for what I'm doing at the moment, I work for Oxford University Press and a lot of the work I do at the moment is on the platform, my math platform. What that is in itself is an application for school kids to do homework, maths homework. So they have lessons on everything in there. And I'm using it to test that in terms of, you know, what the page look like when it now load it up if you put some marks on it and come back to it if the marks are still there. And so far in terms of 100% I would say it's not all that like happy tools. It's nowhere near to happy but it does what it. What do you want it to do in turban? I mean there are a lot of tweaking you could do with the pixels and get it as close as as possible. But yeah, it's not as like my good friend Angie Jones. It's not, it's not as good as. It's not as good as that. But it does what I do. I mean if you want to really get down into I wouldn't use it for let's say some big industrial easy. I wouldn't use it as a NASA project.

So it just sounds like if you don't have a budget and you need to do a quick and dirty visual validation test, this may be something that might be helpful.

Yeah, I cannot consider that at all.

Very cool. So another part of your framework is API testing and you chose?

Yeah, I chose Request promise.

And why is that Again?

I choose that because I look at a lot of things like Chakra and a few others. A lot of them is written in either the Java base or inside the JavaScript based. They're more for Mocha to use Mocha, but because I run in cucumber and I run in web IO but from a standalone array. I wanted something that could have plugged in to use Cucumber so I could write my test in cucumber and still test it and then call it, you know, so straight out. So I have a feature file that say, I don't know, you know, get this and tell me what the results and the response time and all these things. And for me it was one of the best ones that I came across and I and I sampled a lot of them and was one of the best ones that I came across in terms of a lot of things to it in terms of what it could do, how easy it is to do it and you know, the different ways you could pass in your payload and everything. And I. So hence the reason why I use that.

Cool. So I think another thing that people struggle with and you mentioned you're using BDD is for some reason they can't wrap their heads around that BDD is not just for browser tests. So you just mentioned three different technologies and it sounds like they're all within your cucumber BDD framework. So is there any way that you run a test flow that touches each of those levels of an application? The ui, the API and a visual validation test? Are they completely separate from one another?

No, actually, no. What it is is barring the API.

I write a normal standard bed script that say why is I'm Joe and I want to go and see my account that I suppose have $100 and I, you know, I've standard test like that and I test why is I doing a normal functional test? I grab I doing a visual at the same time so I could visually check every single page that you go on like a check. I could check a page for every single base, a component that on that page that you're going to interact with and grab and grab a picture with. Like I say it's just like in terms of, in terms of data, just like you're using apitools but it's just like on a more manager scale. So it's the same concepts. I will write a test to do all my normal functional testing and I would put the image checking inside of that at the same time. The only thing runs separate but at the same time to what of a better word is the API testing, which means I could write a test that talks to the API.

And I hate to do this, but I can do it where you grab the result from the API and pass that as the data for your functional test. All within the same for you. It's not say you're going to be running one separate from the next. Or you could literally have API testers and functional testers, all of them mixed together and running as one test suite.

Nice. So how do you decide how you're going to make your test? If you're going to make it a UI test or an API test or a visual validation test?

Well, it all depends on what I actually tested at the moment. You know, what is my requirements. Most of my visual tests, most of them are attached to some functional tests. I run them at the same time or I have separate tests. Again, it depends on. So just to draw an example, on the MyMaths website, you have images where, you know, the student log in and let's say they're doing trigs, they log in and they have a set of formulas on the page and the way the page is constructed. Or I could just go directly into it and grab an image on a page to make sure that all the images are they supposed to see is there. Just do a straight without going to test anything else but doing a visual check of the pages. So it all, again, it all depends on what it is I'm looking at. I would write a test accordingly. I might be doing. I might have to fill out a form and the best way to do that in a sense of what I'm doing is by the API. So I'll write an API test to fill the form out and then grab the results from somewhere else.

Cool. So are you an independent contractor or do you, are you consultant or do you work full time for, for the school?

At the moment I am a, I'm a full time employee of a consultant firm and I work at Oxford University Press. That's at the moment, but in the next month I'll be back contracting. I'll be back working for myself.

All right, so how do you go into our company and say this is the approach of testing we're going to use? Is that something that they tell you, hey, we're hiring you, you just automate this? Or do you actually have input in.

Some instances, like although I work for a company at the moment, when I went to the Oxford University Press to work for them, I think, I think I could go back a little bit. When I went to work for Cambridge University, which is Cambridge Assessment, when I got there, they had a particular problem and I was given a green slate, more or less a clean slate to put something in place and to get things done. And so there are times where I would actually go sit with the people, find out what it is they're trying to achieve and come up with a strategy for it. And in some instances I would come and I would just work with what you have because you already have something in place. So I will just take that and enhance on it or just follow it as a call. It all depends on what the situation is at any given time.

Awesome. Yeah. The reason why I ask is you're also using bdd and a lot of times teams will use BDD to try to help with their communication as well. Is that, did you start this project and they already were using cucumber or is that something that you recommended using?

Okay, yeah, no, when I started, I could. Now I could come back to Oxford University Press. When I started there over a year ago. No, they were all waterfall. You know, there's all big stuff. And I was now kind of making a push for agile. So when I went in, in this instance, I was kind of lucky in the sense that I end up on a greenfield project, as they like to call them, where it didn't have anything. So I actually came up with a test strategy. Pass it around the people who needed to see it and they all agree and buy into it. And then what I went on to do and doing and doing now, as we at the moment is I am now doing workshops for the business analysis on them to pick up the whole BDD thing. So you still have. So at the moment, I still have a lot of. Only the most recent projects like the one I started, and a few others actually have the whole cucumber BDD thing going. And a lot of the others, they don't. So I do a lot of workshops at the moment where I am showing them and talking them through how to do it, how to write proper PDD feature, how to write proper feature files and things like that.

Cool. So when you're doing those workshops, is there any common things you see over and over again? You're like, wow, everyone always stumbles on the same exact thing.

I have to say, so far, all the workshops that I have, but because the people I'm working with, they really have no idea. In other words, nobody that have come to one of these sessions so far came with a preconceived idea of what it is and how it's supposed to work. I'm like, God, I'm telling you, yeah, that's the apple there, you know, that's the apple. If you eat that Apple, this is what's gonna happen. But no, no, no, no. You know, so everybody.

Yeah.

So they buy into it and, and I must admit, I'm not, I didn't invent the, I'm not trying to reinvent the wheel. I am using Cucumbers tutorial stack and appeal it to suit, you know.

The people I'm working with and the time frame in which I, I am, I'm doing it. But yeah, they didn't come with no conceived notion of what it was. So it's a kind of a novelty thing to a lot of them in the sense so, and things like that and you know, like where you would save your feature files and, and things like that. So it's all, it's all a whole new concept, a lot of them cool.

So I guess that's even harder than if it's a new concept. Do you even have to start. This is what automation is and this is when you should use and when you don't use it, not at that.

Level because a lot of them know what automation. Well then they have an idea of what automation is and stuff. But what it don't do is they hear about PDD and they hear about cucumber and they hear about Selenium and they hear about, you know, webdriver and all these things, but they've never actually had to use it in any way at all. So in that sense it's, it's a lot easy for me to have to talk to them and explain things to them. And so you find a lot of the questions I would come back is, and what about if I write X, you know, how would that affect it? And just little, just little things like that. And you know, and as, and as, you know, once you start already give me when then and but you know, those are the keywords. So once you start with that, everything after that is what it test is going to look at and ignore the whole given when, then and then but. And all the other little intricate ones along the way. So yes, it's a, it's, it's pretty, let's say not too, too technical but more, this is how you would write it so that I would understand it. So and I don't have to go into okay, you write it now when you write it, this is what I will do with it. I don't have to go through, I don't have to go through that. Thank God for it.

So who runs the test? Do you have it set up in Jenkins or in DevOps or how does that work?

It all depends on the project. It all depends on the project. For my project in particular is fully CI complete CI integrated from the developer checking a branch to going out into productions. A complete CI pipeline. I working on at the moment in my project are using CircleCI well that's what they use. So that's why I use that. They use Circle CI and of course browser stack for multitasking along the way. Yeah, so that's what we use. It's fully. A fully CI integrated thing.

Cool. How about reporting? Does WebDriver IO have any reporting in it?

They have a few. WebDriver IO have a few reporters built in but I don't like none of them. So I just use one called Winston. It's a nice little thing what it does, it passes the. It grabs all the data and save it as a JSON. But it also generates an HTML file with a nice graph on it and it separates out all your functions and your features, your features and your scenarios and all these steps on a take image. And you can do a lot with it. I just have a tick in an image. If it fail, I have a grabbing an image and passing it into the. Into the report at the same. At the same time. So that's why I use as a reporting tool. But yes, WebDriver IE have some reporting tools in them, but I think they have not in it. They have a lot of integration with Allah I think it's called. Yeah, and a few others. But I just like I say I look around and see what is how and what works best for me and what I like and stuff like that. So yeah, I just use Winston at the moment.

So Larry, a lot of times people like oh gosh, what tool do I use? Where do I get started? There's so many options that they don't even get started. So what inspired you to create this framework?

Well, it was the situation at hand at the moment and I was faced with a kind of impossible task. I wanted. The company wanted something where they could reduce the. Of course the whole. When they do a major release it used to take what, two and a week to do a regression test before they could push things live. And also had a next unique thing. What I wanted was something that could also do that could compare the PDFs which would be exam papers because you know, exams, you have people that do maths exams, you have people doing it in the morning and people do it in the afternoon. But it's the same paper, just different questions. And that was one of the little unique things and I did and I started looking around and trying to figure out, you know, what I could use. And I started initially I look at C sharp because what a lot of people was using, it was a dotnet environment with a lot of other bits. And I don't find was too cumbersome, especially when coming to make changes. You'd have to compile and recompile and too much nightmare. And that's what get me into it. And I built it. And funny enough, you know, when you build something working for somebody, it's their property, their intellectual property, which was one of those things. But then I was looking at Cambridge Assessment, which is Cambridge University, and the boss item said, wow, this was really good. You know, this is really, really impressive. What you should do is make it open source. And I will admit up to that point I didn't even know what open source was in, in the scheme of things. I mean, I know what open source was in terms of what I use it, but I didn't know what it was in the scheme of things where I could make something that is open source. So I said yes at the time and I said, yeah, sure, I will do that, definitely do that. And then I, of course I went and do a little research. Oh, how do you do this?

You know, that's what actually make me decide to do it and why it was just a plain thing. And then as time goes by and I learn about different technologies and different things that you could do with a framework, I start adding bits to it and making it one whole thing that could do. I wouldn't say everything, but like in my case, I could do my APIs, I could do visual regression and then normal functional and regression testing all inside the one framework. So that's what it was that inspired me to do this.

What's the name of the framework is a Classy.

Yeah, classy. I know the problem is classy. That's a whole different thing by itself. Classy is, I would call it my brand. I actually trademark the word classy with a logo to go with it. It's something you just a word I hear, I heard many, many, many years ago. It's a German word, by the way. And that is why I started saying one because I already have this and I had to come with a name for it. I, I just decided to, you know, to give it at first. I originally call it Web Driver IO Cucumber dash js. That was my original name for it. And then, then I say, wait a minute, I have my whole classy brand. Why don't I brand it with my, you know, put my brand on it.

Then I named it Classy Cucumber js. And then just like a couple weeks or a month ago, just recently here, I literally name it just classyjs.

That's the whole thing behind that.

Nice. I love branding. That's a great idea. Okay, Larry, before we go, is there one piece of actionable advice you can give someone to help them with their automation efforts and let us know the best way to find or contact you.

What I would say is one piece of advice I will give a lot of people, and I mean a lot of people where automation is concerned is that it is a little saying. I can't remember who said, but this automation should only be used enhanced testing and not replace it. That's one piece of advice I would like to leave for a lot of people because a lot of people have the concept that when you start to do automation, manual testing and all these kind of things goes out the window. But that's not true. You know, that's not. So it's a big misconception to have and the field automation is going to take over the world. That's one thing I would like to say. And contact me. You can contact me via Twitter or again to my normal my personal email which is larryglassytech.co.uk they could follow my brand on Twitter and on Facebook and everything. So I'm very much contacted through most other media.

Thank you Larry for your automation awesomeness. The links to everything of value we covered in this episode, head on over to testtalks.com 2, 4, 3. And while you're there, make sure to click on that. Sign up for a free trial link under the exclusive sponsor section to learn all about Source Lab's awesome products and services. So that's it for this episode of Test Talks. I'm Joe and and my mission is to help you succeed with creating automation awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Talks podcast. Head on over to www.testtalks.com for full show notes, amazing blog articles and other automation awesomeness.