226: Acceptance Testing Using Gauge with Zabil Maliackal

Hit it.

Welcome to the Test Talks podcast, the place to go to geek out on software testing. And now your host whose mission is to help you succeed with test automation, Joe Colantonio.

Hey, it's Joe. And welcome to another episode of Test Talks. Today we're going to be test talking with Zabel, a product manager for gauge at ThoughtWorks. So that's what we're going to be talking about today, Gauge, which is a free open source test automation framework. And as you hear in this episode, Gauge is really flexible. It was designed specifically for acceptance testing. So if you're looking to develop tests that require less code, less maintenance, and focus more on acceptance testing, you don't want to miss this episode. Check it out.

Test Talks is sponsored by the fantastic folks at Sauce Labs, the cloud based automated testing platform that eliminates the need to maintain your own selenium grid and test infrastructure. Try it for free today. Visit testalks.com and click on the Sign up Now link under the homepage Sponsor section.

Hey Zabel, welcome to Test Talks. Hey Joe, it's great to have you on the show today. This has actually been on my to do list for a while to actually talk to someone about Gauge, the open Source framework by ThoughtWorks. And since you're the product manager of Gauge, I thought you'd be the best person to find out more about Gauge. But before we get into it, could you just tell us a little bit more about yourself?

Yeah, Joe. So I am the product manager for Gauge. It's a job that I've been doing for the past two and a half years. Prior to that I was working mostly on continuous delivery. I was a delivery manager for our continuous delivery product, that's GOCD. And I have roughly 18 years of experience mostly building enterprise software and working with as a consultant, agile coach, technical architect on projects in the banking, retail domains. In my current role. And while I've been doing all these projects, I focus on engineering practices, especially around testing. And with Gauge, what I'm trying to do is bring those good practices and build a product so that it's easy to get started to implement good testing practices in your project.

Great. So that's actually a perfect place to start. What are some high level practices you think any automation engineer should follow, regardless of what automation framework they might be using?

Yeah, so the best example of adopting the good practice is following the test pyramid and figuring out where to write your test and what test to write. And many times people overdo their test cases or focus a lot on functional test cases when Writing test cases at the, you know, at the right layer matters. So it's knowing where and how to write tests and using tests as something that gives you the confidence to release. And once you get that idea and philosophy going, things will fall into place.

All right, So I think a lot of teams actually, they'll get tripped up on the confidence to release. So are there any metrics you go by to let you know, hey, this is ready to be released or is, is it just a team decision of when to release?

I mean, there's no metric as such. It's kind of a team decision. They know that, you know, they've put the right tests in their build pipeline and if, when their build pipeline goes green, they are ready to make the release. And that's about it. So it's about knowing that you've written the right kind of tests and you have the right kind of test coverage.

So do you have a rule of thumb of what the right level is for teams to do testing? Because a lot of times, of course, they go straight to the UI as an end to end type automation test.

So the rule of thumb is, I mean, the test method is a good start, but it's about writing acceptance tests that are end to end and then writing complaint tests and all that. So, I mean, there's a nice article on Martin Fowler's blog about the different kind of tests. And if you know that you have these kinds of tests, then you know, I mean, when your test is distributed, that's when you kind of know that you have a healthy test suit.

Now you did mention acceptance testing, and I think this is actually one of the features that Gauge really excels in. So before we get into that though, at a high level, what is the gauge framework?

Yeah, so gauge is an acceptance testing framework. So in Gauge, normally what we've seen is acceptance tests are very hard to maintain, especially, you know, to read, and they run slow. And then it takes a lot of time to figure out what's wrong when your acceptance test suit fails. And so Gauge tries to solve these problems by one is writing acceptance tests in markdown so that they're readable. And when something fails, you know exactly what failed and why it failed. And most importantly, it does it in a way that it removes a lot of code. So it keeps the code in your test suit very small so that it makes it less flaky and less prone to failures.

Cool. So I think most people are familiar with behavior driven development using Gherkin. Is the markdown language the same as Gherkin Yeah.

So, okay, this is a slightly controversial topic. Okay, so GAIN is not a BDD tool. Okay? We focus on acceptance testing. That's the acceptance testing phase. And if you look at many of the literature that BDD and articles about BDD especially, you know, if you go to the Cucumber blogs, they say BDD is not about testing. And they go as far as to say, you know, bdd cucumber is probably the very tool that's misunderstood and people use it exclusively for testing. But that's not the case. So Gage has a different approach. It, it treats acceptance testing as acceptance testing. It's focused on test, it's not focused on collaboration among team members like what BDD is. So it's taken some liberties. For example, Gauge doesn't have a strict given when then format. So the fact that you can use markdown means there's no syntax as such. I mean, we just use some documentation syntax to figure out what the steps to execute are, what the scenarios for the tests are, etc. And that's kind of the main difference from other B2 tools. So again, what I like to say is Gauge is not for solving your communication issues within the team, we believe that's not what testing is for. Gauge is specifically for solving testing issues related to testing.

So can you give us an example like, how does that translate into the way you write your test compared to, say, BDD using Cucumber?

Yeah. So normally when you write acceptance tests, let's say you use plain Selenium for writing acceptance test for testing a web application. And Selenium is a driver for automating tests, automating web applications. You would use a programming language like Java, and then you put in a lot of effort to design your test suit in a way that it's readable and maintainable. So Gauge works on that aspect. It eliminates the need for you to use your programming language for designing readable tests. For example, if you're using Selenium, you don't actually have to use page objects to write your tests. You can organize your tests in Gauge. And it works something like what you do in your flow of testing. For example, if you want to test a web application, you're going to simply write down the steps which are acceptance criteria for the application. And once you've written down the steps, all the steps of how an application must behave, you just go and implement some code to execute those steps. And that code is a simple Selenium click and Selenium scripts to interact with your web application. So that's how it's different. So it doesn't have a prescriptive syntax, it's just a free flowing form of steps or actions to perform.

All right, so give us an example. For example, say you have a login. So rather than using a page object for the login, how do you do that in Gauge? Is it just a reusable function or method using like a keyword?

No, it's not keyword driven yet. So what we have is you write, for example, you're trying to capture the behavior of a login functionality. You just say login with the username and login with a password. That is a step. Those steps can specify parameters. For example, the username or the password is a parameter to a step. And then you implement a code that will execute the step. So the code can actually take in the parameter which is the username, and then use your programming language to interact with the web application. So it's not exactly keyword driven. It will just match the step with the code that needs to be executed for that step and then run it.

So I assume it's using Selenium behind the scenes? I'm not sure. Does it use any other technology to help you with your automation?

Yeah. So Gauge is. So we built Gauge to be like a bring your own tool framework. So you can use Selenium or you can use anything else for driving your Gauge specification. So let's say you have a team delivering APIs and you need to write acceptance tests for APIs, then you can use an API testing tool and plug that into Gauge and have the same benefit. So it's not strictly Selenium. You can plug in any tool that you want. So you can plug in your favorite assertion framework also for the test and what Gates provides is this is a runner which can parse your specifications and run the tests. And then it's got nice reporting feature which will tell you what went wrong and what failed.

That's awesome. Sounds really flexible. So I also assume that Gauge was designed to be easily run within your CI CD systems using that runner.

Yes. So it's a CI first command line, first tool. It's got strong ID integration also, but it's really built to run in the CI. So Gauge is a single binary and it works for any platform and any language.

Cool. So how do you get started with Gauge? You just pull it in like an import statement or is it its own self contained type application that you then develop in?

What Gauge does is you just download the Gauge binary and you just Initialize a Java project. Once you initialize a Java project, that's all again, built into Gauge. So Gauge has templates for all kind of languages where you can initialize a Python project, Java project, or a C Sharp project. So you need the prerequisites for that language. For example, if you're initializing a Java project, you need, you need the JDK installed. And once you initialize the project, you can open it in your favorite editor. We are not very. Again, the specifications are just simple plain text markdown files, and you can open that in the editor of your choice. But if, generally when you use Java, you would want to edit them in IntelliJ, which is one of the most popular IDEs. So we have a plugin for IntelliJ. So you initialize the Java project, you open IntelliJ, install the gauge plugin for IntelliJ, and then you get all the fancy auto completion features. You can run your tests from within IntelliJ Extra and edit Java files like you do a normal Java project.

Awesome. So you did mention that you did work on other projects. And what you want to do is take the best of those projects and bake in those good practices into Gauge. So can you give us some examples of maybe some good practices that you actually built into the Gauge framework?

Yeah. So one big aspect that we're looking at is writing reusable specifications, right? So normally when you use something like Selenium or any other framework and you, you have to build, you have to spend a lot of effort building a test framework so that there's reusability, that you can achieve some kind of modularity or reusability in it. So we've cut down that effort. And that's one best practice that when you start using Gauge, it comes, the reusability part comes out of the box. For example, once you implement a step or you define a step, you can use that step in any scenario.

For example, if you've written how to login into your application, you can use that in other scenarios. Let's say before checking something out of a shopping cart, you need to log in. You can just combine the login step and the checking out a shopping cart step to make, make a new scenario. So you can mix and match over there at the same time. You can combine multiple steps into a concept and then reuse that concept, and then that goes into many levels. So we worked a lot on the reusability aspect of it, so, so that your tests are easily maintainable. And that's what our focus has been. And Then the feedback, how fast you get feedback when your test fails and how you know exactly what fails. For example, we have very good reporting features filled in so that when a test fails, it automatically takes a screenshot and tells you what in the code failed. You can even implement custom screenshots in any part of your applications to know. So we worked on making the reportings really good so that, so that you can easily go and fix either your tests or your application, depending on the failure. So these are the two major things that we're working on. And another thing is we wanted Gauge to be something where it does not dictate the language that you need to use for testing. For example, if you're working on a Python project, you can use the Gauge and the team is comfortable with Python, you can use the Gauge Python plugin to write your tests in Python. So then what happens is the tester or the team does not have to switch contexts to figure out how the test is working and all. So we really integrate with your set of tools, your project's tools, to make it easy to manage your test cases. And that also works in a way that you can use your projects libraries also in your test cases. So it's like it's all this that we've kind of built into Gauge.

So as I said, it sounds really flexible. So it sounds like one of the reasons for its flexibility is this concept of a plugin to help extend its functionality. Can you explain a little bit more about this plugin concept?

Yeah, that's right. So again, how we build gauges, there's a small core gauge and then everything else is a plugin. For example, adding a support for a new language, you can do it as a plugin, adding a support to a new reporting tool, you can implement that as a plugin and you can implement that in the language of your choice. For example, you want to integrate it with jira, you can write a reporting plugin that integrates with Jira quite easily. So we have this kind of modular plugin architecture so that you can build your own plugins and extend Gauge.

Now I know you said Gauge is really flexible and that you can use any technology behind the scenes. So along those lines, can you also use Gauge to test mobile applications using say, Appium?

Yes, it can leverage Appium and there are a few sample projects that people have created and we have to that where they're using Gauge to test mobile applications.

So when I first heard about Gauge maybe two years ago, you actually presented at the first Automation Guild, Gauge was still In Beta. But I believe in June or July you came up with version one. What does that mean for Gauge now that you're in version one?

Yeah. So one of the main things that we wanted to focus on on the 1.0 release is.

Stability.

And the tooling support support for tools. So in 1.0 release.

We support five major languages. Earlier it was just three, that was Java, C and Ruby.

We have now official support, the additional support for Python and JavaScript and we support all major IDEs. That's IntelliJ, Visual Studio, and more importantly Visual Studio code. We built real cool features like all these IDE features into Gauge, so it can actually work in the IDE of your choice. For example, all the auto completion features can work in VI or Emacs. So then we felt that there's good tooling support and there's good stability and that's what pushed it to 1.0.

When we start off the conversation, you mentioned that even Cucumber mentions there's some scenarios where they believe you should not use Cucumber for. So in the same spirit, is there anything that you believe Gauge should, or more importantly shouldn't be used for?

Yeah, so we wouldn't recommend using Gauge for anything other than acceptance testing, for example, writing your unit tests, etc. We really don't recommend Gauge. If you want to do unit testing, use a unit testing framework like Junit or for the Preferred Language Officers for python nose or JavaScript. There are other a lot of testing frameworks there too. So do not use gates where you don't. Where there's no acceptance test.

Another feature I remember seeing when I was evaluating this maybe a few years ago was the strength of your parallel execution feature within Gauge. Can you tell us a little bit more about the parallel execution feature?

Yeah, so it's two aspects. So Gauge, you can write data driven tests in Gauge, where you can define data in tables or CSV and you can run those tests in parallel, where for every row of the data that you're providing, you can kick off your scenarios in parallel.

Because the way we built Gauge, it can utilize all the cores in your hardware effectively to run tests fast.

That's a feature that we have in Gauge in other frameworks. When you run tests in parallel, it's kind of tough to collect the reports on. It's tough to report it correctly. And there are many plugins or workarounds for to get the reports on apparel runs. But Gauge handles that seamlessly. It knows how to, because. Because we built the feature into the reports Also, you can just either run your test sequentially or just pass a simple option to run them parallelly and your reports and everything are going to be handled quite well. The only thing is you need to make sure you write your tests so that they are independent and then they can run in parallel.

Another thing people usually struggle with when they have so called flaky tests is that they don't know how to use the right wait mechanism in order to wait for an element to be available before they start interacting with it. And usually when they're coding from scratch, they don't know how to handle that. So is there anything built into Gauge that actually helps with this type of issue?

It is outside of the realm that Gauge handles and I mean we are actually working on something. We are working on another library that kind of complements Gauge in the browser automation space. We've just come up with an alpha release. We are coming out of the beta release soon. That's called Tyco. It's a browser automation tool. So you can plug that into Gauge to solve those kind of issues. But that's not something gauge handles, it's agnostic of those timing issues.

So if someone's thinking about starting using Gauge, but they're afraid because it's open source, there may not be a lot of contributors to it, or it's not a very lively project. So I guess the question is how committed is ThoughtWorks to gauge and how are your contributors so far helping with the project?

Yes, we have a growing community. It's a popular project on GitHub. We have, if you look at our forums or our chat forums, it's quite active. We have a lot of people using it, reporting issues with it, and ThoughtWorks is committed to Gaze. So if you look at the history of thoughtworks, we've been pioneers in the testing space. A lot of good testing tools came out of ThoughtWorks. Selenium is one. There is, there is Mockito. All these frameworks were kind of the Sahi, another another tool. All these frameworks came from ThoughtWorks and We Care a lot about our testing practices. So yes, ThoughtWorks is committed to Gauge and we really want to improve and make it easier for people to test and be good at testing the projects.

Okay Zebul, before we go, is there one piece of actionable advice you can give someone to help improve their gauge automation efforts? Let us know the best way to find contact you or find more info about Gauge.

Yeah, so one is visiting the website and installing and playing around with Gauge. But then we are very responsive and active on our GitHub channel. So if you go to our website you can find a link to our.

Gita channel, which is like our chat, and that's the best place to find help and support for Gauge. It's kind of pretty easy to install Gauge because it's single binary. You can just download the binary and unzip it and then get running with it. So that's pretty easy. But I think.

It'S best if you download our plugin for Visual Studio Code. Visual Studio Code is a great editor and Gauge.

Integrates extremely well with Visual Studio code. And if you use a JavaScript plugin, it will be quite easy for you to get started and experiment and understand the concepts of Gauge.

Thank you Sable for your Gauge Automation awesomeness. For the links to everything and value we covered in this episode. Help Head on over to testtalks.com 226 and while you're there, click on that Sign up for a free trial link under the Exclusive Sponsor section to learn all about Sauce Labs awesome products and services and to learn more about more Automation Awesomeness in 2019. Make sure to head on over to automationguild.com and sign up for the 2019 Automatic Online Automation Conference. Dedicated 100% to automation, really focusing on leveling up your automation skills in the new year, making you the hero of all your team's automation efforts and making you more employable and helping your career out. Make sure to check that out. Head on over to automationguild.com so that's it for this episode of Test Talks. I'm Joe and my mission is to help you succeed with creating automation awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Talks podcast. Head on over to www.testtalks.com for full show notes, amazing blog articles and other automation awesomeness.