Karate for UI Automation with Peter Thomas

Five, four, three, two, one.

Drop that.

Welcome to the Test Guild Automation Podcast. We're all getting together to learn more about software testing and automation with your host, Joe Calantonio.

Here it is. Here it is.

Here it is.

Hey, it's Joe. And welcome to another episode of the Test Guild Automation Podcast, formerly known as Test Talks. Today we're going to be speaking all about karate. And even though most of you probably know it as an API test automation framework, we have some big news from the creator of karate, Peter Thomas. Peter Thomas will be joining us. Peter is a full stack engineer who loves building and mentoring technology teams, is an avid open source enthusiast and one of the reasons why he created karate. So listen up and discover how karate can now help you with more than just API testing, but also using UI based automation. So listen up, you don't want to miss this episode. Check it out.

The Test Guild Automation podcast is sponsored by the fantastic folks at Source Labs. Their cloud based test platform helps ensure your favorite mobile apps and websites work flawlessly on every browser, operating system and device. Get a free trial. Visit testguild.com sourcelabs and click on the exclusive sponsor section to try it for free for 14 days. Check it out.

Hey, Peter, welcome back to the show.

Hey, thanks, Joe. It's great to be back. This is awesome.

Thanks so much for joining us again. You know, you started karate, I think two years ago and it seems to be evolving constantly. I'm always seeing things on Twitter, seems like you're always working on it. So I thought it'd be great to have you back on the show to talk about everything that's happened to karate since last time we spoke. But what have you been up to since last time we did speak in episode 238, what's new?

Oh, yes, Joe, yeah. And that was awesome, by the way. And by the way, I love your intro music. It's so catchy and so a lot. And I think if I recap, and I think back to how that interview played out, I think it was quite clear that this is something I'm peculiarly passionate about, I think is the word I used. So it's been continuing. I think it's been two and a half years. This was 20. Karate started as an open source project in 2017, February. And as you noticed, and as you said, Joe, it's been evolving pretty rapidly. I'm glad that somehow a lot of things have clicked. All the ideas that I had in mind have come together. I think last time we spoke, the performance testing was new. So to summarize, Karate is a combination of plain vanilla API testing. It has mocking, which is API mocking or test doubles. And it has this, in my mind, what I call one of my best hacks. It's got this very interesting integration with Gatling as a performance test runner, where you can reuse as is your existing functional test cases can be totally reused as performance tests, which I don't think any other open source tool does, and maybe not even commercial ones. So that's where we left off. And I think what's been been happening after we connected and I was on desktops is I have been looking very seriously at UI automation. And yes, that's a big area. There's a lot of players in that space. It's really, really hard. That's something I've realized. I have a lot of interesting thoughts and experiences because coming from the world of API testing, which is actually quite cut and dried, it's very simple. You just make a request, you get response and you need to assert on JSON or XML payloads. It's quite simple. There are very few moving parts and I keep randomly lecturing people on the beauty of REST and JSON. They are very simple, basic technologies and I think that's why REST have been successful. Anyway, that's just this lecture, but suddenly me getting into this wide world of UI automation, it's been really hard. I've been extremely surprised at the amount of, you know what flakiness and instability in Test Script, which is a far cry from what I have been experiencing in the API testing world. So I have a lot to talk about that. In short. Joe. Yeah, I mean, I'm looking actually we've released some early preview version of actually attempting to do web UI autom, and that's what I'm really excited about for the future.

So why did you extend it to ui? Why couldn't you just make it the best API framework you could? Why make it even bigger? What's driving you to make this a larger framework than what you originally intended it for?

A great question, Joe.

Why? One thing is, and.

I sincerely feel that karate actually as an API testing tool, has reached a certain level of maturity. It's being used by a lot of organizations all around the world and I think, thankfully, and it's gotten, in my mind at least gotten a lot of things right, there are very few things that I would change about karate since the day that it was born and all that. What I'm trying to say, Joe, is I think karate has matured and has very few Features left as an API testing tool. It's. It's like, yeah, it's. If you look at the number of issues, for example, on GitHub, there are very few. Right. And I think that's personally, you know, a great thing. I'm very thankful that it kind of panned out that way. So what's the next thing that Karate can look at? And logically, UI automation is something I decided to go after. And of course, the other reason is, you know, me, Joe, and I think this is how karate started in the first place. You know, I look at what's out there and, you know, sometimes, you know, I don't like what I see. I get irritated. You know, I kind of take out some of my frustrations in the form of code and open source. So far it seems to be working out really well. And there's also, there's an ultimate element of ambition here. You know, why not go after. And I'm totally getting into this, fully acknowledging and fully painfully aware in a way that this is a really tough problem to solve. You have Selenium, you have a lot of these other big names in UI and functional test automation. And daring to take karate to compete with these other big names is kind of, in my mind, a brave thing to do. It is going to involve a lot of code. It is certainly going to push some of my personal capabilities to the limits. And I thought, you know what, I would love to take on this challenge.

Very cool. So what are you using to implement automation? I assume it's automating the browser. So are using Selenium. Did you integrate Selenium into Karate?

Yes, that's the obvious. That's a great question and here's the answer. I decided to re implement some of the core of Selenium. Right. And again, Selenium is a big topic, by the way, you know, shameless plug here. I actually, one of the great things that happened at the start of this year was I was accepted to Selenium Conf Tokyo and it was a great trip. And I think that talk has just been released on YouTube. So I go into detail into what makes Selenium tick. Behind the scenes, I talk. My session was pretty much about the W3C WebDriver specification, right? Which kind of recently became a thing. And what I'm getting at, Joe, is since the W3C web driver specification, which is what.

Selenium is based upon, it's now pretty much W3C standard. And in theory, what that means is. Anyone? It's not just the Selenium project, any other project, any, any one person can actually look at that spec and implement it. You know, and, and here's the other thing I love talking about, and this is also part of the answer to your previous question, Joe, as to why did I decide to go after.

Web UI automation. Because when I looked at the W3C WebDriver specification, guess what it is, it is a bunch of rest API calls. And it clicked in my mind, hey, what is Karate? Karate happens to be, at least in my mind, one of the best HTTP API clients that exists. And I thought if I just combine this, I may end up with a pretty compelling combination. So to answer your question, Joe, part of you, the answer is Karate from Scratch does a clean room implementation of the web driver spec, which means that it can now automate a browser provided you put things like the Chrome driver and the Gecko driver and those spec compliant pieces if you mix them into your stack. But here's the other part of the question, Joe. You may have heard nowadays there's a lot of talk about the Chrome DevTools protocol. There's something called Puppeteer which is all the rage, hugely active, yes. Open source project. And Karate has done this interesting thing in my opinion, which, which is it has taken that as well. So it gives you an option. You can.

Choose to automate Chrome directly using the, the Puppeteer or the Chrome Dev Tools protocol. But you can, and this is the cool part in my opinion, you can choose to switch to the Selenium world of webdriver, use that to automate browsers, and now you get cross browser compatibility and it all works over the same unified interface. I have a lot of excitement for this because I think this is unique. I don't see too many other.

Web UI automation options having this particular capability. So I look forward to seeing how that plays out.

So just for the folks that don't know about Puppeteer and the Chrome, it's kind of like Cypress IO. It sounds like Cypress IO only runs in Chrome and it gives you some extra details that you can't get with the webdriver. But it sounds like you implemented this. So it can use both that type of flavor or you can just use straight up webdriver. Am I understanding that correctly?

That is correct, Joe, yes, and absolutely right. In a way, Karate can be, if I dare to say this, a combination of Cypress, at least some of what Cypress does and what you expect from the standard Selenium world of webdriver. Yes, absolutely. It does both.

Very cool. So what would be the Benefits of someone using Puppeteer?

Great question. So.

Of course everyone knows this, that the moment you say Puppeteer and Cypress, you are Chrome only. But what does it give you in terms of benefits? This is a far tighter integration with the browser.

And the second part is this is a bi directional connection. The big difference between how Puppeteer works and how Selenium works is that the Chrome DevTools protocol works over WebSockets. So this is a continuous bi directional connection as opposed to this whole request response and then you're totally disconnected from the server kind of model, which plays out in the whole Selenium webdriver world. And this is very important because now what happens is that since you are so closely connected with the browser, you are now able to listen to things like things happening in the browser and the browser is able to actually call you back because it's a bi directional connection. This actually in theory can solve some of the big problems and test automation today with the Selenium world, which is waiting for elements to appear, waiting for animations to finish, waiting for page loads to be fully complete and the DOM to be ready and all that. The point I'm making here, Joe, is that since you have so much deep.

Introspective control over what the browser is doing, in the case of the Puppeteer or the Chrome Dev Tools protocol, that translates to an advantage in being able to write less flaky tests, in a sense, and simply being able to really make the browser do exactly what you want it to do and less layers in between, less moving parts and so on.

You did mention something about UI automation being flaky and you were kind of surprised by it. What makes UI automation flaky that you've seen like, wow, I didn't think of this, or have you implemented anything in karate to overcome that so called flakiness you've been saying?

So this is a big area and I've been really trying to think over this for a long time and I probably don't have all the answers yet. But why is.

The whole world of browser automation so flaky? And again, you can go back to another fundamental question, Joe, as to why is UI development so hard in the browser? And there are many reasons. Number one, you have so many different browsers. There is a certain history of how browser and HTML and JS technologies or JavaScript technologies have evolved. Remember the old days of the horrors of Internet Explorer compatibility? And my website works in this browser and doesn't work in the other browser and all that. So that's Part of the problem, what I'm trying to say here is that because there is so many things that you can do in a browser in terms of programming and there's a huge number of moving parts. I think that's the key here.

When you load a page in a browser, a web page, right? In theory. And of course nowadays you have all these fancy website, they have all these fancy single page application frameworks like React and whatnot. My point here is that because, and on top of this I just want to add the expectations from end users as to what they expect from a user interface are really, really high, right? People. There used to be this term in the industry like rich user experiences and all that. So all these things combined and you remember the days of Ajax was such a big thing and of course today's Ajax is like taken for granted. My point here is this because of the sheer amount of things that happen when you just do the simple action of just typing in a URL into the address bar and navigating to a page. You load JavaScript that may load other bunches of JavaScript, you may have iframes, you have things that load slowly, you may have some ads come in, some pop ups, maybe something is blocked because of your browser security restrictions. I think the third point I want to add here, besides all the history of browser compatibility because of the number of moving parts, one thing that makes the world of developing anything that works in a browser so challenging is the fact that the browsers are designed to handle many security vulnerabilities or typical ways of attacking websites. Like you've heard of cross site scripting. The most challenging one that you face if you are in the world of trying to automate anything within a browser, is the whole cross origin restriction that you have. You can't load JavaScript which is coming from a different domain from the page that you have just loaded. So I think I've said a lot here Joe, and you got a sense of probably, yes, I'm still probably trying to make sense of this weird world that I just landed in by accident in a sense. But so many things go on in the browser. The various moving parts just keep adding up. And as opposed to the very nice simple world of request response, get a JSON and validate. Automating a browser has really been fascinating and at times really frustrating thing to navigate. So I hope that helps as an answer.

Absolutely. And you did mention, you know, you love API testing, JSON and Rest. It makes it very simple. So because it did start as an API Framework and then you went to web. Did you make it so people could then leverage somehow this API functionality to help them with the UI automation? Is there like integration between the two worlds?

Yes, that's a great topic, Joe, and I have very high hopes for this aspect. I think just because of Karate's pedigree as having started out as an API testing tool, I think this is going to be super useful and super effective. To just give you one example, Joe, and something I've just proven recently, you can achieve.

Logging into your application using the API. And typically nowadays I think most web applications work this way, right? You do need to sign in somewhere, you get some token, it's persisted somehow as a cookie. Typically a cookie. And what you can now do with Karate, Karate is now again, if we really nail this UI automation and I think we have and people can try this out. By the way, right now as we speak there is a release candidate version output, people who want to try out Karate's new UI testing capabilities. So the point I'm making, Joe, is that now, because you have this amazing combination of API testing and UI testing in one unified framework, you can now make a sign in call to your SSO store or whatever using the API, which is going to be a lot faster than having to automate this typing in of a username and a password into a login screen and drop two or three cookies onto the page, which is an API operation using Karate's UI automation capability, similar to Selenium and all that. And you're straight away into your functionality. And this is just the beginning, Joe, because I have another thought which I need to really flesh out, which is you should be able to now start mocking even chunks of pages. And I am actually having a vision where you can start treating UI automation similar to some of the ways we treat API automation. I'm talking specifically about in the world of API test automation, you can use mocks to stub out third party dependencies that you have no control over. And if I were to extend the same example to UI automation testing, I think, Joe, with Karate's existing capabilities, it may be possible to actually start taking chunks of your UI application, your web application, testing only those chunks and being able to replace all the other troublesome chunks and with mocks that have been injected by Karate, and this is going to be a lot more effective. You already got hopefully the example of, hey, you can bypass your login screens, jump straight into your landing page and begin your test. And in my calculations here, that's going to save I think 20% of time just in some of the test suites that I'm seeing here in my organization. So I'm quite excited about that playing out in the future.

Very cool. And there's a common design pattern, I can't remember the name now that people used to recommend to do this, but no one seems to do it. Like I said, mocking pieces of your page and it really speeds things up. So you don't necessarily need the full blown page if you're testing a piece of functionality that has nothing really to do with that. So I think it's a great idea. How far till you see that being implemented? Is it definitely going to happen still in the works?

The API being able to call an API to sign in is done. People can do it right now because we do have the cookie handling capabilities just like you would expect in any browser API testing framework. I am pretty confident about if you're asking, you know, can you actually start stubbing out chunks of a page? I still have actually do it. All the capabilities in my mind are there in Karate. One more point I wanted to make, Joe, on this aspect is if you look for those who care, for those who are interested, if you look at how Karate actually has implemented a regression test suite and this is important and of course one of the most frustrating things in the early days when I started this journey, I'm talking about Karate has a set of web pages. It's like a kitchen sink of all kinds of horrible real Life combinations of JavaScript and slow loading elements. And I have that baked into Karate's test suite. And this is how I test all the browsers, the cross browser stuff and all that. Here's the point I'm trying to make. That particular webpage, which I serve locally, all runs out of localhost in the Karate test suite. And guess where that page is being served from? It just uses Karate's API mocking capabilities. So Karate's API mocking capability is flexible enough that it can even serve plain vanilla HTML pages and JavaScript. So we kind of eating our own dog food in a sense to build out the UI regression testing just for Karate's core. So hopefully that gives you a sense, Joe, that hey, Karate has all these cool things that happen to exist because it started out as an API testing tool and I think that has great potential for the future.

Very nice. So I'm gonna bring up something that's a pet peeve of mine. Maybe I'm not understanding correctly. You know, API tested, you know, when you're doing BDD type Syntax given when, then it makes sense sometimes to say given a request, I get this response back. But when you start doing things like that syntax with UI saying click on this. You know, a lot of times I've seen examples where I've worked on applications where an application has changed and rather than a click it becomes like a tab. And we had 2,000 tests and so if we had hard coded in the test click, we had to go and change it everywhere. I guess my whole point is the syntax. What's the deal with the syntax for UI test? Because when I look at the example on GitHub, it has clicks, it has actual implementation details in that type of BDD type syntax. Am I making any sense?

Totally. I get this a lot. This has been one of the very interesting debates that I get pulled into occasionally.

After having created Karate, which is, you know, you have this whole camp of people who are very passionate about BDD and it's a great thing certainly when it works in the sense that if you look at behavior driven development, it's in its ideal form. You write all your tests or specifications or examples before any a single line of code has been written, right? You have this team of developers and the three me goes and you work closely with your product owners and business experts and business users. And I'm very clear that karate doesn't play in that space. Karate is very clearly you should think of karate as almost like a programming language that you use to specifically address some programming related challenges in the world of test automation. This has been hard to be very honest and sometimes very frustrating because people have this preconceived notion about what BDD should be that they take this one look at Karate, they see given when, then and then they see code on the right hand side, right will be typically JSON. And that's you just mentioned, Joe. You see click and you're doing some very low level operations and it's going to get a lot more interesting in the whole UI space, I'm sure. And people. And this tends to be a great source of concern for people who have come in from a pure DDD mindset. And it takes time to overcome it sometimes takes me a long time to convince and finally make such folks see that karate is not at all trying to be a BDD tool. And now it does have the option of you using Given when, then and but and all that. And it's purely a choice, Joe, I in fact recommend that, hey, when it doesn't make sense, don't use these keywords. Fortunately, the Gherkin syntax on which of course cucumber and BDD tools are based. Most of them are based. Fortunately, it has an option where you can use an asterisk, the star symbol, if you will, to just stub out, you know, that given when, then where you expect to normally put those keywords. And then you can just move on with your test and just focus on actually getting some work done other than worry about hey, should this line have been a given or when? And really, you know, this takes me off because this tends to be a great source of debate and I think a lot of teams waste time worrying about that. And my sincere advice to such folks is please don't.

Yeah, so my greatest fear is people misusing this and not getting reuse. So is there any concept of a page object type deal with the your KARATE implementation of UI automation?

Kind of, Joe. And yes, this could be controversial and because I think I'm known for causing controversy in test automation circles and I'm fine with that. Here's the thing, and I invite people to actually look at the KARATE documentation. You may need to go to the developer.

Branch depending on when you are listening to this, but.

There'S a keyword I want you to look up, which is called locator lookup. And man, I mean, I have a whole lot to say about the page object model, but in short, Joe, and no surprises here, I'm not a big fan of that approach. I personally think it's over engineering and maybe it's a lot of code duplication in a sense. And let me think this through, this is going to be interesting to unpack.

So, Joe, here are the few things I want to talk about. Test automation in general. Right. And while I address your point about the page object model, to me, a true sense of worth of any testing framework or any framework is that it should allow you as a user to achieve the work, to achieve the work that you want to do in as few lines of code or typing of characters as possible. Right? And I believe KARATE is far ahead and it's proved that in API test automation that, you know, with a minimum of syntax and pretty much what we try to do in API testing and now hopefully with UI automation testing, is that you should be able to achieve what you want to do in one line of code. Most of the time you should have a syntax. And that's why the term domain specific language comes about. Right. You know, you have these keywords which are very carefully crafted to do a lot of things that would, would take you maybe Maybe many lines of code in a traditional programming language, but it should be just a one liner in crowding. I feel, and I've seen this on many, you know, homegrown test automation frameworks that, you know, people get a little obsessed with reuse and they invariably start splitting out bits of functionality into reusable files. And here's the thing I think not many people realize. And keep in mind, we are talking about test automation, we're not really talking about production code that's really going to be the heart of your application that users use and all that.

There is actually.

Downside to reuse. And yes, I'm going to say this, and again, coming from, you know, the world of programming, this is probably a very controversial statement to make that, hey, reuse is kind of bad. But the downside to reuse, Joe, is that suddenly you have this extra file that you need to go to and you need to switch between two files to actually do the work.

Right?

And I see this happen kind of with the page object model pattern. So can you avoid that? And I would strongly advise folks to actually avoid it if possible. Anyway, my point here is this. If you look at karate syntax, first of all, as I said earlier, we're trying to make it easy to do the work that you need to do, click input text, use a mouse, whatever, in one line of code. And now the other part of the problem is you have to deal with the whole science in a sense of locators, right? You know, people talk about css locators or XPath or, you know, by text or whatever. And this is where, you know, I've tried to introduce a concept which you can search for in the KARATE documentation called locator lookup. Locators are going to be reused, there's no question about that. So what I recommend if for people who use karate is one, I mean, there is no harm in doing a little bit of repetition, you know, because if you are going to click on that button and you need to click on that button in two different test scripts, in my mind, because it's a one liner repeating that locator is fine in two different tests, right? Yes. The downside is that you may need to, if that locator changes in the future, which obviously is what people worry about, and flaky tests become a whole discussion point. Yes. If that locator happens to change, you will need to change it in two different files. And this is where I've introduced the concept of a locator lookup, where karate has this approach which is still to be proven, I admit. Where you can define this 1 JSON file, which you can think of, Joe, as a map of all the navigation possibilities in your application. I think it's a neat concept. You'll have this one file, it can be nested in a sense, so you can represent.

Your application page hierarchy if that's the way you want to model it. Finally, that is a single source of truth. And we have this approach where all your different test scripts can reuse locators from that one file, which gives you that advantage that maybe you were looking for in the first place, that if your locator changed, you can, you need to change it only in one place. So there's a lot more I can talk about this, but I think you get the gist. I'm not a big fan of the page object model. I think there could be better ways. I certainly think over reuse is kind of a problem that plagues the test automations community in general. And I'm trying to fight a little battle there. Time will tell, Joe. Maybe if I'm successful or not.

So Peter, you just pick up a great point for some reason. You're a lovely guy, I've spoken to you many times. But on social media, sometimes you look at Twitter and it's like a big bonfire for some reason.

Yes, I guess that's good.

I guess we're passionate about automation. But why do you think that is? What's the deal?

Yes, well, partly it's because I do. I think part of the problem is I tend to be, which people have noticed, I tend to be extremely passionate about karate. You know, I take criticism of it very seriously and I'm always trying to do make it better, improve it and all that. I think part of the other reason Joe is very unfortunate is that Twitter happens to be a very terrible medium for having constructive debate. Right. It's. There's so much you can express in. I know how much of the 240 characters or whatever and so much of the nuance, so much of misinterpretation happens that things do tend to spiral out of control. Control. And as you have seen, probably, you know, bonfires result. It's not pretty. You know, I don't suddenly take it well and I wish I could have framed my argument better and all that, but it's very easy for multiple folks to come in and the moment you have all these inflamed passions and all that, it just tends to go in a different direction. Which is unfortunate because I feel coming from a developer background. There's a lot of things I would love to share and try to influence the testing community. And I will have to say this, Joe, in response to this conversation topic, I strongly feel there's a whole lot of blind trust in some of the automation, you know, thought leaders, which you know, and I've at least I've personally experienced that if I just dare to try to say something that may change what is a popular belief or contemporary thinking, I tend to be attacked, you know, very brutally in some cases. I'm kind of fine with that, but hopefully I've given you a slight flavor of the way I'm thinking about it. But yeah, Twitter is to blame. Joe, Twitter is terrible for discussions. I would far prefer it, and maybe we can do it one of these days. You could probably host a roundtable and we can kind of duke it out in a debate format, which would be a lot more meaningful in my opinion.

Okay Peter, before we go, is there one piece of actionable advice you can give someone to help improve their karate test automation efforts and let us know the best way to find contact you or learn more about karate.

One of the best ways, Joe, is to look at my Twitter feed or the Karate DSL official Twitter handle, right? So it should be easy for you to find the official karate Twitter account. And if you just look at the latest few tweets or posting from this Twitter account, you would see very clearly one of the posts with clear instructions, even with a video as to how you can actually do what I just talked about, which is download this one zip file, install it. It's not even install, it's just a decompress into some folder of your choice. And you can run literally, and I have video proof of this, you can literally run your first web browser test in less than a minute. So yes, look for that.

Thanks Peter for your Karate Automation awesomeness. Thanks to everything of value we covered in this episode and head on over to test Guild.com alike Automation A273 and while you're there, make sure to click on that Try for Free Today link under the exclusive Sponsor link to learn all about Sauce Lab's awesome products and services. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe and my mission is to help you succeed with creating ads and automation awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Automation Podcast. Head on over to testguild.com for full show notes, amazing blog articles and online testing conferences. Make sure to subscribe to join the guild and continue your testing journey. This has been a Joe Calantonio production.