Challenge Accepted: Rethinking Automation with Benjamin Bischoff

In a land of testers far and wide they journey seeking answers, seeking skills, seeking a better way through the hills they wandered through treacherous terrain but then they heard a tale of podcasts they had to obey O the Test Kill Animation Testing Podcast Guiding testers with our nation awesomeness from ancient realms to modern days they lead the way O the Tusculon Oasion Testing Podcast with sweeps and lyres the bards began their song A tune of knowledge, a melody of code through the air it spread like wildfire through the land Guiding testers, showing them secrets to behold O the chesskill Automation Testing Podcast Guiding testers with our nation awesomeness from ancient realms to modern days they lead the way oh the Testill Automation Testing Podcast.

Oh the Tescal Automation Testing Podcast with lutes and liars the bards began their song A tune of knowledge, a melody of code through the air it spread like wildfire through the land Guiding tester showing us secrets to behold.

Hey, it's Joe and welcome to another episode of the Test Guild Automation Podcast. Today we'll be talking with Benjamin all about a LinkedIn post he recently had where he made a comment where he said, when it comes to software testing and automation, I disagree with everything. Just invite me to your podcast and find out for yourself. So challenge accepted. We'll also talk about a bunch of other things. He's an API expert, a testing expert in general. Before we get into it though, if you don't know, Benjamin has spent over 15 years as a software developer and trainer. In 2016 he transitioned to test automation after being captivated by Selenium. Currently he works as a test automation engineer in Germany, focusing on back end and API testing. And Benjamin also authored the book Writing API Tests with Karate. If you haven't gotten it, we'll have a link to it in the show notes for sure. And he also maintains the open source project for cucumber BDD parallel test execution and reporting. So he really knows his stuff. He's also a renowned conference speaker and shares insights on testing automation and software craftsmanship on his website softwaretester Blog. And it's not here, but he is an awesome magician. If you ever see him at a live event. He does magic on the fly. Very cool. You want to miss this episode? Check it out. Hey Benjamin, welcome back to the Guild.

Thank you for having me back.

Awesome.

Glad to be here.

Great. Great. Yeah, it's been a little over a year now. I had you here in April last year. So what have you been up to since then? What's New?

Oh, I've been doing a lot of API testing, but not only. I mean recently in the last month I've been focusing on rewriting our internal test framework for front end tests. Even though I'm more in a backend role now. But I took the chance and a little time that I had off to do that because that was needed.

Awesome. So that's a good place to start then. As a magician though, before we get into it, do you work out new tricks all the time or is there anything new that you've been working on?

Oh, absolutely. I've been doing few little shows here and there. Last time was our internal QA meetup that we had and I always try to integrate some magic, at least magic principles into the talks. I do also the future talk.

Love it. So you're saying you're rewriting a front end test framework. Were there any surprises you found when you saw this as a complete rewrite? Is this what made you write that post? You disagree with everything you know?

Oh, oh, that. That was actually another trigger. This post.

Or multiple ones. But the test framework I rewrote because we, this had been in existence for about eight years. I wrote the one before too. And then we had a lot of learning through constant usage of it. And also technology is evolving and selenium and cucumber and all the parts that we're using. Also things like Junit and Maven had significant advancements, so it was just needed to be up to date.

So after doing this, if someone has a, if they're creating a framework now, do you have any tips on how they can make it so it's easier to upgrade? Because obviously over time you need to extend it, you need to make changes. Is there anything you could do to create, I guess, a maintainable framework over time?

Absolutely. I mean the first one was maintainable as well, otherwise it wouldn't have lasted eight years.

Exactly. Yes, yes.

I mean we could have also gone on using it. The tips I have is stick to clean code principles, read a lot about design patterns, do things like not building everything into a core framework, but also having a plugin architecture to be able to just have on the fly functionality in the framework that you need. So yeah, keep it very modular but also don't over engineer it. That's not worth it.

Absolutely. So do you recommend someone owns the framework or does like, like do you have multiple Sprint teams that are constantly checking in code? So the framework creating not crud over time, but it's like growing and growing and probably getting bloated how does that work for us?

We are a pretty big QA team. It is belonging to qa. I, along with two of my colleagues who are also automation engineers, are the core maintainers of it. So we definitely keep an eye on the features and the pull requests. So we treat it as not open source, but like internal source. So people are absolutely invited to contribute to it, but they have to pass us because it's used in multiple projects and we don't want the framework to become unstable in any way.

Nice. How do you resist the urge of just rewriting it just so you can learn, Maybe say you want to learn Playwright. So like, I'm going to rewrite the whole framework and do it all in Playwright or whatever, or WebDriver IO, whatever it may be. Like, how do you resist the urge of. A lot of times people take this as a chance. I'm going to rewrite in the latest and greatest, whatever that may be at that time.

It's funny that you mentioned that, because my latest talk that is going to premiere tonight at a meetup is exactly about that. Why it can be better choice to use old tools that you already know and not go with the shiny new things that everybody raves about or hypes about. I think for us, because we have been using those tools already for such a long time, everyone is very familiar with it. So it makes it way easier, way faster to rewrite this stuff in the same kind of technologies, but deal with all the learnings we had outside of the technology, if that makes sense.

Well, I guess you could expand on that. So, meaning, like a lot of times people use a technology and they think it doesn't do something because they don't know enough about it, and then they create something to do something that it might already do. I don't know if that makes sense, but is that what you mean by that? Learnings?

Yeah, but also things like how you use the framework. So thinking about the customers. So our QA engineers who have to write tests with it and create glue code and create components, we just wanted to make this easier for them. So this is not really dependent on the technologies we use internally, but more how we expose APIs or ways to create the tests themselves.

All right, so can you give an example, like, how did you make it easier then, based on what you learned?

So for this iteration of the framework, Selenium based framework, I got rid of all the page component factory and page object factories we had before. I mean, even Simon Stewart, the creator of Selenium, says it Might not be the best idea in any case. Also, we had pages and components completely separated, so oftentimes we had to implement the same functionality in both. So if we wanted to find an element in a page, there were special functions in the page class. If there were components that we wanted to find sub elements in, we would have to recreate this. But now everything is just one. So a page is a component, sub component is a component. So it's just you can have unlimited hierarchies and everything works exactly the same way. So just little things like that makes just much easier to implement.

Nice. So I know probably years ago Selenium 4 wasn't out, and I think Selenium 4 adds a lot of new features. That makes it a lot more modern, I guess you could say. Did you add any new functionality into your framework based on selenium 4? Was that one of the factors as well?

Yeah, it was less about Selenium because we're keeping the cutting edge version of Selenium all the time also in the old framework, but it was more about the other parts like Junit 5 is now in place and not 4 anymore. Also, we are ready for any future of Cucumber. Before we were stuck at a specific Cucumber version because we did some hacks around it and this is not needed anymore. But still, I mean, we're experimenting a lot at the moment with Selenium bidi. I know it's far from being ready and accepted, but still we're doing a lot in this direction to just be ready when it's official.

So once it is official, what do you think? If someone's just hearing webdriver bydi for the first time, what is that going to allow you to do?

This is basically allowing us to get messages from the browser directly, independently of the Chrome DevTools protocol. So at the moment we can do things like that, like getting specific logs and messages from Chromium based browsers. But Bidi will allow pretty much all the browsers to do that. So having a websocket connection right to the browser, not relying on the typical Selenium polling mechanism to get some insights into what's happening in the browser. But the browser can directly send messages back to Selenium.

Awesome. How did you refactor this without breaking anything, do you have like a process?

I also incorporated my own learnings because I found that one of the pain points of the old framework was documentation and having good examples. So I actually started with a demo project. It was like test driven development, but not On a unit test level, but on the whole framework level. So I created this demo.

How would you say, this demo test project, and then implemented the framework functionalities I wanted to showcase. And yeah, basically, I mean, the old test framework is still running in all of our applications, but in parallel. We started implementing the new one and porting tests over, so they're now running at the same time, basically. So we can compare exactly what's happening and if we reach the feature parity to the old framework and also measure things like performance and speed and resource consumption of both.

All right, I'm going to move off this topic soon, but I'm just. I think this is something everyone struggles with. So after you did the refactor, you now have two running in parallel. Is anything you learned based on an assumption you may have had before the refactor, you're like, oh, wow, that didn't do what I thought it was going to do. Or that worked better than I thought it was going to work. Or that worked worse than I thought it was going to work.

Actually worked a lot smoother than I thought. The whole rewrite. Now we're hitting some things that we hacked in the old framework that I'm currently thinking about how to do it in the new one. But in general, what surprised me the most when going from Junit 4 to 5 and all the new versions of the dependencies was the resource consumption, especially when running them in CI CD systems. It just takes way less resources, especially when running in parallel. One thing I have to mention too, because I forgot, one of the main reasons for the rewrite was also that we basically added parallel runs as an afterthought on the old framework. And now we started with parallel runs and build everything around it.

And what allows you to do that? Is it Junit 5? What allows you to do the parallel runs better?

Junit 5.

Okay.

Definitely much, much nicer for parallel runs. Also, cucumber and junit 5 work really well together, so. Yeah. And also, of course, the framework itself is now built to be thread safe, so all the components we have cannot mess each other up anymore.

Nice. So if someone's listening, they're using Junit 4. Do you recommend that they do upgrade to Junit 5 as soon as possible?

I would certainly recommend it. I mean, there's also the Junit 5, I don't know what it is called, legacy or something that allows you to run Junit 4 tests with Junit 5. This is already a good step in the right direction, but I'm actually surprised how little it is still used. I mean, a lot of people are still relying on number four.

Right, Right. They're probably afraid we don't want to touch thing that's working. But you know, who knows, that might be.

But sometimes it makes sense.

Right, Right. All right, so I want to dive into a little bit. Like I said, what caught my attention was that LinkedIn post where you said when it comes to software tests and automation, you disagree with everything. So maybe could we break that down? Why did you write that? What were you talking about? And I'm just curious to know if I agree with you or disagree with you.

If you look at LinkedIn, you see a lot of posts, especially.

In the spaces of test frameworks and test automation and general QA that are plain wrong, that are misleading. People spread old information. People compare test approaches and tools that shouldn't be compared to. So it's really, really rare that I see a LinkedIn post, test related or software related post that I say, okay, I agree with that. Especially when it comes to comparisons of Selenium versus Cypress, Selenium versus Playwright, which is not a comparison that makes any sense. Or just today I saw a post with the different types of testing where somebody created a graph of how you can test what test approaches there are. And there was manual testing and then all the different kinds of tests like performance tests and integration tests, all under manual testing. And then there was another branch, right, besides this that said test automation. So this person considered test automation as something completely separate and also called the other side manual testing, which is always something that triggers me.

Yeah, so that's something that triggers me as well. But the opposite is when people rename things. And so when you have a conversation, you pretend like you don't know what manual testing means, when everyone knows what manual testing means. But I get the gist of the concept of not calling a manual or even automation what they call it checks or verifications. But I mean, you know, it's automation testing. Why do we pretend? I don't know anyway, but I don't want to get into that. But I understand the point. How deep do you get into these LinkedIn articles? A lot of times I'll write an article and people just go by the title like Slim vs Cypress. And the first if someone read the article, the first thing would said, clearly you can't really compare Slam versus Cypress, but this is going to compare, you know, whatever the features and whatever how to decide between tools, you need to know how to decide between the tools. So you need to do some Sort of comparison. So how do you address it then? So that you are able to say, all right, it's not slim versus Cyprus, but here are some options for automation. Or here are pros and cons. Like, how would you address that?

If there are articles and I have time, I definitely read them. Otherwise I won't comment on that. I mean, there are people that I will not name on LinkedIn.

Zemika.

No, no, no, not you.

But like very well known people of testers that jump on every conversation.

Yes. Yeah, yeah.

So every conversation you will see the same. I would say two names. Both of them I blocked because of that.

Yeah.

And I don't want to be like this.

Right, right.

I don't want to just. If people spend an hour writing long posts on LinkedIn that I don't agree with, I don't want to say, no, you're wrong. Goodbye. If I write something, I try to give reasons why I think they might be wrong, but I'm always willing to learn. If people say, no, I think he meant it this way, or people give me good reasons, then of course I will change my mind.

So what's the best way for people to find out more about good automation and testing? If you think, you know, LinkedIn can be kind of a minefield. Do you have any resources that you go, okay, just go here, you're probably better off. Is it just more like, do your own poc, get your hands dirty yourself type deal?

That definitely also find people that you trust that are experts in your field. Don't try to just blindly follow people that don't know how to communicate. Read a lot. I mean, read documentation of technologies. Yeah. Follow websites like yours. Of course. Follow your podcast, definitely. There are so many good resources in terms of testing. A lot of books, there is Ministry of Testing. There are a lot of very nice communities. I'm a member of the Cucumber and Selenium communities and also a member of the Karate Slack channels. So just carefully choose who you talk with and about what.

Yeah, so maybe we can talk a little more about communication because I may know the people you're talking about. And it's funny, a lot of times people come off really poorly. They do comments, but when you speak to them in person, you're like, this person's lovely. They make sense. But all of a sudden you get them in a comment, you're like, oh, my gosh, this is horrendous. This. Why would you write this? But do you have any suggestions? You just recommend, like, not getting into it. Like when you communicate, like you said, just have an open mind. Like, any suggestions for someone? That's okay. I just want to come off better when I communicate that I don't come off so harshly when I'm leaving a comment.

Yeah, I've definitely fallen into this trap before. Just if I'm in a bad mood and I read something that triggers me, then I can snap to and write something. But if that happens and I find it out later that and think about it, then I will also apologize. But yeah, I don't know what it is with some people that maybe they just think they know it all and everyone else, I don't know everyone else doesn't. So they're like in this ivory tower looking down on everyone and saying, what are they doing? Yep.

I get in trouble with a lot of people because I like to just present information. I have a speaker, speaker speaks their truth. People listen. I want them to decide. I try not to inject my opinions on it. And sometimes people like, well, you should have, you should have said something or you should have, but it's not my personality to do so. But anyway, that's how I like to approach it. I give the information and let people decide on their own where to go.

Yeah, that's why I like this podcast.

So you did mention karate. I know Peter. Peter's awesome, but sometimes Peter could come off kind of. Peter's great, but sometimes some of his comments could be taken the wrong way. So a little bit more about karate. If people maybe haven't tried it for whatever reason. You, I think you, you. I don't know if people know Automation Guild, we have monthly sessions. You're going to be doing something on karate UI testing. I think when people hear karate, they think API testing and they don't know that has all this other functionality. So I guess maybe a quick plug about karate UI testing.

Okay, so yeah, Karate UI is a way to do UI testing using karate, but with all the benefits that karate API testing has, like mocking things or authorization and authentication. Yeah. What's extremely powerful is just mocking APIs while doing UI tests. I will show that. And finally, Peter told me that the UI testing was actually the first part that he did in karate. And basically when you look at Selenium, it is an API in the background. So it's making API calls to the web driver. So it is just API communication. So that's why it works. And it can be really, really powerful if you combine all the disciplines that karate has.

So I don't want to put labels on you, but it seems like you're a big karate advocate. Did you plug this into your work framework? Is this just something you like to work on on the side?

I did not plug it into our framework. This is always kind of hard decision what you put in a framework and whatnot. For us, we have very specific jobs for the frameworks. So we have the UI part that we use my framework for and we have API tests and TRPC and GraphQL tests that we use karate for. Because this is just better. I mean, I could potentially build an API test framework myself, but why should I when there is one of the best out there already that is free?

All right, so there's a lesson there. Because I would think most people would say, I'm creating a framework, I'm going to create my own API integration. A lot of times they don't realize, let me check out these other solutions that are out there. I mean, I used to do something called Serenity that was like a wrapper on top of a lot of things. Made things thousands times easier. When I'm not focusing on coding, I'm just focusing on using a tool to help me test.

Yeah, this is the thing. You have to know exactly what tool you use for which job. And there is this famous Maslow's hammer that when you have hammer, all your problems look like nails.

Right? Right.

Yeah. You need to be very careful. And this way we can actually have full control over our UI framework and the API stuff that we do in karate. This is also. This is so advanced already that we don't really need to change anything there. Right.

And I guess as a fallacy, everyone's always looking for the one tool that does it all and not realize maybe you can have depending on even teams. Maybe one team is better suited for Cypress, another team is better suited for Selenium or whatever you can use to get the job done. I guess sometimes.

Absolutely. Yeah. And I always tell my team, if there is anything that is better for our use case for UI tests, then let's use this. I mean, I used to be different. I used to protect my software at all costs and consider it my baby. But not anymore. It's just about the value this solution creates and this is it.

Yeah, it's hard not to fall in love with the solution though. And sometimes that becomes you. Like when you get in LinkedIn, you're like, no, this is the only way, the only tool. So good. Nonetheless, don't fall in love with tools, I guess, or solutions.

Yeah, absolutely. But still, try our tools. I mean, I tried out Playwright, really like it, tried out Cypress. I really like it. But for us this just works a little better.

Exactly. Love it. And you have good reasons for it, so that's what I like. So, you know, I've been trolling you a little bit more and one other thing I want to touch base on. I know we're almost running out of time here. I like to keep this to half hours. You wrote something about you were doing a hackathon, I think, at work and you did something with AI and you scored first place. So I have to ask you about AI. What are your thoughts on AI? What did you do with AI with your, you know, what was it?

It was basically a project to improve how we show images on our website and which images we show where. Fortunately, I cannot give you too many details there, but I like using AI tools. I'm very critical about what they can and what they can't do. I think they're very much overhyped, especially when you play a lot with different ones and you realize that they can help you if you know what you're talking about and if you know what you want to know already, then they can be a super cool rubber duck. I mean, you can give them your problem, they can give you ideas. Oftentimes they will not work, but when you give them why they not work, they give you better ideas. So I treat AI as like co developer that I have a conversation with, but I would never trust it to just generate code for me and then put it into production. We're far, far away from that.

I love that. I wonder if someone has AI. Rubberduck.com that'd be a great domain.

So yeah, I've been asking people about AI. Is it overhyped, underhyped and underhyped? And to you it seems like it's overhyped at this point, but you're not, you're not ignoring it though. Oh, it's overhyped, Forget about it.

No, no, I think it's overhyped, but it can be pretty helpful. But it's just another tool at this moment.

So if someone's listening and they're like, okay, you mentioned some tools, but do you have any AI tools you recommend people playing with to get familiar if they want to see what it can do and what it can't do.

I really like GitHub Copilot because it's also integrated into different IDEs like IntelliJ Idea and VS code. Both work really well. I mean, you just. And it learns locally from your code so it can actually suggest you things in your coding style. So this is really nice. Also I found that Gemini, the Google one is can be quite good if you prompt it well. And for more general things, I also like Microsoft Copilot, the one that is in Bing but set it to strict mode if you want to know something about coding.

Great advice. Okay Benjamin, before we go, is there one piece of actual advice you can give to someone to help them with their automation efforts? What's the best way to find or contact you?

Keep on learning. It's my advice for you. Don't fall into the trap of just following one person on LinkedIn and just taking everything they say for granted without questioning it. Find different opinions. I think that is the best way to just get your own opinion. The best way to contact me is via my website softwaretester blog. It has all the social media stuff in there so this is good starting point.

Thanks again for your automation awesomeness. The links to everything of value we covered in this episode. Head on over to testguild.com a499 and if the show has helped you in any way, why not rate it and review it in itunes? Reviews really help in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack automation awesomeness. As always, test everything and keep the good. Cheers. Hey, thanks again for listening. If you're not already part of our awesome community of 27,000 of the smartest testers, DevOps and automation professionals in the world, we'd love to have you join the fam@testguild.com and if you're in the DevOps automation software testing space or you're a test tool provider and want to offer real world value that can improve the skills or solve a problem for the Guild community, I'd love to hear from you. Head ON over to testguild.info and let's make it happen.