Unifying Manual and Automated Testing Efforts with Javier Alejandro

Foreign.

Get ready to discover the most actionable end to end automation advice for some of the smartest testers on the planet. Hey, I'm Joe Calantonio, host of the Test Guild Automation Podcast and my goal is to help you succeed with creating automation awesomeness.

Hey it's Joe and welcome to another episode of the Test Guild Automation Podcast. Today we're going to explore the synergy between manual and automated testing with expert Javier, who is the CEO of Crowder and founder of Lippia IO. In this episode we discuss understanding manual processes along with automation to ensure both testing methods align for consistent functionality. Javier also introduces an open source framework using Gherkin to test APIs, emphasizing Gherkin's role in and behavior driven development. We also touch on the importance of syncing both manual and automated changes and dive into Lippia, which is a platform enhancing test management system. And we also go over the benefits of integrating both manual and automated tests on one platform using a unifying cucumber implementation. And Javier highlights the broader role of Gherkin involving not just testers but also product owners and developers. So tune in for insights on harmonizing your tests and for better efficiencies and quality. And a little more about Javier. He is dedicated to continuous growth in IT consulting, software quality and test automation with high quality standards and has more than 15 years of experience in technology applied to business. He is also the co creator of Crowder which he created in 2013 after a series of successful projects in Argentina and has expanded its operations to Manchester UK developing the first ever Crowder integrated BDD automation testing framework for the cloud. Really a lot of stuff in this episode. Jam packed, you don't want to miss it. Check it out. Hey Javier, welcome to the Guild.

Hi. Hello Joe. Nice to be here. Thank you for inviting me.

Great to have you. Really excited to talk to you about this topic. Before we get into it though, I'm always curious to know when I talk to founders why they got into test automation or why they created a tool to help people with test automation or testing in general.

Okay, yeah, well briefly I started in the QA world as a CTO before in Thompson Reuters I used to have a development team without any QAs in my team. So I decided to bring some QA people and make the things work better in the team. So that was a very good experience for me. That makes me think for the future to improve that team and then with the time we we grow that team with more automation tools and different things. And after my experience in at Thompson Reuters I. I decided to found Crowther and I choose the QA team because it's a matter of some needs in the market that needs QA processes and many teams in the development area and even in our customers that has internal teams need that process. So we started working on that in that area and was funny because at the beginning we didn't start it as a normal QA services company. We started just from the beginning doing automation, which is kind of a strange thing, but we started in that way directly in automation, which is not the usual, I think kind of a startup. But that's in general my how is enter into the QA world.

Nice. So yeah, like you mentioned, it's not normal to start with automation. So why did you start with automation then? Was that like one of the biggest market needs you saw at the time?

No, I think that's because of, you know, when you just start, you pick the first project that you can find. Right, that's true. And at that time, what's a project that needs to accomplish? It was maybe a muscle kind of project. So you. We need to put effort to complete test suite that a development team has a commitment to accomplish in a period of six months, have to reach. I remember they have to reach a thousand automated test cases in six months. And the team that they have was pretty small at that time. So we added effort to that team to accomplish that milestone that they have committed with the corporation.

Nice. So I would think if you started with a lot of automated tests over time you'd have to have processes that couldn't be automated. And then all of a sudden you have more manual processes and more automated. How do you kind of bridge that gap? Did you find any gaps between. Okay, now we have to bring in information from our manual testing efforts as well.

Yeah, I think that. Well, I have two different topics about that. One is, as we start doing automation, we don't know too much about the manual process that this customer follows. Right. But then when we started to do more and more automation projects because suddenly the customers request us just do automation, they bring the manual process and we execute the automation process. So sometimes we need to match the effort and to correlate the activities to make sure that the same things that the manual QA team executing in normal test cycles, we automate the same or at least in accordance which test you want to automate and make sure that the functionality keep the same. And one of the challenges that we find is as we use open source frameworks like Selenium, Cucumber, et Cetera, we support the test cases into gherkin format and sometimes the customers use a regular tool in a step by step basic kind of format. So the main challenge with that is to match and to maintain the correlation between the scenarios that they are supported in one format for the manual process and in the automation process. So that correlation was critical because as soon as we change something, the manual testing has to do the same and vice versa. Right. As soon as a manual tester add a new scenario with their test cases, we need to change that automation. Obviously, with good communication that is possible and no problem about that, but we find that difficult to maintain that correlation clean and smoothly. So yeah, then we started to experiment some things.

Nice. So just in case, I know this is really tester focused, but if anyone's listening, they don't know what gherkin even is like. What is Gherkin?

Well, Gherkin is a language that supports or was invented to describe behaviors. Basically it's not for testing in particular, it's for supporting a specific kind of way of building software called BDD Behavior Driven Development. So the Gherkin format basically is a kind of natural language with specific keywords that you use to describe behavior of your system. So basically it cuts for five keywords like given, when, then, and then logical connectors like and, or, etc. But it's pretty straightforward. So given a condition, then when I do some interaction with the system, then I expect this behavior. So with that format you can describe the behavior of a system or a software and then test that accordingly. So for us, it's basically a good common language that all the development team can use, not just the testers, but also the product owners, the developers, the architect testers, et cetera. So with that single description of the software, you can make that everyone talk about the same thing. Basically one of the issues that I mentioned is that when you describe scenarios that you have the testing in a specific tool, you have to describe in a certain way or in the way that the tester wants to describe. Right. So that is a matter of interpretation. So I think that the gherkin language reduce the level of interpretation, at least leverage the same kind of writing the things. Right. Obviously you can write gherkins with specific, with more detail, with less detail. It's not an exact science, it's not mathematical, but it's a good way to approach a common language for the whole team.

Great. It sounds like. So is that, what do you mean when you unify or you get the correlated activities is everyone is using the same terminology, regardless if it's a manual activity or manual tests or automated tests. So you know what's going on with both efforts?

Yeah, that's one thing. But also using Gherkin, as I mentioned, include the whole team also the developers, the product owners, the analysts, et cetera. So as soon as you start using Gherkin or BDD to support your requirements in behavior, you can basically easily code that as a product, then you can easily test that. And as soon as you start testing that, as we start in the opposite way, as I mentioned, doing automation instead of doing manual testing, what we learned is that our team, when they need to find a bug or a flaky test in an automation project with Gherkin, the scenarios, they use the same Gherkin scenario to test it manually. So when they find the bug, for example in a login scenario, they okay, I find a bug here, let's replicate the behavior manually to understand where is the bug and then correct or report the bug or correct the flaky test into the code. And when we realize that, we realized that we can use the description of the scenarios that we did in Gherkin to do manual testing, which is not a common behavior basically for the industry. So that's why we started to say, okay, let's start writing Gherkin from the very beginning in any testing process, no matter if we will automate that later or not. And with that mindset we started doing that and we reduced a lot therefore, to maintain that synchronization that I mentioned at the beginning. And also we have some barriers, right, because the manifesters usually are not accustomed or easily eager to use a development environment or a repository of code. So that is what's barriers that makes the life of the manifesters that has to use Skerkin not so easy. So that's why we start building a tool. Basically originally that tool was for us to do our delivery in the customers, and we start doing that with that approach, how we can make the life more easy of our testers to maintain that single asset as a test. Scenarios written in Gherkin use it for the automation people or developers and for the manual testers as well.

So I know it's a hard time getting the testers use the repository, but is it hard to get the developers and business analysts to actually contribute to the Gherkin effort? Like, how do you know? Is it like you agreed on, this is the terms we're going to use. So if someone writes a manual test using these terms, it also could be used for automation or is it, is this something different just for communication, the way you're using?

Kirk No, I think it's a good tool and as soon as the development team and product owners start to use it, it makes the life more easy for them to specify requirements to code requirements. Because you can use it as a test for your code very early in the process. Not just TDD at the class level testing, but you can use BDD as a functional journey level, so the developer can use it to test their development. So in some cases, some teams start writing the test in Gherkin format from the very beginning before rewriting the code. So it's a TDD approach, but based on bdd. Right, right, right.

So you mentioned repositories. So you said then you created a tool to help with the repository. So maybe you can expand on that. What does that mean? And just for folks that are listening to this, definitely check out the show notes where we'll have the embedded video for you there as well.

Yeah, basically as soon as we started to do this process, what we did, we started from the end of the story, as I mentioned, instead of starting from a tool that lets you support your test scenarios and then automate that in the tool that you choose, we do the opposite. As soon as we have our code repository, we can start writing Gherkin from the very beginning using the same repository that the developers use. So in this case I'm showing a little bit a sample of Libyan where I have different projects and the first things that I have to connect to use Lippian Test Manager, which is.

The tool that we are talking about. I have to connect a code repository and it's very easy to use because I can have my repository choose the management tool and put my credentials. That is pretty straightforward choosing the settings in any repository. And as soon as I have these credentials in place, I can see different things in my screen. One of the things that I can see is all the runs that I can do with Gherkin files. Right. I can build a new manual run and put some Gherkin files on it, which is the most difficult part. In general, usually when you use a test management solution, you support your test cases into the application and you depend on that application to build your run, to report your bugs and to report the progress of that run. In this case, what we do is to connect to the code repository, collect all the features files that are included in that repository, and with a single pickup. I can see, I'm showing right now in my screen, how can I see a Gherkin file as a test case, right? So what we simply did is to put the Gherkin files as scenarios, to use it as normal test case and use it as a guidance for the tester to execute those actions into the application and the test. And simply you can report a pass or failed status into the solution, a blockade, a retest, you can specify different status for each test. And in the same way that I did this, I can also include any automated test that I run from my automation code. So using an API that is part of the solution, any developer that do their test using Gherkin can inject results into the same platform and combine in a dashboard or in a very straightforward way, how was the results of my automated runs and my manual runs. So I can combine both things in one solution. So both teams. If I have the task divided between manual testers and automated testers, or simply I have a kind of hybrid tester that can code and make automation.

He or she can have both the results in a single platform using the same format. And the good thing is despite that, is that we are not a lock in approach in terms of for the people that wants to use Libya because they still keep their test in their repositories. So you can edit any Gherkin file from the repositories without using this tool. But if you use the tool, you have the advantage of making the life easy for the testers.

And also just a quick thing because you're using Gherkin then is that why you can. You could use it in any programming language because it's just a text file, so you don't have to worry about Python.

Yeah, yes, yeah. You have many implementations because we are based on Cucumber, which is the most widely used platform for Gherkin. And Cucumber has many implementations From Python, Java, JavaScript, even products that are very popular like Cypress right now supports Gherkin files. Carate is another platform or frameworks that use BDD as a platform. And Gherkin files obviously most of them are standards. Karate has an additional things that add to the Gherkin files mix some specific features. But in general terms any feature file including Behave from php, which is one of our customers use extensively PHP and they have a lot of scenarios written in Behave can be collected and edited directly into the tool.

One other quick question I'm thinking of when I'm seeing this is I used to work for a huge corporation, we had divisions in that division. We'd have multiple teams working on different products and so you wouldn't necessarily know what team B was doing if you were in team A. So it looks like this because if someone's using Cypress, if another team was using Java, another team is using JavaScript, if they all used a cucumber implementation for that, across the different platform, different tools, or the different solutions that you're doing within your business division, say like a senior engineer can go in or a senior manager could go in and then see what the effort is across all those different programs, all within this one dash. Am I thinking this correctly as well as one of the benefits?

Yes. Yeah, absolutely. You don't have to use the same architecture solution to automate to use this platform. Any platform at this stage that can support VDD Gherkin can be connected to this. So that's. Yeah, we are very fond of using Gherkin. Some of, there are some, obviously some, some taste on that some people don't like too much, but we think that it's a good solution and based on that we built this. Yes. And just to quick look about that as soon as you as you can connect to a repository, you can edit directly in the same way that a developer with the difference that you don't touch real code, Java or Cypress or JavaScript, whatever, you just focus on the behavior, which is what we try to approach with the solution.

And because once again it's in a format everyone understands, developers and testers are able to communicate better as well. So if a developer creates a test, the tester knows what they did to test whatever they tested because it's in a readable format.

Yeah, yeah. In an ideal scenario and a developer can start writing the test and then the tester could add more different scenarios to the same basic test probably. So it's a matter of collaboration, writing and improving those scenarios with it's not just responsibility for the tester, it's something that the whole team has to be understanding. And also we always say with our customers that the quality is a team responsibility. It's not the QA responsibility. So we try to encourage that.

Nice. So I think in the pre show you may have mentioned that, you know, Lippia has been in development for two years. So I'm just curious to know when you started and where you are now. What are some insights or any insights into maybe the journey into building a tool or lessons learned along the way?

Yeah, well some important things is that we as a QA services company, we are not sitting all the time in the seats of the developers and we understand with this completely how the developers feel because we need to build a tool. Right. And it happens at the very beginning that we have some issues in the same that the customers we helped at the beginning. We don't have good QA process at the beginning. So we have to learn how to build a software. Because you know, when you focus too much on something, probably you lose perspective of the other things. So we learn a lot about that. And the other thing that we learned about that integration process of testing a tool that is for testers, you know, it's important to improve the process of building the solution, acquiring modern practices. We are very fond of DevOps practices and agile practices and we only pass for all the problems of bad implementation of those practices. So we learn on how to improve our product development process. Right. Which is not the normal things that we did as a service provider. So we have to learn how to build software products being testers, basically.

Absolutely. So I assume that one thing you probably learned also is it's tough when you start with the tool and you're vendor locked with it. And it seems like that is a feature. You mentioned it briefly how if someone started using this solution, they're not locked to this, it's not vendor locked per se. If they created the testing gherkin, those gherkin can then live on after if they had to move over to another system. Did I understand that correctly?

Yes, that's correct. Is that correct? Because one of the things that happened being a small company is sometimes the customers, when you show the solution, they are, they like the solution, but they are not so sure that if you can keep the support of the platform for many years and they don't, they have the risk of taking the decision of buy a solution like this and then depend on a company for many years or for many times. So that is on purpose. So we try to do that on purpose. Obviously there are some information that remains in the solution, like the results of the runs, et cetera. But your asset, which is the main asset because the runs are volatile, right? You make a run a cycle or a test cycle and then you can repeat that many times as you need. But we said that the scenarios and the guardian files are an assets like code, right? So your application is part of your application. So you have to be ownership or have ownership of that we cannot keep on our solution.

That's great. I know that's one thing a lot of people is a stumbling block. So I really appreciate you building that in. That's a great benefit for sure. So you know, it's been in development for two years. But I know you also have customers, so I'm just curious to know, do you have any like success stories or feedback from your customers on what they've used it for?

Yeah, we have a couple of customers that are using right now are five to six customers that we start using early as a beta testers. And with that approach we improve the features of the products because they are real testers using the product. One of the main customers that use is a bank that in Argentina that we provide services for them for like five years now. And we started doing automation for a CRM product for dynamic CRM from Microsoft, which is probably not the normal path. Usually you start building a new application with microservices and start doing automation. In this case they have this big installation of 3,000 users in the whole bank and they want to automate because the process of setting up paths for resolution of claims in the bank are very complex and if they don't automate that, every time that they make some changes, they break some rules. So we started with that customer and at the very beginning when we don't have the tool, we used to read a repository and run do that process that I described below.

Right.

And as soon as we have the tool we offer to the bank and we start using that and showing the customer how it works that process. And despite that the tool is not well known as more commercial products. They're very happy with that and they planning to expand the usage of the tool or Flippia for the whole team. They are a hundred testers in the bank and not just the people that work with us, it's for the other vendors that provides people to the bank, even internal employees of the bank that do testing. So very happy of that because it was a learning process. Obviously they have the openness to try the solution before it's ready to the market because they start to using the solution a year ago more or less and based on the strict regulations that the banks has to comply, we have an on premise installation in that case specific for the bank. It's not a cloud installation, but it's the same architecture based on Kubernetes and using all the same infrastructure, but internally into the bank's cloud. Basically.

Awesome. So how would you explain what Lithia is right now? Like is it a test management solution as a test tool, is it a collaboration platform? Like what is it?

Well, Libya for me is a. Or at least for us is defined as a test management solution, but a different test management solution. It's a test management solution based on Gherkin.

Nice. So you know, I know it's, you know, it's somewhat new, but I know you're always working on new things, new features. Do you see any new features and enhancements that you're currently working on that are going to make it even more powerful?

Yeah, well, basically we have a very big roadmap to improve. Obviously we want to make it better as the time passes. Basically one is the usability. Usability is key, especially for a tool like this where it has to be agile, has to be easy to probably the concept will remain the same, but giving more accessibility, easy to use for the testers. The other things that we also write an open source framework that let the users write gherkin test for APIs without writing code. So basically what we did is to add more terms or more words to use Gherkin as API testing without writing code, without writing Java or without writing the implementation of each step. So with that tool, basically what we expect is to avoid much more code into the automation process and make the life more easy for the testing that is another tool or another feature that we try to incorporate. And eventually we will planning to make more smoothly the integration between CI CD Tools and Lipia. Basically what we do is to build some adapters to add more frameworks that makes the life for the developers easy to inject results. Right now we are building well, we have Cucumber, we have Java, we have Karate, we are adding Cypress adapters and the idea is to make the Libya has an API but to avoid dealing with the API. We are trying to build adapters that makes EC2 automation frameworks to ingest data.

So it sounds like it's extensible if someone can someone use that API or is it something they'd say hey, we need this integration, can you do it for us? Or like how does that work?

No, it's open the API basically to make. To avoid the people dealing with the API.

Exactly.

We build the adapter so we have the adapter you can download from GitHub and start using. But obviously new frameworks appears all the time. That's why the API is open to avoid.

That's a good feature. So it can grow as new tooling comes out, it's easy to copy and.

Also we are open to have more people writing adapters to inject results. Yeah.

Awesome. So did you say that what was the name of the open source tool? Is that available now or is that something you're working on for the API?

No, yeah, well originally as I mentioned, Libya started as a framework, but in fact it's not a new framework, it's a combination of frameworks, open source. So include if you want to test web application. It's a framework that has Selenium, Cucumber and among other things. Right now what we did is with the API version of that framework, which is basically based on Rest assure. What we did is to overcharge the Gherkin file to have some specific steps, standard steps that you can use to test API that is available in GitHub under CrowdApp Repository. So there are some to do testing with web API and mobile and this low code version of API testing.

Awesome. So if someone's listening to this, they're like, oh this sounds great. How easy is it to get started? What do they need to do to check it out or give it a spin themselves?

No, basically to use or to try. Leap Year is just very easy to go to leapia IO and you can request a demo for a trial period and we can set up an instance very quickly.

Awesome caveat. Before we go, is there one piece of actual advice you can give to someone to help them with their BDD automation testing efforts? And what's the best way to find, contact or learn more about Lippia?

Well, basically to know about Lippia is the page that I mentioned Lipia IO with two Ps Lippia. The advice for PDD is I think that sometimes it's not easy to start using because it seems like it's an overhead in terms of effort to to use it as a platform, as a solution. But I think that it's good to see the wider picture, not just seen as a solution to the automation, but it's a process to do better development and to describe behavior. And with that in mind it's better to have that approach in terms of how your software will be documented and tested and developed. So I think it's a good combination of of a tool for the whole team, not just for the testers.

Thanks again for your automation awesomeness. The links to everything of value we covered in this episode. Head on over to testguild.com a471 and if the show has helped you in any way, why not rate it and review it in itunes? Reviews really help in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack automation awesomeness. As always, test everything and keep the good. Cheers. Hey, thanks again for listening. If you're not already part of our awesome community of 27,000 of the smartest testers, DevOps and automation professionals in the world, we'd love to have you join the fam@test guilds.com and if you're in the DevOps automation software testing space, or you're a test tool provider and want to offer real world value that can improve the skills or solve a problem for the guild community, I'd love to hear from you. Head ON over to testguild.info and let's make it happen.