265: TestProject a Community Testing Platform with Mark Kardashov

Hit it.

Welcome to the Test Talks podcast, the place to go to geek out on software testing. And now your host whose mission is to help you succeed with test automation, Joe Colantonio.

Hey, it's Joe. And welcome to another episode of what in a few weeks hopefully is going to be called the Test Guild Automation and Testing Podcast. In this episode we're going to be talking with Mark Kardashev, CEO and co founder of Test Project, which is a free automation framework that's built on top of Selenium and Appium. So out of the box you get a whole bunch of features that you don't get when you're implementing Selenium and Appium APIs by yourself. You might have heard also recently that Tricentis has acquired Test Project and that is one of the things we'll talk about, as well as all the other features you get when you use Test Project. So listen up and discover what all the buzz is around Test Project as well as how Tricentis plans to make it even better. You don't want to miss this episode. Check it out. Test Talks is sponsored by the fantastic folks at Sauce Labs, the cloud based automated testing platform that eliminates the need to maintain your own Selenium grid and test infrastructure. Try it for free today. Visit testtalks.com and click on the sign up now link under the homepage Sponsor section.

Hey Mark, welcome to the show.

Hey Joe, nice to meet you. Thank you.

Today I'd like to talk about Test Project and the big Tri Census announcement that just came out. But before we get into Mark, can you just tell us a little bit more about yourself?

Yeah. So my name is Mark Kardashov and I'm CEO and co founder of Test Project. I'm working in a test testing industry for the last 10 years and I have some deep knowledge in test automation and testing in general. In 2015 we found a test project and since there I'm there.

Very cool. So Mark, there are a lot of automation solutions out there and so in 2015 what made you want to start a project and specifically around test or test automation?

Yeah, there are bunch of solutions out there and there was a bunch of solutions back there in 2015. From my experience we saw that there is a great passion about open source, especially Selenium and Appium and for those days there is no solutions that were able or allow manual testers to use those tools efficiently. Basically those tools were mainly targeted for testers who possessed developing skills and those manual testers who didn't possess those skills. Basically could not use those tools.

So Mark, at a high level then how would you explain Test Project to someone?

So on a high level, Test Project framework was created to allow mode testers and organization.

To benefit of the two major open source tools, which are Appium and Selenium, and collaborate with each other effortlessly.

As you mentioned, a lot of newer tools coming out nowadays for some reason don't use Selenium and Appium behind the covers. But the best practice or the standard seems to be Selenium and Appium and out of the box. Using Selenium or Appium like you mentioned, is for someone that's new to automation or testing can be a challenge to get up to speed really quickly. So it sounds like the solution is almost like a quick starter kit where it does all the plumbing for you behind the scenes and you can just download it and get started automating. Is that how it works?

Right, exactly. So basically Test Project is a framework that fully equipped with all the features that every test automation engineer needs basically and provides them a complete tool chain around those Appium and Selenium frameworks, from, you know, built in reports.

To distributed execution capabilities, recording capabilities, everything you need, you have it there. And it's all built around Selenium and Appium.

So Mark, how does it work? I just tried it really quickly on one of my machines. It seems like you log into one of your servers and then you download an agent and then somehow that agent is communicating with your local machine and your servers up in the cloud. Is that how like there was no install really other than installing that agent?

Right, right. So basically Test Project framework contains two major parts. One of them as you mentioned, is Test Project Agent, that is you need to install and run it locally. So basically all of your test execution or test automation done locally on your personal machine and Test project cloud, which is basically allows you to manage everything. And the test project agent is communicate with Test Project Cloud and basically provide you a complete solution. So the test project agent is already come with Selenium in Appium and all other configurations. So basically it's a basic next next installation. We made it very very simple and we basically configure and install all of the drivers and all the configurations for the test automation engineer. So basically you don't need to do anything except this next next installation.

So Microsoft, because of that like the UI interface where you would create your test if there's a new version of Selenium or Appium, because I didn't install anything, is that something that's handled by your servers. So whenever they log in, it would already be updated with the latest and greatest drivers and versions.

Yeah, exactly. So basically, as you probably know, sometimes Selenium and Appium drivers are not that stable and sometimes there are some. We have some bug fixes that should be done there before. So what we are doing, we're testing and basically using the most stable version of Selenium in Appium driver and once the new agent is released, is already.

Contains the most updated and most stable version of Selenium in Appium drivers that we actually test and see that they're stable and would not make any regressions.

Nice. Another cool feature I noticed is there's this jobs piece within the ui. Normally to run Selenium tests or Appium, you would use Jenkins and Jenkins would coordinate what machines are available and where to run. It does this because every person on your team would have to have an agent on their machine. You could then tell it to run on certain machines. Is that also a benefit of using the agent using this job feature?

Yeah. Imagine that every account basically have this huge grid of agents that can be installed everywhere around the globe. They don't need to be necessarily installed on the same physical location. So for instance, if you have a remote team that you're working with and they're part of your account, they will be part of your execution grid. So the jobs section provides you the capability to execute those test cases that you recorded on each and every of those agents around the globe. And you can even schedule those execution using the job section, the test project.

So how does it integrate something with a CICD system With Jenkins? Would. Would the job piece replace that or is that just another feature that you can use this or when you scale up, you could then just point it to Jenkins and Jenkins could handle that for you.

Yeah. So basically we have a dedicated plugin for Jenkins. So those who have a Jenkins as the CI CD orchestrator, they can use this plugin. On top of that we have an API which basically allows you to perform any action that done through the UI but by the API. So you can trigger job, you can trigger tests, you can get full set of your reports as a JSON. The Jobs monitoring screen is additional capability that teams who do not possess Jenkins or any other CI CD tool can use as a good starter.

Nice. So you did mention there's a plugin or there's an add in for Jenkins. That's another unique feature I noticed. Like I said, I just downloaded this recently. Instead of taking a look at it, you have this feature called Add ons and in the add ons it looks like the community can contribute certain functionality. So if someone's logging in, they have an access to all these other libraries that other people have worked on. For example, I had to give like a rest agent that someone could download and start using right away. Could you just tell us a little bit more, I guess about that feature?

Yeah, I think that the community approach of Test Project and the Add ons is one of the core features of Test Project. Basically it allow testers around the globe utilize capabilities that other testers shared. Basically Add ons are building blocks of test automation that can be reused by others, as you mentioned, as a REST API add on or for instance, recently someone shared image comparison add on and we have some J round add ons. So basically if you develop an add on that can be useful for someone else, you can share it with the entire community. And I'm not directly related to your account, can download and use this Add on as part of my recording scenario or even coded scenario that I create in Test Project. So as for today, we have more than 100 add ons and it's expanding.

That's really cool. So do you know how easy it is to create an add on? Is it just a matter of submitting it? Does someone ever look at it? Does it go through a security scan? How does someone contribute? I guess the question, yeah, it's a.

Question that is raised a lot. So basically the process of sharing Add on not within your account, because sharing an add on within your account is simple. Basically it's developed in Java using Test Project SDK and they're uploaded and shared to your account. But if you want to share it with the entire community, you can submit a share request and then our support team will be in touch with you. And in most cases we are reviewing the source code and do some code analysis and once we identify that the add on does not has any malicious code, we allow him to share it. So basically it's a process. The idea behind developing Add ons and Test Project is that users are developing everything within their local ide. So if they're using Visual Studio code or Eclipse, they're developing everything locally and then upload those binaries to Test Project.

So another feature that once again, it's almost going back in time. Everyone's coming out with this record and playback feature again. And this seems to be one of the main features of Test Project as well. Just talk a little bit about you. I think you call it a smart Test recorder, why you call it a smart test recorder and what that gives you and how it is different than say I know you've been in the industry 10 years but say maybe 20 years ago when Windrunner or something was promoting just record and playback.

Yeah, the record and playback are in town for a while. I agree. The reason we call our recorder as a smart test project recorder is it basically relies on the actions that we created and it also relies on the action that community contribute. So we're using some AI based techniques to identify the elements that user is interacting with and suggest the user the most appropriate add ons or actions that are part of add ons based on those elements. So our goal is to extend the number of actions that user can be can down on specific element and that what makes the recorder much smarter.

So typically I didn't get into, I know you've been around since 2015, how large is the community and how reliable have you seen the test recorder be? Do people usually like what's a flow? Do they usually start with the test recorder and then they modify the test with code themselves or do you know what your typical flow is for your typical user for a test project?

Yeah, I can share this information with you. So our community currently is more than 5,000 companies and individuals and most of them the majority of them using the recording capabilities. We also have some, we provide some SDKs and coding capabilities as well. So basically you can use test project SDK with standard Selenium and Appium APIs to migrate your existing tests or use them as well, or develop new tests. And even we provide an advanced feature of generating code based on recorded test. So for instance, if you recorded several test cases, you can generate the code from them and then upgrade and enhance them with additional code or logic that you want to put there. But most of the users are using the record and playback feature and it's working pretty well.

Does it help you? I guess. Can you write code in any SDK? What is the code generation feature that I'm seeing?

So the code generation feature basically generates your code from the recorded test to a Java and we also.

Experimenting with C. Currently there are several accounts or users that using C it will be publicly available within several weeks. So you will be able to export your recorded test to C basically one of the two languages, programming languages is allowed. And I think what makes the recorder even smarter that sometimes I'm sure you know that sometimes there are some elements which is bit tricky to treat with standard recording capabilities such as grids or date Pickers or tables. So those UI elements, what the organization who use test project are doing, basically they developing an add ons for those elements and then upload those add ons to test project and then manual testers once they use the recorder, basically the add ons and the code that is uploaded is extend the recorder capabilities and they can do some advanced actions on those UI elements, which is not primitive actions such as a click or enter text, but more advanced action that were coded inside those add ons by the test automation developers and later on used by the organization testers.

So it does sound like there's a lot of recording being done. Is there any features that helps you make it so it's more module so that you can reshare like say a login function so you don't have a thousand scripts all having their own login feature. How do you extend the record and playback to reuse existing code?

Yeah, so basically there are some capabilities of nested tests.

For instance, if you already have login flow that you created, you can use it as part of your more larger test. So basically what the users are doing, they're creating building blocks of recorded tests and then created a master test that use those building blocks with parameterization of course, and eventually it become very scalable.

Nice. So I also, I think I may have glossed over the fact that because you are in the cloud and all your users have agents, you can see what everyone's working on. I guess it's. It seems it's made for agile teams for collaboration. Can you expand on anything I may have missed? Like I said, what kind of features do you get when you log into a central portal and your whole team is using it and you're able to see all their assets.

Right. So basically you can make some aggregation within your account as well. So you can create, you can allow or add different users to different projects. But once a user is part of a specific project, he can see all of the test cases that were created there. You can see all of the agents that are publicly available. He can see all of the jobs, all the reportings or reports that you have there. He can see all of the coded tests that uploaded to this collaborative space, all of the elements and data sources that were uploaded and created. Basically you can fully collaborate.

With all matters that related to test automation.

Nice. And one of those things that you mentioned already is reports. But could you just tell us a little bit more about the reports since it is in a separate tab? What kind of reports can people generate? Does it help with debugging things like that.

Yes, of course. Basically we have a dashboard that monitors and provides you all of the reports and the executions that you did within your account from a single test through a bunch of tests Or a job. Job is basically a test suit. So you can see screenshots, you can see their error logs, you can see messages that you had. If you had a data driven test that were executed, you will see how many iterations were executed, what kind of parameters were there, if there are some parameters that were failed. For instance, you tried to perform a login action with a set of 10 different logins and three of them were failed. You will see all the failed actions separately. You will see everything there. By the way, you can also generate a PDF based on this report and download it if you want to send it to someone. Of course those reports are also available through the API. So once you have API access, you can have those reports in the JSON format. You can also use Slack or you can configure your jobs to get those reports.

To your email or your colleagues emails. For instance, if you're running a daily execution and you want a report to be sent to some specific distribution, you can add your managers, your colleagues there and by the end of execution you will have it rather on your Slack channel or on your email and all of those capabilities are coming out of the box.

Very cool. So Mark, you know, like I said, there are a lot of frameworks and when someone asks me what's the best one, I never say there is a best one, I say there's the best one for your team. So what criteria do you think would make a team a perfect fit to use Test Project? And what scenarios do you think it may not be the best fit depending on the team's needs.

So I can only tell from what I see.

We see a lot of startups and medium sized organizations that don't have any test automation framework. And by using Test Project they can start immediately, within days. Teams who don't possess developing skills, testers who want to create automation but do not possess developing skills can use Test Project to boost up their automation and to expand their test coverage. So basically our early adopters are startups and small medium teams. But surprisingly enough we also see several corporates who are using this project as well.

Very cool. So I guess the big announcement that just came out, I think it just started yesterday at the time of this recording, was the announcement that Tricentis has acquired Test Project. I failed to mention Test Project is free. Am I understanding that correctly. And if so, what does this Tricentis acquisition mean for you?

Yeah, and for us, I think it's a great news for us and for entire test automation community because Test Project basically expands Tricenti's support for the open source community and it's great to be part of a larger team with a lot of test automation frameworks and test automation solutions inside. And I think it's a great news for our community. Test Project will retain its free community based offering, which is also great and I think it will encourage a lot of testers to jump in to the test automation world, improve their test automation skills and make the coverage or test coverage within the organization.

So Mark, I guess what some people might be afraid of is since you're being acquired by a company that.

Charges for services normally and products, is this going to become a paid solution then at some point or what does it get, you know, what does Tricentis get out of this? Are they able then to sell other services that you don't have to build from scratch?

No, I think, no, as I mentioned, Test Project will retain free community based offering. I think that from my perspective Tricentis are starting to be involved more in open source community and supporting open source community. And this is the way of Tricentis to, to provide this support through Test Project and expand their support for the open source community.

Okay Mark, before we go, is there one piece of actual advice you can give someone to help improve their Test Project automation efforts and let us know the best way to find contact you or to find out more about Test Project.

Yeah. So first I welcome everyone to join Test Project Community. Create an account, it's free. Download an agent, start working. It has some great wizards inside of it. It's built in a way that it starts as a simple solution but then when you start digging you will find some advanced features inside of it. And as always we have this chat that all of the users are welcome to ask our questions there. One of the great feedback that we get back from the community that we are always available on chat. So if someone of you have any questions, don't be afraid, just start automating and we will be there to help you to success.

Thank you Mark for your automation awesomeness. For links to everything of value we covered in this episode, head on over to testtalks.com264 and while you're there make sure to click on that sign up for a free trial link under the exclusive sponsor section to learn all about Source Labs awesome products and services. So that's it for this episode of Test Talks, which will soon be called the Test Guild Automation Podcast. I'm Joe and my mission is to help you succeed with creating automation awesomeness. As always, test everything and keep to good Cheers.

Thanks for listening to the Test Talks podcast. Head on over to triple W.TestTalks.com for full show notes, amazing blog articles, and other automation awesomeness.