222: AI-powered Test Automation Dashboard with Dzmitry Humianiuk

Hit it.

Welcome to the Test Talks podcast, the place to go to geek out on software testing. And now your host whose mission is to help you succeed with test automation, Joe Colantonio.

Hey, it's Joe. And welcome to another episode of Test Talks. Today we're going to be test talking with Dimitri, who's a product manager for a really cool application called Reporter Portal IO, which is AI powered test automation dashboarding that I think you need to check out. It's open source and it works for pretty much every type of test automation tool out there. Out of the box allows you to do a whole bunch of things, especially being able to quickly analyze your test run results using a machine learning and algorithm to really bubble up real issues with your test runs. But that's just one of many features we cover in this episode, so listen up to learn all about AI based test automation dashboarding awesomeness. Check it out. Test Talks is sponsored by the fantastic folks at Sauce Labs, the cloud based automated testing platform that eliminates the need to maintain your own selenium grid and test infrastructure. Try it for free today. Visit testtalks.com and click on the sign up now link under the homepage sponsor section.

Hey Dimitri, welcome to Test Talks. Today I'd like to talk about Test automation dashboarding as well as Report Portal IO, but before we get into it, could you just tell us a bit more about yourself?

Yeah, sure. So I was 11 years within the IT and I actually software developer and just like after being back end developer I also had an experience with front end development and at the end of it I find myself helping inside the big test automation project which had something about 100 of thousand test cases. And based on this experience I saw a lot of gaps within the test automation and I start to bring the experience from the development into the test automation with all the best practices and all this stuff. So currently I work in the EPAM system which is in Minsk, Belarus, small country in cis region. Yeah, 11 years in it.

Awesome. I love your background and I love how you were a developer that got into automation. I'm just curious to know, is that one of the reasons what made you want to start Report Portal? Like what were the reasons for focusing your energy on an AI based dashboard?

Yeah, it actually was one of the reasons why I've started with it, because all the stuff, all the things I was fixing inside this huge project, all this amount of code and overall the process around it, at the end of the day I found that I need something to collect all this huge amount of the results. This was the idea which I was looking to implement to fix at this moment one of our department leaders find out me and propose me. Okay, let's we have the same things, the same ideas, let's just start and implement it. So this was the point like five years ago we started it with the idea to implement something which will be pretty robust and let's say the standard for all test automation results within our company. And at that moment we call that as solution accelerator which actually helps inside the projects to start the test automation with like good fancy and valuable reporting. And just like after the three and a half years we find out that well the thing is pretty good, the thing is helping to our projects. They also clients start to be interested in our solution. And then we decided guys, we need to go on open source with that. Because the open source and all this community which is currently growing around the report portal is the unlimited source of the ideas for our solution. And just like one and a half year ago we have been open sourced and start our big road within the open source with the community around the world. And based on the latest results of our downloads, number of users in Slack chat, number of the feature requests on the GitHub we are really like maybe successful in this case. Community is interested in our solution and we growing pretty fast. So even sometimes it start to became a problem because too many requests, too many ideas that we just have not enough capacity to to cover all the needs, all the ideas from the community.

Awesome. So my company just implemented a Kibana elasticsearch stack for dashboarding and when I saw this I was like oh my gosh, out of the box. It was really easy to install and really easy to get started with. So how did you determine what features to bake into it at first?

Well, you're completely right from one point of view. It can be replaced with Elcastack plus Grafana, all these dashboards, but still the team who will use it have to be pretty expertise in this stack. And with the report portal you got all those benefits just like a magic in one click, let's say everything out of the box. I would say that the key benefit what we have inside of it is the real time reporting. And the key for that is for example if you have pretty huge test automation scope and you have to re execute a lot of test cases, there is no need to wait until the full execution will be finalized. For your test cases it's like eight, six, ten hours. Just like you will start to see results within the report portal in A few seconds after the execution have been started. And just like.

Sometimes appears the situation when the test cases start to fail because of different and various problems. And in usual case you will find out these problems only next day. But with the report portal you just can check in a few seconds and make it click and fix and re execute it once again.

First of all, the real time reporting, then the idea that the report portal can be used as a single entry point for all the test automation and testing results within your project. And basically you can use it as a single point of decision for your product, let's say or build go or not go in the pipeline. So basically report portal will collect the results from different sources, aggregate those results, analyze them and via the API you can receive back the status. Is it like build past or no, how many critical or product issues you have inside your build and so on. Then comes the next point which I mentioned, which called analysis. And this is what the function which we started maybe like four or three years ago. And initially it was planned to help engineers to analyze the results which appears within the latest execution. So basically that means that if you have initially analyzed your fails and just like execute the test automation scope the next day and some of your test cases failing because of the same reason which was yesterday. This feature helps you to like by copying all this related analysis from the previous run. And in case if you have assigned a linked issue in JIRA or any other backtracking system, you will see that the bug already exists. It also can show you for example for Jira, is it open or closed? You also will see all the comments. Also you'll see the history line of changing status from past to failed. And with the latest version, currently it's four plus versions, we actually introduced the machine learning algorithms which help you to really be confident about the status of your fails, status of your test cases. So basically we find out that machine learning.

Is exactly the case for ML and basically implemented the algorithms which utilize all the historical data we have in our database for your project and basically analyze your latest execution proposing or let's say marking up is it like product bug, is it automation bug? Or let's say have suggestion it can be a product bug, just like checked here or here and as a result a different project we did collect Some, let's say KPIs or metrics. And for some projects we found out that this feature reduced the analysis of the fails up to 90% of the time which team previously spent for reporting and for result analysis. Well, having all these Statistics in our database as an outcome for the managers. We have all the dashboards, widgets which represent you, the main KPI and metrics for the test automation. So basically initially the report portal started as a tool by the automation engineers for the automation engineers. And once again since we had all the statistics in database, we decided to add more functionality for managers. And then all these dashboards with widgets and metrics appeared and currently pretty big list of the projects start to use the report portal as a single dashboard for the result in testing, especially in test automation and even more within our company. Since we are following the concept of the dev test ops culture, we started to consider the report portal as a basement let's say like touch base for the for our way to the continuous testing. And since it utilizes the capabilities of the reporting, we can collect all these statuses within the let's say pipeline of the test automation and really bring all the testing results in one single point where you can just find everything in one click and many, many other small supplementary features. Once again, it was built by the test automation engineers for test automation engineers. So all those small features and functions really saves a lot of time for the for the engineers.

So I love the idea how it saved. I think you said one of your teams 90% of their time with debugging. A lot of times, you know, when you work with large enterprise teams, you may have a lot of failures, they don't know what to focus on. So how did it save the team time? Does it put things in buckets? Like this is a real assertion error. So you should definitely look at it where this other one is like the element time dose that's more like flaky.

We use actually the base for the ML. The base for this analysis is the stack trace and log. So you have for your test automation for your failed test cases. Basically initially the report portal works the next way. When you have your execution finalized, the report portal will mark with to investigate, flag all the points, all the fails where the input of the engineers required. And when you execute the analysis, the report portal just takes your failed test case and start to look for the most similar fail within the historical data of your previously analyzed fails. And basically it can group it like an assign that this fail was like a product bug in previous execution. This fail was because of the test automation problems. And since within the report portal you also can add the custom defect types, at the end of the day you can find out that it will grope your fails according to specific reason of the fail based on the historical data and those one which are not recognized but can be grouped based on the stack trace. You just can see that okay, those issues look similar and you just like make a bulk edit, select them and assign them for example to the system issues. Just like to give you an example, if 50% of your test cases just failed because of the unavailable environment, you can just open one such a failed test case, look through it, set the system issue for it and just click to make analysis within one launch. And basically the report portal will find all the similar fails within this one launch and mark them with the same reason. So for you just like maybe few fields will be left without the analysis and you will spend your attention only for those test cases which are really require attention of the engineer and not just like double checking all the existing issues.

Awesome. So before I interviewed you I asked my Slack channel for my guild conferences if they had any questions. So of course a lot of the questions were around AI. So I just want to pick your brain a little bit around this to see if we can address some of their concerns. One of them wants to know, you know what is the best way to ensure that the data is processed in the best way for your machine learning algorithm? Is there anything they need to do to the data in order to get the best results?

Well, all the data automatically will be indexed for the machine learning and maybe like the starting point which let's say we have the suggestion for such for newly started project, just send at least seven launches within the report portal and analyze those seven launches manually. And after the IT report portal will have enough learned data to start the analysis. So basically we also leverage the approach like leaning with each new execution. So basically as more data you have in it, the more basement IT have for the analysis. But as a starting point just give them seven launches and analyze them to have like basement for the ML.

Cool. I think you answered someone else's concern. I guess their problem was they're not seeing AI was not working for them. They wanted to know if there was any specific criteria they needed to do to get it to work, any specific way or anything they needed to do to arrange the launch data for the AI to start working. But it sounds like they probably need to run multiple iterations of tests in order to really get it to kick in.

Yeah, because once again if you have no new data within your project or just few executions, how the ML will decide? Is it even for humans? Sometimes it will be hard if this engineer is not inside this project. So just give it more details and it will start to work as soon as it has enough data.

Great. So how customizable is a report portal for folks that haven't seen it yet?

Well, this was the pretty crucial and important point when we start to implement it because from day one we was assuming that it will be open sourced one day. So currently it even built on microservices architecture first of all. So basically it's a perfect extension point for the report portal and you can just implement your own small teeny microservice which can use our API and let's say do any changes within the API and put it inside the role of all our microservices just like put it within the all application and it will work with any other microservices which is inside. And at the same time since we are open source, you can see all the code, it's available for everyone and everyone can, let's say, contribute. And once again, assuming that it will be open source one day, we will try to make everything extensible as much as possible. So like in most cases when we add some specific functionality, we just not put few lines in code, we make some sort of the buses or some sort of the interfaces which can be extended or let's say replace it with another functionality which will be required by another company. So projects the same for the machine learning it actually for analysis itself you can put different analyzers for the report portal, just changing the priority for them via the configuration and just make another type of. So like ML is just one of the options. You can also put any regexps to compare your fails. You can put just strict equal to compare your result, just put another class point it to extension points and it will work. There may be one problem which we faced with the contributors from the open source that our main user group is test automation engineers. And since we start to grow pretty fast and our solution currently used within the big company with a huge cluster, we have to build it according to the practices of the good and stable software. So this brings a list of the requirements of the let's say pretty skilled engineers or developers can do some changes. So yeah, from one point of view you still will see pretty let's say hard or difficult parts within the report portal inside which helps us to make up and running huge clusters. But at the same time we left as much extension points as possible for any functionality which is inside. And at the same time we still have open API. So if you have the report portal and you'll click on the profile picture on the right Upper corner you will see the button API and under this button you will see this wagger which fully describes all the API and all the methods which we have behind the report portal and basically sending the API request. You can do whatever you want with the report portal and this is also a pretty good extension point. And this is the point how you can use the report portal within your pipeline and basically trigger different actions in the report portal by Jenkins or by any other script and you know, like make a seamless integration with any other tools you have.

I don't know if I mentioned I just installed this recently and I'm trying to get it working for me but like you said, I didn't know that API was there. But when you look under the API you see all the APIs, all the different calls you can make. So awesome. So you guys really did develop it with it seems like you said with open sourcing it eventually and also making it integration friendly. So I love that and I guess that probably why it explains you have so many integrations already with it. I guess you support Cucumber specflow soapui jbehave Is that one of the main reasons I guess you designed it with integrations in mind from the beginning?

Almost yeah. This was also like a key idea to make it able to work with any solution. And I can even say that we have integrated it with Cilk test which is pretty, not like outdated but old fashioned stuff. But still we have the agent for the Cilk test and it helps to send the results from such an old fashioned stuff, old school stuff inside the report portal and make it capable with the modern software or let's say modern way to run the software development cycle. So basically this API and since we have like everything works through the HTTP channel, you can integrate any solution which can just trigger the particular events. So basically when you have this agent or your language, you can wire your internal events within any testing engine or any platform to send those results inside the report portal. It's pretty easy and in most cases the lead engineer just like spend four hours to implement these adapters from scratch. And yeah, we already have a pretty big list of the available integrations. But there is a problem since once again we have a pretty huge demand from the community from different companies, but we still have a limited team and still have just 24 hours a day and we have now enough capacity to make all the integrations up to date every single day we are focusing more on releasing functionality and to fix the main agents to be stable. But sometimes and especially for such a. For example for JavaScript they have new engines every single day and we just have no enough capacity to implement them. And actually if even we did implement really takes a lot of time to update all this big zoo of these integrations. So.

For those guys who are listening us right now, if you will try one of those integrations, just please do not finger point at us saying oh these guys have unstable integrations. We just lack of enough time to fix all these integrations, just spend a few hours and in most cases it's just like few small fixes required for them to be up and running and well we are open sourced for that. We do not have a big team assigned so to this solution and we actually trying to find the outside companies who can let's say support us. And based on this support if we will have it, we can hire the dedicated engineers who will make those integrations and all this functionality, let's say up to date every single day. But still a lot of companies use it. But maybe since we are the engineers in background, we're not the yet perfect product managers and not like still not a startup. We have now enough investments to do all the magic we want to do, but we really trying our best to deliver everything we want.

Awesome. Like you said, it is an open source project. So folks, how can they start contributing then if they want to help out? I think everyone should contribute. This is a, to me this is a game changer if you can get it stable and work with everything. I think everyone needs this.

It's pretty easy to start and the main let's say entry point for that will be our GitHub so you can just find the report portal on GitHub or start from our landing page at ReportPortal IO. There will be the link to the GitHub and where we have the contributing MD file which describes the main steps and at the same time every single integration is allocated inside separated repository and basically you can find out and see the code and start to fix it. And actually for our team we have procedure to review the pull requests from the open source like every two days and as soon as we see it we review it, give a comments, ask for fixes and let's say accept them. Because basically the report portal itself, yes our small core team is sponsored by the EPAM because anyway we close the gaps for epam, we deliver this value for EPAM as well. But the most, let's say the biggest size of our part is the contributors. The contributors within the company, the contributors from the outside of the EPAM as well and we spend time to review and adjust those.

Pull requests to let's say to fit our main goal. So just find us on GitHub, find the thing you need to fix or let's say to implement for you. Contact us on Slack as well. You can find the link to Slack chat on the ReportPortal IE landing page and we are really happy to help you.

Awesome. And of course people should Google things because I'm using a platform called Serenity and I don't see it listed as officially supported but I found a plugin in Maven that allows it to work with Serenity. So a lot of people seem to already be implementing this already. So really cool stuff.

Yeah, but the problem is that somebody maybe did not share it with us. So I know that there are pretty huge list of the different implementations and some of the companies start to adopt us and implement own integrations but the problem is that they did not share back the things they implemented. And just like the main question for such a company is please contribute back, it will help us to grow.

So a lot of times when people want to consume an open source project, a lot of them worry that it's not going to be active or there's not a lot of contributors or it's not popular. So how popular would you say report project Portal IO is right now and where do you see it growing?

Well, we actually started from our CIS region because you know it's more native for us to speak with the let's say native guys and those guys can say us the truth just like okay guys you did something wrong, just reimplement that. And we started with our CIS region as a starting platform to give our first feedback from the open source to give feedback from the community. We actually did some changes and reimplement some stuff and also change our, let's say mindset which is required for the community. And now with all this experience we are trying to meet the market of European market, India market and US market as well. So currently we have a pretty good popularity within the CIS region because it's pretty easy for me to actually just visit the neighbor countries and speak at the conferences. So that was the point to start share the word about the report portal. But at the same time we start to see that the interest start to grow within the European, India and US especially. And currently the US region grows like plus 25% of the users each month. So I guess like just in a few months it will be even bigger than some of all other Users from all other countries. And yeah, you also mentioned pretty good question about the open source and support of the solution because yeah, there is a problem with the open source or let's say open source foundation that somebody did something, put it on open source and just forget about it. And this is the key point for us because when our company and our main sponsor as EPAM see that there is an interest for this foundation, there is an interested interest from different companies, countries and that's it. So this is the point why they sponsor us to develop more. So basically as more clients and as more users use us.

The more day we will stay alive let's say and especially in the case when we will get the sponsorship from another companies and for example, I don't know, whatever they donate to us or buy our support packages, we will live more and more days. And also we considering to try the one way to use the report like to give another possibility to use the report portal. We're also considering the SaaS platform. So basically once again since the report portal is free, you can just download it and start on your own environment. You still need the environment, you need those virtual machine to make it up and running. You need to spend the time of your DevOps engineer or any.

Infrastructure supporters who will deploy it for you. And we just considering that if we will have geo cluster like huge cluster of the report portal which is available for everyone and just ask.

100 bucks a month for example, I don't know how much it will cost but anyway we will count it down and propose it much cheaper than it will cost for you to make your own instance. Maybe this idea will help us to grow, will help us to earn some money and extend our team to implement everything. Basically since the report portal open sourced it will stay for free and can be used for free by anyone. But we also need to search some ways to monetize it and to make it even more capable and.

Provide more features out of the box. And the SaaS is one of the ways we consider it. At the same time we still.

Believe that service offering on top of it with the support packages also will help us to to stay alive as much as possible. And especially the US market I guess since this market can understand the value of such tools which helps to increase the productivity and helps to build the pretty modern continuous testing. Right now we really believe that this market will help us to to grow.

Okay Dmitry, before we go, is there one piece of actionable advice you can give someone to improve their automation dashboarding efforts and let us know the best way to find contact you or learn more about reportportportportal IO well just visit.

The reportportal IO and you will find all the related links. So basically on this page you will find our documentation, our links to the GitHub YouTube channel where we post the trainings and video materials and what's new news. As well there will be a link to this Slack chat where you can find our team and our community and just like talk with us, chat with our users and well find everything you need to start with the report portal.

Thank you Dmitry for your automation dashboarding awesomeness. For links to everything and value we covered in this episode, head ON over to testtalks.com222 and while you're there, make sure to click on that. Sign up for a free trial link under the Exclusive Sponsor section to learn all about Sauce Labs awesome products and services. So that's it for this episode of Test Talks. I'm Joe and my mission has helped you succeed with creating automation awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Talks podcast. Head on over to www.testtalks.com for full show notes, amazing blog articles and other automation awesome.