Keep Track of Your Automated Tests using Delta Reporter with Juan Negrier

Welcome to the Test Guild Automation Podcast where we all get together to learn more about automation and software testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Automation Podcast. Today we'll be talking with Juan all about how to keep track of your automation tests using Delta Reporter. Juan is a QA Engineer with over 7 years of experience automation testing and he is one of the co creators of the new. I think it's new. We'll find out open source reporting tool Delta Reporter. So get ready to discover how to monitor all your automated test results in one place in real time and investigate the root causes by using the Delta Reporter for data analysis. This is a thing that I know a lot of automation engineers struggle with. I'm really curious to know why Juan created this and what his experience was around this particular issue. So you don't want to miss this episode. Check it out.

The Test Guild Automation Podcast is sponsored by the fantastic folks at Sauce Labs. Their cloud based test platform helps ensure your favorite mobile apps and websites work flawlessly on every browser, operating system and device. Get a free trial. Visit testguild.com sourcelabs and click on the exclusive sponsor section to try it for free for 14 days. Check it out.

Hey Juan, welcome to the Guild.

Hey yo. Thank you for inviting me.

Awesome. Awesome to have you. I guess before we get into it, is anything in your bio that I missed that you want the Guild to know more about? No.

Really? No, I'm not sure really about that.

Okay, sure. So I guess as I mentioned in the intro, Delta Reporter, I guess people are curious to know at a high level, what is Delta Reporter?

Oh yeah, it's basically a tool that you can run on a server, on aws, on Docker, on whatever you have. Actually what we do at the moment is deploy it into EKS using helm. Basically when you have it alive in a server, you basically use one of the plugins we have for your framework and then you can send the data from your test directly into DataReporter. The cool thing of it is that you can send data for different type of tests. I mean if you have testing details, test for contract or unit test, you can send those to Delta and also your test end to end. If you are using for instance with Revit IO, you can also send that in Delta. You can basically integrate all these results. We have things something like a pyramid, like the testing pyramid, Testing pyramid. We are actually able to draw that in the launch. So basically you can see your launch and then you can see if you actually are achieving this pyramid of testing with a lot of unit tests, a lot of contract and just a small amount of end to end or maybe SAR reverse or whatever. So basically in just one view, you can check how's your quality doing, how is your institution of tests and how is everything basically nice.

So yeah, I like to drill down into this a little bit more. But you know, what were you struggling with that made you want to create this? Were there any other tools that you were looking for that you couldn't find and you're like, oh, we're going to have to create it ourselves. Like, why take the time to create your own open source project like this?

Yeah, that is a very good question. Thank you. The reason is because we were using Allure before and the unit normally as in Jenkins, but we have the problem of how to keep track of the historic history of the, of the test because it happens is a lot of process. We also have a flaky test. We also have tests that, you know, basically you need to maintain the test and then you need to also when they fail, you need to be fast into figuring out if is the test failing or is the system failing or maybe it's the environment that is an issue. So basically we wanted something that was able to provide us feedback like immediately or maybe that it's really easy to get information we want in order to figure out this kind of stuff, basically the root cause of the failure. And we were looking at some tools. For instance, we really tried to use one tool that is called Report Portal, which is amazing. It's amazing really. But the problem is that when we wanted to implement it, we impact some problems with the, you know, the other teams, like with SRE and the OPS team. Because basically the problem is that the tool was so big that the effort into deploy, it was basically not a priority for them, you know. So basically it was only up to us in que. And in that circumstance we, I don't know, we just believed that probably it was going to be easier for us to just develop something ourselves, just simple. Basically that was it. Basically we didn't think much about it. We just wanted to do it like this, more simple, easy to deploy and everything. And then, I don't know, we just did it.

Really nice. Yeah, I was going to ask about Report Portal because it sounds like it's similar to Report Portal. So it sounds. What makes a difference is that it's easier for you to deploy without getting your other team members involved. Is that the reason why you Went with it again.

Yeah, exactly. For instance, on Delta Reporter, we don't have a control for users at the moment. We are thinking about it, but we don't have that at the moment. So for instance, if maybe if you're using Junit Reporter or Allure or something like that, or Timeline Reporter, which is also very good and you want just something simple, you don't need a huge system to control everything. Probably Delta is good for that because you can basically integrate everything. It's easy to deploy, it's small. So that is the difference we have with Report Portal, basically. But the work portal is amazing by the way. It's really impressive the work the guys are doing there.

Absolutely. I see other people too when they create dashboards is they use something like the Elk stack. So elasticsearch on the back end with the database and Kibana on the front end. Is this something similar or this makes it even easier? More tester focus Delta Reporter. I don't know if you've seen elasticsearch and to know what the difference is between the two or not.

Yeah, I have never used elasticsearch but I know what it is we're using in my company, actually. Yeah, I think that is a lot more simpler really. For instance, all the backend is just running on Python with Flask and the front end is using Next JS in order to store everything. But approach is really, really simple. Like even the screenshots and videos we have, since they are not, you know, they are compressed and everything, we're actually storing those directly on database. So we are always erasing stuff and everything. So it's all really simple, really. We're not using anything fancy for the moment. Like it's all really, really simple to deploy and really simple to use and still don't want to paint, which is the other good thing about it.

That's probably a key feature though, the simplicity. Because you don't want to get bogged down in like making this awesome tech that no one uses. It seems like you create it to be easy, but it gives you all the information your team needs to be successful, is what it sounds like.

Yeah, yeah, pretty much. It's something like that. Yeah. So basically, if it's a wallet and you need a tank, it's basically the simplest tank ever. But that's the job basically.

So I guess the next question is once you do have it implemented, and I guess we should dive into that also is how does it help your organization with automation testing? So you created it and now it's in your organization how is it helping you and your team?

Yeah, basically the thing is that in my company we have basically two systems that we maintain mostly, but you are different on the architecture. One is monolithic and the other is based on microservices. So happens that Delta needs to be is helping us with all these releases. For instance, for the monolithic system, we are basically releasing this big piece of all the components in one place. And we always run all the tests. You know, basically the integration, end to end everything. So basically the cool thing of Delta Task is that when we have a release and something fails and we need to check it to give tell the guys if it's okay to release or not, it's just basically making one click going to Delta and then checking all the history of the tester, see exactly what fail. Also it's cool to see this integration and end to end test. And recently we have a new feature that we call it Smart Links. And these basically are links where you can pass data from the test. So one thing we did with this feature is to create a link to Kibana where we have all the logs about the system. And it's cool because basically when you click on this Kibana button, you are sent to Kibana and we filter on the timestamp where the test was running, an end to end test. So then we can check the logs as well there. So it's pretty amazing because basically something fails, we have access to the logs to, well, screenshots, everything basically. So figuring out what was broken is faster than before. That's the whole thing of it.

Now that is awesome because a lot of times you're like, okay, it broke. Now I need to go into this other system, I need to find it. So it sounds it's timestamp to just click on a link takes you directly there. That's pretty awesome. What other information can you pass with the smartlink? Can you do something like performance, like high level, how long it took the test to pass, things like that, Things like that.

And also on the client, for instance, we have a client for Webdriver IE and in that client we're using an integration for visual regression test with another tool called Spectre, which is very well. So I'm also the main trainer for the visual reduction plugin for Web IO, the one for Spectre. So basically what I did is to create a link that is sent back to Delta with the URL for Spectre. So and basically that is another command. It's a command on the client or whatever IO where you can send Any data back to Delta and then use this data in the smartlink. So in my case what I did was to send the URL for Spectre where the screenshots of test run.

And then I just displayed the button using this data. So it's also like you have the button to go to Deluxe on Kibana and then you have this button to go to Spectre, this visual regression tool. So you also can check the screenshots for the visual regression test.

Nice.

So it sounds like it works with WebDriver IO. What else does it work with then? Like say I'm using another tool.

Yeah.

Can I implement it easily using another tool? Say like Serenity or Test project or something like that?

Yeah, we're facing now is basically creating more plugins for other service, other test frameworks. We're now working on the one for Junit actually Christian Moises is another collaborator and he's working on that. And then we have covert PHP unit we have withwio which helps well testing GI and yeah, basically we are now looking into which one else implemented we're looking at. Yes, actually Jess is going to be the next one. So yeah, basically we're just looking and studying which framework we're using at first. Obviously we are setting priority for the ones we use in our work. But yeah, I mean we are just looking into that actually.

Awesome. Because it is open source, if someone wanted to create their own plugin, how easy is it for them to do that?

Yeah, it's basically from what I saw is basically normally to expand the text framework we're using. For instance, for whatever I joined was using these hooks for when the test start, when the test finish and everything. And then using the test name and the status, stuff like that is sending that information back to Delta and then controlling everything. It's actually complicated really. When I was doing the one for testng, it was actually complicated because when you are running several tests in parallel, they run. You know, you need to use threads and stuff and it's like messy. So we still have issues with tech plugins actually because it's messy. Really with that.

Yeah, it sounds like it would be difficult. So if someone was to try this, then it's. It takes some time. It's not like an easy, easy one and done type of deal. It sounds like.

Yeah, so it can be done. That's the cool thing.

Cool. So also how does it integrate with your system? Is it a call from your CICD system? Like if you're using Jenkins, like how do you integrate it? You said it Works with the pyramid. So how do you know when, how do you get your developers to use their Junit or their unit test, be able to send data to it? Is it. Do they have to code anything or is it just a matter of plugging into say Jenkins or when it runs it just knows like how to collect the data.

Yeah, that is a good one as well because we are actually, we have in our backlog to create an integration with Jenkins, like a direct link or something to Jenkins to Delta. Because what we are doing at the moment is just to use a simple button in Slack. We basically have channels where we have alter lists and stuff and we just have buttons there that link from Slack back to Spectre, sorry, to Delta. That is what we're doing at the moment is part of the plan to actually do something for Jenkins, for Jenkins. What we are doing at the moment is just sending information about the pipeline back to Delta. So basically it's possible to put a button on Delta to move from Delta to Jenkins using this Smart links feature, but not from Jenkins to Delta. That is what we need to do now. And I don't know if actually it's possible maybe to also include Azure or Travis or Oct as well. Probably we need to take a look at that.

Nice. So how long have you been developing this for? Is it fairly new? Like is it years, a year, months?

It's been more than a year, Like a year and a month actually.

And you actually you are using an in house for like a year and you're getting a lot of benefits from it, it sounds like.

Yeah, yeah, we have started using probably around six months ago or something like that.

Cool. So I love the feature of the smartlink. What are some other features in the tool that you think people would really benefit from?

It's also possible to leave nodes in the test. Like for instance, imagine that you are troubleshooting what is happening with the test and then there is something that is probably a bit alien. So it's something good that you can actually leave a node. Since we don't have users or login at the moment, you need to leave your comment and then your name. But it's probably okay for the moment. But that is a feature we have. And also since we have basically a structure that is based on test and then we have the test that is run for the specific test run. So the nodes and all this is going to be stored on the higher test. So basically if you make a new run and the same test is running again and again, you're going to see the node again on the test. It's the same with the test history. Test history is basically you want to show the result for that test in previous runs. So and also if you go into this test history, you can also open the previous test check why a test in the past failed check also screenshot to see maybe it's the same failure or something. So it's not only checking if a previous test failed, it's also checking the information about that test. You can be sure if maybe similar or not. We also have a feature called that is to set a resolution for the test. You can imagine that maybe, I don't know, you're going on vacation, but so. And there is a test failing. So you want to be sure that people don't waste time maybe looking at them again so you can set a resolution like saying that the test is flaky or there is a probe defect or needs more research or maybe it's an environment issue, maybe as you mentioned, last research was not blank today and that was it. So, yeah, basically that are the features we have for Delta at the moment, something we want to implement since we have backend based on Python is to see if we can use something like a Scikit or Pandas in order to automate some of this stuff, basically like Parse, for instance, these logs we have and really give an alert or maybe try to suggest an automatic resolution. That will be amazing to do. And we are trying to examine how we can do that.

So it uses Python behind the scenes. Python has a lot of awesome libraries. So if someone's trying to implement them themselves, I guess they would have to use Python. Is that correct?

Yeah, exactly. I was taking a look and it looks like it's not going to be that difficult. Looks like the libraries we have in Python machine learning in this case are actually really strong. So probably it's not that difficult to. But you know, priorities and things of like, right.

I love the concept of notes and be able to flag things as flaky. When I used to work for a large enterprise, we have like eight Sprint teams and they'd have a test and like, oh, we're going to fix it the next Sprint. And so you'd always have to write down an Excel file and then when you're reviewing every day why it failed, you have to go, I forgot why? Because I couldn't annotate it in Jenkins. Sounds like this type of notes feature is what can help you with that.

Yeah, true. Because for us it was the same. Keeping a spreadsheet with everything and then also keeping that updated is normally messy and everything. So it's like, I don't know, I mean, for us has been great, to be honest, because we can now keep track of all this information in one place. Also, since we have this difference between environments with microservices and objects that are monolithic, basically some approaches that we apply for the monolithic one doesn't work for the microservice ones. It's like I want to keep in the balance as well, so.

Cool. So I like the approach of using Python behind the scenes because you can make it. It sounds like you'd be easy to expand it out to other things because there's so many libraries in Python that can help. So you talked a little bit about plans for the future, but anything else you see that maybe years down the road you'd like to see implemented that you probably can't do right now, but you'd like to definitely get or, you know, in the near future? Basically, yeah. I don't know.

It's complicated to say really, because for one thing, we want to really to keep it simple. So I don't know really. I mean, as you mentioned, Python has so many possibilities that probably an expansion into something else is.

Really doable and probably not that difficult as well. We are still studying if we want to implement users. Probably we want actually so basically, like for years to come. I really don't know. Probably that thing about automatic analysis of locks and everything. Yeah. For instance, in our case, in our company, we are actually storing information in some data sources. So basically we were also thinking about that maybe we can actually connect to some of these sources where we store the logs and get the data from there. I'm not talking about logs from the test, I'm talking about logs from the system when the test is running.

Right.

So that will be.

That'd be killer.

Also amazing to just process and get something like far more smart. Really.

Yeah, that would be very cool. So if someone wants to implement Delta Reporter, what kind of infrastructure do they need? Do they need to have anything in place, like containers or like, what do you need to get running? If someone wants to do this from scratch, just take Delta Reporter and start using it within their companies.

Yeah. What we have at the moment is we have a file docker compose file. That is the easier way to run it, really, because it's just talking docker compose app, cloning, everything. There is actually a start script in our main repo. So that is actually cool because if someone wants to just Test it on his local machine, see how it works and everything. That is the easier way to go. Then for something more enterprise, we have a helm repo. So basically deploy it into Kubernetes is actually really, really easy. It's just pointing, creating a deploy file, pointing to this repo on helm and then just deployed that will deploy a microservice with the backend, another service for the front end and then we'll deploy a database as well. That will be really what you have to comment. I'm thinking about Terrace, but I think yeah, there is a configuration we have as well for to run it serverless, but we haven't tried to be honest. In theory it should work, but I can make promises of it.

Nice. So if someone's using Selenium, say with Java, would they implement this using Junit to be able to plug into Delta Reporter?

Yeah, yeah, exactly. With Junit it actually, sorry, the SNG is the one that is working. JNG is on development at the moment, but it's going to be ready probably during this month. So for Java it's going to be Junit and Tengi. We have both flavors.

Nice. So you are a QA engineer and I'm really impressed. When people have an issue in house, they create a solution and then they open source. It seems like there's a lot of extra effort there that you went out of the way to help the community. So how much work did it take? How much programming did you need? Like how big of a program? Say someone's listening to us, they're like, you know, Juan's inspire me. I want to make my solution open source as well. How much programming knowledge do they need to know to do this? Or how much extra work are you putting in to have this work? And you're working full time and you're working on this now, almost like a side hustle.

Yeah, that is an amazing question because part of the reasons we started with the project was also because we wanted to upscale in our coding abilities basically. And in my case, for me actually it's kind of normal to build tools like this. Like I always end up making something really with Python or JavaScript or whatever. So something I would suggest to someone that is also like kind of like me, that is interesting to do something like that is just to don't be fear about stuff, just do it really. And about open sources and everything is something that I think is good to be honest with the company where you're working, you know, request the time factor. Your plan is actually to open Source to have it independent from the company or maybe with the company and be honest about your intentions like that you really want to improve your coding skills. You want to also fix a problem because at the end it's a win win for everyone. It's a win win for the community, for the company, for yourself. So I think it's something that anyone can do really. In the case of Delta, it's not actually really that complicated because Flask is the thing we're using in our case and it's well known for being really easy to use, really micro web services. It's not really that complicated. So it's actually very good to begin with creating microservices because it's so simple to use. There's a lot of tutorials as well if there is any problem. The community for Flask is also big and amazing. And then the other solution we used was Next Framework and Next is also nice and there is also a community on it. And it was a bit more challenging for me, to be honest, because also it was my first time using React as well. And it's totally different from vanilla JavaScript I can tell you. But still after you basically understand the concept and you apply your knowledge, you apply what you are learning is amazing and it's something, if I need to say something, I think it's something really beautiful as well. Like feel that you're improving yourself, you're giving back to the community and that you are also fixing a problem that was present as well.

Yeah, it really is a beautiful thing like you said, because you are improving yourself and you're giving you. You know, a lot of times people like they're just waiting around to do something. But if you're doing something every day you create a solution, it kind of incentivizes you to keep working on it as well. So absolutely win win for everyone.

Yeah, exactly. That's it. Yeah.

You made it open source now and say someone wanted to contribute. How does that work? Does someone make a pull request? They make a change and then do you review it before you incorporate it into the core? I guess there's a section called Delta Core. How does someone contribute, I guess to Delta Reporter is my main question.

Yeah, it will be something like that basically to create a PR with the change, with explanation of what was changed. Then I think we have some checks that are automatic basically to check that. Well, check that it compiles and stuff like that. Yeah, and basically it can be a thing to me. At the moment we have three persons working on Delta is Juan Ederer, myself, Alexandra Pichaiko and Christian Moisha. So any of us can really review it and basically then it's just merging it and check that everything is okay and create a new release on Helm as well with update. So that will be.

Have you got any feedback since you made it open source from the community?

Not really. Not really. Actually we were like trying to make it more visible in a way because basically to see if we can generate some kind of community behind it, nothing to improve it and see the value of it or everything. So basically yeah, that was our intentions. Now actually when you were asking me about years of the future, yeah, probably this is something like create a community behind it.

Okay Juan, before we go, is there one piece of actionable advice you can give to someone to help them with their automation testing efforts and what's the best way to find or contact you?

Okay, what advice can I do? My best advice is always look for tools. There's always amazing tools out there and I think it's a good practice to actually take some time, check all the news about automation and check the latest technology we have in regarding testing. And probably that's the best advice I can give to anyone. Really the best way to contact me. Normally I'm pretty active in LinkedIn. Really that is something like my main network at the moment. Yeah.

Thanks again for your automation awesomeness. The links to everything of value we covered in this episode. Head on over to testguild.com a337 and while you're there, make sure to click on the Try for Free Today link under the exclusive sponsor section to learn all about Sauce Lab's awesome products and services. And if the show has helped you in any way, why not rate it and review it in itunes? Reviews really help in the rankings of the show and I read each and every one of them.

So that's it for this episode of.

The Test Guild Automation Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack automation awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Automation Podcast. Head on over to testguild.com for full show notes, amazing blog articles and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.