267: Smart Test Execution with Eran Sher

Hit it.

Welcome to the Test Talks podcast, the place to go to geek out on software testing. And now your host whose mission is to help you succeed with test automation, Joe Colantonio.

Hey it's Joe and welcome to the show. Do you ever run your entire automation test suite for every build because you just don't know which ones to run based on the checked in changes? Running all your tests for every check in is really time consuming as you know and it creates a lot of noise if tests stop failing. Have nothing to do with the checked in code just because you have flaky test so in this episode we'll be talking with Aran Sher from C Lights. Aran shares his thoughts on Test Impact analytics. This approach allows you to find the minimum subset of tests that needs to be executed to help increase your velocity without sacrificing quality. You don't want to miss this episode. Listen up. So Automation Guild 2020 is coming. I'm planning it now and I need your help. If you haven't already, can you head over to GuildConferences.com and just fill out that embedded survey form that's on guildconferences.com it really will help me to find speakers and topics you need to know in order to help you succeed in 2020. Guildconference.com thanks again for your awesomeness.

Test Talks is sponsored by the fantastic folks at Sauce Learning Labs, the cloud based automated testing platform that eliminates the need to maintain your own selenium grid and test infrastructure. Try it for free today. Visit testtalks.com and click on the sign up now link under the homepage Sponsor section. Hey Ron, welcome back to the show.

Thank you Joe. Happy to be here.

It's always great to have you on. We had you on episode I think 147 and 224 about C lights and about AI and all types of things to do it. More newer approaches to test automation and analysis and analytics and things like that. So that's what we're going to talk about today. I think you have a cool new feature that we're going to touch on. But the first thing before we get into it for the folks that didn't catch you in episodes 147 or 224, you just give us a little background about you.

Iran sure. So my name is Iran Share I've been in the enterprise software business for the past 20 something years, mainly in startup companies that were successfully acquired and bringing innovative solutions to the market. Part of it spent few years at Mercury Interactive Those of you who remember which was one of the, I would say breakthrough companies in test automation and quality management many years ago and my previous company which I co founded in 2007 was one of the earliest DevOps solutions was a continuous deployment platform which was acquired by Computer Associates. And a few years ago we started Sea Lights with the vision to modernize quality intelligence and quality management by using data actionable data analytics that will allow you to increase velocity and increase quality.

Very cool. I'll probably end off topic here, but I love Mercury. I loved Mercury. I used to be a Mercury maven, one of the elite people that always contributed to the knowledge base. Back in early 2000 it just seems like a lot of people and a lot of startups now in test automation, a lot of them have routes back to Mercury. Any reason why that is? It seems like such a hotbed of innovation that came out of there?

Yeah, first it was a very innovative company and a lot of the innovation came from the engineering team.

There is a very good understanding of the enterprise ecosystem. Not just quality management, test automation, but also developers tools. APM application monitoring solutions came from there. I think what we're seeing today is that there is a need for new solutions. The existing solution, the old solutions, they don't actually apply to the high velocity of software development and delivery that exists today. It's a distributed architectures, it's a different infrastructure, different pace. There is a need for new solutions and I think this is why we are seeing a lot of new solutions in the testing market category. And also a lot of money is being invested into startups companies in that area.

Absolutely. And one of the things I liked about Mercury was it had a full product suite so I could kind of keep track of what my tests were doing and put requirements against them and also put them in scenarios where they went against certain releases. And I kind of lost that going to open source. And the problem with that, it was stuck to Mercury's product suite. But newer tools seem to be working more with open source and filling in that gap. I think your company is one of them and one of those gaps I've been seeing a lot of is over time. I've been working on a project for five years. We have more and more and more and more tests and and I just don't think the answer is just run all your tests for every check in is really working for a lot of companies or even feasible. So I've been reading a lot on your blog about something called test impact analytics. So for the folks that don't know at a high level. What is Test Impact Analytics?

So Test impact analytics is the ability to actually, I like to call it smart test execution and exactly change those old practices of running all your tests for every build, running all your tests all the time. The reason you don't know which tests you need to run, so you simply run all of them because you don't want to compromise quality. But Test Impact analytics actually puts some brain into the test selection process and allows you to identify what is the minimum subset of tests that needs to be executed in order to increase velocity without compromising quality.

Very cool. And I'd like to probably drill in a little bit more on the brain aspect of it before that though.

Why do you think testimpact analytics is important? I've spoken to some people that are like, oh, I'll just run on a grid to affinity. It doesn't matter if I run all my tests.

Yeah, that's another solution. You can start, start running tests in parallel. But as you mentioned, as you have more and more tests, it means that you will need more and more environments to build and wait for these environments to be built. And it's also very, very costly and you also need a lot of DevOps resources to help you with that. And when I'm talking about Test Impact analytics, it's not just for one type of test, let's say browser test, where you have some great grid tools that will help you with that. What about the integration test and the unit test that you are doing on each pull request? And you need to wait for even an hour instead of waiting minutes. What about your functional test? What about your integration test? What about your system test, your end to end test? The cool things with Test Impact analytics or smart test execution, you can apply to any test by the way, also to manual tests. So you get a full solution and apply this analytics engine and simply save time to shorten the regression test cycle. So your releases are going to be faster, your productivity is going to increase. Your developers and QA engineers, they don't need to wait a lot of time for the feedback. If you're doing a code change and now you need to wait an hour for your regression test to be completed, that's not a productive way to work. I mean, even if you do something else in that hour, the that's a context switch, it's not the way you want to develop. But imagine you're doing a code change and after two minutes you're getting all the feedback that you need, not just from the Unit test, but also on the integration test, the component test, the API test, the system test, it's actually also enabled you to do a real shift left because now you can select those more complex tests which are longer to run, but because you can select just few of them that are related to the code changes that you did, enable you to get not just faster feedback, but higher quality feedback about those code changes that you are doing.

Absolutely. I work for a company, we have to release patches sometimes. Like you said, I love how it covers more than just automated tests. If you're just running all your automated tests, all of them, you get a lot of noise if some stop failing and you don't know if it's failing because of the code checked in or they just feel like they're flaky. So I think this helps really pin that down. But on top of it, like you mentioned, we also have manual tests. Because it is a complicated product, you don't want to just release it to the customer. It's a healthcare product and it takes a long time because we have about 900 manual tests. And so it sounds like this solution would help you say not only here are the automated tests, but here are the tests, the manual tests you can run. And we probably could get the hotfix out the door a lot quicker using the this approach.

Absolutely. And as you mentioned, when we're looking at test impact analytics, we're looking at as a holistic solution. It means that we need to provide a solution that would work for companies that can release very, very fast. But on the other end, there are some large enterprises like you mentioned, healthcare, where you don't want to take any chance. So here you can use the data of the impacted test, not to do a selection of tests, but actually to prioritize the order of the test that you want to run. In this way, those impact tests have a much higher probability to find bugs. So that means you're going to fail fast and you're going to save time by that. We also provide, we were thinking about all type of solutions and all the features that you need in order to actually get these smart test execution out of the box connected to your test frameworks. Actually you need to connect it to your CI and start running with it.

Very cool. So you did mention the brain aspects of it and I guess a lot of people may be skeptical unless they see it in action. How do you know or how are you able to identify impacted tests?

Okay, so I'll explain that There are several layers. So the first layer is as you mentioned, to identify impact test. So in order to identify impact test, the first thing you need to do is actually the ability to identify which code, which code elements, methods, functions.

Are being executed for each of your tests. So you need to isolate, I would say.

Or to correlate each single test in your test suite to the actual code that this single test is actually testing.

So does it use based on history and how much history do you need in order to know that?

So what we're doing, we are using an instrumentation.

Technique. And while your tests are running, we know how to identify for each test what is the actual code that this single test is actually testing. And we keep it in a metadata form in a database. The second thing we are doing, we are scanning all your builds automatically and we are analyzing which code was added or modified for each build. Then we are correlating the data. So we know for a certain build there were certain code changes. And now we can identify which of your tests are actually testing those code changes. Because we have the footprint of each of your tests that can be, as I mentioned, any type of test, the integration test, the functional test, the end to end test, the manual test. So for each of your test suite, for each of your test stages, we are identifying the impacted test.

That's the first level. But it's not enough. It's not enough. What we are doing, we are adding or enhancing the test impact analytics to a more, I would say complete recommendation list, which including on top of the impacted test, also your latest failed tests. So we are also tracking those tests that didn't pass in the last few runs. And those tests are also being added to the recommendation list. And another important element to add to the recommendation list is new tests. So it could be that for this build you develop some new tests and you need to run them as well. And the fourth element is what we are calling pinned or flag test. So that's your ability to mark certain tests that you're saying, I must run those tests regardless of what was the code change. And then with all these four elements of impacted test, fail test, pin test and new test, you're getting the full recommendation list.

So once you get all this information.

How does it present it is in a dashboard. And then someone needs to manually then create a test suite based on those recommendations and then run it. Or is it some sort of automated process that takes your recommendations and automatically talks to your CI CD backend?

Yeah, that's a great question. There are three ways to consume the data of the recommendation list or the smartest selection, the first one, we are providing an out of the box integration with the CI. And specifically if you're using TestNG, Junit and Mocha, this is for the current release that we have. We are also supporting an automated execution. So you don't need to do anything. You just click on a button for clights to do the test execution. We will do the selection and the execution based on that selection automatically from your CI. That's the first way to consume the data. The second one is you can execute the recommended list and consume the data through our API. So for any test stage, any test with you, simply have a simple API call and you are getting the recommended list and then you can take that list, you can change the order if you would like, or take that list and apply it to any maybe custom test frameworks that you might have. And the third way is typically that would be for manual tests. You can consume the recommended list via our ui. So you have a list after each build. We have a list of the recommended tests so you can consume it through our ui, our dashboard and also through a downloadable file.

Another thing that comes up all the time when you speak with management about these tools, they're like, oh, what's the real time savings we get out of this? So any features to help show them how much time was saved or improvements based on this, this technology?

Yeah, very early in the process when we were working with our design partners and interviewing, I would say probably dozens and dozens of QA managers, QA Directors, VP of Engineering. The time saving analysis is critical and actually that's one element that we added to the product where you can see the time saving analysis for each build per, each day, per week, per month and per year. So it's very easy to see the time saving. And you know, we were so surprised with the results. You know, it's amazing. You know, you can see within a day you can cut three hours.

Typically what we're seeing, 50 to 80% of your time can be saved simply by not waiting for tests that you don't need to run.

Wow. So that's a real percentage you've been seeing for your active customers. 50 to 80% time saved.

That's the initial results that we are seeing. Yes, it's not applied to, you know, it really depends on your test architecture. But this is another thing. If we're seeing that you don't have those saving, actually it's a good indication that probably you need to change your regression test architecture. Because what is the problem if you have too many tests that are testing the same things. We will recommend all your tests and that means you're just wasting time. And it means that your testing architecture is not modernized and you actually, it's a good indication for you now to go and build a new testing strategy where you can have a deterministic test. That's very, very important because you don't want to. You know, in the past a lot of organization were.

Developing their test automation strategy by running tests that are actually dependent on the previous test, in many cases on the previous test data. Because in the past you didn't have this technology of test impact analytics or smart test selection. But now that you do have it, what you really need to do is to make sure that each of your tests is a deterministic test where it has its data and you can run it on its own. And then you can really take the, or use the benefit of the test impact analysis.

That's great advice. So that makes me think of another question though. When you do give the recommendations, if you have say a lot of end to end tests that actually have a lower level test that covers that functionality, is the tool smart enough to know that? So when it makes the recommendations, it says, here are the unit tests you need to run. You can order 50 of these UI tests because they're basically doing everything that unit test I already have covered.

Currently not, but we have all this data and in the future this is something that we will add just to add on top of that, it also means that with this data we can also give you an analysis about your, you know, redundant tests as well in general. So you can start maintaining your test by removing tests that you, you don't need. Today we are giving this raw data. You can simply download all the data and do any manipulation of the data on your own. In the future, we will look to see to build some additional analytics engines that will show you those redundant tests, redundant test, cross different test stages. As you mentioned, there's a lot of things you can do. This is why I'm so excited about it, because this is just the first step in the new way to manage quality in software development and QA organization. There's so many things you can do on top of this data.

All right, so Eran, I just want to make sure I understand the way it works. You have a recommendation for tests you should run. Does it run automatically for you? I think I asked it before, but I wasn't clear. You give an example of how that would work?

Yeah.

The C Light Smartest execution actually.

Identified the recommended test and also execute those tests for you through the CI at this point that is enabled if you have a testng, Junit or Mocha test frameworks. If you don't have these, actually you can consume the data from our API and simply apply it to your test automation framework. And the third way to use it is through our UI or download the file of the recommended test. This is typically done for manual tests.

Very cool. I think you actually have a very extensive REST API backend that people can consume. They're not stuck to your dashboard. They can actually, like you said, hit a REST API, maybe write a Python script to do whatever they need to do in one of their funky backend systems.

Exactly, exactly. Actually with the test Impact analytics, if you don't have outside from manual test, you don't need to go through the dashboard. It's actually you integrated to your infrastructure, to your CI and that's it. The only impact that you will have is that you will wait less time for your test to be completed.

Very nice, Arun. When I first talked to you, I think it was maybe two years ago, a little over two years ago. And you seem to continually be adding features to this product. I guess the first thing is when you add a new feature, is that like a paid, Is it always a paid component or is it always different depending on how big of the new feature is?

It depends on the feature, but currently for our customers, when they buy the product, we are giving them all the features.

Nice. We may have spoken about this before. How do you know? Like, how do you organize? Like, there's probably so many things you want to do and it sounds like you have on your back, like, how do you choose which is the next highest priority to work on?

So a lot, a lot, a lot of conversations and meetings with our customers, that's one thing. And second.

We have a very talented team and I would say they're very smart and very curious and they always like to learn. So we are reading a lot of materials on the subject. But I can tell you with the Test Impact analytics, with the smartest selection, that was.

Outstanding, I would say feedback that we got from everyone that they needed. It's a critical problem. They've been dreaming about such a functionality for many, many years and they're very excited about it. We are very excited about it.

Yeah. This has been on my want list forever, so this is very cool. So previously you did mention parallel tests. Does this help at all? But parallel testing, does it have any sort of features? That helps you do parallel testing better.

Yeah. One of the hardest things in Test Impact analytics is to isolate and identify.

I would say, the code footprint of each test. Now when you're having tests that are running in parallel, it's very hard to know which test is covering which code.

We are actually using several algorithms, including machine learning to identify it. And the more that we are running the, the accurate data that we have.

So you don't need to, it means you don't need to change anything. You don't need to do a special calibration run, you don't need to set another environment and run your test in a serial mode. Simply run your test. Most of the organization are running the test in parallel. We know how to identify each test and what it's for footprint and so you can get the test impact analytics. So that, that's one of the, I would say, features that we are very proud of because it's very hard to do. And of course we also issued a patent on it. So that's the, I would say the parallel test support.

Nice. Now, I was speaking with someone the other day and they had a cool solution and then the question came up, oh, it only supports Java and someone wanted to use it with C. So is this technology like, does it only work with Java and Git or how flexible is it?

Very flexible. We understand that when you want to apply Test impact Analytics to integration test to end to end test, in many cases those testing configurations are much more complex, meaning they would have multiple programming languages. You might have JavaScript for the front end and Java or C, C or Node for the backend. So we need to support those distributed environments, microservices environments, multi programming languages environments. So that was all designed in the solution.

All right. So Ron, I know a lot of times people get a technology and sometimes they overuse it or they don't know where it applies or they say it doesn't apply to me. So where would you say Test impact Analytics would, where would it be used best used for? What are the scenarios, the situations you think someone would get Big bang for the benefit using it?

Yeah. So I see three main use cases. The first one is for developers who feel and see that they are waiting too long for their pull request or for their ongoing builds or when they're submitting code changes to their CI and they need to wait for their automated test to be completed. Not just the unit test, but you're running through the CI, the service test, component test, API test, some integration test. In those cases where you Do a code change and you need to wait for the CI to complete your test. That's an excellent place where you can use the test impact analytics and cut the the waiting time from I would say 20 minutes, 50 minutes, an hour to few minutes. That's one use case. The second use case is I would say after you merge to the master. This is where you're running your more high level.

Tests. The QA automated test, the functional test, the system test, end to end test or you have a lot of automation there. That's another great place to use the test impact analytics. Because these tests are very expensive to execute. They take a lot of time to execute, they require special environment setup. So you can save lots of time on this type of test of the QA end to end system test. And the third place where you can get a lot of benefits from test impact analytics is for manual tests. Because these tests, you know, can take you many hours, sometimes multiple days. And using a smart test selection for this can, you know, get those days into few hours.

This sounds really cool. I think it's going to help a lot of folks. I think what some people may be thinking though it sounds complicated. So how hard would it be to get started? Do you have some sort of onboarding process or something to help guide someone implementing this solution?

We have an onboarding, a simple onboarding process and actually we also have our customer success team which will be happy to do a remote session. We typically start with a basic poc. So the cool thing you just need to integrate with the CI. There is a simple plugin for Jenkins or for the other CIS and you just need to drop our test listener and add it to your deployment scripts. This is where your the deployment script that you're using to deploy the new builds to your testing labs. That's practically it. And we're just analyzing the data. So the cool thing is that we are just analyzing few builds and immediately we are showing you the the time saving analysis, the potential time saving analysis for each of your build and you can do an assessment over a week or over a month and see the result. And if you're happy with the result, then you can turn on the test execution feature or use the API to consume the data and move from recommendation to actual execution.

Okay Ron, before we go, is there one piece of actual advice you can give someone to maybe help them with their test analytics in general or test analytics using your tools and the best way to find or contact you or learn more about Sea Lights?

So the best thing is to Visit our website www.clights.IO. there's lots of data there. We're actually writing a lot of white papers and blogs and if you're interested in a demo there is a demo request.

Form and.

We will run a demo with an expert and going over the use cases a personalized demo. So.

That'S the best way to learn more about test impact analytics and smart test execution.

Thank you Aran for your automation awesomeness. Flinks everything to value we covered in this episode. You head on over to test talks.com267 and while you're there, make sure to click on that Sign up for a free trial link under the exclusive Sponsor section to learn all about Sauce Lab's awesome products and services. So that's it for this episode of what's going to be the Test Guild Automation Podcast. I'm Joe and my mission is to help you succeed with creating automation. Awesome Ms. As always, test everything and keep the good. Cheers.