224: The Need For Velocity with Eran Sher

Main thing that will happen in 2019, we will see this transition to not just AI and machine learning in test automation, but managing quality through data analytics and not just opinions and feelings and experience.

Welcome to the Test Talks podcast, the place to go to geek out on software testing. And now your host whose mission is to help you succeed with tests, Joe Colantonio.

Hey, it's Joe. And welcome to another episode of Test Talks. In this episode we'll be test talking with Aaron Sher, the CEO and co founder of C Lights IO, All About Software quality, analytics and why analytics are so important in today's fast paced software development and testing world where we all find ourselves in. So sit back and get ready to discover actionable tips on how you can track and optimize your testing efforts to increase your team's velocity without sacrificing quality. You don't want to miss this episode. Check it out.

Test Talks is sponsored by the fantastic folks at Sauce Labs, the cloud based automated testing platform that eliminates the need to maintain your own selenium grid and and test infrastructure. Try it for free today. Visit testtalks.com and click on the sign up now link under the homepage Sponsor section.

Hey Ryan, welcome back to Test Talks.

Hey Joe, happy to be here.

Awesome. So today I'd like to talk about software quality intelligence. Before we get into it though, you were on the show about a year ago in episode 147. What have you been up to and any major changes to C Light since that time?

Actually it was a very interesting year because we managed to speak with hundreds of engineering software engineering leaders.

Quality assurance leaders, DevOps leaders, and I think that the main.

Message to the market, what we are hearing from the market, is the need for velocity. We are seeing that software development is changing and most of the changes is around velocity of development and then of course velocity of testing as well. And that impose a lot of new challenges on software development teams and on quality assurance teams.

Awesome. So along those lines, were there any features then that you've added to the product that you thought, wow, there's really a need for this and maybe we need to tweak something to get that handled. For all the type of requests we're getting to handle Velocity, I think there.

Are two main new offerings that we've released while understanding the need in the market. One of them is the problem of efficiency.

We see that too many unnecessary tests are actually being developed and you need to invest a lot of resources in develop those unnecessary tests, executing those unnecessary tests and then maintaining Those unnecessary tests. And the reason is that there is no clear data or report that can show you where are your gaps. You know, a data analytics that can scan all your builds, all your tests, the unit test, the component test, the API test, the manual test, the automated test, and then provide you a picture that will give you the data. Where are your testing gaps, where are your quality gaps, where you need to develop tests, and also where you don't need to develop tests. So this is.

One of the major new capabilities that we release. We call it test gap analytics.

Awesome. So actually I should ask before we get into test gap analytics, what is software quality intelligence? I think that's a a term I've been seeing on your website. And is that a different piece of the software or is that a higher level that also includes test gap analysis?

That's a higher level that also includes test gap analytics. So software quality intelligence is a term that we came up with. And the idea is that.

Software development is becoming so fast and so distributed that it's becoming very hard to determine to determine what's your quality. Actually, there are three main challenges with quality. It's very hard to determine what's your quality. It's very hard to determine which test you need to execute. This is why everyone execute all their tests all the time. And we know that it's going to be a bottleneck because you keep adding tests all the time, especially in continuous integration, continuous delivery environment. And the third one is that it's very hard to determine which test you need to develop and which tests you don't need to develop.

So quality intelligence, the concept is the ability to use data analytics that can provide you real time data and offline data to answer those questions. So it's a platform that automatically generate and collect all kind of data sources and provide you with a real time view to show you where are your testing gaps, where you need to develop tests and where you don't need to develop tests that can show you or can analyze which tests you need to execute.

According to code changes in builds. And it can also help you through data analytics, determine the quality of your release by comparing it to previous releases, by looking at lot of metrics over time and compare between them. So I would say quality intelligence platform is the ability to use data analytics to help you with quality analytics. So we see, I would say, two major planning processes in organizations today. One is Sprint planning.

And I think everyone would agree. We've seen it probably with 99% of the calls that we did that Sprint planning is Mainly focused on defining the user stories for the coming Sprint. And then the planning of the quality activities or the test development activities are pretty straightforward. What the automation engineers or the developers who need to develop the functional test, the integration test, not just the unit test, there have been tasked by guys go develop test to those user stories. And the definition of done would be that these new tests for those user stories will pass, will pass successfully.

But we also found that there is no way today to actually verify that those tests were developed properly, that there were no code changes that were done as part of the Sprint, that those tests were not actually testing it at all. You don't have this visibility, that's one thing. The second thing, and that's even more important that it's not been handled at all. No one is actually handling in the Sprint planning the quality technical debt, the testing technical debt, because everyone knows that they have testing gaps, but they don't know exactly where are those gaps. So once we introduced the test gap analytics, now the scrum masters, the product owner, the dev team leader, the QA architect, the test architect. Now as part of the spring planning, they're able to pull the test gap analytics report, understand where are the high risk gaps, and then decide if they're going to develop specific tests in the that specific sprint to start closing the gap.

I, I speak with a lot of, I have a sprint, a team with a lot of sprint teams and so eight to 10 sprint teams. And a lot of times even when they're working on the same features, they don't know how the other person is going to affect what they're the part they're working on. So is this something that would help people also when you said playing their Sprint, to know that they may have requirements and they may have tests for those requirements, but they may not know what they don't know. I don't know if that makes any sense, but it's almost like a lot of times when you're creating code, you don't realize that maybe another piece was touched that you didn't plan for. Even though you think you're actually testing the requirements that you had ahead of time.

Yeah, it does. Because if you generate the test, get analytics, actually you generate two data points, you can see the last sprint end result and understand all the code changes that happened in the past Sprint and are still not tested. That would be that retrospective that you're talking about now the data is visible and you can use that retrospective data in order to plan the next sprint.

Awesome. So what are some Other challenges you've seen when people try to implement software quality intelligence.

So I would say the first challenge, as I mentioned, is understanding the gaps. The second challenge, challenging actually.

It'S a change in the mindset. We're starting to see lots of development teams as well as software engineering team leaders that they've been tasked with quality assignment. They don't really know what it is because they are not coming from quality assurance background, they're coming from software development background. And now they have quality.

Automation engineers and they need to manage them. And it's very hard for them to validate what is the scope or to validate the test results or to validate the test themselves if they're actually testing the code that is being changed. So this is another major change in how the, the planning and the retrospective is actually done. But if the data is now accessible and visible, you can understand better how both to guide your testing activities or how to better plan your testing activities and as well as how to validate that they were done properly.

Awesome. So this is pretty much when a team's assessing how good I guess their the test are, the quality of the test are.

Yeah, because now everything is, is moving very, very fast. You have lots of code changes done by different people, you know, across different micro services. You know that Velocity is creating new challenges and you need to get the the data in order to better plan your test. Think about how tests are being planned today. The test start that the planning starts by specification or requirements. You start writing or develop the tests according to the functional spec. But then after the test is done a week later, the code can not the code of the test, the code of the tested application can be changed dozens of times. Is your test still valid?

You don't know. You need data to help you to validate it.

Awesome. So how does this help Velocity? Does it tell you then you're wasting time running these tests or this test has no value. So therefore you're able to really have a targeted set of tests that really are testing the changes that are being checked in by your developers.

Yeah, that's one capability, we call it test impact analytics that help you to analyze code changes in a specific release and then match or select which tests would.

Are suitable or have the high probability to find bugs.

According to the code change. But I think there is a simple, simpler way because think about.

If you can analyze the gaps in your tests. So for example, you know, think about a system that can analyze all your tests and let you know what's not tested, what skipped all your tests. So you ran hundreds of thousands of unit tests and you ran hundreds of automated tests, and you run dozens of manual tests and probably hundreds of component tests and so on. After all those test execution, what's still not tested? Cross all your tests. If you can get this data as well as correlated with all the code changes that happened in the past month, in the past two weeks, you will get a view of all the code changes that happened and none of your tests are testing it. So if you will be able to get this data, you now will be able to develop tests to those recently code changes. Now, what's interesting about it, you're going to gain two things. One, we know that if someone is doing a code change and that code change is not tested at all, you have a high risk for a quality issue. So if you develop tests to those areas, you're going to immediately improve your quality or decrease your quality risks. But more interesting than that, think about it. You now have a way to direct your test, your regression test, your integration test, your end to end test, your functional test. I'm not talking about the unit test, I'm talking about the high quality test. If you can direct those tests to areas that your developers are changing frequently, you're actually putting now a very robust infrastructure for regression safety net. Because those integration tests now are focused to test areas that your developers are changing frequently. And if you have no integration test for those areas, you are allowing your developers to change the code and to deploy it very rapidly. Now to production, because your integration test, your end to end tests are now covering that and that's what's enabled Velocity.

Awesome. So a lot of times, even when you have good dashboards, it's hard to get the teams to pay attention to them. So in this type of solution, how does all this information bubble up? Do you have to do a lot of analysis or is it just be able to look at a certain tab to know, oh, I need more integration tests in this component or something like that.

So originally we had dashboard that shows you all your builds, all your components, all your micro services. And you can look at the last build and see exactly how it was tested and what's the status. But for this Sprint planning, what we develop is a simple report. And actually you don't need to do anything, you just need to click on it and it will show you all the gaps that you have. And based on that report, probably it will take you 10 to 15 minutes to review the report. Think about, you're in a sprint planning, you set 15 minute minutes with the team to understand your gaps and then based on that, you can take a decision which test you need to develop for the next sprint. And within few sprints that you're working with the test gap analytics, you will see that you are closing those important gaps, those risky gaps.

And then you enable your developers to move faster. It's actually raising another point. And I think this is a major change of mind.

You know, today people look at quality and they try to tie it to coverage level. So they say, you know, my unit test needs to be at 60% or I want to see my coverage increases from 40% to 80%. And that's an indication for quality. General speaking, it's an indication. But we all know that coverage is not the most important metric. Actually the way we are looking at it is you can be at 30% code coverage, but still you can have high quality. Why? Because those 30% are covering the interesting, the important, the high risk areas. So we want to change, you know, our customers, our users are changing their mindset from looking out, just coverage on looking at how to maintain a low risk testing technical debt or quality technical debt. So the key here, it's not to close the gap, is to maintain it in a low risk mode. And this is, you can, in that way you can be very efficient.

Awesome. And that's a great point because a lot of times the coverage will still exist. Even if the code is dead or it's unused in the product, it still shows us as being covered. So does this allow you to know that hey, you may have a lot of code coverage, but it happens to be in dead code or in features that you users don't even use.

Yeah, we can show you the areas where you have your gaps. And we correlated with the recent code changes. So you can see that you might have test to areas that are.

Not changing at all, but on the other end to areas that are recently being changed by your developers. You, you don't have test. Or we can show you that for, let's say critical files maybe you have, you know, core calculations, you have logic, you have lots of APIs. You don't have coverage at all. You don't have coverage for end to end test. You don't have, you have gaps in your end to end test, you have gaps in your component test. You can slice and dice the data across all these for each test stage.

Cool. So I think we may have covered this in episode 147, but for the folks that haven't heard that you Mentioned end to end testing a few times. And most people, like you said, are familiar with cold coverage for unit tests. But can you tell us again how you actually get coverage for UI functional tests? Is that more of a model that someone needs to come up with beforehand or is it actually smart enough to know that you have these selenium tests and these selenium tests map to this piece of the application?

So it's not functional spec coverage. It's not coverage in the way that QA teams or testers are used to. It's actually what we believe is that everyone is moving to automation. It's transitioning to automation. QA engineers today are programmers and actually what we provide them is test code coverage. So when you're running your end to end test, or you're running your UI test, or you're running your functional test, most of those tests can start in the UI and then.

Go to the backend and cross microservices. We know how to capture the actual code that was executed. So now the same as developers today know what is their unit test coverage. We develop the capability to show every QA engineer, every automation engineer, the coverage of older tests after it's deployed, after you're running testing in complex environment, cross microservices, cross distributed servers, cross services, whatever, you can get the actual code coverage as well as for manual tests.

Now how does it get that though? Do you have to instrument your test or does it know based on.

You don't need to instrument your test. We are using it's called dynamic instrumentation technology. It means that once you deploy your build, you have a wrapper C lights wrapper. We call it test listener. And that's it. You just continue to run your test as you're used to.

So just add to your deployment. When you deploy to your testing labs, you also deploy our listener together with the code change. We have an integral with the new build that you're testing and we have a simple plugin to the most common continuous integration servers like Jenkins and TFS and others. And it's seamless.

Does it add any overhead that you've seen when you run this technology on the machine?

Yes, there is some minimal overhead as any instrumentation technology would have if it's monitoring, if it's regular code coverage, but nothing that is, you know, too serious to prevent you from doing functional tests. We do have customers running it with performance tests and we didn't see any issues. So there might be, but it's not significant.

Cool. So I know most people listening are probably very familiar with their CI systems. Again, statistics about the unit test with coverage, pass, fail rate, all that type of information. But where I see a gap is finding that exact information for manual test or functional test. Where I see that gap is with the functional Selenium test. So does your technology also work with functional tests using technologies like Selenium?

Yes. So the technology that we developed is actually allowing now developers, QA engineers, automation engineers to really get their actual code coverage for any test, not just like developers used to have. Unit test, code coverage. Think about the capability that you can have code coverage for any test that you're running. Functional test, component test, performance test, and even manual tests. And no matter how complex is the deployment or the configuration, you might have a complex deployment with the UI and databases and backend servers and bus servers, whatever. And now you're running a test that's running through all those microservices and distributed servers, you will know what's your actual code coverage across all your components. So that's the technology that we develop. You don't need to do any changes to your code. So this is also very cool. All you need to do is to use our plugin to the CI system. So you just add our test listener to your deployment script, the deployment script that you're using for your testing labs, and that's it. And you're getting the data keep you do the test as you're used to execute your automated and manual test. And we're collecting all this data and immediately presented in the dashboard.

Awesome. That's the next thing I was going to ask. The machine's already instrumented and it's collecting data. Does it do anything else besides coverage? But that's your thing. But like performance metrics or anything like that. Any other functionality that's included that people may not be aware of.

So we're collecting lots of data. It could be the test status and the test duration and the other thing, we are collecting this data over time so we can do lots of correlations and trend analysis. And of course we are also collecting all the code changes data so we can correlate the test data, the coverage data with the code changes data and do everything over time. I'll just give an example today, if you want to know what's your, you know, what your tests are covering, you need to run all your tests on a single bill and then understand what you're covering or not covering. Our technology allows you to collect it over multiple builds. You know, the first build you ran, your unit test, the second build you may run some other manual tests. The third build, you run the unit test, but build the build failed. So only half of them been executed. And then the other build, you run the functional test. We know how to read all these data and by the end of the week or over a period of time, understand exactly what you're testing and what you're not testing without making any special, you know, changes to your processes.

Awesome. So I also, as I was researching, I noticed the phrase the test less approach. So it sounds like you're testing less, but you're testing more. The more critical things. Is that how with the test less approach is all about.

Exactly. And the approach means that you don't need to develop too many tests, you need to develop the right amount of tests. And if you develop the right amount of tests, it means that you're developing less tests, but you're actually testing more. But when you develop less test, it, it means you're developing much faster because you have less tests to develop, you have less tests to execute. So you're improving the execution time, the testing cycle time, and you have less tests to maintain. The one condition is you're writing less tests, but they have higher impact than what you have today.

No, I think that's awesome because a lot of times I've also seen people that have their test cases keep growing and growing and growing and they never prune it. So it sounds like using this technology you can say, okay, maybe two years ago these tests were valid, but now they're not adding any value, we can just get rid of them. Like I said, you were on last year and you have spoken to a lot of different, lot of different users and a lot of different developers and such. So you did sound like you made some sort of pivot. So is there one core thing you learned during that year that made you change the direction or is it just you kept hearing the same thing over and over again about these quality gaps and that's what made you stop focusing more on that aspect?

We keep hearing it more and more. If originally we were focused on the release quality analytics in real time, we understood that there is a big need to get data analytics that will help development teams and quality assurance teams with their planning activities. And this is the big addition that we made to help get data analytics that will help them with Sprint planning, with quality technical debt planning, with integration, and end to end test development planning.

And so.

What do you think the new role will be for QA managers as we go forward in this kind of. It's almost like AI, machine learning type technology, where we're getting all these analytics and we need to make decisions based on that?

Yeah, it's a good question. Actually. We've seen it in a lot of organizations that the QA manager is actually transitioning into a new role. On one aspect, the automation engineers. In many cases we see that they've been transitioned and now being managed by the software development manager. And the QA manager is responsible for standards. And we think that by using data analytics, the role of the QA manager can be actually changed. Because when you're using data analytics, you're able to provide to the development teams and to the automation engineering teams a better guide, not just about standards and how to develop this, but to really direct them. Where are the gaps where you need to develop this? How can you be more efficient? How can you develop tests that will create this regression safety net that will allow developers to move faster? So using data analytics, the QA manager role, a QA manager can use this data to actually make more impact in the organization, provide quality infrastructure that will help development teams and automation teams to develop again to be more efficient with the test development, develop the right test to the actual important gaps, and to use data analytics to assess the quality of the release.

So there is a whole new, I would say, space here and responsibility and impact that QA managers can now make in their organizations.

Yeah, I really see it empowering them and also helping them deliver quicker and faster. Because a lot of times where I work we have a process called VNV where we actually put it into a blessed environment by IQA people and we run the test and some of the tests may fail and we have to have the developer teams get out of this sprint and start debugging why it's failing. And a lot of times that test has nothing to do with that particular release. So this sounds like almost like not only will it tell you what to run, but once again you're only running what matters. So when it does fail, you know, this is high priority. And the QA manager is then has real power to say no, we really do need their time, because this is an actual issue.

Yeah, absolutely. That's one of the. I think it coexists because on one hand, you know, the AI and machine learning test automation.

Software, which is mainly focused on generating new tests, is great if you will combine also the analytics data that we provide, those tests can be directed to develop tests to the right areas and not too many tests, only the tests that you need. So you will gain from both worlds.

Okay, Byron, before we go, is there one piece of actual advice you can give someone to help improve their software quality intelligence and let us know the best way to find or or learn more about the solution.

So my best advice, not just for software quality, is use data. It's much, much easier to make decisions where you have data and you make better decisions. And if it's in software engineering and quality, now you have the data, so use it. And for more information you can also Visit our website www.sealights.

And there are a lot of resources there and white papers and lots of webinars that we're doing on quality in general, on software development, on test metrics, quality metrics. So lots of info there.

Thank you Aran for your software quality intelligence awesomeness. The links to everything of value we covered in this episode, head on over to test talks.com2two4 and while you're there, make sure to click on that Sign up for a free trial link under the exclusive Sponsor section to learn all about Sauce Labs awesome products and services. And while you're there, make sure to click on our 2019 Automation Guild banner. Automation Guild is the largest conference in the world. On my dedicated 100% to helping you create test Automation Awesomeness. I have an awesome lineup coming together. I think you're really going to dig it. Check it out@automationguild.com so that's it for this episode of Test Talks. I'm Joe. My mission is help you succeed with creating Automation awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Talks podcast. Head on over to www.testtalks.com for full show notes, amazing blog articles and other automation awesomeness.