Scientifically Optimize Your Processes using AI with Aviram Shotten

Welcome to the Test Guild Automation Podcast where we all get together to learn more about automation and software testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Automation Podcast. Today we'll be talking with Avram Shannon all about scientifically optimizing your process with AI and probably other quality testing related topics. Avram has been in the quality for over 20 years and he's filled many roles. So he has a lot of experience, a lot of areas. Really, really excited to dig into this with him, I believe. He is currently the Chief Solution Engineering Officer at qualitest. And if you want to know anything about quality and especially about how quality is changing with these tools that can help you now, you don't want to miss this episode. Check it out.

The Test Guild Automation Podcast is sponsored by the fantastic folks at Sauce Labs. Their cloud based test platform helps your favorite mobile apps and websites work flawlessly on every browser, operating system and device. Get a free trial. Visit testguild.com sourcelabs and click on the exclusive sponsor section to try it for free for 14 days. Check it out.

Hey Avram, welcome to the show.

Hey, good to be here.

Awesome. So, before we get into it, is there anything I missed in your bio that you want the Guild to know more about?

I think he did a splendid job. I do think that experience is important and I'm here for a long, long time. I've seen things that have been rather successful and I've seen things sinking faster than a Led Zeppelin. So no, just appreciation of I'm here for a long time.

Awesome. Well, that's what I like. You've been here for a long time and there used to be a joke that development had all the innovations going on, but I've spoken to over 300 people on my podcast. It seems like the past few years though, it seems like testing and quality, there's been more and more innovation. So with all your years of experience, can you give maybe a little bit explain how you see quality today and how it might be different from where it was when we both started back in the day?

Sure. Look, I think that in reality, testing used to be perceived as a necessary evil, while testing is actually an enabler to make sure that you delight your customers with an amazing software. The way that testing transformed in the recent years starts with agreeing on what I just said. It's not a necessary evil. It's basically a key process you would like to install. That on the one hand will not slow you down because you need to deliver very fast and you need to deliver a lot, but still to maintain a level of inspection and testing, to assure that whatever you go to market with.

Is fit for purpose and your customer will love it. Once you understand that, all of a sudden the dynamics, the psychology of testing has shifted from a necessary evil to let's do the best thing we can given constraints. And that enabled us to, to be at the position today where testing is fast, testing is innovative, testing enables faster, safer launches of software, and basically testing enables innovation because in the end of the day, innovation needs to be safe. You cannot risk your business, and that's exactly what quality assures you.

So I think actually that's probably not many people think of is that testing is innovation. I've never heard that term before. Can you just talk a little bit more about that? Because I think a lot of people see it as the opposite, especially as you talk about creating software faster. The first thing that comes to mind in most large organizations, that we need to get rid of QA and streamline it. Not necessarily that this testing actually is innovating and it's helping us create better software for our customers.

There are two types of innovation. First of all, test innovation. Those are really exciting things that are happening right now. From your sponsored source Labs will bring in the ability to test across multiple devices, multiple browsers very quickly and to get feedback very fast. AI in testing, which I'm sure we will be discussing later on in the conversation. And this is all about innovation in testing. Testing is enabler for innovation. It's testing in the process of innovation represents the balance. We can't be innovative without being safe or at least be able to analyze and understand what level of risk we assume when going to market very fast. And that's exactly what I meant. I meant that testing is a facilitator of innovation by addressing and presenting the risk we assume by going to market very quickly.

Absolutely. And I think I saw a webinar that you gave and it talked about QA optimization. So I definitely want to dive into AI. I think AI is a piece of this, but at a high level. What is QA optimization how you see it?

So QA optimization is basically the opposite of vogue standard approaches to fix testing gaps or quality engineering gaps. When we say QA optimization, meaning that we apply a standard process to evaluate perceptions of what are the gaps around software delivery lifecycle, the testing lifecycle, and compare that with reality. So it might be that one can think that test automation is its biggest problem, but if we take the perceptions and we will compare them with the data that this specific project or department has. We will be able to approve or disapprove their perceptions. So I always come back to this great example that we were challenged that there isn't enough test automation. When we applied the QA optimization process, we realized that actually there was healthy amount of automation, but it could never reach to a point of utilization because lack of environment and lack of data and lack of ability to analyze the results of automation. So basically, even if you had 2x or twice as much automation, you would still not benefit, not in terms of quality, not in terms of velocity. So Q optimization takes the perceptions, put them against reality, and we use very advanced method to establish reality. We use machine learning and advanced analytics to identify where the problems are. And then we're showing how to get to a place where we educate people about the challenges. We show what the challenges are in almost a scientific manner. It's no longer suited and booted consultant. It's about the data and the implications of the data and the roadmap to fix it. And every customer and every project we address with QA optimization, the results are slightly different. We don't believe that there is one size fits all. There's probably few sizes fits all. And you have to it not only the things that are right, but you also need to address the biggest gaps and the showstopper first. And those are all identified and visualized by QA optimization.

I think this would actually help a lot of companies because I worked for a large enterprise. Whenever we did planning for an upcoming release, it seemed like there were a lot of assumptions right? Just like testers had some assumptions, but they couldn't really tell you why they felt that way. And the assumptions ultimately usually ended up being wrong, like once it got in the wild. So it seems like you're taking a different approach. It's not just like quality optimization is writing more automated tests, it's actually gathering all this data that's probably available throughout your whole software development lifecycle and pulling out maybe insights that are unique to your organization that you need to address in order to optimize your QA strategy. Am I understanding this correctly?

You're absolutely right. And when I was under speaking or I was under appreciating how important it is to be very contextualized, those assumptions we use, they are helpful, but sometimes we miss the mark in terms of what we actually have, how efficient we are. Let me give you an example of a very large UK enterprise we surveyed. We applied the Q optimization process. And basically there was a big mismatch between management perception of how much testing needs to be conducted, the test team understanding of how efficient they are, and what are the most value adding test activities. And you could see it in the survey, right? There was a big gap in terms of what people thought about efficiency, etc. And when we analyzed the data, we analyzed test cases and defects, and we used the NLP to do that. And the syntax that was used to describe the test cases and the syntax that was used to describe the defects was almost a complete mismatch. That showed us that while they invested thousands of hours in planning and test design, they actually found all the defects elsewhere. They found them using exploratory techniques, they found them using automation. And if we will all be a little bit more scientific in how we operate and strive to get better data to base our decision on, I think we will find ourselves working hell of a lot less. I hope it makes sense.

Yeah, like how do you get better data? Because there's a lot of data being produced and a lot of people just either getting overwhelmed or like you said, they're looking in the wrong place for the data.

I don't want to be inclusive, I don't want to say everyone are doing the same mistakes, but I think that sometimes we give too much weight to our gut feeling and our what happened last time around and our subjective understanding of what happened last time around. We should be more careful with that. The basics is get your, you know, testing business intelligence right, make sure you have your testing dashboard installed, make sure you have visibility into defects versus test cases execution, where the defects are clustered, who finds the defects, which method, once you have that, you're much, much smarter. I'm not even talking about apply nlp. This is for advanced organization, who can afford that. But doing the basics right does not require a lot of investment. It more requires the discipline and understanding that you want to be data powered rather than gut powered.

So if you are analyzing the data, then how do you find correlations, maybe in different data points to know that you're maybe using the data to create a correct strategy?

We created a framework to look at all areas of testing and we don't think that we will ever be able to drive consensus without, or drive things forward without consensus. If we won't be able to show people that there is a structured process to review all the interfaces of testing and within testing that we touched on all the key areas and we actually collaborated and connected with individuals. Whatever process you will install will not be Successful. What I'm trying to say is that yes, we developed a framework that looks at testing through the perspective of external user, external client to testing, if you'd like to, and the internal test stakeholders, the managers and the engineers. We're doing that on the one hand, we're using data to make it less objective, more objective. This is what you guys are having and it's stated in your data. But we are doing this by driving consensus and by communicating frequently throughout the strategy process to the findings, to the changes we recommend we make it somewhat democratic so people can vote about priorities. As an example, there isn't a process that is complete without technology and people and vice versa. It's about getting all things harnessed together so whatever we will come up with will be executable and easy to understand.

So how do you implement this though in an organization? It sounds like it's a lot of work, your framework, other steps that are easy to implement? Or is it gradual? Is it a big bang type of implementation?

So this is our biggest concern, that people will lose interest. They will bring on a team of people like myself or my team or my company and they would like us to advise them on how they can be better in quality. And all of a sudden you come with this very fancy report and people lose interest halfway because it takes time to see the outcomes and it takes time to get the return on investment. The first thing we apply is a very ROI oriented framework. Our plan usually ends in six months time and more than 70% of it needs to be demonstrating value within 90 days. So basically we're working with 30, 60, 90 and 180 day plan. And if it's not fixable in six months time then you might want to take it aside. If you need to drive a bigger change in the organization, a longer change, you don't know what will happen in six months time. So we are limiting ourselves to something that will keep people engaged. If we want to drive, for example, to drive more in Sprint automation, there has to be a process that shows value in 30 days and probably converges in three months so people can see the value of in Sprint automation, how can we do day one automation if we are going to sell this promise to someone and it will show it will bear fruits in a year from now, I promise you, too many things will change and this initiative will dissolve into error.

Absolutely. I guess a lot of things that ship people up though at this stage is being able to measure like in 90 days, like figuring out what the correct APIs, KPIs or metrics are. So any tips on what to track with KPIs and metrics when you're implementing this type of strategy?

I'm a big believer in looking after the trend. I don't think that and a lot of things are relative. Some people demonstrate higher degree of satisfaction versus other people that a certain period will show lower degree of satisfaction. So I would like both subjective and objective measurements to be checked over time. To check over the trend. I would look at methodologies like the knowledge innovation assessment that we developed in Qualitest, but also there are other industry standards like TMMI and like tpi. And wherever you start from, just make sure that when you engage again after a year you got better. The key areas to look at of course are defects, test case utilization. You need to come up with meaningful productivity metrics. How many test cases executed is less important, probably because it can be anything between 100 to 10,000. What is the value of test cases? How many test cases have identified issues? How many test cases were successfully executed in the last 30 days versus previous 30 days? Those are things that you can analyze and make insight out of. I think there's also a big difference if we're measuring test automation and manual testing. Manual testing tends to be a little bit more stable because in the end of the day, if something is changed, the manual tester will be able to get over it and the test will be successful. Automation. However, it's not the case. Automation tends to break if software changes and if we don't have a really succinct process to capture changes, identify them, inform the test automation expert or the SDET within the insprintors, anything in that shape or form and make sure that the automation, when executed, is already up to date with the change. Automation will break. And measuring the success and the stability of automation over time is a key metric to prove the ROI on test automation is heavy lifting investment and people want to see the fruits of it via their, you know, peace of mind. If I had my automation running and nothing was found and it was successful, I can deploy safely. This is the best return on investment. So automation stability will be something that I encourage people to look at.

Absolutely. And also on the webinar, you had a webinar on scientifically optimizing your process with AI, I'll have a link to it in the show notes. But one of your team members, I think brought up the point that a lot of teams or they see a lot of companies not looking at change with the quality lens, so they may have KPIs and metrics, but they're just looking at like a technology or a process, not necessarily the quality output. So can you talk a little bit? Why a quality lens when you're thinking of change is so important when you're doing this type of framework?

If I remember correctly and if I understood the question correctly, I think that in the end of the day, you need to have the kind of measurement that will demonstrate the right things for your organization. Not all test business intelligence means everything to all people. Sometimes you're in the process of change and you would like to measure things that relates to successful transformation. And sometimes you just finished your automation depth, your backlog of automation, and now you want to look forward into how to integrate automation from day one. At that moment in time, your KPIs might need to be a little bit different. I also urge people to come up with the right metrics for the right stakeholders. Unfortunately, the things that are very meaningful to an ass, that can be less meaningful to a decision maker two levels above that person, literally in the same building. So a contextualized measurement coming up with the right KPIs for the right situation, the right project and the right Persona, I think will drive the most value and most alignment. Alignment is so important when it comes to software delivery. Alignment is important in general, but especially in software delivery.

Absolutely. So you didn't mention dashboarding. Is there any other tools that can help teams come together to really make quality bubble up to everyone? Not just an sda, but like you said, a high level CEO, a cfo, to make sure everyone's on board and speaking the same language, and that what one person thinks is important is important to the whole organization.

With taking the chances of coming across as too salesy For a company like Quality, the ability to be transparent and to show exactly where testing is, where development is. We use our own tool, surprisingly called Quality View. It's ready, just ready made with all the interfaces and it's so good to use. But the tool is less of the issue. The most important thing to drive is consistent use. And when you launch a test KPI or a test BI tool, make sure it integrates with the test cases, with the defects that sometimes stored in two different systems or managed in two different systems. Wherever you maintain your user stories, production defects, production tickets, if you can have access to all those into a single repository, you can drive very meaningful insights like escape defects, stability, metrics, etc. There are many great tools out there in the market. Ours is equally good.

Absolutely. I know you don't want to come off as a sales pitch and I appreciate that. But if someone is interested in learning more about the AI piece because a lot of times when you hear about AI, it's just AI focused on helping people write better automated tests, not necessarily helping the organization create better quality software. So can you maybe talk about is there AI built into this product that can help them with these type of goals that they might have that are beyond just help them write better automated tests?

So far we covered business intelligence Qualitest have a very well defined approach to artificial intelligence in testing. The reason is that is because we don't want to be everything to all people. There is no such thing. AI is very complex. It comprises of so many subfield and expertise. What we do with AI, we have the approach of use cases and so far Quality Test have launched three use cases that leverages AI in one shape or the other and they are very focused. It's not everything, but we made a lot of studies and we invested a lot of brain power in coming up with the most meaningful tools or use cases. And the three use cases we're addressing, or the three challenges we are addressing using AI are the following. Failure prediction. Because in 99% of the cases our customers are sitting on goldmine of data. And AI or machine learning has so much capability to take historical data like defects, test case execution software changes that are registered through logs and compile logs and Jenkins etc. And to come up with a prioritized list of what to test first and what's the probability of that test case to fail. This is so helpful if you want to drive velocity. And it's not everything. It's not just automation, but this is something that we know and we proved I think 20 times in the last two years that it's doable with very high accuracy and it's working. And it's not a fake promise. It's not everything there is to do about testing, it's just failure prediction. Where will we break next? The second one is about efficiency, our ability to harness algorithms to make sure we don't have overlap between defects, test cases and requirements. We call it dev consolidator and basically crunches your test cases on the one hand, your requirements on the other end and your defects. And it removes duplications. If you have the same defects twice, it prompts that you have it twice or it looks the same and then you can check why a test case all of a sudden reappear? We closed this test case three months ago. And all of a sudden we have a new test case that looks and walks just like the previous. What happened? And the manual person will struggle to do that. NLP doesn't Same goes to test case duplication. We usually find anything between 22 to 45% duplications in test design. That's an incredible number. The maintenance effort associated with so many test cases is incredible. The cost and deforestation is incredible and we are able to capitalize on that. I don't want to to take too much time here, but definitely machine learning and AI hold so much promise and we are carefully launching more and more offerings to make the lives of our teams, but also our customers and anyone life much easier by harnessing those powers to the benefits of quality engineer professionals.

As I mentioned once again on the webinar, what I love about it, I think you mentioned this also on the webinar is your approaches. You know, pitching AI is like this magical tool. You or someone on your team said AI is there to help with low level decisions to allow engineers to focus on more high level decisions. And these low level decisions where like you said, there's a gold mine of data. So it seems like that is the approach that you are focusing on, which I think is really helpful for people to follow as well.

Yeah, low order tests needs to be handled for higher order tests like user experience testing, those are the things we want professionals to handle. We don't want them to do regression all day and all night long because there's a potential risk hiding somewhere. Handle the risk through AI. Free up a significant amount of your day to focus on the things that makes your customer the happiest. And this is what we're trying to do here.

Love it. Okay everyone, before we go, is there one piece of actionable advice you can give to someone to help them with their quality strategy initiatives? And what's the best way to find contact you or learn more about qualitest?

So first of all I'm always available. My email is my first name.my surname@quitiesgroup.com Our website has just been revamped and relaunched and I really urge you guys to look@quietusgroup.com There's a lot of information there and look me up in LinkedIn and Twitter. I promise to be responsive.

Thanks again for your automation awesomeness. The links to everything and value we covered in this episode. Head on over to testguild.com a340 and while you're there, make sure to click on the Try for free today link under the exclusive sponsor section to learn all about Sauce Lab's awesome products and services. And if the show has helped you in any way, why not rate it and review it in itunes? Reviews really help in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe and my mission is to help you succeed with creating end to end full stack Automation awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Automation Podcast. Head on over to Test Guild's for full show notes, amazing blog articles, and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.