225: WEKA Machine Learning Automation Testing with Bhavani R

Hit it.

Welcome to the Test Talks podcast, the place to go to geek out on software testing. And now your host whose mission is to help you succeed with test automation, Joe Colantonio.

Hey, it's Joe. And welcome to another episode of Test Talks. As you all know, there are more and more tools available and that are leveraging machine learning algorithms to help with UI based software automation testing. But what about leveraging AI for helping in other areas of testing like test data management? That's what we'll be test talking all about today with Bhavani R. Bhavani is going to share her insights into how her team uses an open source tool called weka. Hope I'm saying that right to help her make predictions for her team's tested activities like test estimation, test coverage and issue forecasting. So listen up to learn all about weka and also other ways in which Bhavani leverages automation within her team's software development lifecycle. You don't want to miss it. Check it out. Just a heads up. For some reason in this particular episode I had audio issues that I could not clean up. But I think I got it good enough for listening and I just want to apologize for that beforehand. As I mentioned before, I am moving to Tennessee. I'm in Tennessee so for some reason my equipment got something. Something's off so I think I resolved it. But anyway, listen up and enjoy. Check it out.

Test Talks is sponsored by the fantastic folks at Sauce Labs, the cloud based automated testing platform that eliminates the need to maintain your own selenium grid and test infrastructure. Try it for free today. Visit testtalks.com and click on the sign up now link under the homepage Sponsor section.

Hey Bhavani, welcome to Test Talks.

Hi Joe, thank you for the opportunity and I am a longtime listener of Test Talks and I am happy to participate in Test Talks.

Well, it's actually my honor Bhavani, because you actually were a speaker at this year's Testing Guild. You did an awesome job around weka, which is an AI based machine learning type technology that helps you analyze your test results. So I'd like to talk about that today and all things test automation related with you. But before we get into it, can you just tell us a little bit more about yourself?

Actually I have started my career as in testing before 15 years in Zoniac Software Private Limited and after that I have joined DC CAP. From there I have joined as a tester and I have grown up to team lead there. After that I have joined to D.C. cAP and here I am working as a quality architect now and I'm adding QA team as well as managing our product. Key touch in personal life. I am having a small family, my husband and my son, only son.

I actually didn't know you manage a large group of people as well. So any tips on how you can help manage folks to succeed with their automation projects or any advice to getting large groups of people together to do automation correctly?

Yeah, for automation implementation we have actually how we build the automation test strategy is we will automate that repeated task only and we will split the big tasks into smaller and then do the automation. And we, we have. We are having the best practices to implement like writing small shell scripts or batch scripts to analyze log files to debug the issues.

More and give the quality delivery of the project or product and the next one thing I am I always ensure to my team is we should always keep learning and we should not never stop learning at all.

Very cool. And that's one of the reasons why I really love that approach of continuous learning that you're building into your team. I think that's really critical to succeeding with any type of project software related. So I know you mentioned using batch files to analyze your test results as that's something you're actually using WEKA for. And I guess before we get into.

It, what is WEKA is a machine learning library. It is an open source tool. It is developed by University of Waikato and.

It is actually first tool which is having the geographical user interface machine learning tool. And it is easier to start with because I am not having large programming experience. I'm having the automation program experience but not kind of a developer. So it was easy for me to learn WEKA fast compared to TensorFlow and other things.

Funny. Can you elaborate a little bit more or give an example of how or why you are using WEKA and how it's helping you and your team?

Yeah, actually I have tried.

VACA for my test management efforts like for defect prediction and project estimation and test coverage prediction. I have used for example if.

We are having the history of database already for the test coverages similar kind of projects with that we can predict how fast we can deliver that product to the customer. That kind of. For that kind of task I have explored and used backup.

Very cool. So how accurate is it with the predictions? Are you able to say this is what it predicts and then it actually turns out within a range that it was accurate or not?

Actually it ranges we have to. That depends on the data, how much data we are training with the.

With the tool. The lot of test data need to be there. So we have to how. How much we are training that much accuracy we will get the output. So actually what we have tried is we have tried up to multiple test test data.

So we have got reached just now up to 70 to 80%. We have to reach more. We have to train the test sets with more data. Then we will get more accuracy.

Now I know you said it was easier to learn than tensorflow but it still looks kind of complicated when you just download it for the first time. So how easy is it actually to get started with weka?

Actually it is pretty easier for the business since machine learning other machine learning libraries needs your extensive knowledge in programming like Python OR R or TensorFlow. So compared to that it is having an GUI tool and the libraries are pretty easier. Only you can start with and it is easier to start with. Then you can explore to other machine Learning libraries like TensorFlow or Python programming to achieve more.

Are there any resources you recommend to people to help them get up to speed quicker? Is there anything that you. You tried when you were first starting that threw you off that you said oh, I wish I knew that before I started.

They have provided courses in their website itself. It is.

And they are having the picture with a handle also. And we can if we are following that and the professor Ian Bitten, he who has given the lectures, it is available in YouTube first I have started with that and I have. Then I have explored it myself also.

Do you know if you could programmatically interact with weka? Is it just through the GUI ui?

No, actually the programmatic way also I have explored. I have, I am working on that first I have tried with the GUI thing and I am working on the programming part now write the Java program and we can call the class files from the weka library.

Very cool. So Bhavani also maybe can we go over some of the pieces that make up weka. Like for example, what's a knowledge flow in weka?

Knowledge flow is data prediction library only. First, first thing we first in the first session I have shown the basic option with the explorer. It is a simple one. The knowledge flow is having the detailed one here we can do multiple things and in in depth analysis we can do it with the knowledge flow. Actually only I have selected the linear regression. We can select the 0 or algorithm or kind of decision tree. All the things that we can do it with this knowledge also like how.

Would you share this with a stakeholder like a product owner Would they be able to understand it or do you just output the data of the analysis and anyone would be able to read it?

Yeah, actually the non technical people only look for the results. So we will formulate the flow and we will explain. We will present it to the team like how the data is predicted and they have. They will keen. They are mostly keen about the results and the prediction data.

We actually use this for defect analysis. Like is it able to predict how many defects you might have in your next release or anything like that?

Yeah, we can do that. That's why as the sample we have taken I have shown that test coverage prediction. Similarly we can do with that same thing. We can do with the different set of test data. What is the file? We are passing the CSV file. We can give the defect data means we have to train with the defect data and then do the prediction. We have to do the multiple. We have to do the training with the multiple sets of test data. Then only the predicted result will be accurate.

How do you trust once you implement in WEKA and you get the results, how do you trust those results? Do you have like a set of training data you use first to verify? Like how do you know that what it's telling you is even correct?

Yeah, actually in the initial phase we will not get. We have not got the exact result. Like what is the outcome we are expecting. After some time it will. We are having the more data. Right how many times we are training that model. It is important to get the accurate results.

Is there any inflated confidence level using this? Sometimes people say oh we have automated tests so we don't need to worry about it. Do you see this? With this AI based tool out it told us what we have. Don't need to really worry about it or think about it anymore.

No, actually it depends on the size of data and.

The number of projects we are working and how. How much.

Does the size of the project as well as the test data. That is very keen on this part. To get the confidence it it will take.

Around 3, 3 months data or 6 months data based on the company size and how many projects are running at the phase same time. At the same time.

Where do you see AI going? What are your thoughts on AI and testing? Do you think this is going to replace testers?

I won't agree. It will replace or test replace the testers are testing the our role will be changed like we will be the intelligent task will be done by humans. So we can do that trainer part. We can do the other part. The Machines can do. That's what my vision.

So Bhavani, I think if I remember correctly you did speak at once again at this year's Testing Guild and you did mention your company is actually going to try to implement WEKR into one of your actual products. So could you just tell us a bit more about what that looks like or how that would work?

Actually yeah. Thank you for that opportunity. Speaking opportunity and yeah, it is in a still in incubation stage. We are trying to implement in our test management product itself for prediction and we are working on that. It is in initial phases.

Also if I remember correctly, I think you mentioned you use webdriver IO for your Selenium testing efforts. Just curious to know why you chose that particular language binding. I know a lot of people use a lot of different flavors, but why did you choose Webdriver IO for your automation?

Actually I have used both Selenium Webdriver as well as Selenium Webdriver IO. Selenium Webdriver IO is.

Written in JavaScript and it is easier to learn and write fast. So for my team members we have implemented, I have trained them and we have implemented in number of projects with Jenkins Continuous automation. We have integrated the Selenium webdriver IO. The scripting is easier compared to Java.

So who on your team actually does the automation? Is it the front end developers? Is it your testers? Can you just tell us a little bit more about how that works in your organization?

Actually more of QA testers, we will do both manual as well as automation testing. Our team is empowered like that.

So if you do have a tester that maybe doesn't have a programming background using Java or JavaScript, how do you encourage your team to maybe learn some of these newer skills that can help them with your automation efforts?

Yeah, actually for that we are having weekly learning sessions and apart from that we are encouraging the team to explore new tools and give training to the other team members. The trainings will be happen happen in monthly ones and the weekly knowledge sharing session will have a small con small new concept. Whoever learned about the new technology or something they have learned about a new tips, testing tips or automation tip they will explain. And this way we are encouraging the team to team and motivate the team to do the automation learn the automation.

So during those learnings with your team members, did you actually learn anything and go wow, I never knew that. And because your team member took initiative your whole team was able to grow based on that knowledge?

Yeah, actually my team is all the team Members are very great and I'm happy to work with them. And one thing is they have come up with Cypress. It is a new concept. So that tool I haven't used before.

I've actually heard a lot from people recently saying they love Cypress IO especially if you're working with JavaScript. So to the person who recommended it to you give reasons why they really enjoyed using that tool or any benefits of using Cypress.

Actually for.

Mobile responsiveness and apps testing it will be useful. So they have explored and they come back and we have done for one project now.

Bhavani one of your talking points was about QA Touch. Is that a product you guys actually develop?

Yeah, it is a product developed by us and we have launched our product in last year. Last year by November and currently we are having the test management and product which which will cover the.

Functional test cases as well as we. You can map the requirements and test cases. Everything we can do that we are for giving back to the community. Our CEO Karthik Chidambaram has given an idea to option for free for small teams forever with the essential features. This is our.

Unique value we are giving and we apart from that.

We are having the new options like.

We are having the new features in Roadmap like automation integration and continuous integration as well as SAS Labs browser stack also. And then we are working on that prediction with the AI.

So do you actually use your own test management tool to manage the test management of your own test management tool?

Yeah, we are using as a test management tool in DC Cap. We have implemented that and we are using and our clients are also using for whichever project we are delivering. We are putting all the test cases into the system and the clients are having the. We are giving the access to the client for user acceptance testing also they will enter their user acceptance testing issues in the Q Touch.

So since you are creating a test management tool, what features you think every test management tool should have?

Every test management tool should have the option to have a single repository to maintain all the test assets. And apart from all of this, it should be very user friendly. The user interface should be very simple and sleek. It should not confuse the users. That is the one basic thing I am looking for it always.

Now you said a lot of your clients actually use it. Did you have you learned anything that your customers really wanted that you'd even think about as you were testing and developing the software?

Yeah, actually a few few feedback we have received like that. So.

Actually for the continuous integration we have, we have set it in the really in depth roadmap. But one customer is asking about AI and all. He is not a technical person at all, but he has asked about the implementation of AI.

So Bhavani, can you actually walk us through what your life cycle might be as you're developing software? I know you mentioned you do manual testing. How do you decide what to automate, what not to automate, things like that?

Actually in our project what we will do is we will actually when we will get the build, we will.

Discuss and build the test strategy for that we will segregate the.

Test cases into automation as well as manual test cases. So based on that we will, we will devise into parts and first we will focus on the core functional test cases, then the UA test cases. We will do the repeated tasks like for example for the data population and the things we will do the scripting in at the initial stage. Then once the project is stabilized, then we will write the write automation script for the whole project.

Whole project. The all the bug fixes are done and then for regression testing we are using the automation script.

It has actually.

While writing the regression scripts and implementing the automation for regression scripts, we have reduced 30% of our testing time.

Awesome. And how much are developers involved? Like how do you know that they're not like are they doing unit test and that you're not just doing an end to end test because they're not doing testing. Like how involved are your developers, I guess in your automation, if at all?

Actually developers are not doing any automation unit testing. They will do the testing and.

They will do the.

Unit testing and they will pass on to us for the next level of testing.

So you did mention Jenkins. So I assume you're doing some sort of continuous integration, continuous delivery type process. Any suggestions on how to get people to pay attention to when a build or test actually fails within your CI CD system?

Yeah, we can. We are having the option to receive an email from Jenkins if the job is failed. We can get an email.

That in that way.

We can identify the issue and sometimes the one if one critical test is failed means we have to pay the attention otherwise it will be problem in the later stage. So the email notification will help.

How often do you actually find bugs with your automation that your automation actually found a legitimate bug? I know a lot of times we do it for peace of mind and a lot of times it doesn't actually find a bug. So how effective are your automated tests?

I guess the question, yeah, that is a great question. One time we found one interesting bug in One of my project, it is a voting application with the three regions. So first for two regions, the vote will open up for.

Number of days for each region. For example, region one for two days and it will close. Then the voting for region two will be open. And next region three, what was the issue we faced is in region one it has completed, it was worked successfully and in region two also the votes are placed.

Successfully and it is voted against that particular region. But when we go to the when we execute the script for region three, what has happened is the voting, the votes were posted to region 2 again, not to region 3. It was an interesting bug. We found it through automation.

Cool. I wonder if you could have even replicated that without automation. So that seems like a cool user case that you could uncover with automation. So nice.

Yeah.

So Bhavani, you are creating a test management tool. So I assume you'll have more than one user on the system at one time. Do you do any sort of performance testing? Like how do you handle to make sure that your application not only works for one person, but it's going to scale for like a team of five or 10 that probably will be using it at the same time?

Yeah, actually we have done a base performance testing and then only we have deployed our product into cloud and apart from that we are having a monitoring system to monitor the application always to track the.

Downtime or if the application has go down. In any case we will get notified and we will check. As of now.

No downtime has happened. I'll hope for the best.

Cool. Okay Bhavani, before we go, is there one piece of actionable advice you can give someone to improve the test automation efforts and let us know the best way to find or contact you.

Actually for the automation, we can't automate everything we have to, then it will become flaky test. So we have to be cautious and we have to identify the use cases best use cases and it should be minimize our overall testing time. Based on that we have to build the automation test strategy. And the best way to reach out is my Twitter and in my LinkedIn too. You can reach out to you.

Thank you Bhavani for your WEKA automation awesomeness. For links to everything of value we covered in this episode, head on over to testtalks.com 225 and while you're there, make sure to click on that. Sign up for a free trial link under the exclusive sponsor section to learn all about Source Lab's awesome products and services. And if this episode has helped you in any way, why not share with a friend or coworker and together we can help spread automation awesomeness to the masses. So that's it for this episode of Test Talks. I'm Joe. My mission is to help you succeed with creating automation awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Talks podcast. Head on over to www.testtalks.com for full show notes, amazing blogger articles, and other automation awesomeness.