Advanced Test Automation with Anthony Venditti

Foreign.

Get ready to discover the most actionable end to end automation advice for some of the smartest testers on the planet. Hey, I'm Joe Calantonio, host of the Test Guild Automation Podcast and my goal is to help you succeed with creating Automation awesomeness.

Hey it's Joe and welcome to another episode of the Test Guild Automation Podcast. Today I want to share with you one of the highest rated sessions at this year's Automation Guild Online Conference that took place February 6th to the 10th. If you missed it, you can still purchase an instant access ticket and get your hands on all the sessions live Q and A recordings in our private Slack channel. And as a bonus for being a Test Guild Automation Podcast listener, you can get 30% off when you use the code POD30. That's P O D. So head on over to automationguild.com now and register to get instant access to all the awesomeness that occurred at the seventh annual online event. So here is Tony Vendetti's session on Advanced Test Automation. Check it out.

The Test Guild Automation Podcast is sponsored by the fantastic folks at Source Labs, the cloud based test platform that helps ensure you can develop with confidence at every step from code to deployment. For Every framework, browser, OS, mobile device and API. Get a free trial. Visit testguild.comSourceLabs and click on the Exclusive Sponsor section to try it for free today. Check it out.

Hey, this is the Advanced Test Automation Automation Guild session using Selenium and other open source tools to build powerful test automation solutions. By Tony Venditti.

I'm Tony Venditti and I am the Test Automation Manager at a company called advantager.

Advantager is a subsidiary of Blue Cross Blue Shield.

And today I'm going to be talking about the following items.

My background with test automation test automation tools and approach applications and test types using Jenkins as your test automation hub and then if time allows, we'll be talking about test automation, team structure and organization, possibly the future of test automation and Question and Answer Session Question and Answer will be at the Automation Guild on February 7th at 2:30 I believe. So I'm looking forward to.

Talking to you then and answering questions about this presentation then.

My background with Test automation I started out a long time ago with with a company called Lotus Development Corporation.

Lotus was just a great place to work coming right out of school at the time it was in the in Cambridge, Massachusetts. It was Lotus Development was bigger than Microsoft at the time. They had their flagship spreadsheet product called Lotus one two three. They subsequently came out with another big Hit product called Lotus Notes.

I started as a QA engineer there and in the late 80s, early 90s, long, long time ago. But it was an exciting time because the Lotus 1, 2, 3 spreadsheet was the biggest application on the IBM PCs out there, which had come out in the mid-80s. And the spreadsheet was the application or the reason to purchase a personal computer. So it was really a, a burgeoning industry at the time. And so it was so fun to be a QA engineer at that time in the industry.

With regard to test automation tools, there really wasn't anything out there. We had these massive teams, you know, two, it wasn't unusual to have a 200 person QA team, you know, testing a single software application because it was almost purely manual testing.

So, you know, we eventually started using things like the macros that were built into the products to do some automated smoke tests and things like that. There were a couple of engineers at the time that were working on, they were developers working on in house tools to basically develop some automation. You know, I remember one tool was called the journaling tool, you know, to basically script out or simulate or automate test processes and test cases. But it was all, it was all new.

Then there was a company called Segway Software and they had a product called QA Partner. And I believe that Segway started with seed money from Lotus. I think Lotus was looking for tools like this to, you know, to.

Test their products, to do simulated testing, test automation with their products. So I believe that Segway software was started with seed money from Lotus Development. And in fact I was one of the people using beta versions of QA Partner at the time and loved it. It was a great tool. It had a language, a proprietary language called Fortest. And the Fortest language was very C like, but it was extremely powerful and you could do a lot of things, you could do a lot of test automation with QA Partner. And eventually QA Partner became Silk Test when the Internet came along and browser testing became a big thing and Segway changed its branding and called basically a Silk Test suite of products that they had. And I use that as well. IBM purchased Lotus Development or acquired Lotus Development and I became an IBMer, ultimately working for IBM for about 20 years with, you know, in combination with my Lotus Development time and loved it.

So, and loved and very passionate about software testing, specifically Test Automation. So loved using Cilk Test.

I even wrote a letter to Lou Gerstner, who was the CEO of the time at IBM and asking him, you Know, telling him that I thought that Segway would be a great acquisition for IBM.

You know, simply we were using the tools, it was very expensive, proprietary tool. I love the tool.

And so I doubt he even saw the email. But.

Not long after that I found out that IBM had acquired Rational, a company called Rational. Rational's big flagship product was a source control product called clearcase. But they also had a Java based test automation tool called Rational Functional Tester.

So I knew that when IBM purchased Rational, I could see the writing on the wall that IBM wasn't going to pay to have this expensive proprietary software from Segway when they owned Rational Functional Tester that basically did the same thing. Now rational again, RFT was a Java based scripting language and CilkTest was a C based language or C like language for test language. So I saw the writing on the wall and started to experiment with, with Rational Functional Tester and Java and start to basically learn Java. The thing I missed was in my evaluation of Rational Functional Tester is it didn't have a lot of the tools and capabilities that were built into cilk test. CilkTest had this powerful user library with hundreds of already written functions for file, IO, date time, database, all kinds of functionality built into the product that Rational Functional Testers simply didn't have. However, Java was a powerful language and we could write those, or mimic those, or basically develop those File I O operations and write them in Java. So that's what I set about doing. And it also was a way for me to kind of learn how to program in Java. So I took the libraries that cilktest offered, the File I O libraries, the database libraries, the datetime libraries, et cetera, and rewrote them in Java and ultimately created this large framework for Rational Functional Tester that was used by a lot of products and a lot of teams at IBM to test our products at that time. So I tried to make it as similar to CilkTest. I tried to make Rational Functional Tester as similar to cilktest as I could by creating these libraries.

That ended up being what we were using at IBM, was Rational Functional Tester with these Java based, ultimately developed a framework around those function libraries. Then when Selenium came along in the early 2000s.

Developers started experimenting with the open source Selenium tool to test web based applications. And they embedded, they basically embedded the Selenium wrappers.

The selenium APIs, and created wrapper classes for them and embedded them into our framework so that we could basically run the automation using rft. Our RFT framework or we could use this new Selenium model that was implemented in the framework. So it was very powerful and very adaptable. I ultimately left IBM in 2011 and worked at a payment software company called ACI in Waltham, Massachusetts and basically took all my experience in test automation and my approach to the framework.

And they were using various tools. There were people there using Cilk Test as well, but they were looking to phase that out and move on to Selenium which was becoming an industry standard and it's open source so it was very high value.

Prospect for, you know, for developing test automation. So I stripped out, I basically developed the, redeveloped the framework for, for ACI and incorporated the, you know, Selenium. You stripped out the RFT components and in court and made it a full blown Selenium framework. And ultimately that's where the beginnings of the, the approach that we use now started, started there.

So now I'd like to talk about test automation, test automation tools and approach that we use.

And I know that Joe wants, you know, the automation guild likes, you know, deep dive, you know, technical demos and you know, this, this is a PowerPoint presentation, an overview of the approach that we use. But in the future I'd like to come back again and maybe do a deep dive where we kind of go into the code. But for today's purposes in this presentation it's a very high level overview of the approach that we use.

So this is the, the, these are the open, so we basically use a collection of open source tools. They're shown here and we wrap these up in a custom framework that you know, ties all these pieces together. We use Java for our, our program, very powerful scripting language. We use Eclipse as our idea. We use Selenium for our web based testing the approaches and the framework is adaptable enough that you know, if, if we had to use.

If, if we had to test mobile applications or Windows desktop applications, we could do that by swapping out the, what we call a test engine. So Selenium is our test engine tool that we use for web based applications. If we needed to test mobile applications, we could swap in something like Appium for desktop applications. I have swapped in cilk4j in the past. So as long as it's Java based we can plug it into this framework and the same APIs will work for those applications. We use Testng as basically our framework. We use its data provider.

Provisioning and its annotations and structure for our scripts. We use Maven to build our projects and we Use Jenkins as our test automation hub. Now, Jenkins is a CICD build and deploy automation tool, but it also makes for an incredibly powerful test automation tool as well.

So we use open source solutions because it gives us the freedom to select the tools that we need to get the job done.

There's flexibility. They're not, you know, open source tools typically aren't limited to specific operating systems or file formats. And the example I have here is in Cilk Test had a great result file called a dot, res or res file and it was a really slick result file for test automation results. And if, if I wanted to share those results with developers or other QA engineers, they would have to have, you know, a $5,000 copy of Cilk Test on the desk to view it. So, you know, that's, that's the problem with proprietary formats. You know, because we use open source formats, we can produce the results in open source formats like HTML or a text base or xml or you know, we, our solutions can generate result files in.

Extent dashboards, we can even produce Excel sheets, things like that. So, you know, we can do all sorts of things to provide flexibility and that's what open source gives us. If you have the skills, you can develop the solutions that you need and you're not dependent on a single proprietary solution provider.

So that's why, and ultimately, you know, cost savings. The open source tools are typically free and you know, you can, you can build upon them to create very powerful solutions that can compete with the best of the proprietary tools on the market.

And over time, that's, that's what I think we've done.

Our pro, our approach, our framework is what I call a programmatic approach.

It's, we're coding, it's not a scriptless approach or a codeless approach or record playback approach. And I've, I've been doing this for such a long time. I have discovered and I have failed so often that I've learned, you know, what works best with, with test automation over the years and, and what's, you know, the best approach to be maintainable for the long term. And I believe a programmatic approach using a framework like we have is the best approach for developing maintainable, reusable.

Software. Test automation software that can run a test regression suite now and run that same regression suite two years from now.

This next slide goes over the structure of the framework. As I mentioned earlier, we have the, you know, the test tool. So that would be, in our case, we have all web based applications. So Selenium is our, is our Test engine or test tool of.

You know, that we, that we use for all of our test automation. Like I said, if we had mobile applications or if they, they start to develop mobile applications or desktop applications, Windows desktop applications, we can plug in different test engines or tools and we can use the same approach and the same APIs and.

And this, and the same scripts basically for testing those applications. The, the next level is the, the common core framework. And this is this powerful framework that's been, that contains those massive function libraries that we converted to Java. It has common logging and result reporting. It has common error handling, generic function libraries that do all the file IO operations, datetime operations.

Database operations, et cetera. You don't need to reinvent these.

Functions and these methods. It has a standard directory structure which all the tests use this standard directory structure and it uses the same script formats and templates and APIs. The APIs are the same for all applications. So you can see that this core can be used for multiple applications. Here in this slide we have three applications. But at where I am now at Blue Cross Bushia, we have over a dozen applications that plug into this common framework to provide test automation and test cases, automated test cases for over a dozen applications there. But this is essentially the approach that we use. We have each application uses the common framework to develop application specific libraries for that application. So this is where all the business functionality and workflows are developed and we try to break things down into small methods modules that can be reused for multiple test cases. So this is where we store all those test libraries. The next level is the application test data. So we use the testng data provider so we can read and externalize data from spreadsheets, from databases, from text files, from CSV files. And the data can be embedded in the script itself. So the test data is very important to the automation. So it's this next level here, the final level is the test script. And the test script is basically a harness that executes.

The methods that you wrote at the application level that you're calling in your test case in your script and then you're passing the test data into those test methods, those test method parameters, which I'll have a slide on in a minute.

And executes the scripts. And basically it's, you know, it's kind of a batch process that executes the script. So it's a very straightforward process for automation, but probably very similar to what a lot of companies are using out there.

However, I think, you know, I've seen tutorials, Selenium Tutorials online and our approach ultimately at the core level is very different and I'll try to talk a little bit more about that.

So this goes back, it harkens back to the Silk Test days. By the way, I have my such a big fan of Soak Test. I have my, my copy of Soak Test from.

From. From January 5th, 2000 here.

Great product. But I think that this open source solution and it definitely draws a lot from the power of cilktest, but I think this open source solution is even more powerful. So we use a similar model that CilkTest used in that it had a parent page, a child object, and then an action. So for example, you'll have the login page and you'll have the password, and then the action would be to set the text and the password or set the password or the username child object and the action would be to set the text of the username. The login button. The action would be to click the login button.

The whole approach.

Is like that.

Selenium basically identifies all of the web elements as just that, web elements. They're just generic web elements. So everything's a web element. Our approach defines it as a class type, a text field. We define that web element as a text field. If it's a username text field or whatever that we define as a text field object or a button is a button object, a checkbox is a checkbox object, a list box is a list object. With each of those class types, you have a web widget library that has dozens of methods specific to that class type.

So in Eclipse, by defining every element as a Java object with a specific class type, when you're creating your script and you hit the dot, you get the context sensitive text field methods that are available to that text field. So settext, gettext, cleartext, and all those functions.

Are built into the text field class. We call it a web widget class. And we've also defined those are HTML web widget classes. We've also defined widgets for AngularJS and ext JS elements, things like that. We have classes for those elements as well.

This is a graphic right here of the Java doc for our utility classes. So you can see the text is pretty small here. You know, hopefully you can get a hold of this presentation and you can see it a little bit more close up here. But you know, we have, you know, the browser class, datetime class, Encryption class, Excel class, File I O class, Images class, FTP class, PDF class, randomization class, string class, Timer class, XML class.

There'S no end to the support that you can build into your framework.

There are just dozens of Java libraries out there, open source Java libraries that you can use. You develop wrappers to build them into your framework if you need. I'm sure there are lots of people out there that use the Apache POI classes, Java libraries for Excel so that they can read and write from Excel files. So.

We do that all the time. If we need support for some other file type or some other methodology, if there are Java libraries out there for that we can build it in. We had somebody create, we wanted to incorporate extent dashboards into our framework and somebody, you know, wrapped created a wrapper class for the extent capabilities. And if we enable that in our property settings, our automation will generate the extent dashboards with each result. So there's no end to what you can do with open source tools. You know, it's just your skill and what's available out there.

This slide right here goes over the externalized test data files and how we use the testng data provider to plug into our method, our application specific methods. So on the left hand side of the screen you have an application specific method. On the right hand side you have the external data in a spreadsheet. And the way that testng works is its data provider functionality is it plugs in the data for each row into the parameters of this method here. And you can see right here, so the group ID gets plugged in here, the group name gets plugged in here, etc. The invoice level gets plugged in here and it executes that, that first row of data.

In this test case here automatically and executes it as an individual test case, then it moves on to the next row and it executes those. The beauty of this approach is that if management comes up and says, hey, you've got 10 test cases here. We need 100 more combinations to test this. You can easily go to the spreadsheet, plug in all the different combinations, and you don't have to change a single line of code here to test 100 additional test cases. So it's very powerful and flexible and maintainable.

I'd like to talk a little bit about applications and test types.

So when I started at Advantage or they were using, every application used a different tool. They were using qtp, they were using Selenium, they were using other tools and, and even with Selenium they were using different frameworks for different types of tests in the same application. It was just a mess. And the way this, when we implemented this common custom framework and approach that we have, the beauty of it is that you can test n number of applications with it. So for all the different types of testing we have, we have things like link testing and smoke testing and regression testing. And all of that is done through this same approach. So it's a very. And that the same approach for every application. So we're not using different tools, different approaches, different frameworks for every application. Every application uses the same approach, same tool. So we're not reinventing new logging methods, new error handling methods, new database methods, new file IO methods. For every application that we're testing, everything uses that common framework and uses the same I O, same database, same date time functions as each application uses the same function libraries.

We have. So different types. I mentioned link, smoke, and regression. We have something else called a heartbeat test. A heartbeat test quickly launches the application, logs in, validates that it's up and running, basically. And we do this for hundreds of servers, production environments that we have, and we run it every five hours to make sure that they're up and running. It's just a quick ping that we get these reports every five hours that tell us that, yep, everything's up and running. If some, if, if, if there is something occurs that it can't load the page or it can't log in, or it doesn't display the homepage correctly, it'll get flagged in the report in red and basically is telling you that there's a potential outage here. And so then we go in and we manually investigate to see if it's a real reproducible outage or just a password expiration or something like that.

The link test goes a little bit further than the heartbeat test in that it logs into the application and then clicks all of the links and modules in the sidebar and then validates that all of the elements appear correctly with each component. And that's about it. It doesn't modify any data. So we can run a link test on a production environment without.

Causing any problems with.

Data modification and things like that. And we execute the link tests every day on all of our client environments, production environments, QA environments, all environments.

Next is the smoke test. And the smoke Test executes between 10 and 50 test cases basically work, you know, the basic workflow functionality of the product.

It does modify data. So we can only run test smoke tests in QA environments. We can't run them in production environments, obviously.

And we run these on a daily basis as well. And these tests, link tests maybe take 15 minutes to run Heartbeat tests maybe take a couple of minutes. Smoke tests can take, you know, up to an hour, even more than an hour. And regression testing, Regression testing, finally, is the largest test that we have and that you can, you know, have suites for each module of 500 test cases per module. And these tests can run for, if they run sequentially, can run for 20, 24 hours.

So now I'd like to talk about using Jenkins as your test automation hub.

And Jenkins is part of our overall strategy and our open source tool set. It's an incredibly, while I mentioned earlier, it is a build and deployment automation tool. It makes for an incredibly powerful test automation tool as well. And what we've done is, you know, for all of the products that we support.

You know, again, not a great graphic. I'd like to come back and do a second presentation on a deep dive that we can go further into detail and demos in Jenkins. But.

For today's overview, we have a tab for each of the applications that we support. And if you click on the tab, you'll see subsequent environments that the test can be run on. And then if you click on the environment, you'll see the types of tests like I mentioned earlier, link test, smoke test, heartbeat test test, regression tests. We have API tests, we have database tests, we have data generation jobs. So those you'll find for each application, you click on the test type that you want. Now, these can be executed on demand by anybody in qa. So our automation team develops the automation, essentially checks it into Jenkins, creating these jobs. We have over 1,000 jobs in Jenkins for all the different products that we support. And you would, you know, anybody qa, anybody that has access to Jenkins QA development can then execute the automation that they require for their particular applications. So it's an incredibly powerful way for us to have a hub for our automation and interface with QA and development. So it's very easy for them to log into Jenkins, go to the application of choice that they're working on, find the type of test that they want and execute it. The results get pumped back into Jenkins, they can review the results.

So we have a pool of.

Right now we have 40 test nodes that the dozen or so applications that we support, and somebody kicks off a job, it will go find one of these 40 nodes that we have and run the job on those nodes. We have stuff that gets executed on demand, stuff that gets triggered by deployments, stuff that is scheduled to run overnight. Literally thousands and thousands of tests every day that are executed and managed through Jenkins. So I really love the tool. It's just a fantastic open source tool and it's also built on plugins so you can add just like our framework, you can add all kinds of capabilities to it through the use of plugins into Jenkins.

One of the things that we're looking to add to Jenkins is the ability to rather than have 40 test nodes sitting idle 24. Seven in our Azure cloud based environment.

Ringing up bills, we want to be able to have a dynamic scale set environment where we spin up environments dynamically and execute the test. And then when the test is done the resources shut down and go back into the pool and wait for additional tests to run. That is in our future. Working on that currently right now.

This graphic right here shows the nodes that we have running at any given time. You can see most of them are running tests and then you can see the ones with the empty desktops that are idle and just waiting for additional calls to Jobs. This is our result. This is our sample HTML result file that we have. And again it's very similar to a silk test res file where you click on a node you click on. So there's a suite at the highest level suite contains a series of scripts. Scripts contain a series of test cases n number of test cases. You click on a test case and you can see the steps that the test case executed and it goes down to the, you know, entered text in this text field. Click this button, you know selected this list item and I have sent the log files here. I've copied and pasted them into defect reports for developers because that's how thorough they are in a step by step process tracking the automation process that we have. If something fails, it's highlighted in red. You can see a stack trace and then an image of the the failure on the screen. So it's much easier to debug what's going on or just to distinguish between a product failure and just an automation issue.

So this concludes the main portion of my presentation on our framework approach and tools. It's really an overview I would love, if there's any interest, I would love to come back and kind of do a deep dive slash demo of some of the, of the framework and the tools, etc. There's a couple other sections I wanted to talk about. There's. There's not enough time to do it so I will. I spoke to Joe, he said we could, we could go into more detail on those topics at the live question and answer session on February 7th at 2:30 I will briefly introduce them.

One section is on how our automation team is structured. And.

We like to have a we're a shared service team, a central automation team. We work with the QA organization. I like to have a dedicated automation lead for every application that we support and then also support staff for that lead of every product that we support. So depending upon the complexity and the size of the applications can have anywhere from one to five or six support staff for each automation team or product lead. And that's the structure that we like to have. The next slide is kind of the workflow and how we work with the QA organization. Not enough time to go into, into detail there. And lastly, I wanted to just to talk about the future of test automation. My thoughts on the future of test automation.

And you know, I and I think near term it's cloud based, it's dynamically spinning up environments, it's, you know, containerization.

But I think in the next few years we're going to start to see AI become more and more integrated in test automation. You already see it now with tools like OpenAI, ChatGPT, incredibly powerful artificial intelligence tools. In fact, the right side of this slide right here, I typed into ChatGPT, what is the future of software test automation? And this is what it spit out. And when I reviewed it I was like, it's incredibly amazingly accurate.

So I think the future is bright with AI and test automation tools, at least until the killer robots arrive. And finally, I just want to put a plug in for the Automation Guild 2023.

Live Q&A sessions for me. Tuesday, February 7th at 2:30 to 3:30pm Looking forward to talking to everybody there and discussing all things automation.

Thank you, thank you Tony for your automation awesomeness. For links to everything of value we covered in this episode, heading over to test guild.com forward/a437 and while you're there, make sure to click on the Try it for free Today link under the exclusive Sponsor section to learn all about Sauce Labs awesome products and services. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe. My mission is help you succeed in creating end to end full stack automation awesomeness. As always, test everything and keep the good. Cheers. Hey, thanks again for listening. If you're not already part of our awesome community of 27,000 of the smartest testers, DevOps and automation professionals in the world, we'd love to have you join the fam@testguild.com and if you're in the DevOps automation software testing space or you're a test tool provider and want to offer real world value that can improve the skills or solve a problem for the Guild community. I love to hear from you. Head on over to test guild.info and let's make it happen.