4 Reasons Enterprise Testing Fails with Autumn Ciliberto

Foreign.

Get ready to discover the most actionable end to end automation advice for some of the smartest testers on the planet. Hey, I'm Joe Calantonio, host of the Test Guild Automation Podcast and my goal is to help you succeed with creating automation awesomeness.

Hey, it's Joe and welcome to another episode of the Test Guild Automation Podcast. Today we'll be talking all about four top reasons enterprise testing fails. I think enterprise testing is critical. A lot of times people look to smaller companies to try to implement techniques that may work there but don't work at the enterprise. So I'm really excited to have an enterprise expert joining us today. Autumn, who's a enterprise account director at keysights Technologies and she supports clients with AI powered test automation solutions like Akplant software. You probably heard of a plant software, They've been around for a while. They're one of the leading test automation solutions that specialize in in digital transformation. Also, Autumn has over 10 years of experience selling into the enterprise space. So she knows the types of things you're probably dealing with when you're trying to interact with your say, C suite or executives when you're trying to get buy in for a tool. And she also recently spoke at keysight's corporate conference in Las Vegas on the subject of identifying key Personas and delivering test solutions to C level executives. Quantum provides leading testing teams as solutions to support organizational objectives to shift left what you need to do nowadays as we try to create software more, quicker, faster or higher quality and to keep up with the ever growing client demands in a digital world. You don't want to miss this episode. Check it out. Hey Autumn, welcome to the Guild.

Hi Joe, it's great to be here. Thanks for having me.

Awesome. Great to have you. I'm always excited when I talk to folks that really kind of specialize with enterprise solutions and problems. So I guess before we get into it though, is there anything I missed in your bio that you want the Guild to know more about?

No, I think you nailed it and we'll just share some of my own personal tips and best practices to help when we are directing executives to get the buy in when it comes to testing. So awesome. I think it was great.

Great. So as I mentioned, you know you have a lot of experience with the C level executive suite. I know there's sometimes a disconnect between do tools bubble down to up, but it's come from top down. So how do you get buy in from, you know, higher level CTOs C level executives to let them know, hey, here's a solution that's really going to help you with. Is that something that usually is driven from the bottom up or from the top down?

So it's a great question and it depends. I've seen it both. So depending on the organization and who you're working with, I've seen it where it's the direction from the C level executives to let their team know, hey, test automation and these advancements in testing are a priority for a business get on this, find the right tool. Whereas from the bottom up it's kind of can be more challenging at times if you don't have that buy in. So ways in which to do that. If these QA teams or development teams are struggling, is that making sure that their executive team is brought in early on make them a part of that evaluation process. That's something I ensure to do with my clients. So it limits some of these roadblocks that they have later on because you go through all this work through an evaluation, spend months testing a tool. But if you don't have that initial buy in from those key stakeholders, it will be very challenging for them. And a lot of times projects will be denied because they just don't see the actual purpose and vision that's staring them right in the face to help them when it comes to these testing challenges.

So what are they looking for? I know it's people laugh at roi, but for an executive like what do you put in an executive summary you think really would pop it to say okay this is important, let's look into this more.

Yeah, well something many executives are, there's ever growing client demands and if you're not keeping up with that, you're going to fail as an organization. And that's something that any C level executive would agree to that we need to get the products that we need out to market faster. So through test automation and identifying the best solutions to advance that testing or eliminate manual testing to get those things rolled out. And by showing that and connecting that the synergy between the two, that hey, this tool will get us to this launch of a product faster, less bugs and defects. So that can cost the organization millions of dollars at times. That's something that would really resonate with the C level executive in a way to position it that would be a way that touches on what they care about.

Great advice. So you know, I don't know if you've seen the shift. It goes back and forth. When I started everything was enterprise level software, especially when you work for healthcare. Things have been Kind of vetted, improved and then a lot of companies made the shift to open source and kind of like mixing and matching and creating their own solutions. Now I'm seeing a swing back to like more platform type of solution. Are you seeing the same thing? Is that something common or what are your thoughts on that?

It is common. It's coming up more and more. In many discussions we've been having over the last couple years with many of the enterprise organizations I've spoken to is that they're noticing that open source just isn't a viable solution that's withstanding these client demands and testing for these the long term. And so they're coming to solutions like Eggplant or they're leading test automation solutions out there to say what is a better way to do this that we can be incorporating into our own testing? And so we're noticing the shift because the ways in which through the open source, it's not providing them the necessary means to test complex end to end testing or supporting more of testing coverage through the automation so for parallel execution. So a lot of that is limited when there's manual testing or even through open source that there's a lot of complex systems they're looking to test and you're just not going to be able to do that with an improper solution. And to your point, more of a unification instead of having this disjointed seven different solutions across various different departments within QA is just not an optimal path forward. Again, going back to my earlier point, there needs to be that unification across the team of what's the best solution for us, that all of these individuals are able to utilize in a way in which that will promote best results for the business.

Absolutely. I also saw a recent study by Forrester maybe a week ago that talked about how enterprises are really looking for this, almost like an all in one platform just to help encourage everything in one place. Collaboration, it seems like a big trend for sure. So when you talk to executives that have been around for a while, do they talk about things they've been burned by, like, oh, we tried automation before, it doesn't work or you know, do they have. Are there barriers that a tester needs to overcome to convince them that you think that they've heard something before and realize that maybe the executive is kind of like a little hesitant to go forward again with something?

Joe, Too often we hear that. Right? That's why you're met with these. A lot of executives, they try to go down this path. They didn't properly plan for it, it they weren't a unified front for these objectives and it fails. And then they associate the failure with test automation when that's not the case. There's so many other factors that go into why it failed. And one I think it goes back to even outside the testing tool is are you doing that proper planning up front for identifying what are the primary objectives for this? And more of a, especially if task automation is newly adopted into an organization, you need to have the right individuals that are testing it, that have some sort of experience or at least the proper training because if you don't have that, it's going to fail. So I think if you're reintroducing test automation generalization, make sure you know who you're speaking to. And are they already equipped to utilize test automation? Are the experience with it, do they have the necessary training and the proper project planning and rollout plan to adopt this? And one of those business objectives, if you don't have that course set concrete up front and it doesn't matter the solution that you're adopting, it's most likely going to fail. So I think if you're empowering your QA team as the representative coming from these test automation tools with those, with that guidance, they'll be much more have the buy in from those key stakeholders to adopt a solution.

Again, I think this segues in pretty well with the four top reasons enterprise testing fails. You mentioned technical resources. Lacking technical resources once again, this goes back to a shift I've seen back and forth before. It was someone uses a tool, you do record and playback and it's pitched as, it's easy as that and that didn't work. Then it's like, oh, testers are developers, they need to have hardcore development skills. And then you try to fill these roles for a tester that knows Java and it's almost like a unicorn, you never find them. So does that feed into the lack of technical resources, that type of approach?

It does. And it's putting this pressure on these individuals that it's, you're really setting them up for failure because they don't have the necessary means for testing or just the bandwidth. I mean so many times I speak to clients, they're like, we need support to roll this out. We just don't have enough resources to actually be scripting all of this or building out this tool. So it's one thing that you really need to be assessing is is there just enough bodies to withstand these projects and then do they actually have the proper training for this and going back to your question around the resources with that. So if they're not equipped with any knowledge around test automation, how do we get them to accelerate that training and adopt those mythologies when it comes to new testing tools and platforms and what's the best way to do that?

I think what adds even more difficulty is time. So time seems to be speeding up and companies seem to be doing more pivots. So maybe a company was focusing on approach one approach and then all of a sudden like, oh, the business is investing in this other area. Let's go over here. Now, do you see that as something that causes failures as well with time, lack of automation because it's all over the place rather than focus. Good process that's been in there for a while.

Yeah, no, it's a great question and a good point you bring up because again, it'll be this, not to beat a dead horse, but if you don't, if you're on that unified front for like here's the primary objectives and that falls off. We see scope creep all the time or other priorities will rise because there's already things in production. I mean, we have to keep in mind when you're going through this valuation to bring on a new solution, you have to be supporting the existing testing that you have going on in production today. So there's a fine balance between the two to make those shifts and advancements while withstanding what is existing and they're supporting. So I think a way to prevent this or at least help with finding new solutions again is ensuring that those key stakeholders have a clear vision and that is properly related to those QA teams and everyone is unified on what are the phase one high priority items that we need to get rolled out. A lot of times I'll work with clients and through the initial discovery, they want to boil the ocean with everything they're looking to automate or they're really, I think overshooting what would be as a phase one when bringing on a new solution or for testing. And it's more, it's an investment of time and dollars that the organization needs to understand that you're not going to be able to fix everything overnight. But if you do the most critical things early on, you'll have better success.

Great. So definitely time is one of the top reasons enterprise testing fails. I guess another one is you mentioned this is and testing complexity. So a lot of times at enterprise level you need to test a mainframe. Still, people don't realize that there's still mainframes you need to test a web browser, you need to test a mobile device, you need to test a hardware sometimes and I think a lot of times with open source tools is fragmentation. So you have Selenium, it only does browsers, you have Appium, it only does mobile. And then you get this kind of Frankenstein approach then to testing because it adds more complexity because you need to do end to end testing. And probably one workflow. Is that something you've seen as well as an issue?

We do and I see this often because so I specifically support our enterprise banking enterprise clients, but it can apply to all leading enterprise organizations because a lot of times you have this legacy products applications or mainframe green screens that you need to be testing and then more modernized technology stacks. So these applications that you have or UX UI testing for browser stacks and they just again going back to manual testing or Selenium like open source, it's not robust enough to be testing all those complexities end to end. And so you're losing out on what actually the user experience could be. And that's a gap, that's a miss. When it comes to testing. It can be a significant challenge. Working with a leading POS organization, we found that they weren't able to test across their POS systems. And so without doing that they weren't really seeing okay, from soup to nuts. Here is some of the areas that are gaps that we have. Here's how an individual is actually using this from a hardware to the actual software. So within the inner workings of that and by allowing for a test automation tool that's more robust, you can capture that more adequately. And I think it goes back to one size, doesn't fit all at times. Again it depends on the infrastructure that you're dealing with. What are the pre existing tools, what's going to be the complexity for replacing any of those existing architecture and really understanding that plan up front again to do a proper rollout.

That's a great point. Especially as more companies go cloud native it seems like you have to skip from one system to another and interact cross systems almost. And that could cause issues as well. Being able to communicate with a third party system when you start off maybe on a web browser and then you have to do an authentication, then go somewhere else.

Yeah, and look at going from okay, mobile to desktop and all of the different ways that as users we can interact with these organizations now. And if they don't have the proper ways to test both external but internal workflows as well. Because really if those internal workflows for those individuals, like if they're their CRM systems or anything from tracking into like robotic process automation. So if they're filling out a form externally and then that data is coming in internally, is that properly getting transcended and loaded and what is that process? So a lot of times it's not captured or there's limitations when it comes to reporting insights. So not really sure what areas within the applications are getting most utilized from a user. And so we're going in kind of blind. And assuming here is the way which that a user is using our application, but really you haven't tested so many other areas.

How do you help testers or companies get over this end to end testing complexity? No, this isn't a promotion, it's just, you know, I've used Eggplant, the company I worked for. I don't know if Eggplant still liked this. It used a really interesting approach where it was more image based. So then it did allow you to have like almost a discrete environment image based and because it was image based was able to interact with all these type of technologies. Is that the type of approach you've seen kind of work or is that still the vision of Eggplant, of what you all have been doing to help with that end to end testing?

It is. And I think that's been really beneficial because the method in which we're testing through OCR Optimal Care Technician and Immunovation Tech Technician, it's allowing us to actually anything that's on the screen, we're then able to test just like a user would instead of like an open source where there can be limitations because you're relying upon the DOM and if anything is shifted or changed, which of course there's going to be those changes. And again if there's any disjointedness between the dev team and product and something gets rolled out, instead of not being able to capture that or having to start from scratch, you have a more withstanding way to be testing that. The other differentiator that we're seeing, not just through edplant, but through other testing tools, is a model based approach to testing. So creating a digital twin, an emulation of the application in which you're testing. And so what we'll find is through the complexities of end to end testing, you can create this primary model that's testing an application, but then sub models that then test that end to end workflow and it helps these individuals run their directed tests but also exploratory testing. So it's opening up a greater scale to which here's the actual coverage and interactions and workflows through all of these different tools making up within our organization.

So can you talk a little bit more about this model based approach? Does it sniff the interaction of your application then knows that the paths that you need to test is it that smart or is it more you test once and then you can just add things visually to add on to that model you already have.

So once you create it and you've run this so individuals will go through and say okay, will identify to the machine here is the directed test. But if you let it run then freely through exploratory testing, it will continue to build upon the intelligence through AI that here's the ways in which. So instead of manual testing, okay, here's 400 tests that we know we've tested. We have to test has taken us hundreds of hours, if not more. By letting this run through the exploratory testing, if we look at eggplants, you're capturing 4,500 different user journeys that are identified that you and it is no fault of the developers through intelligence or anything. It's that there's just not enough time in the day or manpower to be testing all of those possible user paths. So by letting the tool run, you're identifying all of the other possible different user journeys. But also more intelligent testing. There'll be a heat map to identify hey, within this particular area of the application. You've actually not tested this at all and we're finding that this is getting critical interactions from your users.

It's almost risk based testing. Like hey, you have a risk here you didn't even know about. Probably now you do and you know it automatically because it bubbles up that insight.

Correct. So you're catching that even before you're getting to the next stage of okay, this is actually a bug.

It can be.

Did you even know we're going to identify any bugs? Because now you're going to be able to test that area. That was a mission.

So end to end testing. So we went over time as one of the killers of enterprise projects, end to end testing complexity. A lot of times when I see a company starting and planning, they don't look at their full roadmap. So they may start on a greenfield application and not realize two years down the line you're going to have to integrate with, I don't know, a mainframe or a third party service. Is that something people should also look for as well to help them with their end to end testing capabilities actually.

In the planning phase, of course, I mean, you have to be thinking about, okay, current state for the most important projects we have at hand. But also how will this affect other areas within the business and other teams? I mean, a lot of times it will be siloed because we'll be working with an initial team that's dealing with, okay, this desktop or a new application, like a mobile based application. But you have to be considering if we're adopting this new tool within the organization, can this be utilized across all of these other areas within the business. So I think ensuring that you have a more robust solution like an eggplant to be able to test all those complexities is so important.

So another missing piece, and I think that would be the third most killer of enterprise projects is collaboration and visibility or viability. So can you talk a little bit about how collaboration, especially nowadays with modern software, how it should work or how that helps ensure a project is successful?

Well, I think so important up front is ensuring that as a team you're asking yourselves the right questions is things like, you know, what sort of areas do we need to be testing? What are the highest priority areas? I mean, what kind of skill set is required here to adopt this? So there's a lot of pre discovery that I think needs to go on a part of that planning to be successful. And really as an organization there has to be that willingness to invest in the new methodology or advancements when it comes to testing the right testing tools. And so when you're going through that project planning, all parties really need to be involved to ensure they're choosing the right test automation solution. Because what you want to prevent is you get into that testing phase and you've identified we've selected the wrong tool and you've been there, right? So unfortunately so many teams have and you have to backtrack and do rework. And that's a huge undertaking from many QA teams. And so the ways to prevent that upfront is, which can be painstakingly long but really aggressive planning upfront and again from that key stakeholders down.

So this may sound all gloom and doom, but like in a bad economy, if testers aren't making themselves visible to what they're doing and how they're adding quality to the product to give the C level suite insight into what's happening. Do you see that as a key advantage to testers using a tool that allows multiple people, developers, stakeholders, testers and maybe C level executives have different views into the data and get the right Insight that they need so they can know, oh, okay, I understand. I don't need to go in the nitty gritty. But I know that this team found four bugs before it even went in production. So that's a win or something like that.

Absolutely. I mean as a tester I would want as much visibility as possible being rolled up because I think it really shows also more visibility into all of the work that they've done. Like here's how many hours we've been putting this testing. Here's how many defects are bugs we've identified. A lot of times they don't have that reporting. So it's manual inputs. You're kind of guesswork sometimes. And all of those efforts and success that they've had actually is invisible to the executives. So I think of it's more thinking that as going back to the collaboration. Here's where we are. How do we for the areas that are Mrs. How do we fix this as an organization versus continuing to repeat the same mistakes and also to the QA to have that visibility. Here's all the success that we've had rolling out. Let's repeat this and here's how we do that.

Absolutely. You know, I worked with a team that used an Excel sheet and so every time someone failed in CI they list out the failures, try to find out why. And then every month they'd have to go in front of the executives to say okay, this was not a real issue, this is a testing script. Here are real issues. It was just a mess. So I think that definitely no one wants.

To yeah.

Supporting an Excel doc these days.

Yes, for sure.

They have too much to do and that goes back to any kind of manual effort that you can alleviate to automate is what needs to be adopted in today's digital fast growing world. I mean we have the tools available to us so why not automate these processes? Because you want to alleviate this manual efforts from these brilliant developers that can focus on more important projects at hand that in the long run will drive greater ROI for the business and focus on greater innovation.

I love that I say it all the time. You should be focusing beyond on functional testing. Anything you could do to automate a process to help deliver software better faster is something you should be focusing on. And I think that takes you out of just looking at functional tests. I think that's a great point for sure. So I guess so we went over the time is a killer and to end testing complexity can kill an enterprise project. And collaboration also probably the fourth one I would think is maybe resources and coverage. So trying to get all the challenges that many enterprise organizations have to make sure they have the skilled resources and who is accustomed to agile testing and test automation. How do you handle the resource coverage piece that probably caused a lot of these failures as well for these projects.

So we're seeing this too often. I mean, you nailed it with that. Enterprise testing teams are struggling to have experience, experienced test automation experts available as well. So this is one of the primary challenges. So I think it goes back to understanding the business. Who do we have in place today that can actually adopt and currently use these types of tools? Or where is the miss? Or how do we train them properly and what will those efforts take? You'll be surprised. We'll go through an evaluation, then we get to the implementation phase. I'm like, okay, how do you, how do you plan to implement this? And they're like, we have one guy like Gary and so we need more support. And you're like, okay. So it goes back to that collaborative team effort to identify where will there be gaps once we adopt this? And also what are the ways in which we can acquire additional resources. Maybe it's through that team that we're working with that's providing us with this test automation tool, or maybe it's outsourcing some resources and helping secure those gaps because there's a lot of resources out there to withstand this, these needs. But the maintenance will kill them alone if you don't have that in place. I mean, there's just what they're already supporting today. They're struggling and then it's important to incorporate test automation. But again, if this is newly adopted into a team and they don't have the right resources, it can be a challenge up front. So a lot of that planning needs to be properly vetted out so you can roll that out effectively.

Yeah, I think testers need to take a little more ownership of this. A lot of times, like I was the guy with automation and I couldn't take any days off, people would put me on code reviews and then mock me as a blocker because I was so busy. So it's in your best interest also maybe to train up your team as well, to share the wealth to get the whole team involved. It sounds like.

I agree. I mean that I think our job though, as the individual that are experts within the industry for providing test automation solutions to our clients is helping them understand this. And I feel confident taking that responsibility because typically the individual that we're working with to be adopting that Solution would be a director, QA manager. And so if they're not adequately aligned on this, it can be a challenge and you need to train them. So to your point, like you might have been leading a team and you need to feel confident in the solution to be passing those learnings on to your team. So I think if you're effectively doing that, it will really help your team be successful in the long term.

Absolutely. Also, you mentioned training. Product training is critical. Once again, I worked for a company that used an enterprise tool and they didn't know all the functionality that was in it. So they would spend months creating their own methods to do something that was already built in the tool because they didn't know the tool can handle it for them. So do you see that as a gap as well as like people, they buy a tool, okay, here's a tool, go at it without saying, okay, let's roll this out. So you know exactly what the tool does and how it can help you?

I do, but that's why it's important to know are we selecting, not just are we selecting the right tool, but are we investing in the right organization to help us prevent that? Because going back to it's the organization's responsibility to help those clients to be successful. So here's what this will look like, rolling it out. Here's how we flesh out that plan and here's the way that we continue to support you because you're going to need that. And again, it's no fault to the teams like what you were saying, you had such a large plate already. So bringing on a new test automation solution, that's a heavy lift at times. So you need to have the right resources available to you through those organizations. So it's our job to provide that. So I think ensuring that that is available to the teams is pertinent.

Okay, Autumn, before we go, is there one piece of actionable advice you can give to someone to help them with their automation testing efforts and what's the best way to find contact you or learn about more about keysights, a plant or other products by keysights.

So Joe, I think the most important piece of advice is ensuring that test automation is incorporated at the early stages within testing. So again, we talked about that shift left mentality and I think ensuring that that is brought in as early as possible to help alleviate a lot of that manual testing will greatly help so many teams to prevent them from making the same mistakes. And lastly, I would just say any of the C level execs or key stakeholders tuning in, ensuring that you are supporting these QA teams and developers so you can make the proper advancements when it comes to testing to alleviate these challenges for their team. And to look for more information or contact us, you can go to eggplant.com feel free to reach out if you have any questions.

Thanks again for your automation awesomeness. The links to evident value we covered in this episode. Head on over to testguild.com a433 and if the show has helped you in any way, why not rate it and review it in itunes? Reviews really help in the rankings of the show and I read each and.

Every one of them.

So that's it for this episode of the Test Guild Automation Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack automation awesomeness. As always, test everything and keep the good. Cheers. Hey, thanks again for listening.

If you're not already part of our.

Awesome community of 27,000 of the smartest testers, DevOps and automation professionals in the the world, we'd love to have you join the fam@test guild.com and if you're in the DevOps automation software testing space or you're a test tool provider and want to offer real world value that can improve the skills or solve a problem for the guild community, I'd love to hear from you. Head on over to test guild.info and.

Let'S make it happen.