228: Information Not Automation with Ingo Philipp

Hit it.

Welcome to the Test Talks podcast, the place to go to geek out on software testing. And now your host whose mission is to help you succeed with test automation, Joe Colantonio.

Hey, it's Joe.

And welcome to another episode of Test Talks. In this episode, we'll be talking all about continuous testing. What is continuous testing? What does it mean to you and your team's testing efforts with our guest, Ingo Phillips, a distinguished evangelist at Tricentis. So prepare to discover what the fundamentals of testing are and why it's all.

About information and not automation. Also, I think this is the first.

Time I've ever heard a tool vendor quote James Bock and Michael Bolton when.

Talking about test automation. So.

So really good stuff. You don't want to miss it.

Check it out.

Test Talks is sponsored by the fantastic folks at Sauce Labs, the cloud based automated testing platform that eliminates the need to maintain your own selenium grid and test infrastructure. Try it for free today. Visit testtalks.com and click on the sign up now link under the homepage sponsor section.

Hey, Ingo, welcome to Test Talks.

Hey Joe, thanks so much for having me.

It's great to have you on the show today. And today I thought we'd talk a little bit about how digital transformation requires continuous testing. But before we get into it, can you just tell us a little bit more about yourself?

Yeah, sure. So my name is Ingo Philipp. I'm on the product management team at Tricentis. Now, Tricentis is a software company. We are located in Vienna, the beautiful capital of Austria, in the heart of Europe. And that's, by the way, Joe, the reason for my weird accent and pronunciation. So I'm sorry for that already. And I'm working as an evangelist at Tricentis, so that's, I think, all you need to know about me.

Awesome. So I've been watching a bunch of different presentations you have on YouTube and a lot of themes that come up is continuous testing. And it's actually something that's been coming up more and more on the podcast. So what is continuous testing? Is it DevOps, another name for DevOps, or is it, you know, how would you. I guess the question is, how would you explain what continuous testing is at a high level?

Yeah, that's a good question. And I think Wikipedia already has an article about continuous testing and we also influenced that article. And what we are trying to explain here is that the concept of continuous testing, so the reason it was originally proposed, it was a way of reducing waiting time for feedback to Developers, basically.

Continuous testing as we see it is the process of executing automated test cases as part of the software delivery pipeline. And the reason why we do that is to obtain immediate feedback on the business risks that are associated with the software release candidate. That's in really short terms how I see the topic continuous testing. Hope that helps, Joe.

Yeah, absolutely. So I guess the question is, it seems like as people go towards DevOps continuous delivery, continuous integration, they need to integrate tests earlier and earlier in the development life cycle. But from my experience, I've seen people just struggling with normal automation. So from your experience, how many people are adapting to continuous testing and maybe what are some top issues you've seen them trying to adapt it for or having issues adapting?

Yeah, absolutely.

What we see is that more and more companies are constantly trying to adopt DevOps, trying to adopt continuous delivery, continuous integration. But especially in the enterprise landscape for large organization, there is still a big, big challenge out there.

It goes basically hand in hand with your development approach because the reason why we want to be able to test in an uninterrupted way is we want to be able to keep the pace with our agile development, so to say. So just from a pure testing perspective, and of course a lot of people are struggling with that, a lot of companies struggling with that. And I think the main reason for that, and that's purely based on my data points now, Joe, is that most people first see all those technical challenges before they see the problems they have with the, the people they have in their companies. So what we are trying to also teach those companies is, or to tell those companies is to first figure out, well, do you even have the people with the right skills in your company in order to do that? And this includes people that are able to do automation and to do automation at the speed of development, so to say, so to be able to immediately provide feedback to your development. I hope that somehow answers your question, Joe.

Yeah, so it's a tough one because a lot of people are looking for a silver bullet and like you said, it sometimes comes down to culture or not having the right product mix. So how do you find these developers or testers with this mindset?

It's pretty hard to find them and I mean, it's pretty hard to even create such a culture, such a DevOps culture, because that is what the terms actually means to me. It's just a cultural movement. And in the same way we also need to, to adapt testing, right, in order to keep the pace, as I've said. How do we find those Guys, Well, I don't have the rule for you to find those guys. I think it's just a process of constant improvement. And what we do also internally is we are trying to improve ourselves by participating in what we call community of practices. So this is just a set of people among or across several squads or agile teams or even tribes that you know, have.

The same challenge in their business like software development, like software testing. And you have a huge variety of different people with different skills, like developers, like automation engineers, like manual testers, like business analysts and product owners. And we try to connect those guys in order to make their work more efficiently. Because I am as a tester, I was a test, I'm still are a tester. I can learn so much from my developer friends. I can also learn so much from my product owners. That all drives my testing. So what I learned from my developers as part of those community of practices is basically about automation. What I learned from my product owners is, well, what are the tests I need to automate? So where are the risks? For whom are we building that product? Right, so that's the way basically we roll it internally and in our projects.

The next question then is once you have the people and they understand kind of the process, like what are the benefits you can give someone to tell them, hey, this is why we're doing it. Because a lot of times I see people saying this is what you should do, but they don't necessarily say why they should do it. And I think if more people know why, maybe that would help the culture to actually embrace it better.

No, absolutely. And this is still something that is missing in most of the companies. And so what we are trying to do when we get into those companies, we are trying to connect the dots, right? To connect the dots between the business problems the companies are facing with and the challenges they have in software development and software testing. Because, you know, still terms like digital transformation, terms like DevOps are still some kind of buzzwords out there for some people. These are fuzzy concepts and we really try to break it down into manageable pieces. Like let's start, for example, digital transformation. So almost every company tries to have some kind of digital transformation initiatives to drive those initiatives forward. And I like the way Forrester defined it. I think it was in 2014. In one of their reports, Forrester said that businesses, all the businesses out there, must continuously exploit digital technologies to both create new sources of customer value and increase operational agility in service of customers. Now this means we need just to understand here three terms. The first One is obvious. What is a digital technology? The second one is customer value. How do we create that value? And the second one is all about speed. That's kind of obvious. Now a digital technology, in my point of view, is any kind of technology that generates, stores and processes data in terms of two states. And that's software, right? So I see digital transformation really as a software led transformation. So that's the first thing. And the second one is through all those initiatives, through exploiting those new technologies like mobile or web technologies, we want to create new sources of customer value.

That is again a point where we try to connect the business with testing. What you're doing, especially in product management, is you try to identify the problems or the needs or the wants or the desires of your customers. Now once you did that, you try to find a solution in your product for the problems of your customers, right? And you need to do that fast because as you know, the competition is just a sequel gig away from us. And this usually takes a lot of experiments, a lot of fails to find solutions to the problems of our customers. But we need to be aware of the fact that all the software we build is just a solution, okay? And if the product isn't solved but by our software, then our software simply doesn't work. And that's the reason why we have to do that and also the reason why we have to do that fast, because it's so easy for users out there to just switch to another provider. Now, once we found a solution to the problems of our customers, we ship our software to our customers and then the value of our software is realized once it is in the hand of our customers. And that's how we create value for our customers. So that's how we solve their problems. And of course, the customers then provide value back to us by paying for what we have built. And this kind of cycle is, that's the feedback cycle. And the goal is to accelerate that feedback cycle, to accelerate that value exchange. From a testing and development perspective, the way we support that value exchange is to enable ourselves to.

Wrestle through the entire software delivery pipeline multiple times in ever decreasing cycle times.

That's basically how we do and how we roll and how we connect initiatives like digital transformations to software testing and software development. That is what we try in our product, to give the context. And of course, it depends on the company, it depends on the problems the, the company are trying to solve, and it depends on their market segment. But that is, that is what we're trying to do. So we try to highlight Those problems connect it to all the challenges we have to software development. I hope I had now shed a little bit of more light on how we are doing that. Just let me know if that helped you.

Yeah, so I guess once the product's in the hands of our customer, we need to get their feedback. So in order to get that feedback, it's almost like a, a continuous loop that it goes through a pipeline. So how do we make sure our pipeline is strong through each stage? So when we do get that feedback, it feeds back into the pipeline and it's able to evolve with what we're hearing from our actual customers.

So your question is, how do we get the feedback back from our customers as fast as possible and put it into the pipeline? Yeah, I mean that's the tough job. Right. And it's not a discipline a software development or a software test has to do. I think.

It'S something the whole company needs to work on. Right. It's software development, it's software testing, it's support services, it's sales, for example, it's product management, it's product marketing. So I think all those companies need to be aligned and need to agree on various channels how to get the feedback in. I mean it's for web based application, that's quite an easy task. You can monitor user behavior, you can immediately also monitor the scenarios they are trying or their goals they are trying to achieve in your software. And you can have an almost an automated feedback loop into your software development. And then based on what you monitor in your application, you can decide, well, what do we need to fix, what do we need to implement in order to create more and more customer value. That's fairly easy for those innovative technologies out there. But it's quite hard to do that for large enterprises, for example, where you still have some kind of desktop, big fat applications running on your desktop and you don't have those continuous release cycles. There is a lot of human interaction still involved. What we do for example at Tricentis is we have people who care about that. We are constantly in touch with our customers who monitor also the way our tool is being used and trying to make as many as possible decisions, how we shape our product based on this data and based on the interactions on the communications we have with our customers. And that's basically done by product management.

There's an automated part behind how to.

Handle the feedback and there is a still a human driven part behind it. And I think we will never get rid really of the human driven part because we always build Software for somebody else. Right. So we build software in order to solve problems for someone else. And that someone else is, in most cases it's a human being. And so we need to understand their pain points quite a lot. And I think this is the reason why we still have to have a lot of conversations with our customers in order to figure even out what are your problems. Right. So what do you need? Because what we see is that most customers, you know, say, well, you know, that is the, that is, that is what I need. But they don't tell us the problem. And it's also a long process to really find that out what's the underlying problem. So that's basically my answer to my, to your question.

Awesome. So so far we have touched on the human aspect of continuous testing from the culture to the customer. But in order to have that happen, we also need tooling. And another thing people struggle with is open source is great, but with these tools, a lot of times it's hard to have a cohesive plan where everything's talking with one another and makes sense. So any tips on how to choose tooling? For once someone has the culture mindset for continuous testing to start putting the correct tooling in the right places for the pipeline.

Sure. I mean, you know, I'm part of a software tool vendor, so I can talk for ages about the software we are building, but I don't want to do that. So let me try to tackle that problem from a more conceptual point of view. So I think you need to be first of all aware of, so what kind of projects or products are you working on? And in case you want to see black and white, you can say, well, you have, you're part of an innovative project like you're building a mobile app or a web app.

Which is based on your enterprise system landscape, for example, infrastructure. But you need to get that project out as fast as possible in order to see, well, do we even solve the problems of our customers and can we generate revenue with what we are doing here? In those cases, that is what we see out there. Most of the testing teams are still doing a lot of exploratory testing because automation takes effort and that usually don't have even time to automate test cases. When they want to do it, they mainly go for open source tools like Selenium, for example. Now that's a good way to do that.

It's a nice way to do that and to handle testing and test automation in those smaller short lived projects, I would say. But what we then see is Those projects then turn into products, right? And those products usually evolve. And so that means the complexity of the products gets big and big and sometimes, you know, they go into some kind of a maintenance mode once they are not innovative anymore. And then you are really part of enterprise.

Architecture, an enterprise landscape. And so you need to be able to do proper regression testing also on end to end level, right? And that is what, what we see where, you know, scripting tools like Selenium's fail. Okay, so the reason is fairly simple. I've, I did it also.

All too often in my past. I wrote those frameworks. You start doing that, you start creating your linear frameworks, for example, then you sometimes realize, well, you need to make changes.

That are caused by technical changes in your applications or by changes to business logic. And you need to go into all those scripts in order to make those changes. What you do is you evolve your framework, for example, from a linear framework to a data driven framework to a modular data driven framework. And then at some point in time, well, you realize, well, you're basically the only person on this planet who understands what your framework is about. And you need to support from others like business analysts or business experts or people who are driving the business that help you to understand.

What needs to be tested. So you need to follow some kind of risk based approach. And so you need to make your framework somewhat accessible to those others. And then you start building, you know, keyword driven frameworks. There's also what we see quite often now. Then you realize, well, I just did it for a certain technology. I just did it for HTML or for Java or. Net, for example, but I'm not really able to connect all those frameworks with each other. So that is a huge task. And that is where usually the framework implementation stops. And that is the point where large companies then switch over to commercial tools like our tools. Tricentis Tosca, we are a tested platform. We for example, support here out of the box automation across many different technologies and it's fully integrated in a bunch of services we provide through Tricentis Tosca. This is for example, test case design. This is test data management to handle your synthetic test data and to also mask and make use of your production data, for example, in order to make your test automation more robust at the end of the day. Because without proper test data you won't be able to do that. And you know, then you have cases where you don't just need to account for functional testing. There are also, you know, load performance testing aspects you need to cover Security, testing aspects you need to cover. And, and that is what our platform basically provides to you. So you just need to learn one language in order to make use of it. Right. And.

It'S fairly simple, you know, to go from UI automation to API automation or to mobile automation, for example. And it's still a challenge to do that with all those different open source tools because you jump from one tool into another and you basically need to learn a new language. I would say it's still possible. Everything is possible, I would say, but that is how I see, you know, the adoptions of tools evolving over from simple projects over to large and more complex projects. So that's all I have.

All right, so we touched on the culture, the customer, the tooling. At that point, how do you know or how do you tell your team what to know, what to automate and at what level? And what do you mean by testing? Continuous testing? Is testing just a GUI test or do you also consider continuous testing, unit testing?

Yeah, so testing, I think.

Let'S put it that way. So I want to reduce the term testing to search, to unit testing, or to functional testing or to load testing. So testing in my point of view, in its most general form is just a search for information. It's just a search for quality related information. And the reason why we search for this information is to close the gap between what we know about our products and what we don't know about our products. And we want to close that knowledge gap. And the main reason why we want to close the knowledge gap is to reveal risks in our products. Right. And that the risks are the type of quality related information we are mainly looking for in testing. And the risk is nothing more than anything that might threaten the value of our products to the people they matter to, I would say, and these are the stakeholders. So a risk, in short, is just a problem that might happen. So that's how I see testing. And from that perspective, you can do testing in two ways. I would say you can say, well, you can do it by checking or by exploring. So that is the distinction, for example, Michael Bolton, James Bachelor make and by checking that just mean you evaluate your software by applying algorithmic decisions rules to specific observations in your product. Which simply means with jacking, you just try to find an answer to the question does this assertion pass or fail? And the process of finding an answer to that question that always gives the binary result, right? It's always yes or no, it's always one or zero, it's always on or off. And that's machine decidable. And that is what you should automate. But there's another part of testing which is all about exploration. So here you evaluate your software through exploration and experimentation in order to reveal those risks. This is mainly driven by a lot of human activities such as modeling, studying inferences, observation, and a hell of a lot critical thinking that goes into that. And so that is how I see testing. First of all, Joe, so you can separate it in those two areas, I would say. Now how do we then define what to automate and what not to automate? I think this was your second question, right? Or your first one.

Yes.

So we do it really based on risk. So what we do is we are trying to build a model of our software. Okay, it's still a model. And what this model consists of are all the requirements that basically describe our model. And a requirement for me is nothing more than a service we provide through our customers, we provide to our customers through our products so that they can achieve their goals, whatever these goals are. So that's how I define a requirement. And we built then a model of our software based on the requirements. And we are well aware that this does not reflect our real software product, does not reflect reality. But at the end of the day we have some kind of a tree like structured set of requirements and then we prioritize those requirements. So the first step is we capture those requirements, we structure those requirements and then we try to prioritize requirements. And that is where the business perspective that gets into the game. And here we just, for each requirement in relation to all the other, we just ask the question, well.

What is the frequency of usage of that requirements? So in every requirements there's a certain amount of functionality or non functionality hidden. And so how often do those users really make use of that functionality? And that can be supported by data, by monitoring your application. That's the first dimension of the risk. The second one, and really keep it easy here and simple, is the damage, the potential damage that might occur when that functionality described at that requirement won't work at all. And that's an estimate we do and based on the, you know, the frequency of usage and the potential damage. Then we calculate the risk, we estimate the risk which is part of that model. And having that, we then assign test cases to those requirements. What we have at the end today is not just a list of prioritized requirements weighted according to the risk contribution, but also we have a prioritized list of test cases. And from that list it's like having a product backlog for your product owner. We have a prioritized list of test cases, and then we start with the high priority test case and we start to automate them. So that's in rough terms, how we roll it, basically how we decide what to automate and what not to automate. It's purely risk driven.

Awesome. Now you've mentioned risk maybe four or five times. Well, why is risk so important? Does that help us when we get to release that we only need to run the most critical tests and not get bogged down with tests that really aren't testing anything of real value?

Yeah, you know, what risk quantifies is just the potential of losing something of value. I think that is how I would define risk in its most general form. And of course, I mean, if you have a user scenario, for example, in what we tries to reflect a certain user scenario by a test case.

Especially in end to end testing, and then we ask ourselves the question, well, what would happen to our business if the user wouldn't be able to do that? Right. And here we make a distinction between, you know, the edge cases, the edge test cases, and our straight through test cases. So the straight through test cases are those test cases that reflect those user scenarios that has the biggest business impact. And.

So that's basically. I would. I think I forgot your question, Joe.

I was just saying how you can use risk to determine what tests you need to run when you're delivering. And it would cut down your test suite so your tests run faster and be more targeted.

Okay, so no, we can't use that information to be able to run our test portfolio faster. It just will tell us, in which order do we have to run our test cases that are, for example, part of our regression test portfolio in order to be able to mitigate risk as fast as possible.

So I guess if you're trying to release software quickly, as fast as possible, you want to automate as much as possible, but you still have manual test. How do you still deliver quickly and be able to execute your manual tests? I know you mentioned exploratory testing, but it sounds like that's not just at the end, it's also in the beginning with modeling. Is that how it would work?

Yeah, I mean, at the moment there is simply no chance to remove the human impact to testing because somebody needs still to write those test cases. Right. And somebody still needs to automate those test cases. And this takes time to do that. Now, in order to reduce that time it takes, we say, well, we will never be able to test everything, exhaustively and we will never be able to automate all of our test cases. And so risk is really just the quantity we make use of in order to figure out what needs to be automated. A concrete example, for example, Joe, let's imagine you have a user story, okay? And that user story is brought in by the product owner in Sprint planning, for example, and you discuss the user story. And here we already start applying some kind of a BDD approach that allows us to first of all understand that user story and the problem we want to solve by that user story. And then we also break it down into a set of acceptance tests, basically examples that help us not to automate that user story, but to first understand that. But we then immediately make use of those acceptance tests in a kind of a given when then structure that we then feed into not just the software testing, but also software development. And for a user story, you then can have hundreds and hundreds of those scenarios, right? And you will never be able to automate that because it will literally take you weeks to do that. So then you will say, well, we just have about one day or two days time in order to finish that user story from an implementation, from a test perspective and from a test automation perspective. And here the risk again comes into the game. Here we say, well, out of those 500 potential test cases we have to automate, how many can we automate? And what are the ones that provide are most valued in terms of risk? And that is where risk again gets into the game. So it might happen that we just say, well, we just automate the straight through scenario. So one test case per user story in order to get it done quickly. And this allows us to finish development and testing in shorter cycle times. At the same time, we are well aware that we probably will miss most of the most important stuff out there. But again, we will never be able to test everything as exhaustively as possible.

Right, Absolutely. That's a good point. So, Ingo, I know digital transformation, a lot of times that comes disruption because people aren't aware of something that's on the horizon. I know you speak with a lot of people, your company has a lot of partners. Is there any trends you see coming up in the next few years you think testers need to be aware of?

I think one of the big topics is AI, artificial intelligence. And I mean, you can discuss this topic in two ways. You can say, well, how do you really test those.

Learning systems, I would call it, in its general form, and how those computer systems, those self learning computer systems can help you in supporting testing. And I think you already had a couple of podcasts out there. I think the last one with Lisa Crispin, right, she talked about Mable and other tools, how these tools support the tester. I think the test itself.

Needs to change a little bit in order to be able to test those systems. And Angie Jones talks a lot about it. And so if a tester wants to learn how to do that, I mean, one way I would recommend right now is just follow Angie, listen to her talks because she gives a lot of examples how she tested, for example, recommendation systems that are based on machine learning, on neural nets, for example. And I think the way we then approach testing and also test automation has to change because we cannot decide anymore in terms of 0 and 1, right? There's a lot of statistical verifications we have to do. And the approach, and the whole testing approach also changes because there's a lot, a lot of learning involved, a lot of learning how those systems even operate. So what are they doing? So what are their internal workings? And so I've learned a lot, by the way, from Angie about that. So that would be my first go to person when I have to give some advice to some tester, especially for that topic. So AI basically is the first thing I see out there.

Yeah, awesome. Yeah, definitely. It's something I've seen more and more of and I think testers need to be ready for. Okay, Ingo, before we go, is there one piece of actionable advice you can give someone to improve their continuous testing efforts and let us know the best way to find contact you or learn more about Tricentis.

Absolutely. So I would really dig into the fundamentals of testing here and my advice for all the testers out there would be please do not forget what the game you're playing software testing is about. As I've tried to explain it before, it's all about the goal here is really information, not automation. So my advice is, guys, please don't confuse testing with automations. These are two totally different things and.

I've seen it a lot, that testing cannot be encapsulated in discrete procedural units. We call test cases, right? And I think Mikey Bolton once put it that way. He said, well, you know, in the.

I think he said it well. A recipe is not cooking or A file of PowerPoint is not a conference talk or a sheet of music is not a musical performance. And so a test case is not testing. So guys, you can test also without test cases and you can definitely test without any tools and without automation. So that's basically my advice.

Yes, you can find me on LinkedIn and you can find me on Twitter. So my Twitter handle is just my name, Ingo Philip. It's the same on LinkedIn.

Thank you Ingo for your continuous testing awesomeness.

For links to everything of value we.

Covered in this episode, head on over to testtalks.com228.

And while you're there, make sure to click on the Sign up for a.

Free trial link under the Exclusive Sponsor section to learn all about Source Labs.

Awesome products and services. And while you're there, make sure to.

Click on our new Guild conference, Automation Guild 2019, which is happening February 4th to the 6th.

It's going to be an awesome time. We have some great speakers.

You want to check that out? Head on over to automationguild.com so that's it for this episode of Test Talks. I'm Joe and my mission is to help you succeed with creating Automation awesomeness. As always, test everything and keep the good.

Cheers.

Thanks for listening to the Test Talks podcast. Head on over to www.testtalks.com for full show notes, amazing blog articles and other automation awesomeness.