Debunk Autonomous Software Testing Myths with Tobias Müller

In a land of testers far and wide they journey seeking answers, seeking skills, seeking a better way through the hills they wandered through treacherous terrain but then they heard a tale of podcasts they had to obey O the test Illumination Testing podcast Guiding testers with our nation awesomeness from ancient realms to modern days they lead the way O the Tuscalon Oasion testing podcast with sweeps and lyres the bards began their song A tune of knowledge, a melody of code through the air it spread like wildfire through the land Guiding testers, showing them secrets to behold O the Chesskill Automation Testing podcast Guiding testers with our nation awesomeness from ancient realms to modern days they lead the way oh, the Testill Automation Testing podcast.

Oh the test scale Automation Testing podcast With lutes and lyres the bards began their song A tune of knowledge, a melody of code through the air it spread like wildfire through the land Guiding testers, showing the secrets to behold.

Hey, it's Joe. And welcome to another episode of the Test Guild Automation Podcast. Today we'll be chugging with Tobias all about debunking autonomous software testing myths. I talk to a lot of people and there's all kinds of things floating around and Tobias is an expert. He's really going to shed some light on a lot of these things that you may not really understand correctly or maybe we're misunderstanding. So really excited to have him back on the show. If you don't know, Tobias has over 25 years of experience in software development and over 11 years in software testing and regulated markets. So he knows all the hard environments like medical and fintech and it's backed by tons of real life experience. And he has a visionary mindset of how automated software testing should work nowadays because he's seen how it worked before and he's part he has his hand in the pulse of AI and all the things that we're going to nowadays. He and his team has built a really cool autonomous software testing platform called Test Results IO, which really helps you increase software quality standards worldwide, Uses a bunch of customers, bunch of clients, works beautifully. So really excited to get his thoughts on this as well. He's also an international speaker, he's actually spoken at Automation Guild a few years ago and an AI trainer at the National Training Week in Malaysia. So he also knows AI. It's not just a buzzword for him. Really excited to have him back on the show. You don't want to miss it. Check it out. Hey, before we get into today's awesome interview, be sure to check out test results. New product user one that Tobias mentions later on in this episode. It's really cool because it aims at helping you to automate non automatable applications and also enhance your end to end testing with automated software testing from the most important part from your user's perspective. To sign up for free now using the link down below and tell them the Test Guild section. Hey Tobias, welcome to the Guild.

Hi Joe, thanks for being back.

Yeah, great to have you finally back. It's been a while. So a lot of things change in the industry. I think last time we spoke was maybe might have been almost two years ago. Like what's the big buzz now going on with AI?

I think the buzz moved on from AI to autonomous testing. Isn't it? So like two years ago it was all about AI, AI and how we integrate AI into product. Meanwhile, it's all autonomous testing.

Absolutely. So that's a good point. So I've been hearing a lot of, like a lot of things. I'm not sure exactly people know what autonomous testing is. I think they just equate it with just the way they understand automated testing. So what's the difference between like automated testing and autonomous testing nowadays?

Yeah, and Joe, that is a good question actually. What is autonomous testing? I mean, in the end I dissected all of the different tools out there that say, like, hey, we do autonomous testing. And most of the time you find something like you already had in the 80s, like a web crawler that is taking screenshots from all of the pages and is comparing the screenshots every single day and highlights if there's a change on the page. And in a sense you can say that is actually autonomous because yeah, it just does it on its own and it finds problems and it can tell you there is a problem. Apparently you need to fix that. But that is nothing that I would actually see as an autonomous testing. It's more or less like a static crawling of web pages. Right. So what else I've seen in the meantime is like things, and that is most of the autonomous tools are wrappers around Selenium or Appium or Playwright that actually take some HTML code, push that to a GPT4 model or any different kind of LLM like Claude, actually get back to Fields and try to identify some locators of that and then generate some code for Playwright and you can run it. And that's also not really autonomous. Right? Because in the end, autonomous. What does it actually mean? What is autonomous testing? Because if you think, think about it. What if you would have autonomous testing? And that is what I always say. That's also what I say it in the training week. It's actually like if you would have autonomous testing, real autonomous testing, then we wouldn't need to have testing at all. Because if I just have a system and tell them, hey, find me all the bugs in that system. Why cannot tell the system in the beginning behave like this system. So if there is autonomous, if there's full autonomous testing, we don't need testing at all. So most of the autonomous part is actually marketing and I'm guilty for that myself because we also claim like our system is autonomous and that's, you might remember from last time, we also use the term artificial intelligence a bit different. So we use AI as a pre abbreviation, but we use it for augmented intelligence. And a lot of competitors also jumped on that wagon and said, I need to have copilots. I mean that is the newest trend right now, right? Yeah, it's like Microsoft started with Microsoft copilots and now everybody has a copilot. And the best part on those is actually there's even one window out there that has a copilot that is actually explaining the test cases that you automated with the tools. So apparently the test automation is so complex that you need to have a copilot on top to explain the test case afterwards after it's automated. And that is where we are right now. Right? And that is actually the problem that people think like autonomous testing, but in reality. So automated testing is a story about automated testing really told. So what is the benefit of automated testing these days? Or is it like, hey, automated testing didn't work like we always promised and now we jump over to AI based automated testing and that doesn't work as well. So now we have autonomous testing. And then that's one of the biggest problems like in the whole market is like all of this marketing blah, blah and these passwords that are you being used and everything is autonomous these days. And none of that is really autonomous. There's like a new player on the market as well that is saying like we do have autonomous testing. And the only thing that I always do with all of those tools is like I go to the tool because most of them can only test web pages, right? That is where it starts. So you can only test web pages and that's really autonomous because sometimes you need to go to an SAP system in the backend or stuff like that, or you need to check your mobile application. And what they actually do is like they have full autonomous testing, 100% autonomous testing. So I give those tools a chance and just enter the URL of the tool in the prompt and tell like, hey, test this system. And it's actually you, so you should know what you are if you are autonomous. So test yourself. And all of them, actually all of them that I tested failed on the login already. So they're not able to even log in if it's a two factor authentication. And I mean that is where autonomous is right now. Right, Right.

So let's break that down a little bit more. So do you believe in autonomous testing or you think it's just a buzzword and it's just automated tested, assisted by AI? How do you define it?

The thing is, for auto, I do believe in autonomous testing, but I think we are not there yet. It's like Elon Musk proclaimed, autonomous driving will be available in 2012. Right now it's 2024 and we still don't have autonomous driving. So I believe in autonomous testing, but not like in the sense of you give the system an application and it will just test it and report all of the bugs. So what I do understand in terms of autonomous testing is like, it just makes testing a lot easier in the sense of there is a use set of data. So what do you use as an LLM, for example, that is actually trained on the different use cases of different kind of applications. So what I have in mind if I talk about autonomous testing is more or less like you give the testing platform a system to test and tell you that this is a core banking application and I want to test you the financial transaction parts. And then the system based on all of the training that it got on core banking systems and financial transactions, it can actually test up to 80% really autonomously and can give you an indication like, hey, so the last 15 times I executed the following test scenario for core banking applications for that specific path, it resulted in this, and for your application you're testing right now, the result is completely different and I cannot make sense out of that. So you need to look into that and I think that is where autonomous testing is going.

Gotcha. So what's the role of a automation engineer or software tester when it comes to either autonomous testing once we get there, or even now? Do you believe in being replaced or do you still see it as needing a human being in the middle of all this?

There's always a human being, right? I mean, did anybody in history of mankind ever get really replaced? Like you hear everybody's Afraid right now, like my lawyer is afraid that he gets replaced by AI and the company that actually support our funding is also afraid that they get replaced by AI because it's all document work. But nobody ever got replaced. I mean, they just extended their skills. Right? There is. So right now you're doing something and now there's the next evolution in technology. Now you're doing whatever you did before and you can now do it in a fraction of the time that it required beforehand. And now you can build up new skills and you can be more efficient in time. I mean, that is actually how the whole economy works, right? Like, how much time can I provide within one hour? And that just needs to increase because otherwise the economy has problems.

It's a good point about lawyers. I actually had a long contract another company gave me. I said, let me just put it through AI and have it read it. And it brought up five points. I'm like, all right, those are valid five points, but I'm not going to trust it. I got an actual lawyer. It just helped me have a better conversation with the lawyer about potential pain points or issues with the contract. So it didn't replace them. It just helped facilitate better lawyering, I guess. Or better.

Exactly. It made you the better customer in the end, right?

Yes.

Because you were able to ask better questions.

Yes.

And that is what AI right now is. It's augmented. It's like it helps you to understand what is there to ask the better questions. So you are more efficient because you have more precise questions. The answers from the lawyer are a lot better in the beginning already. So you don't have this one hour long discussion before you understand the topic because you already got those insights and you are more efficient in the task.

Yeah, exactly. Because I just looked at. I'm like, I don't know. But then once you have the AI have some talking points, it really did facilitate a better and quicker. I saved some money probably because it spent less hours of her having to understand what the heck I was looking for.

The same is true for us, actually. Like all of those financial documents that we went through, it's like over 200 pages. And in the end, I mean, they are more or less similar. Right. And they always have two similar points. And without any experience, it would have taken me days to go through those documents and to even understand them, actually. And then finding hiccups in those documents even takes longer. And that is what AI can do these days. Because, yeah, it's. There is already a lot of data on that, but it doesn't replace the intellectual worker. Right. The intellectual worker is just more efficient based on those tools.

Absolutely. So I've been speaking with a few people and they say that how a lot of times AI systems bubble up insights and then a person needs to look at it and determine whether or not it's a real issue. They think that's going to go away and you're going to have to start trusting the AI to make the right decisions. Do you see that as the right model or is that where you see things going?

I don't know. I'm the kind of person that doesn't even trust my coworker.

So I guess it depends a bit on character. Right. How much are you willing to trust anyone in the first place? And the second one is like, how much am I trusting artificial intelligence? I mean, and that's the point is like, go back like 12, 15 years and everybody was claiming, yeah, we will have to test automation and there will be 100% automation on that. And you just get the results out of all of those tests and you just need to trust it. And where are we right now? We still have test cases as, I mean, as sad as it sounds, but we are still in the beginning of all of that. And now I already hear story about like, yeah, we just, we just need to trust the artificial intelligence to give us the right results. And we've already seen that that might actually be a problem. Again, I'm not on the side that says like, don't use artificial intelligence. It's just like, use artificial intelligence in a way that it helps you. And the part is get the results fast, but control the results. That's my current stance on that and I think that is my stance for the next 10 years as well. Even if you think about the progress that you made in the last two years, I mean, if you remember, OpenAI was start like seven years ago.

Yeah.

And where it went in those seven years is tremendous. From if you see GPT1 in comparison to where we are right now, it is in seven years. And that is tremendous. And the thing is, nevertheless, I would stand my stance that in the next 10 years there won't be the major breakthrough where you can just trust the result of an artificial intelligence. Just keep that in mind because a lot of people believe in Elon Musk and say like, hey, autonomous driving and that will be ready by 20, 2012. And it's not, it was not in 2013, it was not in 2014, it's still not autonomous. And I think as if And I'm claiming that because. But people need to understand driving is an extremely simple problem. If you think about that. A human needs about, I don't know, in average 20 hours of dedicated training on how to drive the car. Right. And how long do you need training for an SAP system? It's for sure more than two and a half days. Because before you can actually use the SAP system or core banking application or even Microsoft Word, I mean how long does it take to master all of those features? And that's the thing. So we are not able apparently to train an artificial intelligence or multiple artificial intelligence in an agent based model that is able to solve a problem that we need training for like about 20 hours. And we already claim that we should trust a system that can do much, much, much more infectious more than that simple system of keeping a car on a lane and making sure that there is no crash.

Right.

Which is difficult but for a human a rather simple task actually.

So what do we do as testers and developers? We're being told all this. Our managers are probably hearing it saying, well just use autonomous testing, you don't need to do anything else. I've asked a few people. This is AI and testing overhyped, properly hyped or underhyped. And how do we know what it is?

Yeah, right now it's overhyped, that's a given. I mean right now it's overhyped and it's still in the hype cycle. I think it's not yet settled down because we still don't know exactly what it actually is because what you hear is like hey, test automation, we only automated the execution. Now we have Genai and we can automate the rest of the pipeline as well. We can generate test cases, we can actually classify results, we can even generate bug reports that are readable and the developer can implement them. And oh, even there we can even extend that bug can be Fixed automatically with GitHub, copilot and stuff like that. So we don't. Right now AI is everywhere and we already know that if something is everywhere, most of the time it doesn't work out. So there are specifics where it will help. And back to your question actually about the manager and that is why we call our platform autonomous testing platform because that is actually what managers want to hear. Right. So you actually, and that is marketing for to get the food in then to show the solution. But what I think what is really the future is more or less like a system where you do exploratory testing. And while you do your exploratory testing. You are actually training the test system in the background so that you can create a model, like an abstracted model of your application, where you can then use prompting in a more sophisticated way today, like simple prompting, actually, to set up all of your use cases. And that is where I see then the first step in autonomous testing and the next is that all of those use cases that are prompted all over the world are combined more or less in an autonomous system.

Nice. So I just had a random thought. You know how we've heard about telemetry being implemented into systems where when a bug occurs, you could kind of trace it exactly to what's causing it. So if I had an automated test that's running, say, in production, and it finds a bug, then not only can it alert someone that says there's a bug, but if they had the telemetry to know what caused it, they can go ahead and fix it as well. Do you see us getting there? Is that crazy thinking or would you trust us?

No, no, that is actually what I hear as well. It's like one of our investors is actually in this OpenTelemetry area, and he's a brilliant mind. The thing on that is also, I think it's also overhyped right now, to be honest, because there's a lot of problems. You get a lot of data and a lot of people have a lot of silos with a lot of data tracing data in there, but they cannot make sense out of that. And we are still in that phase on making sense out of the telemetry data that we receive and then actually getting any benefit out of that data. But you are right, if we do have test cases that can run in production and you can actually monitor them via our telemetry data, so then we can easily identify that test case, actually trigger the bug, and the bug is caused by this problem. I think that is also part of the future. That is a given, and that is closer, actually, because all of the telemetry infrastructure is already there. Microsoft is gathering telemetry data, I think, for the last 25 years already. So we know how to do that. The majority doesn't yet know how to get the semantics out of all of that data. But we do know how to solve the problem. And we also know how to run test cases in production. Right, and you also know how to isolate production. So we can take the production system and isolate that actually as a secondary production system. And that is something that is based on the new technologies that are available. To us right now. And I think that is coming in the next three to five years, that we really use telemetry data first to capture what the test case did and what bugs were actually produced by that test case and where they need to be fixed in the code, but also to generate new test cases. Actually if the telemetry data is not based on test cases being executed, but from real life customers that you actually know. Like, hey, okay, we rolled out that release and apparently 10% of our consumers triggered that bug. And that's actually a chain to trigger the bug based on the telemetry data we received. So we generate a test case out of that. And that's the point. Then actually, do you still need autonomous testing in a sense? Because you have your customers that are running it through the product and are defining exactly the way that needs to be tested. And I think that is a lot closer than full autonomous testing. I think that is happening in the next two to three years, that it's really productive. So it's already available today. You can do that. You can set it up on your own and it works if you do everything right. But I think that is something that you can use out of the box in the next two to three years.

Yeah, I love that. That's going to be awesome.

That will actually be. The thing is, nobody talks about that, right? Yeah, everybody's talking about artificial intelligence and autonomous testing and stuff like that. But that will be the real game changer getting that telemetry data that is already out there in a connection to actually what happened and in connection to the test case and then actually being able to fix the bug because you know exactly where it originated from. That is the major game changer that's coming.

So you heard it here. We're going to replace SRES and developers as well then, right?

That's the point. We replace everyone, right? We replace everyone. And that's, by the way, that's a joke of the history, right? When I was still in college, there were open positions at Microsoft that were called like sdt, like software developer and test. And I back in Germany then I always thought like, hey, that is a software developer that's still being tested. Somehow they still need to prove that they are able to do this stuff. And finally I understood that, no, no, no, that is a software developer that is actually writing automated test cases. And then the term disappeared, right? Because test automation or testing and automated testing was democratized and everybody could do it. The business could test themselves and stuff like that. And hey, SDD Key is coming back as a role. So people, companies are looking for software developers in test again. So that is actually interesting development if you think about that. Everything is getting more artificial intelligence. It's getting a lot easier, it's getting democratized. Nevertheless, the old roles are coming back. That should give us a hint.

Absolutely. So let's talk a little bit more. You know, we talk about autonomous testing, AI. A lot of companies are coming out. I want to talk about, you know, you do work for Test Results IO, you're one of the co founders. So how's your approach different? Like you're pretty a straight up guy. Every time I speak to you, you don't give me vendor pitch. So like how is Test Results IO different? Because you were built differently and I think yeah, it's a unique way. So maybe people who missed the last episode, what is Test Results IO and how do you approach say autonomous testing?

Yeah. So right now Test Results IO is like it's based on a completely different approach to identify elements. Right. Because right now what people often use or always use this, like they try to identify with which input do I need to intact, where do I need to put my text in, where do I need to select the value and they need to find, try to identify a locator for that. That is wrong in every case. So that is why Test results are just differently and what we use, because there is already a locator. There is something that you as a user tells you that is the right field. Right. If you see a field like name, you know exactly that the next field should be my name. And it's not the nearest field by the way. It's. It's a logically next field which is dependent on the culture that you were growing up and stuff like that. And that is actually what we put into Test Results. So it works completely different to any other system in the way of how it locates where it should interact. That's the first part. And then we do have the autonomous part where we are only partially autonomous. So it's not full autonomous testing. What we did is we trained the system to understand how scrolling works. So if it doesn't find an element on the page or in a list, for example, but it can identify that there is a way to scroll and there are a myriad of ways to scroll, actually lists, for example, and you cannot automate that by hand. So we trained the system on what does scrolling actually mean and how does that work and what is the specific on scrolling that you can be at the top, you can be at the bottom already. You are not able to scroll at all. And that is what we trained the system and that is where we do have autonomous functionality in the system that you can just tell it like, hey, I want to go to that, I want to click that element or I want to go to that page or something like that in the system and show us that it can find that element if it's available. And that's a difference. And we also do have a different approach to Gen AI because what you see in Genai is always the typical approach is this like, hey, I do have my requirements or I do have screenshots from my application, generate test cases for that. And I also did a test with somebody on LinkedIn and they said, hey, I want to have test cases for winword. And then you always get those generic test cases like, the performance should be good, you should be able to log in, you should be able to not log in with wrong credentials. It shouldn't allow you to log in without providing a username. So the typical test case is that everybody who is involved in testing can write down out of their head. There's no real benefit. You're generating thousands of test cases without much benefit. So what we do different is like we are generating this model of your application and then you are prompting use cases, like for a used car platform, that's my primary example. It's like you tell test results, say, oh, I want to buy a car, I want to buy a nested mountain, for example, for $6,000. And that is what you put in the prompt. And based on the model that you generate in the background, it will actually generate the interactive steps. So you don't need to understand how your application works. You just give it a use case. And the system is already smart enough in most cases to identify what that sequence is. And that is how you build up your testing, your test scenarios. That is how you build your user journeys. Because we don't talk about test cases anymore. We really think like, okay, what does the user want to achieve and what are the steps to achieve that actually? And that is then auto generated step by step. And that is aware of the context. If you tell the system, hey, I want to like, I tested the system for German Parcel Service and put in the information I want to send a package to Japan and generated all of the steps. And I just said like, yeah, by the way, it has to, has to have four kilograms, which is like, I don't know how many pounds, eight pounds. And it just added that at the right position in the chain of interactions. That is what we are doing different. So the claim is not like, hey, give us the requirements and we generate all of the test cases for you. We democratize the test generation. It's really like, no, we do augmented stuff like give us the use case and we generate the steps for you. Because that's the nasty part, right? Nobody wants to drag and drop those elements and select the right input values. I mean, that's the nasty part. It's exactly like for the execution, I mean, the intellectual part is coming up with a test scenario and with the use cases that you can actually challenge the system with. But not executing a test case or dragging and dropping those elements, keeping the data in, that's the difference in test results.

Also what I've seen is, you know, when you work in an enterprise company, you pop different tech stacks almost. And sometimes I need a test for a web browser now I need a test to test the back end, I need a test to test the main. Because I think using an image based approach, it seems like I think you're able to get over those type of journeys almost. Am I understanding that correctly?

Yeah, completely. And that's the point. I always have to laugh if I see like if we recreated end to end testing and then it's based on Playwright or Selenium, stuff like that, where you say, okay, end to end is for you, you web pages really.

Right? Yeah.

And that's the point. In enterprise, you typically have a huge tech stack. You have web pages, you do have rich clients, you have fat clients that do have some backend systems, you have database systems and stuff like that. And that is if you do have a visual based approach, then you can overcome all of those limitations because the technology doesn't matter anymore. It's even better. I mean also for web pages, you know better than I do because HTML uses Canvas, for example, so you can draw something on the HTML page and you can technically not test that. All of those tools do have fallbacks that they can compare an image, but if you compare pixel perfect image, that it just doesn't work, it's just not reliable. And that's the problem because even within the supported tech stack, like if you say I do have a solution that can test web pages, there are still the corners which cannot test or the next one is Google released Flutter like three years, five years ago, I don't know. And the first rendering for web was pixel perfect. So it actually used Canvas to render the whole interface and all of those typical testing tools Were not able to test it. Now they do have adapters and different renderers for Flutter. It all settled down. But if you do have a universal approach, and the universal approach is more or less like abstract the human eye, if you do have a universal approach, then those limitations will never be there, even in the future. I can now already proclaim that any new user interface framework that you might come up with, with, or anybody else at the big companies come up with, we can only support that today.

Love it. You know, I have in my notes some, for some reason, user1 technology independent. Is that what this. What does that phrase mean?

So that is something that we are working on right now. That is what I mentioned as the technical solution that I see as the future is where you actually have access to the system where you want to test all of the different applications that you want to test and you just click through the applications and the system is learning in the background actually how the application behaves in those different scenarios. And based on that, it is generating an abstract model. And based on that model, you can afterwards then prompt all of the interactions that you want to have automated and that supports you tremendously in your day to day work. And that is actually what we call the user 1, right? Because there's typically an opentelemetry or telemetry area. There's also already you get the telemetry data for development. So it's more or less that the first dashboard that you ever want is the one based on telemetry data. And the first user that you ever wanted is more or less like your tester. And that is why we call that the User one technology that we are coming up with like in the next, I guess in the next six to nine months that will be publicly available. You can already register today for being a design customer for that. And that will exactly be like. Also like. I don't like the term game changer anymore because GPT uses that a lot in the text that it generates.

Yes, game changer in the realm of automation testing.

Exactly. Everything is a game changer these days. And that is just the next evolution because it combines the exploratory testing of a human with intellect with a combination of being able to autonomously define test cases based on a model that you as a human generated and based on the use cases you defined. And based on that, you actually set the boundaries to allow the system in the state that we are right now to understand is the output that I get right or is it wrong? And that is the next evolution in testing for me, it's more or less that you don't do test automation in typical way, but you just test your application like an exploratory tester. And afterwards you can just prompt your test cases pretty simple and get precise results on that. So it's not about trust artificial intelligence anymore. It's about you set the boundaries and you need to trust what you said actually. And that is completely different approach once again to how to use all of that technology that are available today.

Yeah, it seems like the perfect combination of keeping the tester in the loop, using their expertise on the application and their assumptions and having AI just.

Exactly. That's the point. I mean, everybody who worked in testing is like, I still remember the story. It's like the best guy who wrote, whoever wrote a specification was one of the testers in my team because they actually understand the system and they understand that the user and they can actually. They also think extremely precise. So they come up precise requirements that are actually implementable and that are testable. And that is you need somebody like that in the loop. It's always like there's always in robotics there's meanwhile the term like human in the loop. So there's like a human that controls the robot and the robot needs to be sure that it doesn't, it doesn't break the human somehow. That is like those human in the loop robots. And I think that is also required for testing. It's like there will be more and more robotic stuff like automated execution and stuff like that and automated design, but there's always a human in the loop. And that's what I mentioned beforehand is like, I don't see full autonomous testing for the next 10 years.

Wow, very nice. So what's the name of this new product called?

Is it that will actually be Test results are User One.

User One. Okay, cool.

Yeah, it is really User One because that is the first user that you want to like have your system exercise.

Nice. Is there anything else you're working on? I also have written down something called test results IO instant. I don't know where I got that from.

But yeah, that was just a technical term for actually bringing user1 to everyone in the world because we did some. Yeah, as you mentioned in the, in the intro Slack test results are spreading around the world and we noticed that the current delivery process is difficult in different areas of the world because it just works different. And the test result that we instant is more or less like the technology vehicle that we use to deliver the test results. So User One experience to everyone. So if you use a Mac today, even if you use the most bizarre, unique setup that you can ever come up with, you will be able to use user1.

Very cool. Okay Tobias, before I go, is there one piece of actual advice you can give to someone to help them with their autonomous testing efforts and what's the best way to find contact you or get our hands on what sounds like User one from Test Results IO?

Yeah, for user one I guess you can easily register some test results there. You can register to be a design customer. We are always looking for feedback. So if you want to get hands on stuff that is ahead of what you currently see of co pilots and all of the stuff that's currently coming out like the best new thing, if you want to be ahead of that wave, just join the waiting list for being a design customer. We do have some expectations for design customers so you really need to get hands on on the system. You really need to have experience in automated testing and you really need to be dedicated to actually give us your time that you really go into the system and challenge it.

Actually, thanks again for your automation awesomeness. The links to everything of value we covered in this episode. Head on over to testguild.com a496 and if the show has helped you in any way, why not rate it and review it in itunes? Reviews really help in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack automation awesomeness. As always, test everything and keep the good. Cheers.

Hey, thanks again for listening. If you're not already part of our awesome community of 27,000 of the smartest testers, DevOps and automation professionals in the world, we'd love to have you join the fam@testguild.com and if you're in the DevOps automation software testing space or you're a test tool provider and want to offer real world value that can improve the skills or solve a problem for the guild community, I'd love to hear from you. Head on over to test skill.info and let's make it happen.