Shift left in End-to-End Testing using AI with Bertold Kolics

Welcome to the Test Guild Automation Podcast where we all get together to learn more about automation and software testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Automation Podcast. Today we'll be talking with Bertl Kollex all about shift left and end to end testing using AI and all kinds of other automation topics. Bertl has over 20 years of experience with pre sales, professional services development, quality assurance and leadership positions at both small, large and startup companies. Currently he works at mabel which does AI based automation. So he's on the cutting edge as well and he helps them to elevate their software testing practices following modern testing principles. Has a lot of knowledge in a lot of areas, as you can tell. Really excited to talk to him. You don't want to miss this episode. Check it out.

The Test Guild Automation Podcast is sponsored by the fantastic folks at Sauce Labs. Their cloud based test platform helps ensure your favorite mobile apps and websites work flawlessly on every browser, operating system and device. Get a free trial. Visit testguild.com sourcelabs and click on the exclusive sponsor section to try it for free for four. Check it out.

Hey Bertl, welcome to the Guild.

Hi, how are you doing?

Great, Great.

Good to see you.

Awesome to have you on the show. As I mentioned, the pre show, you always give me a lot of great feedback on interviews, so hopefully this lives up to your expectation. But before we get into it, is there anything I missed in your bio that you want the Guild to know more about?

No, you got it all. Thank you, Joe.

Cool. So, you know, we were having a conversation on Slack and you were talking about how a lot of people just focus in on one type of testing when they're talking about automation. It's mostly browser testing. So I thought we'd start off with that. What did you mean by. I think you actually have a blog post on this that says test automation does not equal browser based test automation.

Yeah, I mean, I think most of the time people forget when they talk about test automation that.

It'S not just the tip of the automation pyramid that we should be talking about, but that gets the most visibility nowadays and you have to have the right balance and then it goes into other areas as well. Like browser based testing is not just testing with one part of the browser, but it should be with other browsers as well. So just because something is relevant in your context doesn't mean that it's the end all and be all of everything.

Absolutely. And I think you also did the same Thing about the performance testing and API testing in that post, I guess it's more like a high level approach of all the different techniques, not just to be pigeonholed into one.

Yeah, I feel like also performance testing is also again, just because most of the applications nowadays you see with web based applications, people tend to focus on the page, response times and so on. And there's much more to that. And you've been in this industry for a long time, you all know that there's much more to performance testing, scalability testing. You can have unit tests, level performance tests as well. But people tend to forget about it because that's in their own context. That's the only thing they see as performance tests.

And same with API testing too. There are all kinds of different ways of doing API testing. We live in a world of where a lot of these applications built on top of microservices so you can do API testing between these components. There are several providers that have public facing APIs. And also the API testing there is also a little bit different than you would do internal applications or testing between components as part of your integration testing approach.

Absolutely. I believe you said that you think a lot of vendor tools or tools in general tend to really miss out on an opportunity to collect logs and data as well. How would that help with automation or testing in general? You think if more people look to these other areas like collecting and looking at logs, files for information.

I think what has changed dramatically in the last 10, 15 years is that we have so many more applications that are offered through as a service and software. As a service companies, they have this tremendous advantage that they get all that data from customers as they are using their applications and they can leverage it in so many different ways. And we see a lot of good examples of that that you know, across their customer base, how they can make better use of that data, how they can leverage that. And the same goes on for, you know, and I think this is also an opportunity for a lot of these test automation vendors is that as they run or execute tasks for their clients, customers, they can reveal patterns or find patterns that can be applicable across their entire customer base. You wouldn't have that option if you were on your own and running your own tasks because it's much harder. It's only your data. But think about all the applications that are built with the same UI framework like React. It's very likely that many customers run into the exact same automation problem with a particular component, how you interact with it, how it loads, how it responds under Certain scenarios and, and logs and telemetry can help quite a bit. And again, when you're collecting that from many customers, you can make very good decisions and maybe recognize certain patterns. I believe that's going to be the new frontier later on. And then these vendors finally leverage this data.

Yeah, absolutely. You have these big cloud vendors like Sauce Labs and they have all this information on the back end and I'm sure, I think they release a report actually every year that talks about the most used tools or techniques. And like you said, I guess they could figure out over different verticals. Here are the common issues that occur across all these different people and here's maybe what you want to look out for when you're doing automation as well.

Absolutely. And not to mention, you know, a lot of These companies or SaaS vendors use like third party so often what we see vendor, you know, tests are executed that hey, certain services don't load that are external and you know, they may happen at certain time of the day and that may happen across your entire customer base that it could be an indicator that, oh, it's not your application, there may be a service outage here or there and you know, you can make very good decisions just based on that if you could automate that. The reason why I'm always thinking about this is that just from the end user perspective, who are our clients for test automation? Is the tester just test professionals? If I can reduce the time that they need to take to troubleshoot root cause issues, then I'm saving them time and they can focus on other testing challenges or other testing aspects of their applications.

So you do work for a vendor that actually is an AI based vendor. So do you think that's the reason you see this? A growth area as well is the growth of AI machine learning. I know there's open source solutions like Portal that bubbles up insights to your test runs and kind of handles it using kind of a machine learning type algorithm to tell you, hey, here are your trends and here's things that are probably a real issue and here are the ones that are probably a bug in your test case.

Yeah, I think that there will be more role for machine learning in this space. Even at my current company, Mabel, there's a limited set where we use machine learning for and I feel that it is a little bit overhyped in the industry. How we are leveraging the extent it is leveraged and the thing that.

It almost feels like buzzword almost and then there is already starting to have you see negative connotations with the word like AI ops popping up here and there. I think it is definitely have the potential. But something is just to again to keep in mind is AI is not a new thing. It has been there for a while, but what has made it more available and accessible nowadays is that two things, the cheap computing and the other is the amount of handling this big data and having big data sets. And again that goes back to the we can collect logs, we can observe applications much better. There are lots of clients and customers feeding us their data. So that really caters to and creates this opportunity to use machine learning and AI. There will be always certain aspects of testing and software testing where you don't want AI or didn't want artificial intelligent machine learning going on because you want really deterministic behavior. I know. So there may be context where it's. It's not just you don't want that kind of thing. But I think there will be more and more opportunities to leverage machine learning.

You know, actually as you were speaking, I loved actually I know it sounds like you don't like the term AIOps but that's probably a term that's going to become more popular. And I just grabbed aiops gil.com because I think that's a good domain name. But anyway, so you know, you say it's overhyped but I've actually been seeing more and more of these tools and thinking they actually are living up to some of this hype, especially with self healing. So can you talk a little bit about where you think it's not hype or where you think it is actually adding benefit to testers as they're testing.

To give you like concrete examples. So for example, the auto healing kind of work is at least where we are. It's at my current company, it's not machine learning based, it's just a complex rule set or they have certain strategies that are implemented but it's not artificial intention. But many vendors try to, you know, or some vendors may try to.

Make it appear as it were an AI based solution. So I think that's where I think there is a little bit of hype. You know that you know, just because you have a complex set of rules and you have a special sauce or recipe, it doesn't mean that it's AI. And there are always more places like visual change detection where artificial intelligence or machine learning is a natural fit. It's a good fit to find out differences between two different snapshots of the same application.

So what is AI then? You did mention people aren't taking advantage of logs. Do you see actually leveraging machine learning? That'd be a good case for machine learning because you have all this data that someone could never manually go through even if they wanted to in enough time. But leveraging a real machine learning algorithm can actually help bubble up these insights that you may be missing.

Yeah, yeah, absolutely. That's what I think. I mean, for example, just to, in my very simplistic view, for example, a good example was for DEAs is, you know, several years ago when some of these spam filters came along that, you know, just again you were just using a simple Bayesian filter to figure out if something is a spam or not and assign a score to it. And a lot of these mail programs started filtering spam based on that. It was like, again, you are feeding a lot of data similar.

Thing with the logs or patterns that you can see in this value. You're testing these applications.

So like I said, you do work for a vendor and we talked about looking outside of just doing browser based testing. Does Mable offer anything else that you think people don't know about with functionality that's beyond browser testing?

Yeah, absolutely. I mean what I believe is happening in general. Mable of course supports, for example, testing emails. I don't know how many times you received an email that has their percentage sign, first name percentage sign. You could be testing the email with our platform or you could create assertions inside PDFs. Let's say your website is generating invoices and you want to just make sure that you know all that data shows up in inside your PDF. So these are, or now working on, you know, making API testing a little more accessible and easier to use.

Because at the end of the day, you know, I think wherever the industry is going is like, just like you have seen with the like the source control repositories like GitHub, GitLab. In the beginning they were just again just git repositories. But nowadays they offer many more services around them like CICD integration, security scanning and so on. And then this is the way they try to differentiate themselves. I feel that just like Mable, other vendors also try to go beyond browser testing so that they offer the infrastructure that you can run your tests on the cloud as a service so you don't have to deal with the maintenance and so on, but still do more and more of your testing using their tools and tools. So these vendors try to differentiate themselves by offering these solutions like capturing more logs, visual captures like images, videos of your Sessions, doing more of these again, email testing, API testing, testing, PDF or other kind of content as well. So this is what I believe MABEL is doing too.

Now, this might sound like blasphemy, but how about exploratory testing? When people say exploratory testing, they think you can only do it manually. But could you use an AI based or a machine learning type algorithm to crawl your site and just bubble up things that could be issues just because it's exploring it and saying, hey, we're finding things that we didn't expect and then having a person that's look at it and then make judgments whether or not something's correct or wrong?

Absolutely. I feel like that these AI based tools and any kind of tooling that helps you just to provide you more data.

Can be a really great input. And I'm a huge fan of exploratory testing and we actually do quite a bit of those at mabel because there are always certain aspects that you didn't anticipate or you have a different kind of way of looking at things as you are maybe in a different role. We work with different customers, so you have a different mindset. But the data that you can provide with this AI based, like a link crawler that crawls through your website and gives you insight into where you may have certain problems or load issues or and so on, that can complement that information and then maybe guide your journey as you are doing those sessions.

I look at it as like something that I want to, like a Swiss army knife. I want to carry it in my pocket all the time. That helps me to get through the challenges I need to do my job.

And so I actually recently released a new design of my website, Guild Conferences. So head over to GuildConferences.com and check that out. Buy a ticket also to an upcoming Secure Guild conference. But anyway, I asked my community and you're in my Slack channel, so you're one of the main contributors there as well. And I asked people to take a look at the new site, let me know if they saw issues and then reported back to me. I'm like, how's he finding these issues? Did you actually use Maple's crawler to find some of the issues that you were seeing or like just explain your, your approach to what you used?

Absolutely. And again, this goes back to your previous question, a little bit about exporting testing and you mentioned that you have a new site coming up. So I figured, hey, let me try our tool and see what I can go through. Look at your site but let me do that with Mable and record some tasks along the way, have it run through that task, look at some of the interactions that I can do with the screen and some default tasks that Maple creates is one of them is the link crawler that goes through all your site and looks at broken links, but they also captures a lot of information about the site itself. And we can use that data, for example, to calculate what kind of coverage your own tasks do, as opposed to what the link crawler sees in your application. So that kind of gives you a sense of how much else you could be testing and what you missed out on, which is really important. Again, just another data point for the test.

That may or may not matter because maybe that page is not important to you, but you may want to test certain aspects of your applications that I really want to be able for somebody to go to this site, register for a conference and complete a checkout. So this is like a critical flow for you and you probably want to create a test for it. But again, as I look at a new application, a new site, I usually, just as any tester would do, just driven by curiosity, just try to poke around, look at different aspects, scroll up and down, see if the page loads. How long does it take to load the page? Are there any kind of broken links or are there any kind of outside external dependencies that don't load or point to locations that you didn't expect? Look at the reports that some of these automated tests create for you. Look through the logs. And this is what I like about our platform, is that every step of the way in our test we capture screenshots, we capture network logs so I can look at them. And this is some of the information I shared with you were coming from the tool itself. I didn't have to do it myself. All I had to really do is just navigate through your site and then report some steps.

Yeah, you actually found something called like a same site cookie issue that I don't think I'd ever would have found because I would have been just focused on the UI so much. This is more like an under the COVID error that was bubbling up. And you save me money because my developer starts charging me after two weeks after I find these things. So thank you for that. So besides that crawling, what are your thoughts on accessibility? I've been hearing more and more about accessibility, which I'm happy about. But do you see vendors starting to try to help solutions where accessibility testing? Or is that something you think even AI machine learning is really isn't going to be able to tackle.

That topic is so near dear to my heart because you hear about these three things.

You hear about a few areas that are getting really, really important nowadays. You know, one of them is, of course, you know, testers need to learn data science, you know, learn with a lot of big data. The other areas, security. As you can hear about all these breaches and problems around that. But I feel that it is somewhat, you know, doesn't get enough exposure. Accessibility in itself. And the reason why I feel passionate about it is accessibility in general is that, you know, we have a population that is aging and we have more and more people who have, you know, some disabilities. You and I are getting into an age when we are needing glasses. So, for example, contrast on the size or the font sizes are becoming important to us. And then it can be a differentiator for businesses trying to have a nice web presence or developing applications that cater to a larger crowds. So yes, I think this is going to come up and it is ripe for innovation. That area. A lot of good tools already available.

But yes, I feel that there is going to be a growing need for doing more and more accessibility testing. So to make sure that our sites are.

Catered to a larger crowd and a larger customer base that you can.

Attract to your business. Absolutely. And.

In some other area of accessibility, again, just this recent resurgence on our spotlight on inclusivity as well. You know, just thinking about who can use your service, do you enable all those folks that may have some sort of disadvantage not just because of physical disabilities, but other kind of disadvantages? How do you make your site more accessible and cater to those crowds? I feel that, you know, these are areas that are going to be becoming more and more important.

Absolutely. I forgot who mentioned this, but they said they brought up a quote that at some point everyone's going to be accessibility challenged at some point in their life. As you mentioned, I'm getting older, I'm looking at things, you know, my glasses putting the device away from my face, really far away. So, you know, are there any major offenders you see with accessibility that people completely overlook? That is totally unacceptable now in 2020.

You know, I, for example, I don't see an excuse for having websites with poor contrast.

Because I think it's just there are so many tools available that, you know, simulate how, for example, colorblind or partially colorblind person would see for your website. I don't see an excuse for not having like alternative text for a lot of the images. This is just so simple. To do so that people can navigate through the websites. And again, just thinking about, hey, why do you need mouse control or certain controls for certain websites? Can you navigate through them just using the keyboard, for example? You know, these are very basic things and there are a lot of tools already available. So not allowing that or limiting your user base to those who can fully function and fully control all these devices is just. I don't think it's acceptable now in 2020.

Absolutely. So what are some of those tools that you think are helpful? And does Mable have any functionality or anything on the roadmap that would help with accessibility testing?

Right now, unfortunately, we don't have anything yet.

I don't know about anything on the roadmap, unfortunately, but I use tools that. There are some Chrome extensions like X is one of them that I use. There's a contrast checker extension that is available on both Chrome Firefox and there are some just general.

Tools that include accessibility checks. For example, I think it's called Lightning from Google that in addition just to giving you some information about your website, also gives you some sort of scores on accessibility aspects of your website.

So in your bio you mentioned you follow modern testing principles. So I think a lot of times, especially myself, I love tools, get caught up in the tools. So what are some modern testing principles you think were valid back when I started like in the 90s and what's still going to be valid 30 years from now that people should be focusing on as well?

I mean, to me the most important message from modern testing is again just, you know, the customer is the ultimate judge of quality. So.

You know, the big theme and major theme in modern testing is just again using that data that we have available from customers to make our decisions and drive our decisions around testing priorities. What has changed compared to 20 years ago is that 20 years ago maybe we had the majority of software applications were on prem products and vaulting on telemetry or sending back information to the vendors was the exception. And then nowadays with more and more of these applications and the availability of broadband Internet and the cloud vendors and so on, now collecting these data is, is much, much easier and is the norm. So making decisions and basing information on data and not just gut feelings or a few product managers talking to a few customers.

Can be complemented by a lot of data that you can gather.

All right, so Bertl, the whole point of the show was to talk about shift left and end to end testing as well. So what are your thoughts on shift left testing? I know it's kind of a buzzword. But do you see this as a real shift and what does that mean? Shift left to you.

To meet the big change and the availability of these end to end testing solutions.

What we are also working towards is enable.

Testing application end to end and from very early on in the development stage so that you can for example.

Create your own test as you're developing your application and run end to end test and make it easy so that developers don't have to switch context so that hey now I have this source code in one repository. This is something else some other place I have to switch context in order for me to see my changes tested end to end. How do we enable this developer workflow to be as seamless as possible? So when it comes to like merging your changes into the main code base, then the tests go along with it. So the risk of regression, the amount of rework is lower. That's really ultimately the holy grail so that we can bring this testing on. Because again that also goes back to your modern testing question is that.

The fiction and what was ingrained a long time ago in us is that only testers can do feature testing. And that is just wrong. And I don't believe in this message. But also by enabling developers just to go beyond unit testing in their regular day to day workflow and creating end to end tasks, we are enabling them to do this more of these feature testing from the end user perspective as part of their development routine. That is really the game changing model here.

I agree with you 100%. I guess the problem is, don't mean to put you on the spot, you work for a tool vendor. I'm curious to know, do you actually do this? And if so, any tips to get your developers more involved with testing earlier in the development lifecycle?

Absolutely. We do this. We are dogfooding our products. So we are also testing Mable with Mable and whenever there is a pull request going on for changing our web application, we kick off certain tasks. We can preview our changes in an environment that is set up just for those changes even before they emerge into the main code base. And developers can now run a test from their desktop, from their environment locally, even in headless mode and go through that test themselves before even creating a pull request. So yes, absolutely. And I truly believe in that. And I wouldn't feel confident talking about it if they didn't practice what we preach.

Absolutely. And I guess another thing, when we talk about shift left usually focusing on tools and developers to write code early But I think bdd, when it's used correctly for collaboration, really helps before bugs even get into the requirement, before a developer starts writing code, that's the ultimate shift left. Do you see that as a process that can help folks or one that teams are embracing or using correctly?

I do really like the concept of bdd. Absolutely. To me the bdd, what it enforces and why I like it is that the notion of putting yourself in the shoes of a customer and the end user, that is ultimately the challenge even when I was a developer, is to maintain that mindset. How can I put myself into somebody else's shoes? How can I empathize with the end user so that I'm going through the trouble using our application and it is working the way it does. I like the fact that BDD enforces this kind of thinking, this outside in thinking and again by creating these end to end tests that are mostly end user focused and as you say, very much in line with what BDD would do. I like that from this perspective again because we are focusing on the, on the end user, on the customer, how they would be using your product.

So early in the show you mentioned about SaaS providers, how they're able to get all this data and collect logs and such. I think Meeble is almost like a SaaS provider as well because it runs in the cloud, right? So it's capturing this data. So I don't know how much insight you have into this, but are there any best practices or things you've seen that teams are doing that are working for them that you could share or tips around that?

To me the, the important thing is again, just getting the whole team on board and making sure that they understand what this provides. How it is different running your tests in the cloud versus running them locally. But if you didn't have tests before, of course that's a whole different set of problems as opposed to when you're migrating from an on prem to the cloud and also working with your team and getting. I think the more successful it is in places where you can get not just the testers that create and maintain these tasks, but you can get the developers and the whole team to look at tests or test failures and analyze as problems because there's a lot of information that these platforms provide a lot of data and that you wouldn't normally be able to do and capturing often part of the, and the hard part for the tester's job is just to capture all that information and when you see a problem and then share that with the rest of your team and tools, including mabel. What it allows you to do and makes it easy for you to do is to share that information with other members on your team so that they can look at the, they can look at the exact step, look at the screenshots, look at the HAR file that we capture. You can look at the chrome step trace for each step in your test. So you know there's a lot of data. So again, you don't have to reproduce because it's already there. So that can save a lot of time. But so you need to get your whole team involved. That's where it works best.

Great advice. Okay Bertl, before we go, is there one piece of actual advice you can give to someone to help them with their automation testing efforts? And what's the best way to find or contact you?

Yeah, I mean, my best advice is that we should all remember that we are all knowledge workers. So keep learning. You know, I think continuous learning is super critical for everybody in this profession. And there are a lot of great resources, you know, podcasts like yours, Joe, just doing great service for this community. You know, there are meetups, there are online courses, online providers that are, there are a lot of them nowadays, great books, blogs to follow. So just keep learning, keep learning new things. And you know, there are areas that are pretty hot right now that you may want to, you know, even if your current employee doesn't let you do that in your during your job, you can, you know, take an hour or two on the weekend to learn about these trends. And you can contact me via Twitter. I'm on LinkedIn as well.

Thank you Bertl, for your automation awesomeness. For links to everything of value we covered in this episode, head on over to test guild.com forward/A319 and while you're there, make sure to click on the try it for free today link under the exclusive sponsor section to learn all about Sauce Lab's awesome products and services. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack automation awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Automation podcast. Head on over to testguild.com for full show notes, amazing blog articles and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.