Chatbot Test Automation with Christoph Boerner

Welcome to the Test Guild Automation Podcast where we all get together to learn more about automation and software testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Automation Podcast. Today we'll be talking with Christoph all about chatbot test automation and a cool tool you can use to help you with your chatbot automation. Christoph is the founder of Body and former CEO of the company TestPlus and his function as the leader of the department's development and testing. He focused on mobile testing automation, crowd and community testing, and a lot more. Together with Florin, he's actually invented Bottium and it's becoming the industry standard for test automation for chatbots. So after Selenium for websites and Appium for apps, I think this is the next tool you need to have in your toolkit and that's Bottium, which is the next step for bots and aut. So if you have anything to do with any type of automation, especially bot based automation, you don't want to miss this episode. Check it out.

The Test Guild Automation Podcast is sponsored by the fantastic folks at Sauce Labs. Their cloud based test platform helps ensure your favorite mobile apps and websites work flawlessly on every browser, operating system and device. Get a free trial. Visit testguild.com sourcelabs and click on the exclusive sponsor section to try it for free for 14 days. Check it out.

Hey, Kristoff, welcome to the show.

Yeah.

Hi, Joe. Hi guys out there. Thank you very much. It's a pleasure to be part of your guild today.

Really excited to have you on the show. I've been hearing about Botium for a while, so it's great to have actually the inventor of Botium joining us. Before we get into it though, Kristoff, is anything I missed in your bio that you want the guild to know more about?

Well, it was maybe the best bio.

I've heard about myself.

Pretty cool. Thank you very much.

Cool. I guess. Before we get into it, how common do you think chatbot testing is nowadays? Is it a fad? Is it something you see as a growing area that you think is missing testing and resources to help testers test it?

So first of all, it's something new. Yeah. So testers, test automation engineers have to adopt. In fact, all the big enterprises out there have in their future strategies conversational AI, meaning bots and virtual assistants. So all this stuff is coming up more and more, but in fact, I still get these questions in my inbox asking me what is the value of testing bots? And then well, if you are a passionate test out there, this, this really hurts. Yeah, but I can compare it. As you said in my bio, I was running before test consulting company and then we were doing a lot of test automation for, yeah, maybe 15 years ago for websites and then the last five to 10 years, mainly mobile test automation. And also there, there were these times where we had to convince people that you need to test. And we see the same thing right now with bots. It's a topic that is really coming up, a lot of attention, but still people asking me, what is the value in testing chatbots?

So what is the value of testing chatbots?

I was not expecting this.

Well, the value of testing is derived from quality and quality means at the end of the day the satisfaction your user can feel every time he talks to your chatbot or virtual assistant. This is why, I would say why we call our whole discipline here quality assurance.

It's all about quality.

And from company side, it's of course also about confidence. So you need a high confidence level to ship your products into production environments. And this is I think from utmost important, especially in agile project. And yeah, of course when we are talking about values and agile projects, then automation is the key enabler there. Test automation usually leads to an early identification of defects and of unwanted behavior, simply.

So there's.

This is not different when we're talking about bots.

Yeah.

And at the end of the day all this stuff we know, decreasing time to market as a result, costs of quality are decreasing and at the end you amplify new end user satisfaction. All this stuff at the same time. Now this, this is how I would explain the value of testing bots.

So, and I wonder with your experience over the past year with COVID and everything, I would think there'd be. There was probably an uptick in people utilizing chatbots. Chatbots, because more people are going on websites and they're not going out to maybe a bank physically, but virtually. Maybe they're going on and trying to get their questions asked. Did you see that as a trend that more companies were like, well, we didn't even think about how many people are going to start using chatbot. And now we kind of need to definitely put some resources in to test it.

Yes, definitely. So as bad as this whole pandemic thing is for the topic of conversational AI, it was definitely a boost, to be honest, because starting with this pandemic, so many companies were knocking on our doors and telling us, well, we get 20,000 calls in our support centers every day and it's impossible to answer them. Can you build a bot for us? And in those days we were usually. Or the first reply was well, we are delivering here a test automation tool set for bots. We are not the bot developers. But yeah, one year ago in March we even, we helped a lot of companies also to develop some kind of Corona bots. I remember a lot of especially when it comes to financial institutions and so on. People were very unsure what is happening with my mom money at my bank account and so on. Will the ATM work in the upcoming days? So there were so many questions coming in. And then on the other side, a lot of these bots were also used internally. So employees were asking these questions, Do I have to stay home?

Will I be fired? How is this whole thing now going on? And yes, it was definitely a boost for conversation in the items.

How exposed could a chatbot leave a company? I'm just thinking of, you know, when you SQL attacks or security attacks the same like when you're testing a website. Is it the same concerns you need to be think about when you're testing a chatbot?

Yes, definitely. So that's a very important point and also one we are following here at bottom. So what we are delivering is, let's call it a holistic test approach. And in this approach all these topics we know from testing websites on mobile apps are coming again. It's about different test types, approaches, it's about functional, non functional, it's smoke regression, acceptance and it's of course also about security and performance. Or a very big one is also GDPR for example in Europe and there we have a quote of 89 whatever percent when we run our GDPR test that it fails with the first question. The first question inside is what are you doing with my data? And 89% of all bots we are asking cannot answer this question at 8. In fact security is a very important topic and we have seen cases where we testing with Botiom. Well, let's say that the case you want to avoid as a company is that your bot cannot resolve an intent that reacts or replies with something like sorry, I don't understand. This is a typical case and as a user you don't want to see this. But the worst case is when the bot breaks and exposes some security relevant data and so on. For example, and this usually happens when bots break. You see sometimes what is the NMP engine behind what, what is the bot building framework used and so on. And this is for an attacker already a first option. And I mean More than that, we are doing our security testing is running for different stages and the first one is testing on an API layer, for example. This can be compared to API testing. When you're talking to a bot building platform or a chatbot, usually you start to run your test against the API because there we can really execute this enormous test sets. And this is what we see in testing chatbots that quantity really makes a difference. This is when, when I was testing web and mobile, I would have said it's more about doing the right tests than doing a lot of enormous tests. When it comes to conversational AI, this is different because people can ask bots everything. So in theory, test sets are infinitely large. So all this discussion about test coverage.

How many or how much can we.

Reach is even wor compared to what we had before. But yeah, coming back, we usually start an API layer because there we can execute something like 100,000 conversations with this. Typically regression tested for chatbot V like this takes a few seconds. And in big teams with 30, 40, 50 developers, conversation designers and so on, I see in Sprints stuff happening like from a few hundred to even a few thousand builds, meaning a few thousand times you run a smoke test. And if your test automation is that fast, you can run also the whole regression 1,000 times in sprint. Nothing stops you from. And yeah, therefore we start on this API level and especially there we are using for security testing a set attack proxy from. So the OVASP is pretty well known from from the OVASP guys out there. So everything we fire to a chatbot and everything that comes back as a response we send through this proxy. And people are pretty, yeah, they go crazy when we do something like a conversation where we say hello to a bot and check if he detects the welcome intent because this is 2min thing to automate this. And the result is the shows the conversation is green. But then we go on the results to a security tab and it shows 30 different security vulnerabilities or something that fired. So this could happen. The cool thing here, and this is kind of our philosophy we're trying to make, I don't want to say demystify test automation, but we are trying to make it fast and easy and security testing with what box is switching on the do security testing switch. And then every request, all these kind of all these 100,000 questions going to the bot and all responses coming back are going through our ZET attack proxy and are tested through hundreds or four hundreds of possible vulnerabilities. And therefore you don't need a lot of security knowledge in your team. So this can be done by everyone. But to add here, this, this, this is not the whole story. So especially when it comes to, I don't know, a financial institution, a bank or something, I wouldn't say if you do this and you don't have any errors there anymore, then you can ship to production. You are totally safe. This is not the whole truth. The whole truth is this is taking the low hanging fruits, let's call it like this is this stuff that you really can fix on your own easily. And when you have done this and you go for an external consultant for a real pen test or whatever, you can already tell them, guys, we have continuous security testing here in place. We are doing 1000 builds in the sprint. We are running 1000 times in the Sprint, the security test set because it's that fast.

So why not let's always run it.

And These are the 300 security vulnerabilities we already identified and we fixed. And then usually an external audit, an external pen test company, an external security test can start on a different level.

So there's a lot of info there. And I love this approach. It sounds like we keep mentioning it's a conversational AI type chatbot, could be what you're trying to test. How do you know what to test? There seems like so many possibilities. Is it because you're using an API, there's schema you can return and then you just test that schema or is it like how do you approach testing an application that could have numerous type of paths or responses based on numerous criteria?

Yeah, so actually this is what and how should we test is one of the most important questions to ask.

On a very low level, I would say on the left side we are.

Testing the dialogues, meaning the conversation flows itself that have been designed by QSL designers out there.

And on the right side we are.

Testing the NLP performance, meaning more or less the health state of your AI engine. So testing the optimization flows is I would say pretty easy because usually you have some list of intents you define somewhere in the planning phase. What should our bot be able to answer? I don't know. For a telecom company it would be something like how can I quit my.

Contract and is the new iPhone in the store available?

Stuff like this. And usually so what we see at our customers, usually they have roundabout hundred or the bigger bots, a few hundred intents. And usually you have these a few hundred user examples behind every intent. And if you multiply those two values together you end up with these Thousands or even 100,000 conversations or something like this. And this, this is a good starting point for a regression test. But this is only the left part. This is your dialogues. This is really checking. If I ask A, do I get back B exactly like intended to be.

The other thing is, as mentioned before.

Your whole NLP performance, your NLP score. So talking about natural language progressing this new word.

Yeah there it's about more or less we are getting there into data science as testers. So in general with conversational AI there is a new vocabulary, there are these intense utterances, AI models, there is NLP, NLU, F1 score, K fold analysis and so on. So a lot of new vocabulary and it's a lot of data science. What we are doing there is we are showing what is your confidence distribution of your intents. And therefore you see well for stuff like how can I quit my contract, the NLP behind. Or if you want to say like that the AI or the chatbot does resolve to it with 98% or whatever. So you don't need to touch this intent. Everything is cool. But on the other side we are highlighting for the. Let's go for the same example. Is the new iPhone in the store available? It may be resolved just with 38% or something. And this is definitely then an intent where you have to take action, where you might have to add more user examples, meaning more training data, retrain the bottom and retest the whole thing. And this is just the beginning. We have distributions on an utterance level.

On an entity level.

We are creating whole confusion matrices and they are more or less a combination of all possible intents and so on. And yeah, as mentioned before, still with all this new vocabulary and so on, we're trying to make it fast and easy to use. Like always in testing we are using red and green and if you have these red boxes popping up, it's usually a good thing to click onto it and you will see all oh, here we are missing training data and so on. And in this case then we even go the next step. If we have a case like well here we cannot solve resolve to the intent with more than 30, 40, whatever percent and you just have a few user examples behind. Then then we are even doing test data augmentation. Now you just hit the button and then we are using a paraphraser to generate test data and so on. So in theory with bottom you can.

Do codeless test test automation.

Very nice. So you know we mentioned Botium. I want to get into botting. But before I just had a random thought you talked about vocabulary and people think of like you mentioned bots separately than Chatbot a bunch of times. So can you use botting to test other types of bots? Like for example, I know audio is becoming more and more popular. Can you use bot aim to test audio type bots or is it just text based chat boxes?

Definitely yes. So we are going here really fully end to end and we are testing all aspects and end to end means voice based testing of course. And even they're really fully end to end because voice based could be, I.

Don'T know the Alexa you are talking.

To on the one hand. But on the other hand voice based could be also an IVR system you're talking to. So interactive voice response. These are these typical hotlines that are telling you if you want to talk to support, press 1. So they are disappearing, they're going to be replaced, thanks God. But they are there, they need to be tested. With Botium you can do this and still without any coding required.

So you know, I guess I should have asked this earlier, but what is Botium then?

Well actually Bottium everything started here with an open source test automation framework for bots that we called Botium Core. This was as you mentioned, I really like this in your introduction or my bio when you said something like next logically step after Selenium and Appium. This is exactly what was our intention in 2017. Florian, my co founder and I, we.

Were doing a consulting project and the.

Customer wanted to build a bot and he wanted to have the whole thing having bot development, lifecycle setup integrated into the IDB pipeline. We had these requirements like a few hundred intents ending up with with thousands of conversations to test. And we knew we need to test this. I don't think it was around about 10,000 conversations. We knew we need to test them permanently every day a few times. So it was clear no way of doing this manually. So we started to look for what is out there and yeah, we found nothing. So end of 2017 we are based in Indiana, Austria, so in the heart of Europe. And we were taking part here in startup contest, immediately won this one, then went to the same one for Europe and also there won the prize. And yeah, this was the beginning of option. Founded the company and two months later we launched Optum Core. And yeah, still there is an open source test automation framework and it's for.

All guys out there who are used.

To Selenium and Appium. It feels very familiar. Instead of a Web driver, you are creating a bot driver, you're setting your capabilities, writing test. So this was our first intention. Of course, technically there is something different in the background and Selenium Adapium. Now there's other stuff going on, but for people who are used to automating with those technologies, Optium feels very familiar. Yeah, as you mentioned, we then pretty soon became, at least in the open source market, we are for sure the industry standard. But having an enormous community growing, we got a lot of requests, like people who are. This is cool, testing the conversation flows. But exactly. What about this NLP stuff? We need to monitor the health of our AI. What about performance testing? I just done a demo this week for a customer who has in parallel 50,000 users every minute in his chatbot. So this is an enormous number.

So all this stuff, like I mentioned.

In the beginning, it's a holistic test approach. You need to do low stress testing, you need to do security, you need to test your nlp, you need some monitoring, you need a pipeline integration. Yeah.

Therefore we decided, well, Altium Core is.

Cool, let's keep it open source for the community. But for all this stuff around, we need something bigger. And yeah, this was then our second product called OptionBox. And yeah, this is more or less very nice UI. At least we think it's nice.

Depends on the people using it. But yeah, there you really, with a few clicks you connect to your bot, you write your first tests and you.

Have them integrated in your pipeline. It's.

It's usually when I'm doing demos, it's like at the beginning I ask is there someone who has set up a test automation framework or project with Selenium or Appium? And usually people say yes. And I'm like, so just, just think now somewhere back in your head what this means, if you would do the same thing with Selenium. And then I show them. Well, we export from the bot building platform the customers are using to develop their bots. The credentials are usually a JSON file from helloflow or Watson or whatever. Yeah, we are supporting more than 40 platforms. So this is a drag and drop into our quick Start menu and then you are connected to the bot forever. Takes one second versus setting up a webdriver, setting capabilities and stuff. Then you hit. Next and easiest way maybe to compose. Test is our live chat where you just say, well, I talk now to the bot. Yeah, so you open your microphone and you talk to the bot like I'm talking to you. And on the other side the bot does the speech to text recognizes the intense replies. At the end you hit save this whole thing as a new test case and then that's it, the test project is finished. So after, after five minutes we have a repeatable test project that we can execute now 100 times a day if we want to. Full pipeline integration is there. It's just a webhook to call full reporting. And I'm always like, maybe I'm most excited myself about it because I was doing this for so many years, but especially in big companies setting up something like, like testing end to end a mobile app and Android and iOS in parallel, maybe on real physical devices with some device cloud provider and so on, integrating this into a pipeline, trigger it with every build and so on. Well, to be honest, this took me at least weeks in all these companies, in some of them even months. And the same thing we are doing here in five minutes and, and it's codeless. This, this at least ourselves. It's exciting.

Yeah, it sounds a really cool tech. That was going to be one of my questions. How's this different than someone just using Selenium? Maybe, rest assured to test the API. This is built specifically for bot testing. And it sounds like almost like not unit test, but it's almost like you would test the bot away from a website per se and then you still would use Selenium to do your browser based automation. But this seems more like a solution targeted specifically for the bot functionality to get up and running quickly so you don't have all these other distractions maybe, or had.

Yeah, exactly, that's it. And as mentioned, we're really trying to cover all aspects of testing a bot. So taking this easy example that I was drawing now, where you have a small conversation with your bot recorded and then you can execute it how many times as you want. The next step is you say, well, these tests are running now against the API of your bot. Let's run them end to end. And this means just selecting out of a dropdown. I want to run them now against the browser farm, against all possible operating system and browser combination. And I also want to run them on real physical devices like smartphones or an Alexa. But the test set stays the same. So you have recorded a simple conversation that runs against the API, but it will also run in the web browser. And we take care in the background of all this stuff. So we take away technical complexity. No one has to deal there with Selenium indicators or something to find an input field where you can send text to the bot or whatever. Botium Takes is the way so you can really concentrate on the main purpose defining your tests, what should be tested, what should be the content and so on. And the same is as mentioned before, then you go the next step and say, oh, let's test against an IVR system. It's no change, it's the same test. It's just telling botu, well now run against an IVR system.

So I know people always going to ask this, do you ever have like a test that Selenium interacts with Bodium or is it always separate? Is there ever an end to end test where you would have to leverage Selenium, Appium and Bodium together in one type of flow?

Yeah, so definitely there is interaction. But I mean it in case like bottom box is using Selenium and Appium in the background. So when we are talking to a bot through a website or mobile app, we are of course using Selenium and Appium. So those two things out there are other industry standards. I'm working, working with them since 15 years.

They are perfect.

And why should we reinvent this thing? Yes, we are using them, but as a user you don't have to write Selenium code direction I tend to call it. This is the magic that happens in the background.

Right?

Right.

Yeah. So I guess what are the components that make up Bonium then? Well, what's the bottom stack? Like, what are people getting when they install Bonium? Like, is there like main areas that do certain things that people need to know about?

Yeah, so. Well, there's a tech stack and there is what you get on a low level.

You get a web application, you have.

A web ui, you log in and it's really easy, quick starts wizards. And when you want to start with the NLP testing, not with the conversation flow testing, then we also have very low entry levels. Like instead of thinking how can we test now, our nlp, how can we check this? You can just use our conversation model downloader that directly connects to the AI model, downloads it and automatically generates test sets, test cases based on the training data. And this is of course then when you're running these test sets, you are purely testing your training data, which is not a fulfilling approach, of course, but it's a very good starting point. So we see when customers are developing sincere spots and they realize now, wow, we need testing and go for bottom box. Then again we have here automatically generated test sets with hundreds, thousands of conversations in it. So it would take a lot of time, time to produce them manually. And then usually you just Add on top your own specific tests. So this is one thing you get on the other side from a tech level. So for the operations or the FOX teams who would run Botchin box, in theory it's a containerized application, so it comes more or less in a Docker container. And if you have hosting it on premise, the only thing you need is call it the standard server, something with four cores or something, I don't know, 16 gigs of RAM and the docker, it's a Docker compose app command and 10 minutes later everything is installed. You point your browser to The UL and BlockchainBox is there. That's it supporting of course when we're talking about talking about containerization also Kubernetes.

Or OpenShift stuff like that.

But yeah, from a tech stack, that's it in the background inside these Docker containers. Blockchain box was mainly built with JavaScript React Redis so that there's of course a lot of things working together and inside a lot of containers talking to each other. But once again we are taking away all this complexity from users and we are even packing inside this container stuff like browsers and so on. Headless if you don't have a contract with a browser farm or something like that to have a starting point, we have a few headless browsers inside for smoke test or something. It's really another enough just to check if the bot is there and you can talk to him now for fragmentation.

Of course it's another topic.

Or we are even packing in device emulators for Android and so on. So we need to do too and people just need at the end to say well run tests against this Google Pixel emulator and that's it.

So I think you mentioned it started off as an open source project and of course to get all this other functionality it takes time and effort. You have paid features for enterprise, enterprise for security and performance testing. So I guess is this still open source? Like I'm on your website, is the mini free the same as open source? Is that like a different offering?

Yes, it is. So what is and what will always be open source is Blockchain core. This is mentioned at the beginning the comparable to Selenium or Appium and I really like someone in the community in some blog posts called us the Selenium for chatbots really like this technique line because this really is. You can do a lot of stuff with it but you need all this scripting, you need the knowledge and you can mainly test conversation flows. All the stuff around is not Possible. This is open source and it will stay open source. What is there now? The bottom box Mini, as you saw, is the replacement of our Community Edition, our former one that was there of the bottom box there. We saw that people were struggling a bit with the hosting on their side. So it's for DevOps and so on. When we're talking about Docker and containers and so on, this is all complete and easy, cheesy, but there are still people who want to try out something and who just want to register somewhere and two minutes later you can try out stuff. And I mean, I was in this situation a lot in big companies, well, you could download the Community Edition, but to install Docker, to run these containers, you don't have sufficient rights and you're waiting for a week. So. So we wanted to avoid this situation. So therefore we said, let's take more or less our free Community Edition that was there, let's make it deprecated and just host it for our users for free. And this is at the end now, the botchinbox Mini, we are hosting it. We are even paying for all the hosting costs out there in the cloud and users can use it. It's eligible for company usage. Of course, it's from the feature set perspective. There are differences to our paid plans. There's of course a different support level, different feature set level. But we see, to be honest, a lot of big companies using it at the beginning of their project because they are like, well, we need some kind of testing solution, but let's start somewhere, let's try this out and yeah, you can try it out without any risk.

Okay, Christophe, before we go, is there one piece of actual advice you can give to someone to help them with their bot automation testing? And what's the best way to find contact you or learn more about bottium?

Yeah, so. So for the first question, my advice is of course start testing using botiam.

I mean, being the founder and CEO.

And also without choking, it's. Well, there's absolutely no risk because getting started with the Mini is completely free. And as I mentioned before, this testing and quality assurance really makes the difference. This makes the difference between, between high and low end user satisfaction. And we all know when it comes to stuff like chatbots, then we are in the Internet and it's mainly used by these millennials, generation set and so on. So people who are used to stuff that works, if it doesn't work. Exactly. This generation is kicking your ass in shitstorms and so on. Well, let me give you an example. We were testing, testing the chatbot of an airline who changed their ticket booking system to a chatbot. And it was, to be honest, before Corona. At the moment, I think not so many tickets are bought. But the thing is their chatbot couldn't understand the user request, I want to book a ticket or I want to book a flight. This was his main purpose. And they didn't realize this for two days or something. After two days you can say, well, it's such a big airline that sells so many tickets, so you can put a number behind the financial law. This is no big thing. But the loss of reputation was enormous. I saw all these postings and so on. How can these guys maintain an Airbus or Boeing or whatever if they cannot produce the chatbot and so on.

That works.

So reputation loss is a very big thing. So therefore my very first advice is, guys, start testing your bots. And therefore I don't have to say use bot because anyway, you will end up with us.

It's just the best option you have.

Yeah. The best way to find our contact you.

Yeah, sure. Just Google for us, hit our website, you can book there a demo session. This is actually still directly connected to my calendar. So anyone who wants to block my calendar and have a demo session can book one. So a lot of ways social media, we are everywhere.

Yeah.

So if you search and all this.

Search engine for us you will find content.

Thanks again for your automation awesomeness. The links to everything of value we covered in this episode, head on over to testguild.com a342 and while you're there, make sure to click on the try for free today link under the exclusive sponsor section to learn all about Sauce Lab's awesome products and services. And if the show has helped you in any way, why not rate it and review it in itunes? Reviews really help in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Automation podcast. I'm Joe and my mission is to help you succeed with creating end to end full stack automation awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Automation podcast. Head on over to test guild.com for full show notes, amazing blog articles and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.