Evolution Of Testing to AI and TestOps with Nikita Fedorov

Get ready to discover the most actionable end to end automation advice for some of the smartest testers on the planet. Hey, I'm Joe Calantonio, host of the Test Guild Automation Podcast and my goal is to help you succeed with creating automation awesomeness.

Hey, it's Joe and welcome to another episode of the Test Guild Automation Podcast. And today we'll be joined by Nikita, talking all about AI, test ops, all kinds of new things going on in the industry and what you need to know and how things are changing and what to know all about so you don't get caught off guard and the type of skills you're going to need for the future. You don't want to miss this episode if you don't know. Nikita is the CTO and founder of case. It's a cool solution and we're going to dive into that as well. You don't want to miss this episode. Check it out.

This episode of the Test Guild Automation Podcast is sponsored by the Test Guild. Test Guild offers amazing partnership plans that cater to your brand awareness, lead generation and thought leadership goals. Get your products and services in front of your ideal target audience. Our satisfied clients rave about the results they've seen from partnering with us, from boosted event attendance to impressive roi. Visit our website and let's talk about how Test Guild can take your brand to the next level. Head over to testguild.info and let's talk.

Hey Nikita, welcome to the Guild.

Hi Joe, thanks for inviting.

Great to have you. So I actually met you all in person at Selenium Conf and I got to see see a little bit more of what your solution does. I was really excited to get you on this call finally. I guess. Before we do though, is there anything I missed in your bio that you want the Guild to know more about?

Yes. My name is Nikita. I'm Case IO, CTO and Founder. I'm developer myself. I'm the guy who have started a company as a pet project and occasionally it became a global business with customers from 85 countries. And yes, I'm a fan of SkyFi. I love my family. I have two wonderful kids and I still write code because I love it.

Awesome. What kind of code? What language?

Different. It doesn't matter for me. So I use code to solve problems. Not so I can write in python, in Java, JavaScript, in PHP. I don't know very well Ruby, but can do some stuff.

Nice. In the pre show you mentioned something that was interesting and I thought people would be interested to hear the name of the company is Case. And I always see it incorrectly. Why did you come up with the name Case?

Yeah, so initially I was building a solution for manual testing. And I've used a combination of QA and test case. That's how Case game and play set up. But yes, right now we are not only for manual testing.

Great. Cool. So I guess the first thing I want to dive into is, you know, over the years we've seen a big shift in testing and trends, maybe go over where we started and where we are now. We keep hearing about AI, generative AI, all these type of new tools and technology. So I'm just curious to get your input on where we were and where we are now and then where you see us. So maybe a little background that way.

Yes, great question. Because right now a lot of people, they are afraid of AI. They think that AI will take our jobs, people are not necessary anymore, et cetera. No, long story short, no. But before we talk about the AI, I want to discuss the evolution of software. As I know, correct me if I'm wrong. First, software testing has a discipline period in 50s and established in early 60s and it was mostly manual because we had big mainframes, we had cards to to put machine code instructions. So. And people were trying to solve bugs by manually cutting holes in the cards. So that was a pure manual testing because yes, people were fixing bugs with their real hands then. Yes, we got more complex programming languages, we got more complex frameworks and et cetera. And in the end of previous century, we saw more solution for automated testing. So because people were tired, they didn't want to do a lot of repetitive tasks by their hands. They want to automate. People are lazy. We don't want to do the same job twice. Yeah. And in the beginning of 20th of this century, we saw a boom of tools for test automation frameworks, test management platforms, reporting tools and etc. Right now, as I think we are near a new shift, a new step of this evolution ladder. So AI is here. So the world has changed, AI is here. So we have to adapt how it affects our work. I think a lot. But let's go back again and I talk about existing landscape. For my opinion, there are three types of software testing nowadays. The first one is manual. There are a lot of companies who are doing manual testing who doesn't even have test automation in place. The second type, I call it manual automated. Let me explain why. Because we have people who write code to make manual. Testers don't do their work with their hands. But we still have people who write this code. And the third type of testing is AI testing. No code testing, low code testing and etc. Where we can generate end to end scenarios from customers tracks, where we can generate scenarios from product description, where we can record clicks made on the website, on a mobile phone. And in the future there will be all of these software testing types will survive. We still need a manual testing because we have to teach AI how to test to give AI context about product. The second type is automated testing code written by developers. There will be a lot of space for that because we have complex products. For example as I remember Netflix have more than 1,000 of microservices in their cloud side and it is very hard to provide context for AI about the whole system. So we still have to rely on developers and now some parts who knows how to automate complex scenarios and etc. And AI no code, low code testing will cover the biggest part of tests in place, for example, but that will be simple tests, for example verify shopping cart and etc. And there is another thing. So there are three types, major types of software tests, let's call it end to end tests, integration tests and unit tests. Different layers. Yeah. And over the last 10 years we have seen a lot of tools that said hey, Codless test automation, forget about writing code and etc. Use our tool and you'll get everything automated in minutes. But the problem is that they are focused on end to end scenarios mostly. To be honest, I haven't seen any tool that generates unit tests for my code. I have seen only a few tools who says hey, we can generate API tests for your smart machine. But most of the tools they're focused on end to end tests. And we all know that how looks software testing pyramid. In a perfect situation it looks like a pyramid. The problem when we have a waffle cone, we have a lot of slow end to end tests and we don't have unit tests, et cetera. And right now a lot of tools they're focused on end to end. But the real life.

Why is that? Why do you think it's they focus.

On end to end more because it is very simple. You don't have to dive into the context of the application. You can just copy repetitive actions, record them and play. That's simple. But here we have ChatGPT, OpenAI who can write code, who can do. I've tried it by myself. ChatGPT can, if you provide methods, it can write pretty good unit tests. And sometimes chatgpt surprised me. She or he, I don't know, it.

Wrote, covered some cases, I didn't think about that. And it brought and covered and provided very good coverage. So ChatGPT can write a pretty good code right now if we are talking about the fourth version. But the main problem that ChatGPT, it can't execute these tests. So it can write code, but it can execute. You have to copy this code, execute and provide feedback back to ChatGPT.

Is that just because it's a limitation now though? Do you see like. Because I see they have an API now and a lot of tools are, you know, starting to integrate the API. Is that going to be just something that goes away then? It will do that for you.

Yeah, and we do that right now. In fact a couple weeks ago we have released a feature that can generate automated tests from your manual test in different languages, frameworks using ChatGPT. And according to user feedback it provides pretty good results. But ChatGPT doesn't know anything about the context about the web page. It can only guess what locators, what XPath you have. But it can be a very good base that reduce time to market. So you can just take the script generated by ChatGPT, adopt it, change locators and et cetera and execute. Yes, and in the future we will see how ChatGPT will generate code, but execution of these tests will be on third party tools because yeah, ChatGPT is its main purpose to generate output from your prompt.

Right. So do you see a time when you know each company would have their own type of AI model that has learned about their application so it would know about context and then you could use generative AI and it would understand how to create a test. Or is it always going to be, it's always going to be almost like a black box to AI.

No, I don't think that every company will have their own chatgpt. That's related to an old war between cloud solutions and self hosted solutions. Yeah, because it is extremely expensive and hard to train your own model. As I know companies like OpenAI spend billions of dollars to train these models. I think Microsoft with their copilot spent almost the same and it is very hard to get Roy back, get your investments back. That's why people will use shared model, because the more context, more information it has. It was teached by on thousands of repositories, the better output will be. That's why a lot of test management solutions will go cloud. For example, we all know Cypress and they building their own cloud execution platform. You still have your own your code but you execute in a Cypress cloud. That's why we will see a strong motion and I see it right now that people will move from self hosted solutions to cloud solutions in a devtools space. That's if that transition has already finished in issue management solutions. Do you remember Jira Jira they have removed their Jira server version, self hosted version because it is so expensive to support customers who have different versions. They don't have release cycle so some of them can have super old versions and you have to support them because you have committed to that. That's why Jira Atlassian switched to cloud tools for example. Yes, as I know so working on a BitBucket but I haven't heard about the updates for Bamboo CI Gotcha.

So you know what are some of the advantages? Because I think once again the pre show you mentioned how pretty much that's the next thing developers tools are going to the cloud because you mentioned something about GitHub Actions that you don't need to local host solutions anymore. So what does that mean for developers and testers then? Is it make it more collaborative or does it make it where you can then use I think you said shared models of AI to figure things out.

In fact Pandemic made us collaborative. Yes, because people went out from offices, so they worked from home, some of them moved in other countries, other series so. And yes, cloud solutions helped a lot with that. But in fact the biggest benefit of cloud solution is shortened release cycle. Cloud providers can release updates almost several times a day. When you own a cloud solution, you can make an experiment, you can test some feature on a small part of your users, you can do canary release, you can do A B tests to see how it works without affecting biggest part of your customers. So that's why I think cloud solutions will win.

And so would that then accelerate the need for more automated tests then because you can release quicker and faster?

Yes. And as we know software testing is a significant part of software development life cycle. It affects time to market super heavily, very heavily because you have, you want to build a new feature, you have to write requirements in Jira for example in another issue management tool then you have to write code, you have to test it, you have to release it. So and if something, as you know something goes wrong on a testing site, you have to return ticket to developers to things bugs, then get back to test again and et cetera software test. Automated automation solved this problem and reduce time to market. So cloud solutions CI City shortens it as well, the next step will be AI self healing tests so you don't have to return issue back to developer. Because right now platform knows the change you've made to the code, it can relate it to the tests you have. If you have pretty good reporting, you can understand what was wrong and create a patch fix for the bug in the code that is going to production are prepared for release. That's what we see in GitHub actions with their dependency checks. Wait. Oh, you have a new vulnerability in your package. Let's update. Here is pull request. Please review it.

So does that also do you see that then accelerating test ops? I've been hearing more about test ops the past few years with the need for all this information because the more tests you're in the cloud, more info, more results, you're going to need to be able to triage them quicker, have metrics and measure a lot of different things. Do you see that as a rise for the term test ops I've been hearing more and more about as well?

Yes. And I think I've heard TestOps about four or five years ago and right now this is a trend. So I see a lot of companies who name their products desktops. I see a lot of blog posts about desktops and I think that we will see an evolution of software developer and test of QA engineers from QA to desktops engineer. Because AI will take more and more in future. It will be able to not only to write a card, it will be able to execute this card. Maybe not ChatGPT but other tools based on ChatGPT and some but AI doesn't know if it works or not. It still needs a feedback loop from a human. For example, we know that there are so many AI tools that generate automated tasks, but how do we know if we reduced our time to market? We don't know. Maybe we have made worth. So okay, we have switched from we have fired all key engineers, we have brought in place a new AI tool but we have increased end market wise. So do we want that? I don't think no. And we will need people to build metrics to monitor these metrics and give feedback how it goes.

So metrics is a big thing. It's been around forever. But what are the modern metrics then? Are there newer metrics that people are going to need to rely on as we go Cloud native test ops first type of approaches to testing? Yeah.

In fact from the metric sides nothing will nothing but we will have the same metrics in place. So when we want to refactor some code. So we write tests, we have for example, a very good coverage. We have reached 100 unbelievable 100% of coverage. And we want to refactor our code, for example, rebuilt all the functions under the hood and etc. How do we know that it works? So we use tests, we run these tests and we get expected result. So the product works the same, but internal structure was rewritten. The same will be with the AI tools and with generative AI and etc. So we have the same metrics, but we want them to increase or decrease depends on the metric. For example, we want to decrease execution time and increase.

Reliability. But we will watch them, monitor them and see how it goes, how this transition happens. And we can use this information when we find gaps to provide feedback to AI generative system.

So, you know, we kind of skipped over this. And you mentioned with case you have a new feature that you added, ChatGPT. It created automated tests or manual tests. Could you just explain that a little more? Maybe explain how that works, what that looks like? Because I don't know why we just skimmed over it, but I think that's kind of a big, big deal.

Yeah, we have a product that is used by different companies. Somebody have only manual testing, somebody have only automated testing. Somebody have, for example 50% manual, 50% automated. And as we have millions of manual tests in our system, we can generate automated tests from them. We have a description, we have test description, we have steps on how to reproduce this test, we have maybe screenshots how it should like. And we have built a prompt that push this information to ChatGPT and it generates COD as an output based on test information. And it works. And it works pretty good.

Yeah. So how accurate is it? How does it know? It just reads all the different assets.

So we provide information about the project. So for example, projects can be different. It can be a mobile application, website, microservice and etc. In case every project has a description, then we have test. We know about its priority severity, we know about layer, we know about, for example, we know about other, for example domain model and et cetera. And we pass all this information to ChatGPT and it guess and we can provide feedback. Oh, it's not what I have expected. Can you regenerate and it generates again? Yes, sometimes. Right now it is not super perfect, but it works very good and we are going to improve this experience.

So I guess, you know, how does this tie into case and what is case for people that don't know and is This a feature that comes with it or is it just a separate feature? Maybe to understand how this works, people need to understand what CASE is, how things are set up in order for ChatGPT to work with it.

Yeah. Case is a test ops platform for manual and automated testing. It was born as a test management solution. I was working as an engineering manager in a company, I had several teams and I didn't have software testing at all. We had a lot of bugs, we had a lot of complaints from customers. So I've hired people and I was looking for solution for my team. But occasionally in 2016, all of them were not super cool. I thought, wow, how that could be possible in 2016. Why do we have big print buttons and etc. So I've decided to build the tool for myself because I had necessary skills, I knew what I want. And as every developer, I had a small graveyard of bad projects that were never released. So I was working after the day job. I released first version, got positive responses. Some people started using it every day, so there was no way back. And then I realized that we have a bigger problem. People were using different tools for manual and automated testing. They were using reporting tools for automated tests. They were using for example test management systems, test case management systems for manual tests. And it was super hard to tell what is going on, how looks the process of test automation, how much progress did we do over the last, for example, week, two weeks and et cetera. And that's why we have built integrations with almost all popular software testing frameworks. We have custom reporters that can support live documentation so we can create tests in CASE based on the results. You sell results in your repository so we can add additional context with tests using decorators and etc. Depends on the language. And we provide analytics based on historical data. So how your soft tests are performing, how they work, what problems do you have, how you can improve your software testing process and reduce time to market.

So does it bubble up insights? Say hey, we noticed over the past month, I don't know, this automated test in this area keeps failing every third run or something.

You will see it from the dashboards. So we have built Grafana like dashboards and you can see all the fluctuations and et cetera. So Grafana doesn't tell you about, but it allows you to build information dashboard for yourself. What metrics do you want to track?

So almost sounds like a go to place for orchestrating all QA activities.

Exactly. Test ops.

Awesome. All right, so I guess before we wrap it up, two last Questions. The first one is where do you see case going? You had the first sounds like you started to integrate ChatGPT into it. Do you see using AI to then analyze results as well, to bubble up insights like what's the next step after this? Sounds like one of the first uses of ChatGPT.

Yeah. So right now we can generate code. What we want to do, we want to allow people execute this code in the cloud. So thank you. You've clicked a magic button. Make me an automated test from this manual description it generate tests but you have the main problem of ChatGPT that you have to copy output, execute it somewhere and that provide a feedback. We want to allow people execute it in the cloud, get instant feedback to AI model so it can fix issues or provide a better version.

Nice. So the second one is, you know we've said this before, it doesn't sound like this is going to replace testers. It just seems like there's going to be a lot more testing that's going to be done. A lot of it's going to be automated. However, you need the human mind to go through and say okay, prioritize things. Is this really an issue? Be able to do the thinking of all the activities that are going on behind the scenes that are automated. Is that right?

Yes. AI will reduce amount of manual work we do. It was built to help people and we have to think more. Yes, you are right. We have to understand what is going on and in fact even a couple years ago it wasn't a problem to write a new test. Even a person who doesn't know much about programming languages and etc. Can start writing tests in a couple months. And the main problem is to maintain all these millions of tests we have right now. ChatGPT can generate pretty good code and we will see a lot of tools that will use it to help people to generate more and more code. More code we have more tests there will be in the world. So yes, you're right, we have to test more and we have to analyze what is going on and teach AI models because for example, imagine somebody invents a new programming languages programming language. AI doesn't know anything about it. So we have to as a human, we have to teach AI model to write code using so there always be a place for people who operate these generative models.

Cool. Ok, before we go, is there one piece of actual advice you can give to someone to help them with their AI automated testing efforts and what's the best way to find contact you or learn more?

About Case peacefully. Yes. Learn, get information. So it may sound pretty common. So like, oh, that's obvious. So go and learn. But no. So that's what AI system do. They learn and learn and learn, grab new information. And things are changing very fast nowadays. So half year ago, nobody knew about the AI. Right now we see a new release, a new update almost every week. Okay, AI can do this. AI can write code, AI can produce movies and et cetera. World is changing very fast. That's why we have to keep learning and adapt to this world.

Love it. And the best way to find more about case, if people want to learn.

More, you can find more about CASE at our website, CASE IO Qase.

IO. Yes. And you can try Case for free without any credit card, any upfront payment. So we have a freemium that can be used by anyone if you're a small company or even if you're a student.

And we're gonna have a link to that in the show notes.

Yes. And AI feature is available for everyone.

Really?

Yes.

Sweet. All right, so I will definitely have a link to that in the show notes, so people definitely need to check it out and I'd love to hear what they think about it. So thanks again, Nikita, for this interview. Really appreciate you.

Thank you. It was a pleasure to meet you.

Thanks again for your automation awesomeness. The links to evident value we covered in this episode. Head on over to testguild.com for.

And if the show has helped you in any way, why not rate it and review it in itunes? Reviews really help in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Automation podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack automation awesomeness. As always, test everything and keep the good. Cheers. Hey, thanks again for listening. If you're not already part of our awesome community of 27,000 of the smartest testers, DevOps and automation professionals in the world, we'd love to have you join the fam@testguild.com and if you're in the DevOps automation software testing space or you're a test tool provider and want to offer real world value that can improve the skills or solve a problem problem for the guild community. I'd love to hear from you. Head ON over to testguild.info and let's make it happen.