The Allure of TestOps with Ruslan Akhmetzianov and Andrey Ryoo

Welcome to the Test Guild Automation Podcast where we all get together to learn more about automation and software testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Automation Podcast. Today we'll be talking about Roslyn and Andre, all about next gen Test Ops. If you don't know, Rosalind is the devrel lead at QA meta Software or Kmeta, the Test Ops Ambassador and DevOps Enjoyer Conference Crawler all around all things about QA and testing. So really excited to have him. He's also joined by Andrej who's a DevOps and infrastructure expert at Ka Meta and he has a lot of experience in building pipelines and world class fintech and IT companies who have a lot of great experience here. Want to talk a lot about Test Ops Automation. A whole other bunch of things that you don't want to miss. Stick around. Check it out.

The Test Guild Automation Podcast is sponsored by the fantastic folks at Source Labs, the cloud based test platform that helps ensure you can develop with confidence at every step from code to deployment for Every framework, browser, OS, mobile device and API. Get a free trial. Visit testguild.comSourceLabs and click on the exclusive sponsors section to try it for free today. Check it out. Hey guys, welcome to the Guild.

Hey.

Hey. Great to see you here and happy to be here actually.

Awesome. Awesome. Yeah, I'm really excited about this. Before we jump into it, I'm already off topic I think is are you the folks behind the really popular automation reporting library? Alor.

Yes, that's us. And well, I would say it would be better to say that we are actually people in the team of people behind the ldore report. So QA Meta Software is actually a maintainer of Allure Report and Allure just stops. Yeah.

Great. So I guess the main topic, the reason why I bring that up is a lot of people use the library and I just want to let people know one of the reasons why I came across your company and you folks is through Alar. So one thing that caught my attention is I think you're talking a lot about Test Ops and I know there's a lot of variations of the term Test Ops, what it means and what the connection is between DevOps and TestOps and why we need another ops thing. So I just want to get your own personal opinion is what do you think TestOps is and what's your vision of what TestOps is nowadays?

Yeah, I guess I'll be short and then I will let Andrey say His opinion too. So as far as I understood, I've been working in QA meta for more than a year already. And all this time we've been talking about desktops actually with various people here and there, on conferences, on forums, in messengers and chats and so on. And each time I see some companies speak about desktops, I always see some kind of vendor lock about what is desktops. So I won't like poke with finger at anyone. But in fact the test Ops is considered to be mostly like a tool related thing. But if we dive into what DevOps is reading like Project Phoenix and thinking about that, it's actually a set of processes and people communications patterns and then tooling comes to solve these challenges. And for me Test Ops is also a subset of DevOps actually it's not like some kind of additional ops, it's a subset of DevOps of techniques to communicate and to solve like different problems which come with evolution of company. When it comes through DevOps transformation, QA tends to be a little bit like abandoned. Left behind. That's a good word. So sometimes QA gets left behind because developers, they have already amazing tool set like GitHub, CI CDs, pipelines and so on. So Andrej will tell you more about that. And like ops people, they also have amazing tools like Kubernetes, Dockers, virtualization tools and so on. And when we speak about qa, if the QA team is not like extremely modern, there is a lot of manual testers who actually keep clicking through the web interface because like, well we did it all the time. That's the historical pattern, you know, why would we change it? Why would we believe in automated test? Automated test doesn't understand anything, he doesn't see the page, he is not a user. So we need a person to check it out. But when we need to release like 10 times a day, we can't actually check everything. So at this point here is a problem which desktops should solve. There is disconnection in processes between DevOps and like classical QA. I would not say legacy because it's not legacy at all by now, but classical QA patterns. So the test ops, well for us testops is something but which should help people to break the expertise silo between QA and developers and Ops. I would say something like this.

Well I'll tell in a bit different way. So what is DevOps? Well, because it's the oldest discipline, I would say DevOps is a fusion between developer and OPS operations engineer. So it means the way to deliver the software as fast as possible using various instruments like pipelines, CI CDs and other things like this. And so what is the most important? It's not the tool. I mean the DevOps desktops and others are not tools. It's method. I would say it's disciplines with the various subsets, as Ruslan said. So for example, if we get DevOps, it has subsets like DIVSECOC ops, which means developer Security Ops and well, test ops. It means developers, QAs and operations engineering. So well, what's important about this is a way of faster delivery but not losing in quality. That's what I wanted to say.

So faster delivery but all the loss of quality. Like as Razun was saying, you know, a lot of times you need classical QA and a lot of times that incorporates a person. So how do you get there then? How do you get a marriage of classic QA along with, you know, technology that you need?

Well, some companies like Google, Microsoft, Facebook and others, they made up a word called sdet. Well, software developer in test. So what it means, it means it's a specialist who's good in coding, knows how to deploy things and also has expertise in qa. So it's kind of a mixture. Some companies or experts, they say almost the same. Well, I mean, about worries as did. So what sdeed practice is test ops.

Yeah, and I will just add something from a different point of view. So the whole point of when you are talking about how do we achieve that, like symbiotic or synthesis of collaboration. So just think about that. If we don't make QA, if we don't bring QA to DevOps, developers will test everything by themselves actually. So that was actually a point from a friend of mine, Sergei Egorov, who is actually containers maintainer and Atomic jar, one of co founders. And his point was like what is actually desktops? What desktops is we developers test everything already. So why we need to actually and who are QA people? And I was like, well that's a good point because that's something good to discuss. And well, the main point is that developers, they do a lot of testing, they create unit tests. So they run the software through a lot of quality gates before deploying it to the user. But developers tend to test the happy path. So the idea of developers testing is I want to be sure it works as intended. And the idea of a tester testing software is let's check that it doesn't do something wrong while it does what it is intended. So if we just Leave our QA somewhere behind, the quality will drop.

And the reason will not be because developers can't write any tests, but just they have different mindset. And when we speak about collaboration between automated testing and manual testing. So like a year ago, me and our co founder Artem Yurushenko, we were talking something like, okay, everything will be automated soon. And then every time, each time we say it, somebody raises their hand and says, but you can't automate everything. And we are like, of course we can, but it might be too costy. Well, it might be expensive. Yeah, so like extremely expensive. So then we've been thinking about that a lot and the idea is we should leave manual testing for exploratory tasks. But we don't do regress testing, validation testing, we should automate it and make it as fast paced as possible. And then manual testers or brain UL testers, they will become brand new actually. So they will provide us. So they will tell automation test, like automation engineers, what tests to write, what to test correctly and so on. And at this point automation engineers should work with developers in close collaboration, like I would say in one pull request, like in perfect picture. So that's some kind of testops implementation we believe in.

Yeah.

And there is a small remark. Well, TestOps doesn't exclude manual testing. I'll try to explain why. Well, the point is, mostly in a large company, if you want to test something, you need the testing environment. So it means you need a copy of your working piece of software, well integrated, and then you can start testing. So test ops, I mean SDTs, they also help you to deploy this environment in one click. So to make your manual testing going, not to wait for a whole day and when your operation engineers are going to deploy a whole environment for you.

So I've had this discussion before and there's different extremes. One extreme is people say we don't have a QA team, we don't have a testing team. We have a team. And the team in the Sprint's responsible for all the activities involved in development of the software, including testing, which usually ends up being the developer that does the testing. So in your model of what you've seen, you think I always get asked, oh, you still need qa, is QA dead? So you don't see QA being dead, you see them just no longer being just a separate group, they're part of the team or how does it work?

Well, we're trying not to make their work boring. So what I mean by this boring is when you are doing the same task over and over again. Well, when you test on the same data set. So in this way, after the third or fifth run, you're getting bored, you stop noticing bugs and well, it doesn't work well. But what automation does for this, it covers all the most common cases. And in this way you have only the most interesting things to test.

Yeah, when people say QA is dead, they actually mean they can't use QA artifacts in deployment or it takes a lot of effort. So, and this is actually called by the amount of like crafted work. You know, when people work with their hands, it takes time and it takes like, and it actually causes human factor. So and what Andre says. So if we take all the routine tasks from manual testers, we leave them with amazing job of exploratory testing of new features. So the pipeline would look like, we create a feature branch, we take a couple of days to create a feature. So it goes. Then we create a build, it runs through regress testing. So if it's fine, we like deploy it to the staging or to a testing server. And then a person who is actually a manual tester takes like two hours. Well, they have like this time slot of two hours. Don't create a lot of test cases for like two days working. You've got two hours. So just check the most important things in new stuff. And we get these test cases so the person checks everything if it works and if everything works, so the deployment goes forward. But what happens to these test cases? They drop to the automation engineers as technical requirements for next automated tests. So and when the next feature branch gets created, the previous manual test should be already automated. And at this point manual testing will be able to keep up and it will not be dead at all because, well, as I said, automated tests are dumb. So that means somebody smart should think before them and think through them for them to be like. And the other option is automated tests are dumb and they fail all the time. I mean you just swap the button, 10 pixels left and the automated test says, oh, screenshot is wrong. So I fail. And you get a lot of like false failures and nobody wants to work with them. At this point you need to think about some kind of automated fails, resolution and so on. And that's also a part of test ops because when a new automation engineer comes to a QA team, he creates a lot of automated tests. So these tests start failing. And QA people who are like they are used to people creating fails. And if a person creates fail, it's some kind of smart Fail like button doesn't work. And when an automated test says some bullshit like, oh, the button was moved to 10 pixels. And everyone like, we know, why do you fail? And then people start like, they don't trust automated test. Oh, this automated test again. They are falling again. Nobody cares. And this is like the problem with test in qa because developers actually love their tests. They believe their tests because they know what they do. And when, like a manual QA team sees automated tester creating automated tests, they're like, oh, they will fall again. Okay, that's some kind of another challenge and disconnection here.

So I guess the question is then, how realistic is this? Is this just for startups? Like, it's because you have a small team, you're able to control everything. When you get an enterprise level, I know a lot of times people are just like, it doesn't work. It's, you know, it's too much. Like, what does a company, I guess, need to have in place in order for test ops to be successful? Let's, let's stop there.

Well, there is like, that's a great question actually, Joe. Thank you. So the main idea is, I believe it's absolutely, like, it is absolutely plausible story because I know teams who work this way actually. So. And I've been reading, for example, Uber blog about the way they do their testing. And it actually is desktops. They just don't call it test ops. And that's another question, like, do we actually need a desktops word? Because when we make talks or we talk about desktops, a lot of people comment like, oh, we have this flow. Why do you need a desktops? And my answer is we needed the word desktops for people who actually do not have this flow to explain what they need. Because actually I don't speak about some kind of rocket science. With time, these ideas become like, almost obvious. When you try to build something like this, when you start thinking about like bringing QA back to the DevOps pipeline, these old solutions are obvious, but we need something to like highlight the importance of the topic. And when you ask about like a startup or enterprise, so if it is a modern enterprise, it's easy. So if it is an enterprise that looks like something that has like 400 manual testers department and 20 automation engineers somewhere in the basement writing their tests, well, that's going to be a challenge. And at this point you will need first of all, like huge improvements in processes or changes, if you may not call it improvement, but changes in processes, you will definitely will need somebody to lead this just as with DevOps you always need a person who drives the change because it will be stuck at each step because manual testers, a lot of them don't want to like, I don't know, store their tests in code. That's a great idea, Andrei.

Well, I'd like to share some of my experience because I've used to work in Tinkov which is one of the best fintech in Russia and maybe in the world. And the point is think of is a really various company. What I mean by this there are lots of apartments which have the various level of maturity. So it means like there are some services like an Oracle database with stored procedures and so the testers they just do all the testing manually. So they just call this process comparing with what's expected. And also there are some departments who's selling some travel deals and they've got really advanced DevOps and test ops practices because what they do, they've got great pipeline, it's a bit complicated but it's really effective because they what they do, developers, they write their tests as a proof the product works and then what they do, they use contract testing which checks all the endpoints they've got or declared. So it means if one endpoint is not working as supposed with the previous version so they rework it and then what they do, they deploy it to kubernetes, run some end to end tests in integrated environment and then they release they've got the continuous cycle of development. So it means in one company you can see various methodologies of software development. And so this company uses as beta allure Test Ops and so it works for both cases for the least advanced because they've got familiar interface to track the tests and to the results there and they've got the system which aggregates all the results of whole testing. So if you are a manager of Tinkov you can just log into Allure and see the whole testing in a whole company aggregated and so you can invest more resources in those departments who's not so developed. So by this I believe that Test Ops is the stage of evolution.

Absolutely. Now you mentioned a lot of DevOps and I know you said it's a methodology but I also I'm a big believer in tools because tools will help people do get the job done when they actually have to do it. So a lot of times I see people go down the rabbit hole of we only use open source and they put in a lot of effort and time that I mean I love working with vendors as well, because they could have purchased a solution that was already up and running and works. So like when do you know when to use open source or a vendor base? Is it a combination of both or how do you see that?

Well, I see in a way that open source is great. It's amazing. You can just start up any project you want from scratch in like five minutes. I mean for a prototype. But every open source product requires some customizing. Sometimes it's really difficult to do in this way you have a choice. First choice, you are going to the market and you are trying to find the best engineers on the market to hunt them and to make them work for you. In this way everything is going to work as charm. And the second way, if you don't have this opportunity or budget, what you do, you pay for enterprise edition of this open source product. I'm not telling this about Allure, I can tell this about the Sentry for example there is a open source version and enterprise. So if you don't feel you have enough resources to do open source thing, you just buy the commercial version and like many others.

Well I would say and I would like take a look from a higher level and would think about it like I totally agree with Andre of course but if to look at it by a bird's eye. So if you take a bird's eye look at this question, I would say that you need a tool like you need to buy some kind of ready to go tool when you are not sure you will be able to build a process by yourself. So open source as Andrei said is usually solves some kind of specific task. So there is no like huge. There are of course huge open source projects that solve a lot of tasks. Like I would say Java, it's like why not? But mostly open source projects. Like for example our unlureport is some kind of smaller piece of software and then we get products built on these open source products. I would say again so which work which actually incorporate encapsulate more specific processes and expertise and speaking and answering the question you are actually asked that's a great question again, how did DevOps actually evolve? So if you want to build a DevOps culture in your company, you actually need a lot of expertise. So like breaking the Silas building the pipeline is extremely expensive and is extremely like it takes the best people in the world. I mean like that's why Google for example takes the best people in the world. But then Google creates tooling which allows normal people like US do DevOps in different countries because the Tool actually incorporates the expertise so you can create like tutorials and provide tools for usual people to create extremely hard work. I mean, like take a look at Docker visualization. Before Docker was kind of complicated thing and now I see people. Well, I've got a lot of friends and some of them are not like, they are not engineers at all. So they are more like non IT people. But when they come to some large company to work as a technical writer, for example, so this is their first experience. And then they tell me, okay, it's been half a year, so I got some cool neat stuff to learn. I was like, what? Oh, well, you know, when I need to show the documentation I've created, I need to create a Docker virtualized image, then test everything with it, with environments and so on. And I was like.

Is that really you? You didn't know about Docker half a year ago. And then a person just tells me about virtualization and creating images with environments and so on. And that's how good tooling works. And when you ask me about what's better to take a ready and paid tool or to build something by yourself, well, I would say check your power if you are powerful enough to build by your own and you have time and money for that. Because developing something from scratch is always difficult. Like that's why nobody, almost nobody uses Linux from scratch. You know, everybody uses Ubuntu and that's the same, should I take Ubuntu or should I take lfs? Oh, well, if you ask this, take Ubuntu.

Yeah, Ubuntu lacks the Selinux, which means less security, but that's okay.

Ubuntu is just an example. You know, you can take some kind of more secure distributive, but still.

Yeah, but the major point about the Docker, well, if somebody is going to hear that, I want to say a big thank you to Docker team, especially for their desktop edition, which becomes fatter and fatter with every update and consumes more and more memory as Google Chrome.

So you know, you both brought up interesting points about Test Ops and I think you said it allows things to bubble up from the team. So everyone has a big picture of maybe teams that aren't doing as well, or maybe teams that need investing in, or maybe teams that, you know, automation is up to speed or isn't delivering faster than they should be. So what I've seen with companies when they do use open source, they have all these solutions, they have bamboo, some teams may be using GitHub other teams may be using Selenium and Cypress and there's no way to pull it all together. So as part of Test Ops I would assume that's one of the key agreements. You have this high level overview where you can have insights to all these different groups and technology. So does Allure Test Ops give you that type of insights? Can you just tell us a little more I guess about Allure Test Ops itself as a product?

Well, I can answer this question if you take Allure Open Source so you can easily build such system without paying a penny. I mean just using it in the right way. But if you don't have enough engineers who's capable doing such things, better get Enterprise Edition. It is the same as Kubernetes, for example, there are all instruments to deploy Kubernetes everywhere. For example Kubespray, it's open source, it's free and so you can just deploy it. But you need the knowledge of Ansible first because it provisions the virtual machines of Bare metal before you deploy this. Also you need the knowledge of networking because it requires some expertise in networking. For example, which CNI you are going to use like Calico, Cilium or any other. But you can go to Amazon and get the AKS or eks. It's for elastic container stuff like this. I mean the managed Kubernetes which most company provides. So it is the same. You can use Allure Open source to build a great reporting system and you can get the already tuned.

Yeah, but again so we got some kind of bare metal vision of Andre and now I will just add something above. So speaking of choosing a tool, there is always question what price will you pay from your own toolbox? I mean when you try to adopt some tool you should take a look at how well it works with everything that works for you already. And in terms of Test Ops, as you said, there is already an infrastructure which we would love to.

Be able to work with. And so this is actually one of the first ideas of how Allure Test Tops was born because our team actually created a lureport and they were like so what do people want? So they want these reports to be sustainable, persistent and easy to share. So that was the first idea. But then so and how do you do it? You don't create local runs, you start running your reporting on cicd. And then we just, our guys just started creating all these integrations with TeamCity, with Jenkins, with CircleCI, with GitHub, with GitLab because people actually need this. They want to take the tool then Just plug it with a pipeline. There is actually an anecdotal story about another my friend who works in an IT company who actually wanted to buy alertis Tops but his team were like no, we are not going to take alertest Tops. You know, it's just a reporting system. We'll write by ourselves. We had this conversation three years ago and they still don't have this automated reporting system that takes all the testing because we've started thinking about how many human hours it takes to build something like a Lureport even not even to stops. So it has integrations with 26 frameworks on 11 programming languages. They are officially maintained by us and we officially say that they work well. So These are in GitHub, in a loo framework like repository. That means that you are totally backed by our team when you choose any tool for testing, like almost any. Of course, I guess there are some not really popular tools that will not be supported. But this is open, you can create like it's easy to create integration with a lure report. And if you take allure to Stops, you just ask us. And that's another point of taking of like waiting to take open source or to take some kind of paid software. Because if some company who bought Alert Stops actually needs integration, they come to us, to our support team and they say like we need to integrate some kind of this weird and specific JavaScript testing framework. And we're like oh fine, okay, why not? We have all these API handles available. It's not really hard for us to build such an integration. It takes like a week or two with testing and that's how it works. And that's why I sometimes I'm skeptical about many other tools which I see that are totally that existing closed ecosystem. I've been discussing at a conference yesterday discussing the Cypress tool. So it is an amazing tool for its tasks. So everybody recommends Cypress for the task it was created for. But when people start thinking about variability, Cypress sometimes becomes like a little bit close. So you need to think how to integrate it with your other infrastructure. The same works for example with tmes test management systems. Most of them are great for manual testing, but. And they say almost every TMS tells you that of course we support automated testing, but how does it look? It looks like an API handles from the Timius and they're like, okay, so you take these API handles and build your integration by yourself. So and then all you get from and then everything you get from automated testing is pass or fail because building a detailed step by step integration takes Time and good engineering and nobody wants to get involved in that because like, you know, I'm building some kind of small piece of software that will be used only here with another system, nobody know. And then if I write it today, I will have to maintain it for the end of my days. And that's how it works. So that's something to consider when you are thinking should I take open source or should I take some kind of supported product?

If you write your own software for this, so you'll get frozen until you need it to support it. Then you got defrosted, then coding and then frosted again like engineers of Boeing.

So I know you folks are always working on new technologies, new innovations. You have a lot of reporting, a lot of test ops. Is there anything new on your roadmap or something you're working on that you think people would like to learn more about as well?

Yeah, there are two things. First of all, one week ago we created a first major release actually for Elliott Stops. It has always been 3 point something and now it's 4.1.0. So it has a lot of large improvements and like more than 10 large scale features including actually speaking of integrations, like we've actually made a UI for integration. So now you can just open a window and then Click I want GitHub CI or GitLab CI and then it just works, well, maybe a little bit more complicated, but we tell it works like this and then we just take a look at it. So and the second thing after new release is actually something people have been waiting for a year or more or one or two years already. So on our website we have this forum for if you are a small team or if you don't want to work with your own infrastructure or your own on premise instance, you can get in the waiting list for a cloud version. So we've got about several hundred teams waiting for our cloud which will come to public availability extremely soon, like maybe when this podcast will be already released. So check out Kameta IO and check out our new cloud instance because it is actually much easier with cloud because you know when you take On Premise Alert stops instance, you need to build it, deploy it and so on. So with cloud you just drop us your contacts, we send you the key and the button and you just click the button and Allure is already here, ready to surface your test and just plug in your CI, show the allure, your posture with automated tests and actually you can start working. And actually the cloud version is something which looks like it should be from the very beginning. Like extremely easy to use, extremely fast and extremely DevOps ish. I would say yeah.

And I can tell you why it is because we use the latest cloud technologies for this because during our experience of supporting on self premise versions of Alure desktops we found out that most people are experiencing some problems with the storage because in the CEPH it's a file system and it works not that fast as required. So mostly our instances on Amazon S3 and it works really fast so in this way you'll never see a delay. We've been checking our metrics recently and we found that the average response time of Every request is 0.02 milliseconds.

Okay guys, before we go, is there one piece of actual advice you can give to someone to help them with their automation testing Test Ops efforts? And what's the best way to find contact to learn more about Alord Test Ops?

First, be skeptic about everything you learn and second, try to avoid the pain. So choose the right tools and we've got a nice community of Allure test tops. So why it's important for people to come because they're helping us to become better. They see the things they don't like about Allure testtops and they see the things they like. So it is important to share what you think about that.

Yeah, for me the advice would be like easy one. So if you write tests, run them as frequently as you can. So the more frequently you run your tests the more detail you get. So because tests, as I said they are dumb. So sometimes they fall for weird reasons like time zone night build, weird infrastructure timeouts, hardware problems. So to eliminate all this stuff, your test should run all the time. Like create a separate machine to run the tests all the time. Nonstop. That's the advice. It's Qameta IO our website. So we have actually a blog which tells in a lot more detailed way everything that we've been discussing today. So I guess I would love people to read it and come to me with a feedback as I'm writing. Like there is more than 50 articles I believe. So I would love to get some feedback or collaboration ideas. So come to us, read about alertestopsimeta IO and read our blog.

Thanks again for your automation awesomeness. The links to everything of value we covered in this episode head on over to testguild.com a407 and while you're there make sure to click on the Try for Free Today link under the exclusive sponsor section to learn all about Sauce Labs awesome products and services. And if the show has helped you in any way, why not rate it and review it in itunes? Reviews really help in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack automation awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Automation Podcast. Head on over to testguild.com for full show notes, amazing blog articles, and online testing conferences. Don't forget to subscribe to the Guild to continue.