Automation Engineering Productivity with Dushyant Acharya

Welcome to the Test Guild Automation Podcast where we all get together to learn more about automation and software testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Automation Podcast. And today we'll be talking with Duchant all about engineering, productivity, machine learning, all kinds of things. Technology leader with expertise in building and running high performing engineering teams, delivering quality products at scale, which I think is really unique because a lot of people stumble on this. So we're going to touch on this. He currently runs payment platform engineering teams of engineering, productivity and DevOps at Ripple Labs which aims to provide frictionless experience for global payments on blockchain technology. Another thing we might touch on. So definitely you want to listen to this episode. Deschant also holds a strong academic background with a Master's in software systems and also has an mba. So he has a lot of knowledge. You don't want to miss this episode. Check it out.

The Test Guild Automation Podcast is sponsored by the fantastic folks at Source Labs. Their cloud based test platform helps ensure your favorite mobile apps and websites work flawlessly on every browser, operating system and device. Get a free trial. Visit testguild.com sourcelabs and click on the exclusive sponsor section to try it for free for 14 days. Check it out.

Hey. Shit. Welcome to the Guild.

Hey Joe, it's. I'm really glad to be here and it was very generous introduction. Thank you very much for kind words.

Yeah, I really love your background. A lot of things we could touch on. So the first thing that caught my eye was you talked about something called a productivity paradigm. And when people think of test automation, sometimes I think they think of it just as functional automation. Just curious to know what is the productivity paradigm? How does it relate, if it relates at all, to test automation?

You're right in the gem of the question, Joe. So I really like the question. So let me start by saying that all of the years when I have been doing quality related work, it has been evolving. A lot started with manual testing ages ago. And now people don't like to do manual testing for the right reason. And automation is the right focus on how you are actually testing your feature. But sometimes the testing the feature itself is not sufficient. Because nowadays what we're looking for is a continuous delivery of your product. How fast you can deliver your product. Is it a monthly, is it a weekly? Sometimes it is a daily. And when that happens, you're not just focusing on one automation because you're focusing on the entire pipeline of your code. What happens after the code is written, how quickly it can go to the production and where you have the chances of increasing the productivity. So when I say about the productivity as a paradigm for engineering productivity as a space, I usually try to accommodate that in my understanding.

So I think what would throw a lot of people off is a lot of times they struggle just with functional UI automation. How can they scale? Like, like you said, we have to scale. We're trying to deliver software quicker and faster. But if people are just struggling on just UI automation, how can they actually expand that to be more to help other activities that may be slowing them down?

Well, if I had to give the easy answer to that one is the key to your solution would be to break down your problem into multiple pieces. You cannot solve all the things together, you cannot aim too high and wait for a big delivery to come up to automate everything. If you already have the ui, that's a step one, right. Then think about what is next. Think about what you need to automate next by looking at what is failing first. Right? So start measuring. Being data driven in that kind of situation helps you a lot. Start seeing where your team is going wrong, what your production issues are coming from, where you're spending more time in delivery and trying to automate that part first. Is it the back end, is it database, it is performance or is it just the compatibility with your other systems in general? Right. So it'll become clear if you start measuring data and break down your problem into multiple pieces.

So I know a big piece of the paradigm is getting developers involved because it's like you said, it's not just about focusing on UI automation. So what kind of activities can people be looking at if they're not aware of that? Maybe they're just aware, oh, I couldn't wanna made a browser. But they may not be aware of maybe other opportunities. Are there any other areas you think there are quick wins maybe that people don't think about when things comes to automation.

So as a, as someone who is working on the automation, you look past your base case. That's the idea. So if I am the engineer working on the automation, am I focused on the feature test which is, hey, what is my feature doing and is it doing what it is supposed to do? Or I'm trying to get how I can test my feature faster, how I can test my feature, relax, how I can build my feature in a code faster without any issues, how I can actually get through the finish line of deployment into all the environments, all of the non production environment. Or the production environment faster. So change that thought process for the automation engineer that I'm not only focus on one feature, but I focus on the experience my engineer has or my code has when it is reaching to the customer. On the other hand, if I'm a developer now, we are asking developers to look beyond your code. You're not just responsible to write a code, you're responsible to write a quality code. Which means you do need to do some kind of validation of your code. You need to understand what your feature is and how you are testing it. Yes, your automation team would give you all the tools you need to do your job in that one, but you cannot ignore it. Not anymore.

Great. So a question I get asked all the time, especially, especially as we're doing continuous delivery, how do we measure success? Like how would you measure success? Is it just, I know it's a terrible thing to say, bugs found, because that's always been a terrible metric, but is it delivering software faster? Like, what do you usually measure teams by to see how they're coming along in this paradigm shift?

Oh, that's an interesting question. So you know, I'll go back to my studies of that and sometimes the measure you put creates the wrong incentive for team to focus on that major, that suddenly that major, instead of helping you to do something better, it'll Dr. The wrong behavior. So my first instinction, for example, would be, hey, the good thing to measure is how many releases I'm doing, how many releases I'm doing without any issues. But then maybe if you do not load your release with any feature at all, then you will be doing more releases and not releases without issue. But at the same time you're not actually delivering substantial feature set as well. So now what we'll try to see is what is the experience for the stakeholders, right? So if I am the engineer, what is my experience? So if I created a code, what, what are your metrics? Which actually tells you that hey, I need to spend two days to create the bells, work with the testing, to write test cases, test my feature, which I actually wrote it in three days. So that two days is your overhead on three days of your development. That's not good. But if that two days can become two hours, then that's very good. So if you're looking at the experience of the people involved, that might be a better mattress as well. One of the experience which you always be looking for is definitely your customer. And that's why it will go back to your customers experience that how frictionless deliveries they are getting many of them, how big they are and do they actually know? If I had to tell my customer that, hey, expect you're going to be down because I'm releasing something, then obviously it's not a good experience. But if I can actually release it without even my customer doing it right, that is something which I would like to have.

That's a great metric. I know a trend I've been hearing more about is site reliability engineering. And so they're able to measure calls from customers if something went wrong and that that could be a good way to determine how well the team's delivering value. Maybe as calls going down, customers become more satisfied. Do you do any type of surveys? Is it more just a little, not as scientific to gather that data?

We do, we do surveys. So as I mentioned, the customers is very important. The other aspect which comes in, comes handy is to actually start with who is your customer? So I manage engineering productivity and I manage DevOps. So when I, when I have my engineering productivity hat on, my customers are other engineers because I'm focusing on their productivity. Yes, it is about continual integration, context, delivery. It is about the pipeline, it is about dish automation, it is about the tools to help them. They do their job, but they are my customer, so my focus should be on them. What are their pain points, what they're looking for, technology, tools, everything is included in that one. When I'm winning head over DevOps, I'm working with engineers as well as I'm working with operations and sre. So it changes a little bit. What are they looking for? Right. You don't have to tell it, but sometimes security, reliability, everything comes as part of the package and you actually have to figure out how you do that. The only way to get to that is you need to talk to your customer. Now if you also include your external customer, then service are very, very important. But if you're talking to internal customer, I think talking helps. That's usually something which people avoid many places. And it's surprisingly how easy it is to talk to people in your company and get the info right.

That's interesting talking. That's. Yeah, that's a good, good point. Sometimes the lowest tech thing is usually the hottest thing to do for some reason.

Yeah.

Cool. So you didn't mention you, you manage multiple teams and I assume that you always need to be researching or finding, you know, maybe tools that could help the company or help your teams, as you say, become more productive. How do you do that? How do you know what's worth focusing Your time on with all the tools in the market, all these not exaggerations, but all these claims about AI machine learning, how do you know what tools to focus in on that you think actually are going to add value to your team?

Well, to be honest, I don't spend as much time researching the things as I probably would like myself to do it. You always have to balance how much work you have to deliver on a given day versus how much time you have on your head to research something. At the same time that's not an excuse to I don't want to improve because I don't have time to research. So what I try to do is try to find, find a midway and a couple of things which I try to see. One thing is if there is something which you actually identify as a pain point, then it makes more sense to focus on how we can actually fix that. Is it new tool, external tool? Sometimes internal tools? Sometimes. Maybe you want to write something yourself. You do that. If you actually have no idea, then actually you actually go into real market research. So I did that once when I was working with the machine learning, testing that how do I actually make my test case intelligent? Because I don't know what my output is when I'm testing my machine learning algorithm and we research multiple things and then we realized that hey, my product is using the Google Dialog Pro. Why don't you use the same thing? Because we already have the expertise on that one and then it may or may not be the right taste solution for everyone but for my team, for my contacts, that was the perfect solution because we an expert on that one, it's very easy to integrate and start using.

No, it's interesting point. I think you did a session on AI machine learning and one of the talking points was how do you know, how do you test something where you don't know the outcome? And so any tips around that? Because it seems like you're using blockchain and that seems like a newer technology that people are kind of unsure about. So as you're trying to release software quicker, obviously you need to cover more test cases. But then it used to be deterministic. You could only automate things that are deterministic. But nowadays it sounds like you need to be able to handle maybe workflows that you may not anticipate. So how do you handle that?

That's another interesting question. Yes, I, as part of my previous company, I was responsible to put a JSON automation framework to test the machine learning application. And I did a talk last Year about that one. If I do answer short, what I will tell you is this is a progression of our the test case and test design as we go through. Even before AI machine learning came a lot, many test engineers were starting to think how do I let my test case fail gracefully? So do I expect my taste case to pass 100% and you go few years back, the only way my test case will pass is my asset. A equal to B or A is not equal to B and if it is red and it's just like white or black, right. It's working. It's not working. Very simple. But as you go into the AI machine learning that becomes more complicated because you do not have right or wrong answer. You have possible multiple right answer. And the question is how you let your test framework gradually understand that and fail. So when I started implementing that one, it was initially was just a if else conditions, to be honest. Hey, if this is, it's okay. If this is, it's okay. And I actually had a really good example on that one as well. Like say, for example, I'm testing my login and you want to test if my login is successful, then I'm expecting some kind of welcome message and if it is unsuccessful, I'm guessing I would get some kind of error message. Now I may not know what kind of error message is if if my message is coming from a machine learning algorithm, but I know it would be a message which says hey, this is wrong, invalid something or welcome you logged in. So I can look for those keywords in the message and I can infer what the message would say. Or I can actually go full blonde and use the my machine learning algorithm to take your output as my input and run into my algorithm and tell you the conference level that I agree and I would not say 100% agree is the pass. But I'll say 70% matches also parts. So the second one was what we actually ended up implementing. But the idea here, if I had to summarize, is how do you let your test case fail? It is not between 0 and 1. Your choices are infinite and you start thinking about do I have more than 70% confidence of the match, then my test case will pass.

No, it's a great point. I always worry about not tests that are always passing, but how do they handle what happens when they fail. And this sounds like maybe people should be focusing on how does it handle gracefully like you said. And maybe AI machine machine learning is one way that can help assist with that as well. So how did how did this work for your company? Did it? Because a lot of people say it's overrated, it doesn't work. Did you have success actually implementing this technique?

Very good question. So in my presentation, I wish I had that diagram handy right now. But the machine learning application is just an application. It is just using machine learning. Your input and your output may be different based on the algorithms influence. But at the same time you should think of your system as a normal system. If you actually try to see the actual machine learning, you do have the input data which your machine algorithm, machine learning algorithm is going to use. You do have your machine learning algorithm giving you some kind of rules that this is how they're going to work. Then you do have to collect, put that with your system in place where your input from your customer will go through that algorithm and give you some at the same time you have to capture those output in somewhere where you can actually go through them again. It has to cycle through back into retraining of your algorithm as well. It's a complex pipeline and if you think about it, it's kind of a data pipeline as well because it is very heavy on the data. If you are not recycling your data through the entire cycle, then your algorithm doesn't learn the next iteration. So that's very important. So now once you decide that one and you see the system as a pipeline of data moving across and machine learning is one important key piece of that, your automation becomes a combination of multiple things of automation, which probably you have already done that in your career. Right. If you're good at automation engineering, you probably would have solved each little piece of separately and then the only piece which now you have to solve is your machine learning algorithm becomes very easy to solve. And then you actually go into how would you solve that piece? And you have one, two or three options. So when I said I did implement things, it's like we can implement five, seven different things and you combine them at that and you make sure that you get enough confidence so you can release the product without impacting customer. Right. That's good. If I'm trying to test each and everything, then probably I'll do one release every decade or so. Right. Because there's always infinite possibilities. But I'm trying to make sure that I don't impact my customer. I'm giving myself a reasonable room to failures. If I actually go and deploy something, my customer is happy, I'm happy.

Nice. So a discussion that always comes up when you talk about automation quickly, fast and manual testing, you Mentioned manual testing. We need to minimize it, obviously. But obviously does manual testing have a place in the modern type of paradigm you're thinking of, for example, for exploratory testing for your teams. Is there always going to be a slice that can and only will be done manually by a tester?

That's a very good question. We've come a long way in recent years specifically so you know the earlier days when we used to say that hey, I need to do manual testing. There were a few reasons for that. Reasons like I don't really know how my customer is testing testing, so only my customer would know. So I do have to put a U8 on the schedule and it has to be manual or reason like this is the so complicated integrated system where it has to be deployed on production system and then you only test manually. I can't access it. There's no API. How do I access programmatically? It's I have so complicated flow that I can only access it manually. Right now, fortunately for us, we don't have those kind of limitations. So now I don't say you don't do manual testing. I say why do you need to do manual testing? Because if you can explain that, that I'm pretty sure I don't have to tell you not to do manual testing, you'll tell me not to do manual testing. The idea is, are you asking that question, Are you trying to do manual testing? Because, well, I don't have time to do automation writing. I don't have time to write the automation test script. That is a different problem. That's not about my manual testing is important, not as an organization. I'm not putting enough energy and resources behind writing the better automation. If someone sometimes will say roi, right? That's a very big question which I still hear here, which is a valid argument. If there is one feature which you're deploying and you only have to run this test once for next six months and it takes one day to do a manual testing. And if I ask you to do one month of automation code writing on that one, then probably you do not have ROI to do automation on that one. So then the problem is am I saying that manual testing is good or am I asking the right question that why it takes one month, right, to write that automation. So I think now everyone is going to the better way where you ask the right question and you actually try to automate the things the right way way. And that is actually helping it so long to answer. But I will say it is a transition But I don't see any people now come to me and saying that I absolutely need pendutris. They actually come and tell me that I need it because this is still missing. You get me this and then I stop doing this.

That's a good point because I'm thinking over my career, my long career, a lot of testing was coming up with an infrastructure or an environment or like we need to run it on this configuration. And now cloud and Kubernetes and things spin things up in Docker containers actually take care of a lot of the things that were part of. It had to only be manual because you had a. There was a lot of manual steps involved, but now it sounds like automate it. So in your productivity paradigm, I think you touch on infrastructure. Is this part of where you think maybe this solves some of the issues that maybe people don't know that there's a. You can actually now automate certain things because we have solutions like infrastructure and if so, who handles that piece of the automation in the pipeline?

It does, it does, it does include. So when I think about engineering productivity as a paradigm, I do include infrastructure as well. And that is the reason my DevOps team is tightly coupled with engineering productivity. The way I like to answer that question is if you do not have the right infrastructure and you do not have ability to create the infrastructure on demand, you would not be able to use your automation effectively. Now who does that? Different companies have solved that differently. I would prefer the infrastructure teamworks which DevOps is part of it. You have SRE, you have Platform teams and you have multiple ways of spinning that right idea. But the concept which you mentioned earlier, right, that is my environment immediately. TABLE Am I using cloud? Am I using my Kubernetes or whatever containerization system? I'm using it, but is it something which I deployed and I don't have to change it to test some other feature? And giving that ability to my automation framework is essential because automation would only start when you have the environment. If I'm doing a manual environment setup before I run the automation, then my automation is not automation. It is still a manual and you need to give that ability back to the developer so they can do that within their pipeline. Pipeline as well. Then you need to give them the multiple options. It's not always a luxury to skip some of the testing. Reason for most of my career is related to finance or fintech and when we work with the fintech and finance there's a lot of eyes on compliance and security and you do not want to miss the testing, but you do not want to do manual steps in between as well. And the only way to do is effectively giving you the infrastructure which is as close to production as possible and ability to give you the control using.

Now, it's a great point about regulations. I think I worked for a regulation in healthcare and a lot of times if you audited, they wanted to see the environment you tested and it was really difficult. But now with these type of solutions, you can take a snapshot and actually spin it up. So rather than saying you can't automate it because there are manual steps, you actually sounds like you probably could automate it and would help you because you have an audit of what you did, what everything is that an auditor can actually look at. So it's a good point. So I guess my next question is I always get asked, all right, I'm a tester, I've been a tester for 20 years. Am I becoming obsolete? And you've mentioned developers a bunch of times in the automation pipeline, is there still a need for automation engineer or a tester? And if so, I know on LinkedIn you have a, you hired a few weeks ago and you have an opening. Just like what kind of skills do you look for you think people should be focusing on as we move towards this paradigm that I think really is going to be a shift that more and more people are going to start experiencing.

That's a good question. Again, Joe, shutdown is there's always going to be need, there's always going to be need for the people who have hyper quality. And I for writing something which is effective and repetitive and help reduce the waste in my system. Right, that's what the automation is. Essentially the idea which is changing over last decade is was it manual, was it the UI automation only? Was it the API automation only? Are you actually doing that once a month? Are you running that every night? Is it running 100% successfully or not? Or now what I'm saying is this is running as part of your build every time or not. And how many people are writing test cases versus how many people are supporting writing good test cases? And so thinking versus the actual work, that's what is changing a little, helping it. At the same time, it's also increasing the scope. If you look at four years back, I did not ask developers to write test cases. I would hire new engineers to write test cases. But at the same time, QA engineers would not be able to move past test cases. They'll only have to focus on these Cases only now it is win win because now we create a framework which is efficient for developer to use or at least I hope it is efficient for developer to use. At the same time, automation engineers has more scope. It's now they're not just limited to that feature test. Now they're looking at your product holistically. They are coming up with the tools, they're coming up with the pipeline, they're actually managing the pipeline. They're responsible for putting that code in the hands of your customers without any vendor interaction. So it gives you more options to work on. So yes, there' always going to be a need for the people who have the right mindset and who has the right skill set. Right. I do hire for both of my team, which is engineering productivity, which usually most of the people who come to interview and most of my team who got hired in last one or two years, they came from DIS automation background because that's really helpful to understand this the new paradigm. Right. If you don't have that, if you don't know how you used to do things, it's very difficult as you know how you change the next slide. At the same time, when I work on DevOps recruiting, I look for people who are very focused on introduction as a code as well. So all of your infrastructure should be in your code containerization, how you're managing it, how you are deploying it. It's very important.

Nice. So as we get near 2022, I've been asking speakers what their thoughts are on maybe a tool, a technique or do you think it's going to be big in 2022 that I think people should know more about. Do you have any suggestions for predictions for 2022 on technologies you think are really going to take off?

Oh, that's a tough question. I'm not prepared for that. There's so many things I think on top of my mind. I think there are a couple of things which comes to my mind. One, if I think about the automation, I think we're not as much data driven as we should be as an industry. And anything which help produce the better metrics and focus on data driven dish automation or data driven, even the engine product users would be be a cool skill set to have in 2022. And on my other side of the house, I think infrastructure as a code has gained a lot of popularity and a lot of people are jumping to that one. I won't say it is upcoming because already that everyone is using it already, but I'm seeing a vision where the next year will be using it even more. Even more. As in, not your ops team will be using Fractured as a core. Your automation team would know that as well. Well, your developer would know that as well. If you want to create your local environment using the same logic which you create a production environment, probably it will be possible for most of the company in 2022.

Love it. Love it. Okay Deshaun, before we go, is there one piece of actual advice you can give to someone to help them with their paradigm shift in automation testing efforts? And what's the best way to find or contact to?

Let me start with the easy answer. LinkedIn. I'm very active on LinkedIn, so find me on LinkedIn. I really like connecting to Nipik people. Ripple is hiding a lot right now. We have a lot of open position, really good, amazing people and great work. So if you're interested in Ripple, feel free to look for the openings and reach out to me. I'm always happy to connect with you to the right people. The advice side, I think I'll say not one. Couple of things I'll say actually the one thing which I have learned from my recent network interaction is people get nervous about job descriptions and titles and they don't even think that they would be able to do the job. So don't feel that. Be comfortable if you know half the things in the job description. Be comfortable enough to go and talk to a recruiter and ask actual question on hey, what this job is. I have this skill set already. Do you think the other things I can learn on the job or not? Or is it like prerequisite of me giving interview? Don't assume that you are not qualified because you don't need each and everything in the job description. That's the first thing which I'll say. And the second thing which I'll tell the prospective people who are trying to network the other people, which I have recently as well, is when you reach out to someone, try to see what is your proposition. Tell them what you're looking for. Yes, most of the people do that. But also tell them why they should help you, why you are the right person for whatever job you're thinking about. Give them that one extra paragraph to describe that so they feel it more personal when they actually respond to you. I think that should be all great advice.

I just thought of another question though as you were speaking about. You know, even if you don't match all the requirements, you should definitely apply for the job. But when I look at Ripple, it's like financial transactions using blockchain. Just that alone kind of makes me not scared. But it's like I don't know if I'd be qualified for this. Are there any tools that someone should know about with blockchain for testing blockchain?

Well, before I answer you that Joe, let me tell you. My previous company was an AI machine learning company and before I started working on that day one, I had zero experience of testing machine learning algorithms. Ripple is a crypto blockchain company. Then the first day of the job I started understanding what crypto project blockchain is. Right. So it's not necessary to know everything because you have a specific role. There are things which you need to know and there are things which you can learn on the job. The idea important point for me is are you connecting to the hiring manager? You're connecting with the recruiter and making sure that you know what is expectation for the role. If I'm hiring for a certain role, then I can actually teach you the remaining things, but probably that's not clear in the job description. So if you're looking for a new domain, new technology, don't get started. That's the one thing. And if then nothing else, find out YouTube tutorials. You know, it's like cool people talking in 30 minutes and explaining the big ideas. That usually helps so that you have that fear away and then you actually start talking with recruiter to understand what the role actually means.

Thanks again for your automation awesomeness. The links to everything of value we covered in this episode head on over to testguild.com a369 and while you're there, make sure to click on the Try for Free Today link under the exclusive Sponsor section to learn all about Sauce Lab's awesome products and services. And if the show has helped you in any way, why not rate it and review it in itunes? Reviews really help in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe. My mission is to help you succeed with creating end to end full Stack Automation awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Automation Podcast. Head on over to testguild.com for full show notes, amazing blog articles and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.