How To Optimize your Automation CI/CD Pipelines (and Save Money) with Gaurav Mittal

In a land of testers far and widely journey seeking answers seeking skills seeking a better way o the test illumination testing podcast guiding testers with our nation awesomeness from ancient realms to modern days they lead the way oh the testicle ocean testing podcast.

Hey want to know how to optimize your automation ci cd pipelines and devops and save money doing it if so you're in luck because today we'll speak about guav mattel if you don't know gwav is a cybersecurity data science and it expert with over two decades of experience leading high performing teams in cloud computing machine learning and data security he's also a thought leader in ai and automation and has published numerous articles on optimizing ml deployment security email communications and automating workflows and a bunch more he's also recognized with multiple industry awards and has an aws certified cloud practitioner and a lean six sigma certification professional you don't want to miss this episode check it out hey guav welcome back to the guild.

Yeah hi joe thank you for having me again it's wonderful to catch you again and it's a pleasure to be on your show thank you absolutely so you're like.

A jack of all trades so why why is this topic top of mind now like how to make devops more.

Cost effective so like whether you are working as an automation engineer you are a security professional you are aiml devops is everywhere because ultimately you have to build the pipelines everyone loves to click one button and make sure whatever they are coding it's gone okay they don't want any hassle of deploying it again and again and like rd professionals today they are working on like branches so ten people they have their own branch they want to push the code to the main branch they want to get it deployed as soon as it's done so that like all the checks all the automation tests they are being triggered automatically so devops is everywhere so like that's the reason i also learned it i implemented in my organization and i brought it up this topic so hopefully like i'm happy to share my knowledge.

On this love it and actually you wrote a really cool article on curtailing software automation to licensing costs yeah by curbing licensing needs which i'll have a link for down below but let's dive in a little bit about that what's up with licensing needs what's the roadblock.

There yeah so most of the product based companies they love to proko licensed tools so for example like when it comes to automation okay like there are different engineers working in different kind of domains some engineers they are deployed in salesforce others are in web some are in data each of these domains they are using different language so ultimately you will end up in hiring automation engineers who are expert in this language instead of that the best way what i have seen the trend water organizations they are rooting towards is procore a licensed automation tool the problem is you have procured a license tool you are paying for it but license are limited most of the automation engineers they are sitting at the offshore now at onshore to how many folks you will provide this license so the best outcome is implement a cicd pipeline which will sit on top of this license tool the advantage is let automation engineers they write the test cases but anyone you are giving the flexibility to trigger the licensed automation tools through cicd tool so that's the main benefit of it like how you can curtail the uses of license tool just make like a ci cd tool cicd pipeline implemented which is going to trigger the automation tool and hence automatically like product owner or like managers or anyone they can trigger the automation test too no need to have license so.

When i look at the article i think you mentioned something about using github actions to help with this how does.

Github actions help yeah that's the second thing which i want to touch actually there are several cicd tools in the market okay like for example we have jenkins we have gearset we have copedo for different domains we have different and github actions gitlab so when like i was supposed to like implement a pipeline i have done a research and ultimately narrow it down to jenkins and github actions so jenkins like everyone knows it they love it because it has a very nice ui and it's quite easy to use you have to just write jenkins file and like you can create the jobs if there are some existing jobs you just copy it so manage jenkins is quite easy the problem is jenkins is still paid because you need to pay the server cost which is not much but you have to bear for it and second drawback is like there is a quarterly you need to rotate the security keys which is again a pain point but the benefit with the github actions is it is open source and secondly if the developers they are using github they are very much familiar with the github actions there is no learning involved there is no license nothing so you implement github actions which will trigger your github repo and your pipeline is implemented other main advantages like in most of the companies this github is being maintained by their security team so if in contrast to like jenkins jenkins is calling your automation test tool you are making an api call okay so there is some sort of permissions issue may arise with github actions it is all internal your security team they are handling github actions they are handling github repositories no need to worry about any permission issue it is quite straightforward so github actions is very you know like winner it's easy to implement developers they already know it qa they were writing jenkins file here they have to write a yml script and you can invoke your licensed automation too all right.

So i'm not sure if i'm following all along say i have a paid vendor tool that has licensing yes i have ten licenses and i have a team of thirty correct is it because it's running in ci it's an ephemeral environment and only uses that licensing while it's up or like how does it actually save or help optimize license usage.

Yeah so like you said like suppose there is a thirty team members or.

Maybe i'm not following along yeah so.

Like we have thirty team members and only ten license are available okay you have given those ten license of automation tool to automation engineers now tomorrow when rest twenty they want to execute the automation test how they will execute they have to ask those ten guys hey can you please trigger this particular set of tests instead of that if you are building a ci pipeline which is capable of invoking these automation tests you don't need to ask anyone you have the flexibility so this problem is very helpful when you have offshore and onshore because most of the license they goes to offshore guys they are working there and because of the billing reasons they are being the automation scripts they are being developed there but at onshore like in our time if you want to execute those tests you don't want any dependency for that like just make a ci pipeline which is invoking this automation test and anyone can execute them no need of license very cool so you.

Know you keep mentioning offshore onshore teams there any other ways you can help cut costs besides this approach yeah like.

Even within one team only like the license they are quite expensive suppose your team sits only at onshore and there are two qa resources instead of asking every time like okay can you execute them or even if they are scheduling but you want to run like there was a production incident and now you want to for a sanity check you want to run automation tests so anyone should be capable to trigger it at any time so it's not only about onshore offshore the point here is about the flexibility who can trigger the automation test and for that a ci pipeline is actually very helpful nice so what.

Are some of the benefits it sounds like once again you get rid of the manual oversight of the ui test executions because the ci pipeline takes care of it for you with all you having to monitor it am i understanding.

That yeah so yes that is another advantage what happens like you are running your ui web application automation test and when the tests are running sometimes what happens there are some tools which capture your screen so what i'm trying to say you cannot do anything else the qa has to just sit like this and have to watch the execution but when you execute them through like a ci tool ci tool generally like you need to provide a runner runner is like a virtual machine and ci tool will execute the test on this virtual machine so we are giving the flexibility to the qa or any anyone else who has triggered the automation test to do his or her own work and automation tests they are being triggered on another virtual machine so what happens like in today's world like we are giving aws ec two instance as a virtual machine so when the tests are executing based on the type of ec two instance you have given automation tests will be executed there and the qa they can do their own work so this is another advantage are there any tools.

Where this wouldn't work i would think sometimes companies get kind of sneaky with their licensing like you can't run it in cicd i don't know are there any kind of limitations with this approach.

Limitations with this approach ci i have always seen advantages the limitation is like you need to be thorough somewhere like you need to have knowledge another is like what kind of automation test you want to execute so how to configure like your auto you need to sometimes customize your automation framework accordingly to your like how you are going to set it up in a cicd tool because like okay from your automation tool you can choose okay ten tests you can execute but if you want to execute from this cicd tool you need to club them in some category so for that you need to customize your automation framework that is one limitation basically learning is involved that is something is there and again they are like you are integrating you are integrating cicd tool with automation test so that integration like as i mentioned like we are calling apis so this ci cd tool is going to invoke your automation framework maybe there are some firewall issues gha is going to invoke the github repository only so it's quite easy there is no firewall constraints are there so why wouldn't everyone.

Do this is it hard to set up is it a limitation is it just people that don't know hey i could probably save doing it this way.

Yeah so that's the point that's the advantage of ci and that's what i want to propagate like license is something which every companies they are struggling in today's world everyone is running after the budget cost cutting but this is a very nice way and there are free open source this cicd tools available like gha so i want to like cicd tools they are never free because they require a runner machine and that runner actually procore some cost but i have found a solution for that also so for example like automation engineers they are using a license tool which is quite heavy sometimes company like they provide a virtual machine like windows three hundred sixty five to the qa engineers that you install your automation tool on this machine okay so that you can save the memory of your local machine for the regular work like manual testing or whatever you are doing from gha you can configure windows three hundred sixty five machine as the runner so what we are trying to do we are doubling the usage of this windows three hundred and sixty five machine we have installed the tool automation tool on this windows three hundred sixty five as well as we are also using it as a runner so ultimately gha is totally free of cost gha is invoking the machine which is already available there in the organization you are not using any aws ec two instance or launching any other like virtual machine at any runtime you are using which is already being provided to.

You nice and once again i believe this is all in the article that i'll have a link for down below so this also helps with aws instances i assume as well yeah it will.

Help like if you have already proud any aws ec two instance for twenty four by seven but there is some bandwidth you are not using it then why not to use it the whole idea here is like the ci cd tool requires a runner instead of launching a runner at runtime asking aws to provide a new ec two instance first check within the organization if there is already existing virtual machine which you can use right it will save lot of money for the company that is the main i have put the calculation also like what i was able to save.

What percentage you don't have to give the amount but what was the percentage.

That you saved so percentage depends on the test but like comparing to ec two instance versus like using your virtual machine existing machine it's zero like with the existing machine it's already there you are not paying anything with ec two instance like it depends on number of tests like you are executing five hundred tests and like ec two instance was launched for four hours so for four hours based on the type of ec two instance you are paying suppose fifty dollars now it is zero for one execution so monthly executions by ten teams you can calculate like it's a huge saving it's an innovation actually we are not invoking outside virtual machines we are trying to use what is already being.

Provided to us very cool so once again like i said i'm scanning through the article i think you mentioned other things you can incorporate in your ci pipelines that people probably yes don't take advantage of i think you mentioned sonar cube and codeql yes can you talk a little bit more about that yeah.

So i have actually built the architecture for data projects and when the developers they are pushing the code what i have found like they were directly like once they were like there are time members from their local branch they have pushed it to main and from main they were directly pushing it to the production so this is not a good like devops practice there are several stages involved the first stage is like static code analysis so what happens like when you are writing the code maybe the code is not in a good shape there could be a data leakage or there are some security vulnerabilities which hackers they can hack the code and they can find it for example like i have seen it's very common when the developers they are writing the code they want to print what is the token jwt token value or what is the password or what is the username password they are printing those secret features which is which they should not be print or that should not be go outside outside i am saying because again like when you are writing a web application there are some external tools which are being used for logging so ultimately you are passing this information to third party tools which you should be worried of so how to catch this for this we have static code analysis there are several tools available sonarqube is there codeql is there codeql is a very heavy tool but it's a very you know nice tool for catching the security issues so the first stage should be static code analysis it is making sure like it is basically known as code smell it is checking like your code is in good shape it can be projected to the outside world with no issues once you are done this the second step is like run the automation test because you are going to deploy a new feature when you are deploying it you have to make sure the existing functionality is not getting impacted that is still working fine absolutely and once your automation tests are being passed then you should deploy it to production so this should be the ideal like flow from qa to production and i would assume.

With that flow bacon security into the devops pipelines without having to rely on say really time consuming security tests you could save money kind of backhandingly because you're not relying on yes right yeah.

Even today like this code editor tools which developers they are using they are very smart they are like using ai like but what i have seen still there are some security issues which they miss because these code editor tools they guide you how to write code better they cannot see if there are any injection errors these tools which are security tools which i have spoken that you can implement in static code analysis part these tools they are expert in checking like if there are any injection errors there are any null pointer man in the middle attack could be caused so they catch these kind of things till the time any issue is not happening everyone is happy but sometime like crowd strike issue happened everything come on a halt so it's better to secure your code before you push it to production.

Absolutely so i guess you know you work with a lot of different tools and a lot of companies i know they have a lot of licenses for a lot of different tools they don't necessarily use or need or they have duplicates of everything did you have like a framework or process you recommend to people can go through to say do i really need this tool is it worth having rather than having nice to have and i guess that would save money in the long run if they were doing this type of evaluation all.

The time yeah so this is one thing even which we try to do like but we were not successful because when we talk about different domains there are tools which have expertise for this particular domain for example like github actions is very good it works well for web data but for salesforce there are some other tools like gearset copado which like developers they want to use for deploying the code so it's very hard to say like one tool or one framework is best which can be like which can work for all different kind of domains but depending on the domain like if you are in a data like you are working on python it's better to use sql block sql fluff or rust as a static code analysis and uses unit testing unit testing is one area which i have seen like developers they don't write it but but it's actually very useful so these are some common practices i would say which everyone should use common framework is actually quite tough to say common framework or common tools because different domains have their.

Different needs all right so you mentioned ai briefly what's the role of ai can you use ai now i know a lot of tools like kubernetes yeah the solutions that built on top of that use ai to limit resources a be able to read when it should be up and down do you see those type of ai solutions helping with.

Bringing costs down yes i can cite you a couple of examples ai is useful if we talk in terms of soft savings not hard savings so i give you a couple of examples like in my projects like any data project or like they are very heavily relied on sql and what i did like i created one agent sql optimizer and like it has given me a wonderful reasons for some sqls like existing sql was taking like suppose twenty five seconds the gen ai generated sql was taking seven seconds so it's like more than fifty percent of performance improvement okay on top of it it will also guide you like why genai is making that change so for your learning if you want to enhance your career growth like this is a good uses of genii the problem is it comes with its pros and cons there are a couple of problems one gen ai has a limitation on token like for example it cannot work more than five hundred six hundred lines of sql and secondly it's ai okay ai has a particular confidence level of eighty percent or ninety percent but not one hundred percent one existing sql was giving you one hundred rows the new sql is giving you ninety rows okay it completely changed like instead of improvement it changed joins or anything so you need to be careful what i'm trying to say like ai is good to learn and check the performance but you cannot blindly use it in the to deploy it into the production another example i talked about are these tools security scan tools like codeql and sonarqube are there with gen also like you have written your code you can ask can you provide me any security scans on this and it can give you the output like if there is some hacking hackers can crack the codes and then how you can write it in a better way so you can modify your code so these are some couple of examples ai comes into the picture and one big example is whether it's automation or it's a development people are using it to analyze the production logs like where the error is occurring so it helps you to pinpoint the location from which code line the issue might be occurring so basically trying to save some time for you it's all falls under soft savings yeah so is.

That leveraging opentelemetry with the ai to be able to let you know that it failed in production and what line of code came from so production logs.

Like whatever tools you guys like teams are using for example like in kibana or anywhere and or like in your local and you want to pinpoint like okay from where from which function this error could be pointing out so ai can help you in text extraction like okay this error could be because of this if like there are thousands of files which are getting generated on daily basis it's difficult to manually parts but with ai you can do it a little faster and it can give you some hint like okay maybe this part.

Is failing so are there any open source ai type of tooling you can use in your ci cd pipelines or is this these all probably paid vendor.

Based solutions no no no so there are several like open source like ai i have used i have not used in my devops pipeline but all the work what i have done in ai the models i've used they all are open source so for example like i have spoken about text extraction so nlp is there natural language processing and like it's open source we can write a model around it we can figure it out like okay we can do the text analysis we can do sentiment analysis then for image classification like okay what kind of image it is we can use deep learning ml models like keras tensorflow these are open libraries we can use them so but i have not used them yet in the devops pipeline.

Cool all right guav before we go is there one piece of actual advice you can give to someone to help them with their devops automation testing efforts and what's the best way to find.

Or contact you yeah so my solution will be like right now like penetration testing is there which is a good area and in the devops we should implement it by using this security tools as a qa we all love to find defects and this is one nice area niche area i would say i have found like where i am shining and giving back to the my company like okay yeah these are the issues we should take a look over it so for your learning for your visibility try to use security scans in the pipeline this will be very helpful for you as well as for the company and yes you can reach out to me at my email address gauravmittal one thousand nine hundred eighty five dot com.

Thanks again for your automation awesomeness the links to everything of value we covered in this episode head on over to testguild dot com a five hundred forty one and if the show has helped you in any way why not rate it and review it in itunes reviews really help in the rankings of the show and i read each and every one of them so that's it for this episode of the test guild automation podcast joe and my mission is to help you succeed with creating end to end full stack automation awesomeness as always test everything and keep the good cheers hey thank you for tuning in it's incredible to connect with close to four hundred thousand followers across all our platforms and over forty thousand email subscribers who are at the forefront of automation testing and devops if you haven't yet join us vibrant community at testguild dot com where you become part of our elite circle driving innovation in software testing and automation and if you're a tool provider or have a service looking to empower our guild with solutions that elevate skills and tackle real world challenges we're excited to collaborate visit test guild dot in fo to explore how we can create transformative experiences together let's push the boundaries of what we can achieve.

With lutes.

And lyres the bards began their song a tune of knowledge a melody of code through the air it spread like wildfire through the land guiding tester showing the secrets of to behold.