Automation - First DevOps Focus with Kedar Kulkarni

In a land of testers far and widely journey seeking answers, seeking skills, seeking a better way o the test Illinovation Testing Podcast Guiding testers with our nation awesomeness from ancient realms to modern days, they lead the way. Oh, the testicle of nature. Testing Podcast.

Foreign.

Know more about automation first DevOps and more well, you're in luck because today joining us we have Gadar, who is a DevOps and cloud infrastructure expert and has deep expertise in CICD, Kubernetes and Automation First DevOps. He's written extensively for Open Source.com and Red Hat and also co authored a popular ebook on IT automation and created the AT CASC framework that's now part of Red Hat's Automation stack. He's also known for his unique approach to infrastructure test automation, which has shaped how teams think about testing infrastructure like code. So we'll be diving into GitOps, open source tooling and what it really takes to build DevOps frameworks that matter. You don't want to miss it. Check it out. Hey, before we get into this episode, I want to quickly talk about the silent killer of most DevOps efforts, that is poor user experience. If your app is slow, it's worse than your typical bug. It's frustrating and in my experience and many others I talk to on this podcast, frustrated users don't last long. But since slow performance is as sudden, it's hard for standard error monitoring tools to catch. And that's why I really dig SmartBear's Insight Hub. It's an all in one observability solution that offers front end performance monitoring and distributed tracing. Your developers can easily detect, fix and prevent performance bottlenecks before it affects your users. Sounds cool, right? Don't rely anymore on frustrated user feedback. But I always say try it for yourself. Go to smartbear.com or use our special link down below and try it for free. No credit card required. Hey Katar, welcome to the Guild.

I'm really happy to be here. Thanks Joe.

Awesome to have you. Really excited I guess. Before we dive in, I'm always curious to know how did you get into DevOps?

So it's an interesting story actually. And so basically when I was in the school I was studying for becoming a computer engineer and everybody was running behind making software, writing apps, web apps, everything, right? But my final year project in my Undergrad was in OpenStack. OpenStack is one of the open source cloud projects that were started by NASA jointly with Rackspace I think back in 2010. And so from 2010 it has grown Quite a bit. It's used in large telcos and whatnot nowadays for all the 5G or empowerment. And I got into that for my project and I learned, oh, there is a whole lot more between the software that people write and how it actually runs. And that's what got me into system space and that eventually drove me into DevOps through my career.

Nice. So what I also like is you're an active contributor to Open Source, so this is the first time I actually heard about, hopefully I'm saying it right. Atkask. Just curious to know what is atkask and how did you get involved in it?

So atkask is a name that I came up with. It was inspired from jcask, which is Jenkins Configuration as code. So atkask is Ansible Tower Configuration as code. Currently it is actually known by different names I think it's called. So it was called Ansible Tower when I started, then it started being called as Ansible Controller or AAP officially. And so the way we got started with that is when you use Ansible Tower or Controller, now that it's known as Controller, you need to create job templates and projects and credentials and bunch of other things in the ui. And as an administrator, you're constantly creating these things for your customers to consume those, to make that automation easily accessible. But for you to click around in the ui, it is very cumbersome if you're managing a large set of automation jobs. So I was like, why not use APIs? Why not use Ansible to automate Ansible itself? So Ansible team had some APIs they exposed as Ansible modules which I used to create Ansible Playbooks that could automate creation of all the resources in the Ansible UI in Configuration as code way. And if your configuration is code, then you have all the benefits of version control and GitOps and everything. So that's how the project started. And currently it is hosted in Ansible Galaxy, which is the open source repository of all the Ansible related collections, roles, specific artifacts that you could download. Which is very similar to let's say Docker Hub for Docker Images.

Right.

And this collection that we created at Cast got renamed few times throughout the years, but as of today it has Collectively more than 450k downloads. And not only it has downloads, but it became part of Red Hat's product portfolio and it is now offered through console or cloud.red hat.com to paid customers as well. So that's the journey of Adcast going from a small pet project that I started from my personal pain in the systems to getting it out to the customers, all of the customers.

So I mean, that's a huge adoption rate. So I know a lot of times people create things and it just on GitHub has one star, not many downloads. What does it take to build maybe an open source tool that DevOps teams really are going to rely on? Obviously this is probably a need you saw, but like, are there any other ingredients you see to make a successful open source project?

I think successful open source projects are those projects where you actually have felt that pain yourself, like whatever problem you're solving and that problem has to be in a niche area. Like Ansible is one of the top tools in the industry for DevOps automation. It has more than 30% market share, I think, which is way more than anything else, like Chef Puppet, Terraform, et cetera. So Ansible has a large community and anything that started as an open source project, if you take it to the community and share it, evangelize it, that can actually help you get more eyes on your tool, whatever you're developing. And so that's what I actually did. So I picked a niche area, I was suffering some pain, I saw the problem, I solved it and I took that to Ansible Community and said that, hey, this is what I'm building. And sure enough, there were other people who had started to feel that pain too. And there were two other contributors who also came forward and they said they started their own version of that some time ago. And so all three of us came together. We decided to open source each of our repositories and then combine all three of them into one single repository and then host it on Ansible Galaxy. And so we went from there. So that's how a project like this could come about to be. I think people need to understand when they, when they're talking about open source, it's not about you as an individual, it's about the community. And you have to walk with the community to succeed.

Now that's a great point, but I do see people that start these projects and they're the only ones really contributing or responsible for it. What do you do to get other people to contribute? Because that's also a barrier. I see a lot of teams, a lot of open source projects deal with.

So I have had my fair share of projects which have no stars or contributors. It's very common. You have to just keep going through it. There is no magic bullet, unfortunately, to get the contributors. I think the only thing that you would really do is evangelizing your project and see more and more opportunities to talk about it. More you talk about it more likely that somebody in the field is going to find that, oh, this is important and that somebody could be important enough that they could push to get more contributors. So again, this comes down to like that community building aspect and evangelizing to make your project successful. And as if you look at my GitHub, I have like 170 repositories. Not all of them are really important. A lot of them are like just toy projects. But then I also have to pick which one is actually important and take that one forward to the community. That way you can actually build credibility that you're bringing some nice ideas and not bringing every small thing that you dreaming of.

All right, so which one are you currently working on that's important that you think should get some more?

Love the project that I mentioned, the ATCask, which has been renamed and gotten into other repositories, is most active one I think, and it is benefiting people hundreds of saved hours and maybe millions of dollars in savings. Because think about it this way, if you create some infrastructure in the UI of Ansible and if you mess up something, if somebody else has admin privileges, it's more likely that you're going to have a team who has admin privileges and somebody mess up something and they don't remember what they messed up. And you're blocking thousands of deployments, you're blocking some important bug fix to go out to the customers who knows what. You are blocking out some critical workload that need to be running, that need to be scaled up, and they cannot be because your automation is broken. And this kind of repository that helps you automate the automation, it's like meta level automation really. It helps you save those hours and money and frustration. And that's the whole reason why I feel like that repository is the most important one in my career to highlight from DevOps automation standpoint.

Nice. So, you know, besides this, I know you're also passionate in general about infrastructure as code. I mean, I probably should have led off with this. For people that don't know what is.

Infrastructure as code, that's a really great question. So, infrastructure as code. When I was in my early career and I started realizing that infrastructure is ever expanding and if I keep clicking on the UI and or using CLIS to bring up new VMs, new resources, new networking configurations, whatnot, I'm going to grow out of myself quickly. I'm going to forget everything three days later. If you ask me what was the IP address I set or whatever else for something, I'm going to forget that. That's the time when I got introduced more to Ansible and Terraform and all the other tools that help you write your infrastructure, configure your infrastructure and as code, meaning that this is all the code that you're writing in terms of Terraform or Ansible, whatever you're using, or any other tools out there. But when you write this code, this code lives in your version control, ideally. And that code helps you sleep peacefully every night, knowing that if something burns down tomorrow in a data center where you have deployed something, you will be able to bring that back up within some reasonable amount of time without having to pull your hair. And a funny story regarding Infrastructure as code is like one of the articles that I authored early on for Red Hat, Red Hat Enables the Servian blog is Infrastructure as Code pros and cons. And so when I was in one of the new teams where Team Lead did not know me very well, he asked me, hey kk, I think what do you think about Infrastructure as code? We should start using that. And I was like, totally. I am 100% for that. And he was like, yeah, we need to convince our management about it. I was like, sure, tell me how I can help. And then he went on the Internet, he just searched for the terms IAC pros and cons. And then he found some article, he found a great information in that article and he sent that article to me and said we should use this. These are the key points to use with our management to convince them that we should go infrastructure as code route. And two minutes later he pings me again. Damn it kk, that's your article.

So he found that article organically on Google and it still to date ranks in one of the top results on Google or AI summaries. So I feel very fortunate to be contributing through such kind of early inception of iac.

Very cool. So it was early. So any new pros and cons? You see, I know this is infrastructure. Code isn't a newer term. It's been around for a bit now. So now with your experience after you wrote there, are there any new pros and cons that you that you would add to it or update?

Actually, no. I think maybe one thing that I really feel is the. I think I did not use the term back then in my article, but if I could go back and update the article, I would add bitrot is real. Meaning if you write infrastructure as code, you Exercise it. Once you deploy your infrastructure, three months later, six months later, you want to try and read on the same thing. Something may change in the ecosystem and can break something somewhere. And that is a maintenance and upkeep. Which is one of the cons that I had noted in my article originally as well. But I did not use the term bitrot, which I learned recently from one of my colleagues in the open source community.

So should every company that does DevOps be treating infrastructure as code or are there some cases where you're probably like, probably not a good, good fit for you?

So I have talked about this in my ebook. Like, one of the myths is that sometimes people think it may not be automatable, something is not possible to be automated, or may not be worth automating because you're doing it only once and you're not going to do it again. This is one time thing. Both of those in my opinion, are myths because most of the things, if they're not fully automatable, you can at least partially automate to some extent to reduce the amount of manual work or toil that you have to put in to maintain that thing or build that thing. Right? And then when somebody says it's not worth automating and it's one time thing, I have seen in my career that statement coming to me from various people time to time that hey, we just need to do this once, so just do it manually. And guess what? They come back again and they say, oh, we need to do it slightly differently. The same thing again, but slightly differently. Can we do that again? And like, see, we should have automated this thing.

Do you recommend then developers build the automation in mind? Because sometimes they'll build something. Like I worked with the company that said we're going to cicd, but they didn't make a build that was deployable. You had to do manual steps. And I was like, this doesn't make any sense because I didn't think about it. So any, any tips on how to make your code more automatable? I don't know if that makes sense.

I think when you start thinking about deploying the code right, like so as I said earlier in my introduction, between the software being written and deployed and being used by somebody, there's a lot of things that need to happen. There's systems, there's automation. And for any company who's trying to work on any new project, they need to put that mindset first, like, how are we going to deploy that? And I think that really starts from the point when let's say you have a team of developers, they are working on some project, Each of them need to set up this infrastructure on their local system to some extent to actually develop that project, right? And then from there it needs to be promoted to different environments which are not, not your laptop, but some other shared infrastructure. And so at every step of the way, if you're doing things manually, you're going to just end up wasting a lot of time and money and the resources and it's not good for anybody's, anybody's mental peace, any, any company's financial health or progress and et cetera. So I really think that when you're developing, you should start with, let's say, maybe use a Docker container or some kind of container to create a common shared knowledge of what all needs to be added for this software to actually run on your system. That Docker container, that image, that file container, file can actually serve as a reference point for, let's say, your DevOps persons to know exactly what it all takes to run this software and they can take the automation effort from there forward to automate for the large scale environments. Maybe that's one thing that I recommend, like try to create really good code that can help you spin up the environment quickly. Development environment quickly, because that is the starting point for actually getting that app to the production.

Nice. So we have the starting point. Is there any other steps to create an infrastructure test pipeline? They have a DevOps pipeline, but say they want to include infrastructure test as well into the pipeline. Are there any steps along the way that they would do?

Right. So one of the things I did for my Atkask automation was it was automating the automated platform. Right. So Ansible Tower is a platform that lets consumers, like other team members who are not DevOps engineers to quickly go and provision a resource or deploy something which is a CI CD platform, let's say, for the sake of this discussion. But if I'm modifying the code that is in Atkask to modify an existing job template or a project or a credential, how that is going to affect the actual production jobs that are supposed to be running. So that is that gap that nobody was thinking back then when I started that project. So I created a pipeline where I take the code changes made to the infrastructure code, much like software code, and I had a test pipeline for infrastructure code. So I would actually deploy an FA mirror Ansible tower instance that would basically have all the code that's running in the production plus your changes and be able to Test that there. And I created another project which is also open source, called Genealogist, which had ability to figure out from which job template which playbook needs to be changed or from which playbook which job template will be affected. And there's a whole list of things in between that are cascading failures if you don't pay attention to. And so Genealogist was able to figure out like if you make change in test YAML Playbook, then all of the things that need to be tested for that, it was able to derive that just by looking at the code. And this was way before all the AI stuff, right? So this tool was able to figure out what needs to be tested in infrastructure, deploy an ephemerated infrastructure, run the jobs for you, and tell you that if you merge this, is this going to go good for you or not? And that is basically an SDLC kind of like software development lifecycle, but for infrastructure code.

So where are you learning all this? You're just messing around and saying, can I try to do this? This seems like a lot of stuff you've done before. Like you said, AI is out now. There's a lot of things that make things quicker and easier for developers, like where are you getting these ideas to try this stuff?

So I think the ideas I get are because of my diverse background in terms of like what I've learned in my school. So I was learning for computer engineering, doing a lot of software, then I got into OpenST and OpenStack got me to Red Hat and in between I took data science courses and my masters. So. And at Red Hat I joined as a developer, then I did testing, then I did DevOps and Systems Administration and at VM where I was doing some development and automation, then there was some performance and skill benchmarking I did for kubernetes. So I kept doing different sorts of things for some reason in my career. And that maybe gave me that more well rounded approach for thinking. So that's why I was able to make that connection, like, hey, I'm writing code, be it infrastructure code or software code. It is code. So I can use the same principles I use for SDLC for software development Lifecycle. I write software, I have some unit tests, I deploy some ephemeral environment, test it, make sure everything's working, review the PR and merge it. Why can't I do the same thing for infrastructure? It's code in the end. And that's how it it really came to me. There's nothing more to it.

Right, Very cool. So you did Mention Kubernetes. I know a lot of people do deal with performance issues. Any insights around that? I think I have in my notes. GitOps and Kubernetes maybe some views around that area.

So when I was doing the Kubernetes performance and scale benchmarking at Red Hat, it was for the Kubernetes networking plugins. And we would be running like large scale workloads, which has thousands of nodes, sometimes thousands of pods running hundreds of thousands of different kinds of objects just to overwhelm the Kubernetes controllers. And then one of the things we often found was like, we just need to make sure that we understand what workload we are running and what kind of resource utilization you're going to actually need for that. If you get that sizing correctly, you're more likely than not to be safe with all the performance issues. And that's where everything boils down to. So throw more resources is not actually the solution. But throwing less resources and running production with less resources can also bite you. So you need to try and find that balance. And that balance can only come if you are running your system. Let's say if you have software that you're deploying in production, you need to run whatever kind of operations you're running against that software at large scale. You need to design some tests for that and see what kind of limits it is hitting and then make an informed decision for the underlying resources. Then only you can actually succeed getting the highest performance without seeing any bugs or issues.

So someone's listening. What's the role that usually does the Kubernetes performance testing and the infrastructure as code, are they separate roles or is this something that you would recommend any DevOps engineer or software tester would would know something about?

I think these. Anything that is not software engineering, like writing code for making software is in my opinion honestly is a giant blur of roles because it is called DevOps Systems Administrators, SRES Test Automation Engineers, software engineering in test or whatnot, and really, or even CICD or release engineers. Everybody's actually doing everybody else's job to some extent. And so if you start exploring, there are different career paths you can take in there. And I think to answer your question really well, there is no right answer to see which is the correct path for you. You have to just explore and arrive at a point where, where you feel most comfortable. So in my career I went from developer to QA to system administrator, to DevOps to developer again and to DevOps again and all of the journey I realized that all these years that DevOps is the place where I feel most comfortable. Because every day is a new challenge. Every day I'm supporting a different app maybe or different kind of project that needs different resources in cloud or on prem or whatever. Right. And all of this.

Keeps me feeling afresh about my job versus if I'm writing just software for one project. Software project tend to be like very long. It can take several years before you release first version of your software. So you're coming in day in, day out and writing the code to make one feature work for months at a time. That's great and some people love it. But for me, it was not the path I really wanted to stay on. And that's how I arrived in this area of the career.

Excellent. So what's the latest and greatest thing you're messing around with then? If someone's like, you know, you seem to always be doing something different, is there anything new or any new technology that you're excited about that may not be AI related?

I think I am always intrigued by all of the GitHubs that goes behind Kubernetes like the Argo, CD and Fluxy. I know it's not new technology per se, but. But there's a lot of opportunities to explore it and use it to your personal, personal gains to make your day job more easy to deal with. A lot of kubernetes, namespaces or projects. And I think that's one of the things that I want to spend maybe a little bit more time on in my past time to see how these technologies are evolving with the, with the type.

All right, so speaking a little bit more about ask, I think I saw you, I saw a video on YouTube where you presented at a conferen and I guess it takes a little deeper dive. Is there anything else that you want to talk about when it pertains to atkask?

Sure. I think if a user wants to learn about how to use atkask in the context of Ansible. I have made a presentation in DevConf US 2020. It's up on YouTube and one of the cool things I noticed after I presented ADC was there is similar thing that Puppet does in their offerings called CD for pe, I think Continuous Delivery for Puppet Enterprise. And if you hear that concept, and if you hear my presentation, you'll probably be able to draw very striking parallels between what I presented and what they're currently offering. So I want to argue that I think industry as a whole has caught up to the concept that, hey, we can actually test our infrastructure code before we deploy that infrastructure code into the production infrastructure, which is a great achievement for us as an industry. I feel like.

Or Ansible versus Puppet. Why Ansible?

I think it was just easier for me to learn really. I started learning Ansible when I was in my master's. I was spinning up Hadoop clusters because my master's was in data science and I was using somebody else's AWS account resources because I did not have money to have my own AWS sit from the professor's account. I did not want to keep my Hadoop clusters running 24 by 7 because I was using it maybe few hours every day. So I was like, why should I not automate this? And I came across Ansible as one of the tools that I could use. It was very easy to pick up. And maybe that's when.

That'S the first bug that bit me for Ansible and I kept going at it. I tried looking at Puppet and Chef. It just felt a little bit overwhelming to me at that, at that time. And then Ansible just became my second nature.

Excellent. All right, Doctor, before we go, is there one piece of actual advice you can give to someone to help them with their DevOps efforts and what's the best way to find or contact you?

So one piece of advice would be automate, automate and automate. Really, you really cannot automate yourself out of the job. You just get more productive if you're automating your job and you can actually make impressive things in your career and maybe get that promotion that you wanted or dream role you wanted just by automating and learning how to automate your work and making yourself free for higher things in your career in day to day job. That's how you get more visibility, more work, more meaningful work and maybe your dream job or promotion or I that you're looking for. So that's, that's my advice. You're not going to automate yourself out of the job. That's something that I'm really firm about. And then to find me, you can always contact me on LinkedIn. I think my LinkedIn is open to public and you can always just hit me up there. I'm mostly active and maybe Joe, if you want I could share my email address and people can contact me that way as well.

All right, before we wrap it up, remember, frustrated users quit apps. Don't rely on bad app store reviews. Use SmartBear's Insight Hub to catch, fix and prevent performance bottlenecks and crashes from affecting your users go to smartbear.com or use the link down below and try for free for 14 days. No credit card required. And for links of everything of value we covered in this DevOps Toolchain show, head on over to test guild.com p182 so that's it for this episode of the DevOps Toolchain Show. I'm Joe. My mission is to help you succeed creating end to end full stack DevOps toolchain awesomeness. As always, test everything and keep the good. Cheers. Hey, thank you for tuning in. It's incredible to connect with close to 400,000 followers across all our platforms and over 40,000 email subscribers who are at the forefront of automation, testing and DevOps. If you haven't yet, join our vibrant community@testguild.com where you become part of our elite circle driving innovation in software testing and automation. And if you're a tool provider or have a service looking to empower our guild with solutions that elevate skills and tackle real world challenges, we're excited to collaborate. Visit testguild.info to explore how we can create transformative experiences together. Let's push the boundaries of what we can achieve.

With lutes and lyres, the bards began their song A tune of knowledge, a melody of code through the air it spread like wildfire through the land Guiding tester showing us secrets to behold.