Pipeline as Code Mohamed Labouardy

Welcome to the Test Guild Automation Podcast where we all get together to learn more about automation and software testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Automation Podcast. Today we talk with Mohamed all about Pipeline as Code, his new book by Manning. Really excited to talk to him about this. Mohamed is the CTO and co founder of Crew Work and is a DevSecOps evangelist. He is the founder of Commiser IO and and is an author, open source contributor and regular conference speaker. We actually had him on the Automation Guild this year and everyone loved him. So I thought it'd be awesome to also have an episode on the podcast as well. So you don't want to miss this episode. If you wanna know anything about Pipeline as Code or Jenkins or anything like that, check it out.

The Test Guild Automation Podcast is sponsored by the fantastic folks at Source Labs. Their cloud based test platform helps ensure your fav mobile apps and websites work flawlessly on every browser, operating system and device. Get a free trial. Visit testguild.com sourcelabs and click on the exclusive sponsor section to try it for free for 14 days. Check it out.

Hey Mohammad, welcome to the Guild.

Hey Joe, thanks for having me.

Awesome to have you. So Maha, before we get into it, is there anything I missed in your bio that you want the Guild to know more about?

No, I think you have covered everything.

Sweet. So I really like this book. You actually start off the book. You know, if you've never practiced continuous integration or continuous delivery principles as well as DevOps practitioners, you can learn from this book. So it really starts almost from ground zero and ramps people up all the way to having a full blown pipeline with all these cloud integration providers. So I really love that approach. So I thought we'd start the podcast off with this approach and talk about, you know, what is cicd? I say this term all the time on the podcast. Not sure I've ever really defined it or what actually it is.

Yeah. So basically, CI cd. Let's start with the CI, which basically stands for continuous integration. So basically is the idea of having shared centralized code repository in which all the changes and features of your applications will go through a pipeline before integrating them into this repository. So a classic CI pipeline will be triggered each time you push something to this repository. So for instance, if you push a new commit to this repository which will run the unit test, the quality test, the security, etc. And it will build the artifact or compile the source code of your application and then maybe it will push this result to our remote repository. So for instance, if you are using Docker, it might build in the Docker image and store it on a private registry. If you are working with Java for example, it will might build in JAR and store it on Maven repository, etc. So this is a simple example of continuous activation. As for a continuous deployment, it's basically an extension of continuous integration. So all the changes that will go to continuous integration will be deployed automatically to a staging or a pre production environment. And finally, for continuous delivery, it's basically similar to continuous deployment. However, the deployment is not automatic. It requires a human intervention or a business validation. So for instance, you need to wait for acceptances or business validation from the business departments, et cetera, before deploying the release to production. So this is the definition of CICD pipelines.

Nice. So long. Whenever you talk about CICD pipelines, what is a pipeline then? I guess you mentioned different phases. Is it phases that happens during your software development lifecycle as you're merging code into main or master or whatever.

So basically a pipeline is a set of stages. These stages can be executed sequentially on a parallel. So basically a stage is a set of commands or steps that you will run locally on your machine, just with the CI CD and with the tools that emerge on the market today. You will run these steps on a remote server. So for instance, you can use something like Jenkins or a SaaS platform like Circus to run your steps. So basically as a stage, this set of steps that you run locally, for instance, to compile the source code or to run your. Net test, et cetera. So you just put the steps on something, what we call Pipeline or a Jenkins file, to automate the deployment and integration of applications. So it can be integrated every time you push something to the remote repository in which your source code is stored.

So why would teams want to do this? I just assume everyone is doing this. But for teams that maybe aren't, like, why would someone want to actually use CICD or create a pipeline or any of this type of infrastructure?

Okay, so there is a lot of reason why people need to embrace the CICD approach. One example of that is it will help you detect anomalies at an earlier stage, which will help you reduce the risk of releasing something which will contain a lot of bugs, might impact your business. And also it will also minimize the technical depth because you will have functional tests, unit tests, et cetera. So it will help you improve your code quality and reliability. And also it will gives you a better visibility into your project status and health. So we'll have a better idea about about the product roadmap and the features that you are building. And also it will help you having some kind of pre production environment in which you can do all your tests and also reproduce bugs and do some A B testing before releasing new feature. So to the user. So. So this is examples of why people need embrace CICD approaches.

Great. And you do cover all this in your very first chapter. So there's a few more things I want to pull out of chapter one before we go to the next chapter. You also talk about cloud native apps. So I'm curious to know why you decided to talk about cloud native apps. Are they different from normal apps? Is this a trend you see more people going towards now as well that they need to be ready for?

Yeah, I think the cloud native applications is something which is started to be used intensively in the past two years, especially with the adoption of cloud. So you have a lot of people migrating from on premise to cloud, especially with tools like Kubernetes, service, etc. So and with this new architectures and paradigm of building applications, it changes also the way people do inside CD because when you are playing with the cloud, so you have a lot of services that you need to integrate with. So people also need to adapt their process on their pipelines to be able to build their cloud native applications. So this is the idea why I'm covering a lot of chapters regarding cloud native applications. Because there is a lot of people today are using kubernetes on service applications like lambda functions, cloud functions, etc. So it's something which is very trendy today and there is not a lot of resources that we can find on Internet about these topics. So that's why I have dedicated multiple chapters on how you can build a CICD pipeline from scratch for this kind of applications.

Great. And along those lines you also talk about serverless. I assume that's a new kind of trend as well. What kind of things do you have to handle differently with CICD when you're dealing with serverless?

I think the first challenge is the complexity of pipelines. Because for instance, when you are dealing with microservices, most of the time you are dealing with one pipeline. If you have one pipeline per service, however, we are dealing with service architecture. You deal what we call nano services. So you split your microservices to your microservice to multiple functions. So you now instead of integrating one package, you need to integrate multiple functions. So there is a lot of synchronization to be done and a lot of code that will be shared on these different pipelines. So you need to do a lot of factoring. And also if you have an API that has multiple functions behind it, so you will end up with a pipeline with a lot of stages, with a lot of branching. So it will be difficult to maintain, also difficult to iterate on it on the long run. So this is some examples of challenges that people might face when they start building a CICD pipeline for service applications.

So at what point in the software development life cycle should people start thinking about cicd? Should be from the beginning. So I'm just thinking of a company I worked for on a greenfield application, but they didn't make the application deployable automatically, so you had to manually do it. So it didn't make any sense from CI CD until we were able to actually automate the build. So what teams need to think about or what point should they be thinking about CI CD as they're developing code?

Yeah, so I think for me it's an advice that I give everyone when they ask me about CI CD is just they need to start small and keep it simple in the beginning. I think continuous delivery and continuous deployment, it requires a lot of attention and a lot of skills to achieve that. I think the first thing people need to start with is having a remote repository. So if they are not using yet something like GitHub or Sven or any other version control system. So that might be a good start with to structure the source code of the application, maybe start using multiple branches, something like Git flow to have a master branch feature branch and start doing pull requests and have start having a real review process that might be a good start once they have that. I think one of the things that people can start with is just to write down all the commands that they do when when they build a new features on their applications. So for instance, if you are working on on a JavaScript application project, so I think most of the commands that you might be typing when you will be working on new feature before you push it to your emotional to run your unit test, etc. So start documenting the process. So once you start documenting your process, you can from this documentation generate some kind of pipeline. So you will start seeing some kind of commands that are frequently executed. So these commands, maybe we can automate them by placing them on a central server, something like Jenkins or Circus or whatever size solution can be. So once you have done that, you need to pick your CI server. There is a Lot of choices out there on the market on this book covering Jenkins, but Jenkins is just an example of CI service that exists today. So all the concepts that I have covered in this book can be applied on other CI servers. So once you pick your size server, you just need to define these commands on a template file and this template file, you will put it on your moat repository, you might create some kind of webhook. So every time you push something to a specific branch or if you for instance rise up or request to trigger your site server to execute two commands. So, and the commands can be quite simple. So it can be something like taking out the source code for new changes, running for instance unit test or quality test or any type of test that they are having. So I think the purpose in the first place is not having something which with more than 80% of coverage because it's, it requires a lot of work, a lot of development to maintain that. And this is one of the reasons why people drop out from CI cd because it's kind of cute in the beginning and a lot of people love it because it automates the deployment, etc. But once you get into it, there's a lot of complexities and you don't want it to be some kind of roadblock on your development cycle. It should be something that you will enjoy creating off every week and something that will bring a lot of benefits. So that's why I recommend people to keep it small in the beginning. Just maybe start with some code lenses on the beginning your pipeline to have some kind of coverage reports, something like sonarqube, to have some statistics, etc. So you can have some kind of key points that you might be working on on the long run to improve your quality code, etc. And maybe an additional step can be compiling the source code or building a package. So once you have write that, you will have what we call a continuous integration. So and then from there you can iterate on and add small steps and improvements each week or each sprint until you have something which is working from, from scratch. And from there maybe you will start moving to continuous deployments and start having some kind of staging environments in which you can deploy your changes. So to sum up, I think what people need to remind is CICD is very powerful for people that don't have CICD background in the past. I think the advice I can give to them is to keep it small in the beginning. There's a lot of resources today on the market and also on this book, the first chapters I have tried to keep it simple for people that they don't have background on DevOps or CI CD. So I think that might be also helpful for people to read out this three or four chapters of the book and that give them solid foundation about CI CD before the they jumped onto continuous deployments and delivery.

Great. So you talked about roadblocks. I like how you said it should be helping the team, not making them frustrated. So how do you see most companies implementing it then? Is there someone in charge of CI cd? Is it like once again the developer is responsible for everything? Does it depend?

It depends, but I think most of the time you have some kind of DevOps team which are handling the CI CD. On my current company and also in my previous position, what I have tried to do is to maintain some kind of shared response responsibility between developers and DevOps because that's one of the purposes of DevOps for me is set up practices and also it's a mindset shift of the entire organization. So if you want to have some kind of successful DevOps culture, I think everyone should be involved on this journey. And I think one of the things that you can do is to set up some kind of shared responsibility when it comes to setting up CICD pipelines, especially today with tools like Docker infrastructure as code, et cetera. So I think everyone can be able to get started with anything related to setups and anything related to the cloud or infrastructure, et cetera. So I don't think developers now have any reason to say that it's not my responsibility to build the application or to deploy it, et cetera. So I think now just a simple Docker file, you can build your image, you can test it locally or on a cluster like Kubernetes or Swarm. So but to do so you need some kind of knowledge sharing because at the beginning developers don't have a lot of knowledge about what is CICD, how DevOps works and all the size servers that exist today. And what I have tried to do on my previous project is set up some kind of workshop on which I try to explain particular topic like for instance pipeliners code or how you can write your Docker file or Jenkins file. So once I started doing that, people started to get interested to learn more. And from that I started to create some kind of templates. And these templates developers started to use every time there is a new project or a new feature that you are working on and they need just to clone these templates and they can create their job on Jenkins or whatever the site server without the need, without asking DevOps team for their help, to create a CI CD pipeline. It was very nice in the beginning and also it has a lot of benefits because in the long run developers started challenging the DevOps team on the solution. It wasn't just people that are asking for their help, but they were also people that gave us good ideas about how we can improve the CI CD pipeline. They have suggested new tools that we can integrate to improve the quality or to make the process faster, etc. And I think by doing that and by having this kind of workshop within the organization can help everyone and the team, either on the development side or the upside, to work together to have this kind of shared responsibility when it comes to maintaining CICD pipelines in general.

I love the idea of knowledge sharing and I guess this book almost you can gift your developers this because as I mentioned, it starts from beginning all the way to more advanced topics. And so far we touched on some of the more beginning chapters. What is CI CD pipeline Line as code with Jenkins defining a Jenkins architecture? I guess you go a little more into that, but there's some few things that I want to go over that I haven't heard of before, but maybe just because I'm also a newbie kind of. And that is Packer. You have a whole chapter on Packer. I'm just curious to know what is Packer and how does it help people?

Yeah, so Packer is basically is the concept of imitable infrastructure. So it's similar to when you are dealing with Docker images. So let's say you have an application written on JavaScript, for instance, so you will build a Docker image for this application. If, for instance, you need to update the dependencies of this application, what you will do is you will build another Docker image. So this is what we call imitable infrastructure. So you will always start from scratch. So instead of doing the changes on the existing image, you will build a different image or a new image with all the dependencies. So this is what we call imitable infrastructure. And basically it's the same concept with Baker, just Packer it's for machine images and Docker it's for containers. So Baker, it allows you to build your machine images with all the dependencies, with all the configuration, etc. So this is what we call Baker Machine Image. So you will start with a gold image and then you will provision this image to install all the dependencies, all the tools, all the configuration that you need to have something that will be ready to Use out of the box. On this concept I have used through the book to build a Jenkins cluster. So basically, instead of deploying a new server and then installing Jenkins and all the plugins and all the stuff that I need, what I do is I create an image. And on this image I have Jenkins installed, I have all the plugins that I need for Jenkins, all the credential I need, et cetera. And that way we'll have machine image that I can use anytime I want without going through all the manual stuff while configuring Jenkins. So I just need to launch a server with this image and I will end up with the Jenkins dashboard with everything installed and configured. And another advantage of using this kind of tools like Baker is with the same template file that you have used to create your image. For instance, for aws, you can use it to build your image for GCP or Microsoft Azure or for VMware, et cetera. So you will start with the with the same template file and you will end up with a machine image for different cloud providers. It's a very powerful concept and it's a concept that we see today mainly on Docker.

So just to me, it almost sounds like Maven with a Palm file, where you have a Palm file has all the dependencies and automatically catch up and running, but it's for images, for machines. Nice.

Yeah.

Cool. So another one I've been hearing a lot about is Terraform. I'm in a mastermind with someone that was having a hard time with their customers getting them to build the application, and he was all, oh, I just used Terraform. And he spent like two weeks creating some crazy Terraform script that does the whole build form. So I guess my question is, what is Terraform and how does it help people with cicd?

Yeah. So basically, Terraform is a tool that allows you to describe your infrastructure on template files. So it's an application of what we call infrastructure as code approach. So basically, instead of deploying your servers manually by going to the console or by using the CLI or the API, I think in the beginning it will be easier because you will start with a few servers. But imagine you have a complex architecture which you will have a VPC with subnets, with the servers, with load balancers, with databases, etc. And if you need to deploy this infrastructure manually, it will take a lot of time and you will have a lot of errors because it's really tough to deploy a complex infrastructure without missing something. And also it's hard also to track all the changes that you have done on your infrastructure without documenting the changes that you have done. And that's where the infrastructure as code approach came in. So basically all the stuff that you do manually, you just need to write it down on a template files and these templates files would be read by a tool like Terraform or others. And this tool, Terraform for instance, will just parse this template files and then it will just convert these resources that you have defined on your template files to API calls to the cloud provider. So that way you will have all the infrastructure versioned on template files and you can treat them the same way you treat your source code. So you can for instance, push them to your GitHub repository, you can set up a review process and you can track also the changes, so you can create pull requests, do some merges, have something with an advanced review process, et cetera. So, and you can also do some testing on your infrastructure. So basically this is the the idea behind infrastructure as code and Terraform is just a tool that allows you to achieve that. On this book I have used Terraform to deploy the Jenkins cluster. So instead of deploying the services, the, the server and the load balancer or the the VPCR structure manually, I have automated all this stuff on templates files. That way users can just need to clone their repository, run a command, and they will have the same infrastructure on the same environments that I have used on this book locally or on their cloud provider.

Nice. Now this isn't in the book, but you did speak at this Shares Automation guild and one of the questions that came up was what part of the pipeline is a tester's responsibility? I know you talked about shared responsibility earlier, but are there any stages in the pipeline you think are really tester centric?

Maybe when you talk about tester responsibility you are talking about developers or you are talking about the ops.

It's usually like a software tester, he's kind of like in a Sprint team with developers. He's kind of like straddling both worlds almost.

I think it depends on the structure of your pipeline. So if you have some kind of acceptance test within the pipelines, I think that might be the responsibility of the quality assurance team or the dessert to verify the output and also notify the developers or the product team about the regression or about the bugs that has been spotted. So for me it depends on the structure of the pipeline, on the stages that are defined on the pipeline.

So another question that came up and you actually kind of alluded to it, is not people should start Small because you don't want to get developers frustrated with actual CICD and have them eventually abandon it. And one of the people that mentioned at the conference was they were doing something with security scans and that it was giving a lot of warnings, but everyone was ignoring them. So it's just like almost like automation tests. When they fail, people start ignoring them because they're not. They take up a lot of time and effort. So I guess, do you have any suggestions in your book or in your real world where you can get people to really pay more attention to the signals or the alerts that are coming out of your CICD system?

Yeah. So I think one of the things that you can do is have some kind of notifications. So it depends on the collaborative platform that you are using. I think most of people are using Slack nowadays, so you can implement some kind of Slack notification to notify you about the important stages of the pipeline. And this is one of the topics also that I have covered on the book. And regarding the security, I have covered two ways to implement security or to inject security into your CI CD pipeline. The first one is running security checks on your dependencies. So when you are working on an application you will have, you will be using a lot of external dependencies. So one way to check for the security vulnerabilities is to use some kind of open source tools or Premium or SaaS platform to check your dependencies. And you can configure the pipeline to be filled or to be successful based on the warnings on the number of errors or the severity of the different IBT that has been found. And the second way to inject security is to, for instance, if you are using Docker is to scan your Docker images. So you can scan your document images to see if they contains any kind of vulnerability that has been raised by the public. And at the same way you can also configure your pipeline to be to throw an error if there is some kind of critical security vulnerability that has been found. So to come back to your question, I think the best way to make people involved on CICD and all the stuff that are going through the pipeline is to have some kind of notifications. And I think people also need to pay attention because when you start implementing notification, maybe you will end up spamming people. So you need to find some kind of the sweet spot between bringing the value, bringing the benefits of notifications and also notifying the right people. So that way you can have every time or all the time developers on the products and the Quality and security engineering is involved with your CI CD pipeline. Otherwise they will just avoid it and you will end up with something which is not useful or doesn't add or bring a lot of value to the company.

Awesome. So I thought we should wrap it up. I forgot the answer to this. I asked you this at the conference was is your book ready? Because I have the MEEP version. I guess I have six chapters. Are there going to be other chapters or this the final?

Now there is another eight chapters. Okay, already finished it. They are under review. So I think we will have three more chapters early next week.

Nice. So what are the rest of the chapters maybe that I may be missing that you think people might be interested in knowing more about?

Yeah. So the first part of the book was just the foundation about CICD on all the practices. Also how you can set up your Jenkins cluster because we'll be using it intensively the next chapters and the second and third part of the book is hands on experience. So basically we'll be building CICD pipeline for different cloud native applications. So we'll cover how you can build CI CD pipeline for a microservices application running on Swarm on Kubernetes, how you can build a CICD for a serverless application running aws, how you can run automated tests. We'll be also covering Jenkins X. We'll be also covering some advanced topics like monitoring, logging, how you can do the migration of Jenkins and how you can also set up authorization and arboxes. So yeah, so basically it will take people from the basics until the to the advanced topics about Jenkins and as I said earlier, the book I'm covering Jenkins but the same concepts can be applied on other site servers or other technologies.

Absolutely. I'm calling this a must read. So everyone listening to this definitely check out the show notes to get the link. I actually have a special discount code for people to be able to get get any book on Manning and especially you want to get your hands on this book. I think it's essential and it's a great primer and also get you beginning to more advanced. So I love it. So Mohammed, before we go, is it one piece of actual advice you would give to someone to help them with their pipeline automation efforts and what's the best way to find or contact you?

So I think the best way to start with cicd, I think it's just to grab your copy from the book, clone the repository on GitHub you will find all the resources, all the source code that I'm using on the on the book so you can try everything either locally or on the cloud and just follow with the book. And I think by the end of the book you will be mastery the CICD practices and you'll be ready to build your first CICD pipeline from scratch.

Awesome. And Mohammed, the best way to find or contact you?

I think the best way would be Twitter so you can drop me a message anytime. I have also a blog in which I post regularly new posts about DevOps or DevSecOps in general. Either drop me a message on my blog or on Twitter.

Thanks again for your automation awesomeness. The links to everything of value we covered in this episode. Head on over to testguild.com a 3:45 and while you're there, make sure to click on the Try for Free Today link under the Exclusive Sponsor section to learn all about Sauce Lab's awesome products and services. And if the show has helped you in any way, why not rate it and review it in itunes? Reviews really help in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe. My mission is to help you succeed with creating end to end full Stack Automation Awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Automation Podcast. Head on over to testguild.com for full show notes, amazing blog articles and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.