Building a Real World DevOps Pipeline with Javier Alejandro Re

Get ready to discover some of the most actionable DevOps techniques and tooling, including performance and reliability for some of the world's smartest engineers. Hey, I'm Joe Colantonio, host of the DevOps Toolchain podcast and my goal is to help you create DevOps toolchain awesomeness. In this video, you're going to learn how to build a real world DevOps pipeline step by step, learning how to seamlessly integrate quality in non functional requirements. From code quality checks using tools like sonarqube to advanced security scanning and automated functional testing. Join us as we have expert Javier, CEO of Crowder and founder of Lippia IO, who will explain everything you need to know to improve your software quality continuously. So you want to make sure to watch all the way to the end to see the recommended tools for each stage of your pipeline along with real world code examples. Before we get into it, quick question.

I'll show you.

Are you looking to enhance your automated testing DevOps efforts? If so, I've got something you might be interested in. I recently came across Slippia, which is a powerful automation testing platform that was designed to simplify your testing process and boost your productivity. What's great about Lippia is it has a real user friendly interface and a bunch of awesome features that really cater to both beginners and expert testers. And right now they're offering a free trial giving you a perfect opportunity to to explore its capabilities without any commitment. So whether you're working on web, mobile or API testing, or trying to create a quality DevOps pipeline like the one we're going to cover in today's episode, Lippia has tools that can make your life a lot easier. So if you're curious to see how Libya can fit into your testing workflow, head on over to Test Guild ME pipeline and sign up for the free trial. Now give it a spin and see how it can streamline your DevOps testing efforts. Check it out. Hey Javier, welcome back to the Kild.

Hi Joe, thank you. Thank you for welcoming me back here and this impressive podcast. I'm very, very proud to be here again.

Great to have you. I'm really excited about this episode because I get asked all the time, hey, can you give me a real world example of this or that? And especially for DevSecOps pipelines, a lot of times people don't know how to work so how to create one in the real world. So I'm really excited about this, so I'd say let's take it away.

Okay. Okay. Well, initially I will share A small presentation, just to have a quick overview of what we would see. Basically this is a pretty simple pipeline which is component different stages. But before and even in between these main stages of deployment, a solution like build the code package and deploy, we will include different stages that are important for the quality. As I mentioned, like including quality scan of the code, security scanning, functional testing and performance testing. So we will focus in these four stages which is the quality checks that we will include in pipeline. And the idea of this is that every time that someone makes a commit into specific piece of code, we will execute these check ins that assure the quality or at least improve the quality of the product that we are building. So for this example we will take into account a simple pipeline that is configurated for a set of APIs or microservices that supports in particular a product, a web product. And we will see the different, the different check ins. So the first checks that we will execute is a code quality check basically. So because the quality process is not just making testing or test the functionality of an application we tried, and it's a matter that we try to incorporate in our projects is include the quality in all the stages of the software construction. So the first step is to check the quality of the code. So that's why we include this step. So in these steps we will make in this example three simple steps. First of all, scan the quality of the code looking for two things. One is the vulnerabilities and code smells. Code smells are commonly bad practices in code. That makes the code less secure or less maintainable, or different attributes that we look in the code when we, when we write good code. And then we will check the unit test coverage of that code. So it means that every method, or at least a big percentage of a method, has their own unit test and then run those tests to check if they are passed or there are some bugs in the code. So to do that we use basically a set of tools. One of them is probably a well known tool which is sonarqube. Sonarqube is a scanner that acts in the code that we are analyzing and detect different things depending on how we configure. We can configure sonar to stop a deployment if a certain percentage of unit tests are not accomplished or if they have vulnerabilities in the code, or they have certain rules that are not accomplished in the scanning. And based on that you can stop a deploy or we can just launch an alarm or a report to make the responsible of the code to check that code. And correct after. So in this case I will show briefly an example of the outcome of that stage in a pipeline. So if we look at the pipeline, the pipeline will look like a chain of different steps that are executed, some of them in parallel, some of them one after the other in serial process. And in this case we will review the sonar scan. So the first step, as we can see here, is the code quality where we check linter, which is scanning for code smells, as I mentioned, then the unit test and then the execution of those unit tests. In the unit test we check two things. One is how much coverage we have of unit test and then the result of those unit tests executed properly. So this step, in this example that I showed show in the presentation is passed, so it's passed the steps successfully. But each of these steps generates an outcome. So in the case of the unit test execution, we can see a result of all the unit tests that this piece of code has builded. The coverage is not much at all at this stage, but this is for a sample. But we can see all the results of those unit tests executed. But also we can see how much coverage this piece of code has using Sonarqube. All these components are open source components that are I have configured in a local pipeline on prem, but you can use this product on the cloud using the versions that are available commercially. But in this case, as you can see here, we can see all the results of the scanning that was executed by sonarqu with the different components where we can see the test coverage, duplications, different things that are related to what I mentioned before, the maintainability, reliability and security of the code configured. In this example, obviously these are rules that has to be configured depending on the language that you use, but also depending on the level of risk and the level of configuration you have. Obviously if you put the quality gates too difficult to pass, probably your development team has problems to deploy things, so you need to configure that, customize it to your needs and to your language and the risk that you are open to manage, basically. So going further to the with the next steps we pass the code quality scan and then we move forward. After build and package, we move forward to the security portion. In this section we will pass different sections or different scans that we will execute in parallel. And with those results we inject those results in a report tool that we will see in a second. But basically this has different steps. The first step is to scan to make a static security scanning of the code. And this will analyze also other vulnerabilities that are not checked by Sonar. Sonar could do. There's some overlapping between these tools, but in this scenario I configure sonar for using to detect more language kind of bad practices and I delegate the security portion of that to GitLab. In this case, we are using GitLab on prem. So we have this feature configured in our pipeline to scan vulnerabilities of the code and bad practices. The next step that we will use is to scan for credentials. A common problem of the development process is sometimes we need to configure center credentials to access, I don't know, a token to access microservice that has security database credentials, credentials to access, I don't know, a bucket to store objects in aws, et cetera. All those credentials are managed sometimes in different portions of the code. And this tool has the ability to detect those credentials, spread it into the code. So you know, when the code goes bigger and you have many microservices or many pieces of code, you can accidentally leave credentials in configurations files or even into the proper code in hard coded credentials. So this tool detects that kind of credentials and makes a report of that. So you can then basically correct that, putting those credentials in a proper place like a vault or I don't know, encrypted files, et cetera. The solution that you want to check, but to avoid, basically leave the credentials into the code in clear text.

Okay, so for the folks that are just listening to the audio, the name of the tool is Git Leaks.

Git Leaks, yes. Yeah. All these tools are open source and obviously at the end of the presentation we can share the link to each of the tools to let the people experiment with it. So the next step that we have in the pipeline is the dependency check. Basically the dependency check look for vulnerabilities into the libraries that we use in the code, not just in our code, because remember that we scan the static code that we wrote using GitLab. But if we have a dependency that have a vulnerability, and that vulnerability is reported in a Central repository like WhatsApp or depending on the architecture, different repositories, we can detect with this scanning dependency check which libraries has vulnerabilities. So in the case that we detect vulnerabilities in those libraries, we can replace for a newer version of that library that has that vulnerability corrected, or eventually replace that library if that library is not yet supported by the owner of the tool. So it's expanding the scanning not just to the proper code that I wrote, but also to the libraries that I use which is very common in the open source space where I will use a lot of libraries to do different things. Next scanning that we include in this pipeline is clock. Clock basically to make a lot too sophisticated thing. But the only main thing that clock makes is to count the lines of code and the lines of code. So basically clock gives you an overview of how big is your software. So how big is your code. This is useful basically to prioritize. As we wrote many match on match code. We need to the software goes bigger and sometimes resolving vulnerabilities or attempt correct. These vulnerabilities detected by the scanners are very complicated in terms of effort. So basically with having the size of each component in lines of code, we can prioritize which pieces of code attend first and which of them do it later. So basically clock gives you a. A sense of the complexity of your code. The bigger is the piece of code, the bigger the complexity. And probably will have more priority to fix those issues eventually. When you need to decide which to solve first. As soon as we. This is important because as soon as we put these scanners into the pipeline, it start to generate a lot of. A lot of things. So sometimes it's difficult to reduce the number of vulnerabilities to a level that is acceptable for the business. So this is another tool that helps to prioritize the scope in terms of fixing those vulnerabilities. Then we have 3V. 3V is another open source tools tool that makes basically the ability to scan if we use containers into our solution. Basically this allows to scan a container after is build it. So the previous tools we are just analyzing code or stages prior to the building of an image that can be used later in a container. So this tool has the ability to scan the container after the building. So you know that the containers usually have operating system but also components that added to that operating system to let the code run in those containers. So this, this scanner basically scan the infrastructure components, not just the code, right? So all the operating system dependencies that we can put in an image to build a container later. So after to build a container, we scan the container to detect vulnerabilities in the code of the operating system and all the dependencies and finally all the scannings that I mentioned. In the security stage, Archons generates basically a JSON file or a CSV file with all the results of those scans. Remember that all the scans that I mentioned are open source. So we try to keep that model and to read those scans that usually JSON files are not easy to read easily. We put a tool called Defect Dosho. Defect Dosho is a basically a report tool that consolidates security scanning results. So if we look the pipeline that we had before, we can see in the security section that we have different scans that I mentioned. Dependency check, language check, secret leaks, trivial container. All these stages in the security stage are generating results and those results are consolidating in a specific step that we put into the pipeline core report that consolidate all the results in this tool that I mentioned called defectdojo. So if we look at defectdojo had many, many reports, I will show just a couple of them. But basically the main source of data that Defect Dojo led to analyze its vulnerabilities. But Defect Dojo lets you analyze the vulnerabilities with different aspects by vulnerability. Historically checking which vulnerabilities are more critical, which are high, which are medium, and based on the dimension before about the fixing of these vulnerabilities and using Cloak, which gives you the size of the software, you can prioritize which components of your software you have to fix first based on the complexity, based on how much usage that component has into the application and different aspects. So in this case.

We can see a dashboard with historical finding of vulnerabilities. Then we can see another report that classified how many vulnerabilities each component have. In this case, I will, I'm just looking at one microservice with all the vulnerabilities that has. And here we can see an example of the size of this component. In this case we have a pie chart with the lines of code that has this component, which is 23,000 lines of code in TypeScript and then different lower level in different languages or components. But this gives you the size. Also we can see the vulnerabilities or findings divided by component. In this example we can see each microservice with all the findings that are active. Active findings are vulnerabilities that are not solved. Just to clarify, for example, in this case we have the major level of vulnerabilities in the front end, which is a big piece of code, and then the rest of vulnerabilities spread out in the different microservices that component the application. And also we can see we can prioritize and see vulnerabilities by vulnerabilities, by priority and by age in the report. So the longer the vulnerability are in the report, the bigger the risk to have an issue with that vulnerability if we don't fix in time. In this also we can see the libraries that I mentioned with the dependency check, we can see each of the libraries that we use and see how many vulnerabilities have all these components and we can see in addition a dashboard with metrics with all the statistics about these vulnerabilities. So how many vulnerabilities of each level of critics we have in over the time, over the components and so on. So this is a big reporting tool, so has many, many reports. But it's interesting to manage all the security aspect of the scan. So going back to the presentation, the last, well, the last two steps that we include or we included into the pipeline, right now we made code quality and we made security checks. Right now we will see functional testing and performance testing, the last two things that we like to include into a pipeline. So for functional testing, in this particular example we are using Libya as a framework that we use frequently and is built based on open source components like Cucumber, Selenium, Appium, etc. And in this scenario we will scan, we will execute all the needed tests in this pipeline to do a deploy. So basically in this scenario we can select different stages. For example, if we are merging code into the development environment with a small commit, we can run just a smoke test classifying the set subset of the test that we have in the automation suite. And as soon as we need to make a deploy to, I don't know, QA environment or staging or production, we can run a full regression test. So with the same solution we can run different scope of test in an automated way. So in this example I can show you what are the results of those tests. Okay, so basically we can see in the same way that we see in Defect Dojo the results of the vulnerability, we can see the metrics of the functional automated test executed into this application and see different metrics like how many tests we have in the smoke suite, how many we have in the regression test split by functionality, split by automated coverage or not. So as soon as we automate the test, we can see how many we have automated versus how many we have not automated at this stage of the metric and see the evolution of the different status. Also see the results over the time. So how many tests we executed and how many passed and how many failed in the passing the different stages over the time. So we have a overview of the results but also a detailed view of the results into this section. So basically this is to cover the functional test and going back to the presentation, but also we include at the end of the pipeline a performance test in this case, the performance tests that we include into the pipeline are not stress tests, right? Where you have a controlled environment and you set the application to the limit of the infrastructure or the capacity. In this case we put a performance test which are in charge of testing if the performance of each microservice or each component of the application is generating the response time that we expect. So we can put limit of minimum performance response time and maximum performance time. And with that we check that the test that we executed are between those minimum and maximum value. So we use K6 to write this test. You can write those tests in different tools, but we choose K6 for this specific example and we can see the results in a dashboard, which is pretty simple. But Basically we choose K6 because it's useful to write a specific test in code and not using scripting, which is more powerful in terms of checking the performance. So if we can see the results, it's a pretty simple dashboard and it's the last step of the pipeline. Basically we can check all the requests that we made and how much of them are into the minimum and maximum that we defined for this specific microservices. And we have different additional information, like how many users we have. Obviously remember that in this performance test we don't try to put a bottleneck or to generate a bottleneck into the application like other kind of performance or stress test. In this case we just want to check that the new code is not generation a lack of performance of a reduction in the response time of the services.

How does it know that? Does it keep historical data to say, hey, last run it was like 10 seconds, now it's 20? Or do you have to just do it manually?

No, in this scenario you can see the pipelines. Obviously this tool is an open source report, which is pretty simple. K6 has many other tools in the cloud version that you can store historical data. But eventually using GitLab as we do in the pipeline, you can see the different execution of the pipeline and check how the application perform in each of those pipelines. But this report specifically doesn't have an historical data as we see in the other ones. But indeed all this execution can store it into GitLab in this particular case with all the results stored attached to it. So basically going back to the beginning, just to make a small recap, basically we see four different stages into a pipeline, a real pipeline that check the quality of the software solution, starting from the code quality, just to check good practices in terms of maintainability, unit tests, et cetera. Also checking Security vulnerabilities into the code and dependencies and images that we built for deployment in Kubernetes or Docker and also executing two main steps, one for functional test in an automated way and also performances all in a single bypass. So this is obviously a thing that you can do incrementally so starting for code quality eventually and then add some security scan or all the scans that we have and then increase the complexity of the pipeline as soon as you feel comfortable to maintain the outcomes of these scanning and quality checks.

So Javier, quick question. If someone's just getting started and they're seeing this, is there one piece of actual advice you can give them on where to start rather than being overwhelmed? Oh, I need performance and security and all this right now.

Yeah. Well, for my perspective, a good thing to start is to start making some code quality checks at the beginning because as soon as you start doing that, you improve the code of you improve the way that you build in code, you write code. So first start a code coverage scanning and code smell depending on the language that you use and also do a basic static analysis. So the static analysis gives you a quick overview of how good or how bad you are writing your code basically with those two things I think is the first step. And then if you are doing automation in your functional test, as soon as you start building a single automated test case included into the pipeline. Because a common problem is happens and we see in many projects is that many teams build a lot of effort to build automation but they don't give feedback to the developers of those tests. So as soon as you put those tests in the pipeline that start to give feedback to the developers. So the main purpose of all of this is to give feedback to the development team on how the things can be broken in production later on. So as soon as you, you include this information and this feedback to the developers, you. So these three main steps I think is so scanning the code for code smells, vulnerabilities and functional test after, after you can add more things like performance check ins and more complete security scanning. But I think with these three steps is the main, the basics I think that you can use.

Are there any plans to share the code or repository access?

Yeah, yeah, we can share all the tools that we use and each of the tools has different repositories of sample code. This is obviously part of our an effort that takes some time to build it. So don't overwhelm to put all of these. Because on the other hand the math, the many checks that you put in your pipelines is more difficult to maintain later on. And what usually happen is that the developers then ignore all the reports because there's so many things to take care of. So.

All right, so to wrap it up then, if people want to learn more about Lippia and how from this demonstration how it seemed to really make the functional testing a lot easier and the management, where can they learn more?

Well, basically there's you can check in leap year IO, which is the website but also connected to Lipia IO is.

GitHub.com okay, so it's GitHub.com lipia IO and we'll have a link for that in the show notes down below and for links of everything of value we covered in this DevOps toolchain show. Head on over to Test Guild.

So that's it for this episode of the DevOps Toolchain Show. I'm Joe. My mission is to help you succeed creating end to end full stack DevOps toolchain awesomeness. As always, test everything and keep the good. Cheers. Hey, thanks again for listening. If you're not already part of our awesome community of 27,000 of the smartest testers, DevOps and automation professionals in the world, we'd love to have you join the fam@testguild.com and if you're in the DevOps automation software testing space, or you're a test tool provider and want to offer real world value that can improve the skills or solve a problem for the Guild community, I'd love to hear from you. Head on over to test skill.info and let's make it happen.