How to Build DevSecOps Pipeline as Code with Hasan Yasar

Get ready to discover some of the most actionable DevOps techniques and tooling, including performance and reliability for some of the world's smartest engineers. Hey, I'm Joe Colantonio, host of the DevOps Toolchain podcast, and my goal is to help you create DevOps toolchain awesomeness. Hey, it's Joe, and welcome to another episode of the Test Skill DevOps Toolchain podcast. So, last week I shared a session from a previous automation guild, but this week I'd like to share a session from our previous secure guild. So here's a presentation by Yasan Yassar on how to build a DevSecOps pipeline as code. So maybe this is you. You've heard a bunch of hype and read dozens of blog posts and watch a bunch of different training and videos on DevSecOps. But finally, you and your organization has decided to make the cultural shift to actually take advantage of automation and the benefits of DevOps with security. However, making the shift as an engineering team can often be cumbersome because many tech professionals still need to familiarize themselves with all the technologies required to implement a complete DevOps pipeline, including security automation. So in this talk, Hasan we'll introduce Microcosm, a miniature secure DevOps pipeline he developed at the SEI Institute. Available through infrastructure as code and this microcosm represents a miniature version of a secure DevOps pipeline compared to what you might find in a large enterprise environment. So listen up as Hasan goes over crucial principles of the DevSecOps pipeline and shares his lessons learned with examples with the security community. You don't want to miss this episode. Check it out.

Hey, if your app is slow, it could be worse than an error. It could be frustrating, and in my experience, it's frustrated. Users don't last long, but since slow performance is a sudden, it's hard for standard error monitoring tools to catch. That's why bugsnag, one of the best production visibility solutions in the industry, has a way to automatically watch for these issues. Real user monitoring. It detects and reports real user performance data in real time so you can quickly identify lags, plus get the context of where the lags are and and how to fix them. Don't rely on frustrated user feedback. Find out for yourself. Go to Bugsnag.com and click on the free trial button. No credit card required. Support the show and check them out.

Hello everyone.

I hope you're enjoying the conference so far.

This is Sasan Yossar.

I'm a technical director and faculty member at Software Engine Institute Cardinal University I'm going to Talk about the five common challenges to implement DevSecOps successfully in your environment.

So it's me and I have been.

Doing for software development experience for years, about 25 years and have been enjoying many steps in the software life cycle, different roles and practicing agile, practicing DevSecOps for years. And I also fed the DevSecOps speed in one of the conferences I attended. For me DevSecOps really is driving a car with more than 100 miles per hour and there's no just basically driving open car or no helmet on the motorcycles. So comparing that with the typical software world software development environment now we are going really really fast, really fast in terms of deployment, in terms of delivering into the production environment or a delivery environment. Comparing the couple years ago, even the comparing like When I started 25 years ago I started for engineering we were deploying the software sometimes in a year, sometimes two years. Now we are talking about delivering the software in milliseconds sometime. So comparing the speed relatively that makes us look like that. But biggest challenge in the security perspective we don't want to open our mouth because something may happen. The same is true. We don't want to really drop any guardrails. We don't want to delay the delivery by putting some challenging or some difficulties on the security perspective. So we have to go fast. But when we go fast we have to deliver secure and reliable applications.

It is difficult. So I'm going to start to talk about the overview of DevSecOps first what it is and just have a common vocabularies understanding DevOps and DevSecOps then going to the challenges and actions what needs to be done for each of the steps and finishing up the sustainment problems and what's next. Let's Talk about the DevSecOps what is DevSecOps? So DevOps is the set of principles and practices that enable the better communications and between the dev and opsims but actually all stakeholders the purpose of really getting the business goals apply into the fast cycle using the principles which is.

Continuous integration, continuous delivery, continuous deployment perspective and with the collaboration mindset and deliver the business needs. So that's kind of definition from IEEE, that's official definition from IEEE 26 to 75. Again everybody has their own definition of DevOps or DevSecOps and that's the definition from the formal definition from IEEE. So DevSecOps is driving a security into the DevOps practices. When we get a security, we would like to add a security into the Every phase of life cycle. Every phase of life cycle such as the requirements and design, coding, testing, et cetera. But in reality DevSecOps is DevOps done right? If we are doing a DevOps properly, we should have by default secured edit and based on the survey that we have done this year DevSecOps community survey we find out that mature DevOps organization are able to deploy secure replication much better than the non DevOps organizations. So they are able to integrate any type of secured automations and activities into their DevOps process. So easily we can say that DevSecOps is DevOps done right. So there are a couple principles of DevOps with such as future deployment and has to be iterative and incremental development with the lifecycle. So really talking about the lifecycle from the phases of the requirements and design and delivery and testing it is requiring to have iterative incremental development has to be a small batch size, not a large batch size. We should do the automation every phase of life cycle. That's the common principles. Of course we have to collect the feedbacks and write metrics and measurement based on what we are doing it because data is generated from every phase of life cycle, every components of life cycle. And of course that type of feedback mechanism automation lead us to have a fully transparent and traceable across the lifecycle. So there are key benefits of DevSecOps and I kind of highlighted important ones such as repeatable steps so we can repeat the steps over and over and again. We don't have a manual steps anymore. We have a continuous availability for any needs from user perspective or a tester or a QA or a developer or operation perspective. So we have an availability in every phase of lifecycle for things that we are building, the software we are building. And most important One actually for DevSecOps perspective our system, our platform will be a responsive to business needs including right process that is also including for incident response as well.

Because we have a fully automated infrastructure, we have all dependencies has been keep track in an environment we can respond any outage on time and also effective phase. So we don't have a deal the amount of delayed responses any vulnerability that been discovered in the production environment.

So overall we can say that easily security should be implemented every phase of lifecycle from the beginning, which is inception phases that how we get the feature request that should go through that environment. I know it is very easy to say that. However there are a lot of challenges to achieve that complexity of getting a security into the lifecycle. I know Everybody wants to say that everybody's talking about It's a common sense. Yes, we have to get a secret every life cycle. We are trying but there are a lot of challenges. Let's go through the challenges overall based on our experience and I gather the five challenges. We have many but we can categorize the five challenges of implementing DevSecOps such as the software assurances, organizational barriers and quality perspective and security skills and the security guidance. I'm going to cover all of them in details. Let's start from the first one Secret assurances perspectives what does assurances mean? So we can categorize intercategorization of the assurances such as industry lacks assurances model like industry they don't. As an industry we don't have a standard that we can emerge and we can use. So we are in an organization, maybe healthcare, maybe a finance, maybe some manufacturing and people or organization are defining what they would like to do. So to achieve that lack of assurances model and I'm advising to all of you please attend any industry conferences such as this conferences you're attending and be part of the informal working groups. Talk about your industry peers, understand what type of problem they have, how what is their solution and look at their lesson learned and try to apply that knowledge into your domain that you are working on it. Another one is the business business folks or the business analyst maybe user may not know about what the threats we are dealing. We don't know what type of problems we are dealing. So really educate our business folks and understand and explain what our business needs to our stakeholders such as our analyst and our organization team leads to literally get the business tie into the security needs and assurances, identify the threats, identify our business drivers and have an external audit. Understand our business context and talk about any problem that business risk we need to deal about. Look at the technical risks. Define some of the risk mitigation strategies based on the business needs. So basically tying the business into the an assurances perspective into the software rig so we can really balance the business needs and security needs together when we go to the lower level which is project specific. Now we have the project we are working our business in our organizations and map does the project assurances expectation to the work that we are trying to achieve really tied to any project related data that represented the security perspective. Review the code changes for a security and document approval prior to the release and definitely dedicated testing resources in a project perspective. So basically plan all the security requirements put into your project so we can really keep track the project Requirements or the security requirements at the beginning. Because allocate that resources. Because one of the findings I find that many types of surveys developer didn't have enough time to address the security. Because we don't plan the time at the beginning. That's the place we can think about the time when we do the project planning at the beginning. Basically adding the requirements of the security thinking and also resources at the beginning of the project. The second one is organizational barriers organization barrier. Again we can look at the three tiers and first one is the really really, really poor stakeholder collaborations. So what that means like look at your organizations. Do you really collaborate with rest of the team members like a risk management team, security team, dev team, operational team and how the business teams are aligning with your security goals. How your security teams are aligning with the risk management team. That is really a problem for talking each other. When we look at the collaboration perspective, it's not just sharing the data through the email. I'm talking about looking for more active collaborations, looking for more common ways to talk our problems, learn from our lessons. Basically it's organizational cultural changes that everybody should be aware of in a type of security requirements. And plus the sharing the resources amongst each team members. There's multiple ways to sharing resources. Sharing resources can be done and shared environments or using a chat environments. Or you can use any type of instant communication, maybe set up a weekly call. There is many ways to improve the collaboration with the stakeholders. We can really get everybody putting a secure in their pipelines by experiencing some sharing some lesson learned on the Friday afternoon. As an example, to achieve the good collaborations, we need to have an integrated pipeline. Like when you see the pipeline, what the pipeline means your platform that will help you build the software. The platform should have all the components in it such as any type of software delivery components like issue tracking, system monitoring systems. But that has to be integrated. Why is important? Because we don't want to run any siloed static analysis tools or we don't want to run any specific tools are siloed not connecting other important because the collaboration it is not just the human we are talking about as a team members we are looking for. Tools should be collaborate each other. Tools should generate the data for us. Tools should able to talk each other in other components to sharing artifacts. And then when we talk about artifacts like the risk analysis abuse cases or the outputs or reports and test cases has to be shared then we really think about a design of security in every level of cycle, which we said at the beginning. But how can we achieve? Let's look at the clue on an integrated pipeline. What the pipeline means. Put the source control in a centralized place and have issue tracking systems are aligned together with the communication system, bill systems and monitoring systems, et cetera. You can build up all the components together and connect the dots. And we don't want to be having it just a build server running by itself. We would like to connect the other dots like monitoring systems or integration systems, etc.

So another things I would like to really cover up and the next one is the security priority. How can we make a security priority at the beginning? Like I know we said we would like to collaborate, that we would like to put pipeline in place. But we should have a first prioritization our security that is really changing our organizational cultures. So we have to put a secret as a first class citizen our environment and platform that we can use. Like really embedding secret into the OPS problem into the ops world. But getting into the lifecycle perspective and learn from the ops side and try to transition the knowledge to the business side and let us make that the priority for every basin of life cycle. When we do that, we don't want to really overwhelm everybody. We have to keep it simple. And then simple means like really get a basic and get at the beginning try to really address the low hanging fruit first. Like less risky problem or less risky for organizational applications and then test organization is ready for a culture change. And we know that there is no single technology and tools will help you get to DevSecOps. There is kind of a at building a journey and step from the steps and go up and then add other components. So third one is a lack of quality and it's related to others as well. We always bring the security too late into the life cycle. So when we look at the quality, quality is not just the application related quality. It's like a platform related or we are talking about the application modifiable which is the 12th ill of this modifiable, repeatable, etc. But security should be part of our design or the quality attributes at the beginning. How can we do that? Really we should get the security compliance requirement early in the lifecycle. Putting a security design when we do the design analysis, we surely tie our objectives and providing assurances back to the business again. And then we should really test the compliance against the security policies and to find any gaps again. Why we cannot get the security into the quality perspective because we don't know what the risk that we are dealing. If you don't know the risk. We cannot put the security in place at the beginning. So really get the risk and quality blend together. Put your quality attributes related to risk and security so we can blend it at the beginning of life cycle. Another important element for the quality perspective is we don't really have a good confidence in the release lifecycle we are releasing an application. But how much we have a confidence on the release that we have a program or application running production environment. Again that's also requiring a lot of preparations like having risk based security testings we can do. We can increase the confidence in the release lifecycle by repeating the same process or the same testing or the expanding different type of scenarios run prior to the production environment like in the staging environment. How much we can do automated audit or testing pre production or staging or different test harness will help us to increase the confidence release. So when we talk about the release I'm going to open up a little bit topic here. There is a continuous deployment and continuous delivery that continuous deploy, more automated deliver. There's a manual steps in it. Do deliver it as much as you can with the confidence. Then you have a high degree of assurances on a continuous deployment because you have done many tests, many deployments prior to the production or you have a lot of manual interaction you have done on the deployment phases in a staging environment that will give you a high degree of assurances on a final release. Really having good steps and repeated many, many times. Then you have an assurances in the overall application you're building. Also we should think about the requirements with the traceability here as well. We have requirements which is at the beginning since we added quality attributes into our requirement, which is your tracking system as an example. How can we tie all the test cases against that requirements? That's going to show us some sort of release quality or confidence that we have a requirements. We pulled up security. Then we address the security requirements by doing certain tests. That's going to help us to increase our confidence. But there's another challenge about the complexity. So system is very complex. We are not just building maybe one web applications. If we have a system of systems component. If you're looking for a very big and generic and very big applications it is difficult to achieve the security in the complex world. To solve that problem. Really think about the very small modules or you may have a microservices and try to make the smaller app and smaller size. Then you can add a security into that each of components. Do you remember I said at the beginning about the Interactive Incremental. Requirement. That's one of the the solution. You can add a system complexity level and adding event modularities. You can add more the secret thinking in each of the modules. But we have to really think about the metrics perspective as well. We are adding a secret into the modules but how much we have an assurance is the module is going to work out what we're expecting. We should define your metrics in the complexity level such as time to deploy applications such as defect ratios. You can add all these components into the modularity level. Then let the security policies will drive by integrating security task into the stages of DevSecOps environment. The most important one actually a lack of security skills. So not everybody has enough skills in adding a security into the lifecycle. Business people they're not aware of it. And we should educate our business folks and business team members to understand what the risks are, how they're defining risk and they have to tie with the secret values. It's about the learning and talking with the business teams and dev team and other team members. Then we can help the business folks to understand security risks. Another one is the developers pure developer. Like they're thinking.

They have been doing the work for the work like writing a code or building a systems. But everybody's assuming that developer has a knowledge of security. It is wrong. Not everybody knows in a development lifecycle overall security perspective. I have been teaching at the CMU and my students graduate level. Not everybody has enough knowledge about the security because as a CS department or software engineering or a college we are teaching the students to build up a better systems in terms of algorithms and operating systems. We're not really teaching a lot security because security is a living things. And how can we solve that the security skills let the team engage. Let the team learn from each others. Let the tool help us to learn as a developer what the findings are. If you are doing SQL injection as an example, let the tool direct the developer what needs to be done for a specific solution of the SQL injection as example. And then build up a security culture like a big of a championship program or building up another culture elements. Let it learn from others as a dev team from the security teams and sharing sharing and understanding and also helping each other as a learning. So at the same time and open up the room for developers. Developers will learn by themselves for a better secret applications. The last one and auditors we have auditors are coming and looking for our systems. We have to train them. We have to train them in terms of what needs to be reviewed in the code itself as a developers, as the business owner, as an organization owner. We have to teach our auditors to look at certain security vulnerabilities such as sharing, OSP top 10 or the NIST 853 and explain what needs to be done with the security perspective. So we align with auditors with respect to our DevSecOps environment.

And fifth one is insufficient security guidance. Again it's related to the skills. But who is going to guide in organizations in terms of the security requirements and policies? The first one is the security resources. So we don't have enough resources in our organization. There is a problem if there is not enough resources and organization and knowledge. Developers will go and look at from open source environments. Sometimes open source environment, Internet lead them in the wrong directions. We have to build up organizational knowledge and putting right policies, putting a right knowledge in place and map the policies to domain specific procedures. So our developers will follow up the organizational guidance. Organization security policies. Ask yourself do you have an organizational security policies or not? Then you're going to find out how much is important. Build up overall umbrella so every developers and rest of the stake will follow up the original guidance that you develop from organization. And there are another issues for the security standards. There are so many security standards but when you try to get one of the security standards implement organization will be a big bang for you. Don't do that. Start from the basic want like OWASP top 10. Start from low hanging fruits as the policies as the standards and try to implement it once implemented the first one and you can go the next one. Try to scope it bigger covering up the standards every phases of your life cycle. Start from the CI first continuous integration. Maybe you can do a little bit testings and examples and adding more security requirements and testings to build up based on the standards or top 10 is the good one. And then don't jump to the open SEM right as an example or the ISO standard as an example. Start the basic. But your goal should be more into the high level standards. Last one, the most important one is monitoring. We like to monitor our DevSecOps environment based on the metrics that we align with the site at the beginning. So we would like to really connect the dots from ops to the dev team and then also align with our data that we are collecting. That data will help us to be more proactive, not that reactive mode. So usually we do the reactive and we did the operational mindset or addressing the response and vulnerabilities. If you do the good monitoring without things that are happening and the bad things happening. Production environment we should be really using as proactively as possible by understanding what needs to be done so we can fix the issues prior the bad things happen. That is really based on our monitoring capabilities. The monitoring is not just operational. We have to monitor all our secured activities from beginning to end in the life cycle.

As a parallel, another challenge is sustaining door devsecops environment. There are two ways of sustaining or a problem that you can see in our environment. One is effective usage of pipeline. The other ones maintain the pipeline that we have effective usage. I saw many organization they build up a great devsecops environment but it was very difficult for their dev team to use it. Developers or the stakeholders not able to access it, they don't know what to do. There is no right guidance. There is an onboarding guidance. If it is not used by effectively now we are blaming tools. That tool is not solving our solution. But we have to look at what we have to do as organizational to enable the team members using effective in depth sec ups and widener effective ways such as we can train the users and we will look to get a build up a playbook guidance for developers. We can build up a project startup guidance as an example and we can build up an overall architectural guidance like what is the checklist for going from one gate another one what is architectural expectation to build up Iterative incremental development for a given modules that we are working on and build up a patterns like if you're using microservices build up to what the microservices concept should be aligned with the pipeline perspective. Do we have any shared resources for any given modules? Use that modules in organization that you build it and don't build up again. It has been done already with other developer other team members for a maintaining perspective. That's actually a difficult one. You have to pay a lot of money for a licensing perspective or updating a new tools. And I bet you it is going to be very difficult to upgrade your Jenkins server such as CI server when you have 100 project running in an environment it will very challenging for you to upgrade it. If a running component build up a DevOps mindset to updating any securities any tools that you have in your environment. So you should able to update it in the tools without breaking the DevSecOps environment. Couple example, you can use the base image. You can have a replica version of your DevSecOps environment. Thinking about that staging a dev environment may be testing for your actual DevSecOps environment, maybe a lower number of users, but have a backup scenario so you can use that. You can build up a pipeline orchestration and on top of that you have to build up guardrails to protecting your DevSecOps and which is securing the pipeline. It is different than building a secure application versus securing the pipeline. There are two different components. We're going to make sure that we are protecting our DevSecOps environment as well. How we are protecting and how we're building secure applications. What is next? What we should do? To start, I advise the five steps. The first one is assess your gaps. Understand what type of gaps you have and based on the gaps build up the quick wins for you. The wins is basically maybe organizational culture you would like to do something, maybe a team culture. You will start to build up a collaboration piece. Start with the basic requirements for your DevSecOps environments and MVP and then build up the cultures empowering the champions such as security champions or dev champion for your cultures and measure the result. Where we are, how we started, where we are, what should be the next steps to work wise and always evaluate that and repeat it. So when you repeat the next time, look what you have done in two weeks, maybe a month. So define your sprint for building up DevSecOps environment. How we build up a sprint for our application two weeks to read Sprint. Maybe you can say that common advice. Put a six month sprint for your DevSecOps environment that includes the process and practices and then see where you are right now, where you start six months ago, where you are as of today and what you should be planned for the next six months. Plan it and then let the team believe and also share the same vision and same goals. It is very important to build up a culture as you go building up DevSecOps environment. As I said many times if you bring that a Redis solution from DevSecOps from any cloud vendors or a cloud environment, you may set a lot of frictions either usage or not happy to use the tools or some discrepancies with the tools and architecture to solve that problem. Let the team select what type of tools they are using it and then they will discover what needs to be done in terms of security, in terms of the right tools and metrics. That said, there are a couple examples you can look at from our SLS and SCI DevOps space GitHub space so we have been publishing some exemplary tools or some solution to build up your DevSecOps environment. Just the experimentation before you can play around it and get some familiarization. You may not have a test harness, but this platform will give you a good test harness for you to play around. And also if you get more information about the DevOps and DevSecOps, please constantly and we have been publishing any type of blog post and webinars, please follow up this websites and URLs and get more information. Again it's just the sharing knowledge. So what we learn from our communities or the researchers or the organization we do we have been publishing. Please and connect with us. And lastly connect with me for any question that you have and happy to answer to you for any question that you would like to talk about it. Thank you and I wish you happy good day and safe and healthy days.

Thank you Hasan for your DevSecOps awesomeness. Links to everything of value we covered in this episode Headed over to test guild.com P128 Remember we latency is the silent killer of your app. Don't rely on frustrated user feedback. You can know exactly what's happening and how to fix it with Bugsnag from SmartBear. See it for yourself. Go to Bugsnag.com and click the free trial button. No credit card required. So that's it for this episode of the DevSecOps Toolchain podcast. Once again, I'm Joe. My mission's helped you succeed in creating end to end full stack, full pipeline automation awesomeness. As always, test everything and keep the good. Cheers. Hey, thanks again for listening. If you're not already part of our awesome community of 27,000 of the smartest testers, DevOps and automation professionals in the world, we'd love to have you join the fam@testguild.com and if you're in the DevOps automation software testing space or you're a test tool provider and want to offer real world value that can improve the skills or solve a problem for the Guild community, I'd love to hear from you. Head ON over to testguild.info and let's make it happen.