Challenges Implementing & Sustaining DevSecOps with Hasan Yasar

Welcome to the Test Guild Security Podcast where we all get together to learn more about security testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Security Podcast. Today we'll be talking with Hasan Yasser all about challenges implementing and sustaining DevSecOps. Hasan is the Technical Director of Continuous Deployment of Capabilities Group in Software Engineering Institute cmu. Hasan leads an engineering group to enable, accelerate and assure transformation at the speed of relevance by leveraging DevSecOps, Agile, Lean AI, machine learning and other emerging technologies to create a smart software platform pipeline. He has more than 25 years as a senior Security Engineer, software engineer, software architect and manager in all phases of secure software development and information modeling processes. He's also the adjunct faculty member in CMU Heinz College and Institute of Software Research where he currently teaches software and security and DevOps engineering for deployment and Operations. You don't want to miss this episode. Check it out.

It's no secret that the number and complexity of applications are growing, but what is alarming is that 90% of security incidents result from exploits against defects in software. The need for better application security has never been higher and Micro Focus Fortify is here to help. Fordify is the recognized market leader in application security and it is the most comprehensive and scalable application security solution that works with your current development tools and processes. With Fortify, you can start security applications in a single day including custom code, open source or commercial components and scale as your needs grow with an on premise as a service or hybrid implementation. Check them out. Head on over to micro focus.com Apple Security.

Hey Hassan, welcome to the Guild.

Hey, thanks for having me in Joe. It was nice talking with you. So I was looking forward to have deep dive into the more DevSecOps concept or DevOps or the other challenges as well. Thanks for having me in here.

Awesome, awesome. Great to have you. We talked a little bit earlier in the pre show and you mentioned that you are teaching online now full time because of the situation we find ourselves in.

Correct?

I know a lot of people are trying to do this. Is there any tips you have for being able to teach online more effectively?

This is not the first time for me to teach the online version. I am also delivering an online class during the fall semester. So what I find out that I'm using the similar techniques with this current situation is getting more engaged with the students and when you are in the physical environment, physical settings, it is much easier to talk with the students because they can see each other. But in A virtual setting. It is hard because they don't see each other. The student wise instructors what I'm doing is and actually I'm releasing the content before the class. So let the students look at the contents and during the delivery the remote class delivering just a slide. It's more about a discussion so individual. I'm asking the question to each student as much as I can. Of course depends on the size of the class and number of participants. And then I'm trying to get their engage as talking or writing something to chat window. So I tried that it seems working and after the first week first week was very quiet and we are in the fourth week almost so students they will like they seems they're engaging, they're responding and now they feel that hey I'm able to talk with instructors directly versus in a class environment. So instructors teaching everybody now they are feeling that instructors are dealing with me directly and talking directly. So open up that communication channel and asking the question directly students on the settings and make them comfortable. So force them to kind of engage seems working and also and then be kind of empathy and realize that students are used to having class now they are in the environment. They have some Internet challenge. They may have some other issues in terms of the communication wise. Maybe they don't have the good laptops and stuff. But share the materials first and ask the question during the session. That's what I can say. Joe?

Yeah. Sound like communication is key and being someone. It's been a while since I've been in school but I was always afraid to raise my hand and ask a question. This to me would be like an awesome opportunity for folks that are like that to actually engage more with the instructor. For sure.

Exactly. And also another things I would like to remind them to the folks. Please record your session and share after the session is done through the canvas or the blackboard. Whatever the platform it is share the record recordings. It's going to make them to more comfortable.

So let's change gears now and go into the meat of the topic here. DevSecOps. Before we get into it I guess there's a lot of confusion. People don't know the difference between DevOps and DevSecOps. And now there's site reliability engineering that I heard replaces DevOps. Could you just explain a little bit what is DevOps and how is it different from DevSecOps?

I guess we can start from the DevOps first. What is really DevOps when you look at the definition but there are many definitions is getting from different Industries, different peoples. And when you ask the DevOps what it is, it is really changing from organization to organization or people to people. And as part of IEEE groups I'm part of the vice chair of IEEE DevOps Standardization Group. And based on our research and our write ups, we came up that top the definition of DevOps which is a set of principles and practices emphasizing collaboration and communication between software development teams and IT operations staff, along with acquirers, suppliers and other stakeholders in the lifecycle of software systems. So this is basically from IEEE 266775 documentations. So what really means to DevOps? It's basically having a collaboration between all stakeholders. It is not just the Devon Ops we learned that. But however the industry or the whoever practicing DevOps and they often ignore the security elements. Even the security is the part of the our requirements or part of our work. And as a result of moving fast or deploying fast in DevOps methodologies or the things actually force them to ignore the security. Why it's not an intentional things more about security testings usually done after the integration phases or the before deployments. It is becoming a barrier. So barrier means like it takes a longer time to get approval or block by the security team or blocked by the systems. Like systems maybe not enabling to move fast because of systems finding so many vulnerabilities in a container or finding so many vulnerabilities in a software pieces has to go to life. So which is creating a bottleneck. So what DevSecOps movement it is and get security throughout the life cycle. By design we should have security. But as I said, the nature of the work nature of the business needs are moving fast. It is not enabling us get secured into the pieces with the DevSecOps concept as a community. And we decided to really emphasize security throughout the life cycle because it's not done enough properly. When you just say DevOps it's just, you know, making as important elements. When you do the DevOps you have to think about the secret as well. And we are calling DevSecOps. It's another session actually I said DevSecOps is the doing the right version of DevOps. Specifically the context of security.

So it sounds like that should have been there all along. And the word just helps accentuate what it should have been, I guess from the beginning.

Exactly. We should have it in the beginning. So somehow we ignored or we did not enough pay attention and we did not enough facilitate the security components in throughout the life cycle. So we couldn't spend the time or money or Other dependencies we ignored and that hurt us a lot. If you look at the many bridges in industry right now and because of the fast deployment, because of fast automations, because of the using dependencies, because of getting all the ready images from the Internet or the using somebody else code or like GitHub or anything else, which is opening up a lot of vulnerabilities. So basically the side effect of DevOps forced us to think about the security again. That's how DevSecOps movement started. That's what we are saying that you should have a security. Calling security and explicitly calling them here is part of our process. We have to be aware of it.

Absolutely. So if people are baking in from the beginning of the design. I know in your last year's Secure Guild session you talked about some dimensions of DevSecOps. I don't know if you remember what those are. Can you just give a little bit more?

DevOps is based on like a, you know, maybe maybe three pillars or four pillars which we typically say it's like a culture, automation, measurement, sharing or infrastructure pieces. But all are depends on the fundamental requirements. Like if I am going to implement DevOps, let's do DevSecOps, not DevOps anymore. So it's. It's like we can answer the secret pieces as well when we look at that requirements to establish DevSecOps we need to have a specifically a culture which is organizational culture, organizational and collaboration which is policies and then the how organization acting together, working together. So culture is the one dimension like one requirement I call as maybe the first pillar. And once we have a right cultural environment which enable people are sharing, trusting each other, collaborating on the topics or the problems especially with the security domain. Like if I'm able to as a security person, if I'm able to share any information with my dev team, my architect team, they will have a good information about my system posture as a developer. They can do a better job when they write the code and with respect to domain specific it. So basically building up a culture, building about trust relationship will help them to communicate and other elements is having a automation and measurement pieces like how can we automate all our security testings? How can we automate our threat modeling that we have done it? How can we insert a threat model into the our continuous integration, continuous deployment process? Or how can we do kind of infrastructure check or maybe based on the compliance stack, how can we automate check our configuration of the server? So automation is essential to establish the automation throughout the life cycles. Like we have to Automate the steps which is when we try automated steps that means it's requiring another dimension which is system architectures like system architecture should be compatible enough to so we can develop a specific test cases for this feature that we are deploying or we are delivering to the staging environment. So if our architecture is monolithic architectures we cannot develop a specific test cases for a given features or we cannot see overall systems because we are waiting a lot to get the things build up together so we can do more testing but it will take longer. So if you are building up a continuous integration fashion as a result of every code check into repositories. If my build service starting every time I had automated test case along with the security requirements that have decided at the beginning or I have to do static analysis or I have to do dynamic analysis, I have to do fasting, I have to source software specific pen testing things all are depend on the system architectures. And lastly we need a right practices amongst the stakeholders that will align with our tool stack that will align our automation concepts. So which is kind of an orchestration I call as a process and practices we should have that in place and we know what to do and as a team we know how we're going to work together as a team and then we know what's our game plan. It is as a team. So it's important to have right processes and practices help each other to establish the automation within DevSecOps environments.

Absolutely. And I think the very first dimension you mentioned was culture. And to me that seems like if that's not in place, all the other ones are never going to get in place. So any suggestions on how to really fix what I've seen as culture problems in organizations around this?

Yep, it is. There isn't like a single solution or recipes to fix the culture. It is tough. I mean there are many ways to establish the common collaborations amongst the stakeholder which is starting from understanding what we are doing in the business, what is our business goals, what is our objective and vision. And then everybody should ask a question and from management to the dev team to the security teams, why we are doing it, why we are here, what is our purpose. So basically let them connect our vision and goal together. So people, which is all stakeholder are feeling responsible for the same vision so they can achieve their jobs. Like align with the goals and visions within the team that we would like to do. And after that and building up an environment. It's kind of like a journey. So which comes from the management team and dev team and eliminate any type of like a blame free type of culture. Have an established organization build up a culture. We're not going to blame each other and we would like to look at the solution perspective. We would like to help each other and learn from our lessons which is getting again the how we treat the work and how we are looking for the problem. Instead of having a meetings and penalizing the people that you fail in your build because you're not able to deploy in the securely versus what was wrong. Let's do better the next time. So change that behavior and mindset looking for more the solving more about the learning instead of penalizing the team or penalizing team member and then let them to learn from their mistakes. That's how we start which is from the management support and team level and change the behavior of the our communications and other things. I would like to advise and open up a good communication and good collaboration between all people that part of it like hiding information. It's always breaking the trust relationship. And then if you really apply the DevOps clearly in our organization we can see a good communication all the stakeholders good communication with team members through the tools like using a chat, maybe using a chat apps using a wiki pages using a shared platform shared. So if I share my information with my dev team member and then other person will also share. They will say I'm not hiding anything. I'm working for the organization or a team success and I'm part of it. So sharing actually building up a trust relationship and having a visible building of a trust relationship when we build up a trust between the stakeholders. So we are increasing our communication and collaboration between team members so we can improve the culture by learning, by sharing responsibilities, by sharing our learning and also by sharing the problem solving as well. It's a long topic but that's kind of. I can get a quick advice for you.

Absolutely. So around sharing resources and sharing knowledge, who essentially is responsible responsible for DevSecOps is it a separate team or is it part of the agile sprint teams? If they're doing sprints?

I think it's not a specific team responsibilities. It should be everybody's responsibilities. So everybody should think about their jobs and their role. How can I help my team which is how can we break the silos? So I always ask the question like when I talk my students or when I try to adapt the DevOps into the organization. So I'm not really defining and I don't like that the term of DevOps engineer DevOps team because everybody's responsibilities and establishing the good DevOps practices. When you say everybody means all the stakeholders, we always think about this way. We should say, how can I do my job better? How can I do my job in a way that I can help others? Which is a mindset changes. So if you look at the waterfall approach that we have been talking, you know, now we're blaming the waterfall. It's kind of like a evil things. We have been saying it. We created a silo. So in the silo, that environment we said this is I did my job, I don't care the rest of it. I'm just going to give what has been asked for me. Maybe delivering a code, maybe writing a kind of design documentation or writing your requirements or whatever it is. And then we do a lot of handover. But we never really taught ourselves. We never really said, if I put myself in a situation as a tester or operational person, how can I help that person's job to make it better, to help each other. End of the day we will deliver a software. End of the day our organization will be successful. That's how we can align our goals and objectives and vision together with our work day to day. So changing mindset and saying that what I can do to help you. So which is including the Scrum master, including the dev, including architect, including everybody. And then when we ask that question for ourselves and we will say, okay, this team needs this information. Let me share that, Let me give what that team needs. So I don't want to be a bottleneck for their job. I would like to be enabler, I would like to work for them. They're not working for me.

Great advice. So once you have the culture in place, the mindset in place, I think another roadblock, at least what I've seen with large enterprises is legacy systems. Any advice on how to get this in place even if you're dealing with older systems?

Right. It's a tough challenge. It is really a long journey getting legacy systems because first of all, legacy systems and sometimes maybe operational dependencies, maybe environment dependencies and maybe some language dependencies. So we had to really find out and what type of systems we are in and what are we trying to maintain that. So the one approach is basically look at the organization. They have to look at how much time and money we are spending to maintain that legacy systems, which parts are requiring so much changes or so much maintenance and try to modernize that pieces. So modernize that component. We cannot really modernize everything from scratch because it takes time. But what we should do we should modernize the pieces any module that is getting more requirements. So the basically converting monocle architecture more about the modular architecture, maybe microservices and maybe writing a specific SOAP architecture maybe creating a specific functionality for that features only. So start from each modules, try to convert that modules and then eventually if you do that route and then we can basically convert all the legacy analytical systems into the microservices concept or modular concept actually it's called a strangler pattern. So it's basically getting one element converted and go to next one. Depends on our business needs. So go step by step. And when we do that we have to think always for the features. How can I deploy that capabilities has been modernized or has been rewrite or has been developed again as is supporting the DevSecOps which is incremental iterative mindset. So that should be the overall idea. We cannot really convert everything from scratch. We have to go step by step. And using a strangle pattern is a good option.

No, that sounds like an awesome pattern. I guess another thing, maybe another roadblock is too complexity. It seems like it's not always about tools but we always do tend to get hung up on tools. And then in a large enterprise it's like oh my gosh, yes. What I use is all these options. Any suggestions or advice around that?

Yes, there are but I have been keep checking Joe. The tools just keep coming up in from industries and we would like to. We would like to really generate so many tools because that's kind of like a business needs as well. Because it's a growing areas and everybody in terms of the perspective trying to invest the money and time and build up a tools. So I kind of keep track the number of tools has been released and I think as of probably last month we had more than 300 tools available. For a software delivery perspective. I'm just talking about the software delivery software deployment tools. So within that 300 tools it is really difficult to select which one is good for us. We don't want to be and getting your tools based on somebody else did something. I always ask the question to the team members why we need that tools? What is the purpose of these tools I'm going to use in my environments? Like one analogy is when we go to the Home Depot Joe, do we buy every tools? No, we always buying your tools based on the project what we have. Right. We cannot buy a Home Depot because Home Depot has a lot of tools is available for our projects. We always start from the project. So I'm going To achieve this project what I need and then I will look at my garage and see what is what I needed and get the tools to do that job. Same analogy is also true for the software engineering. And I have to really select the tools based on my tech stack and based on my technical requirements and based on architecture requirements and based on my tool dependencies. I should go from a perspective what I needed, why I need it, for what purposes I'm going to achieve that and start from the step by step and use it. Another things there is no one tool achieve everything. We have to select the tools appropriate for our needs and always, always we have to build up another elements. How can we integrate the tools? I mean I may buy a tools for a software perspective and is it going to be integrated with others and others means like other tools stack in our DevSecOps environments. Because if you're not able to integrate the tools into our software delivery pipeline and tool become a silo. When the tool become a silo and team will be a silo as well. So now we at the beginning we said we're going to build up a culture that's everybody's talking each other. Now suddenly tool is blocking us not able to talk to each other. So we have the relating about if I'm having a tool can I integrate with my other tools? Can I connect it and how can I build up my continuous delivery pipeline? Because we don't want to create any bottleneck as tool is causing us. It should be continuously should be automated. Like automated means. Like we should really create a test script. As an example, connect my build server which may be a Jenkins, maybe the Bamboo and can I put my build scripts and which is creating my test harnesses through the build environment. Is it supported? Is there any plugin available build server, can I connect my issue tracking system and repository management. So we have a fully traceability pipeline. These are things is important to select the tools. We always start from why and then after that what things I'm achieving. How can I achieve my continuous deployment delivery pipeline?

Right. And now you mentioned traceability and the why once you select these tools. I think another side effect is what to measure. Because there's probably multiple things you can measure.

Absolutely.

So either a team lacks measurements or they have too many measurements so they don't know what to measure. So any rule of thumb of key things you may want to measure in a DevSecOps environment.

So we would like to really monitor all the data first of all, like how can we get the data and Then we have the dev step. Let's say we have a DevSecOps environment up and running, right? Then once we have environment up and running issues, how can we monitor our progress? How can we monitor our quality, where we are, what we do. So as the kind of non rule of thumb advice, we have to really measure our deployment frequencies and change lead time and volume, number of work items, stack tickets and MTTR mean time to detection, MTTR mean time to recovery and then we have to look at the time to approval time to patch vulnerabilities and we have to look at the server disk usage. Basically we have to measure our software delivery progress and how we are doing well at the same time what is our quality of application that we are delivering. So which is application performance, response time and then some service log, some disk space, some memory usage with application related performance we have to measure it, but we have to align that pieces with our software delivery process as well and how much we are able to quickly find out any defect in our code base and the push to new version, what is our timeline from the code we committed and that code will go to the production environment. So this type of data will help us to see where we are struggling and do we have lack of skills, person are not developing good code or we're not able to automate the process because we depend on other dependencies. Maybe, maybe you're not able to get the approval process, something like that. So we can really align to improve the process with the team skills and then also the tying with the data sets from the test automations, build automations and deployment frequencies tie back to our process. How can we improve our quality of the process? At the same time really we have to connect our delivery pipeline with the quality elements as well. Like if we are looking for a build process an example and how can we detect any anomalies and throughout the life cycle such as build process, right, that's the way we can capture any anomalies or any type of abnormal activities. It's happening in the life cycle. Maybe one of the dependencies are reaching out to home and pulling some other dependencies. So that's where we can see there is some security problems happening behind the scene because the dependent are reaching out to the home, downloading other dependencies. So now secret person will be freaked out because this is some abnormal activities. But that if you tie that pieces into the architecture work, align with our team members which is dev team, find out what architecture requirements was, was, what are dependencies and align that the monitoring pieces into our work and then we can have a good picture of what we did it, why we did it and we can monitor that progress and time back to our engineering principles. So which technique the organization should decide what is our key value we would like to connect Besides the rule of thumb the wise I give it what we are trying to measure it and align our business goals, align our quality and monitor that and improve better. So collect I advise collect everything and then you can use later on because if you missed some data collection it's hard to go back and collect again. Let's collect everything put into our data lake and we can visualize we can put the metrics based on what we need later.

Great advice. I think it's like culture A lot of times people get hung up on the details where like you just mentioned with measurement you really should be focusing on business goals and even though it sounds like a no brainer I think a lot of people forget that point for sure. So, so what is the RMF process? I have it written down but I don't know why I have it written down. Is that something that can help people with the DevSecOps transformation? Right.

So our method risk management framework so almost every organization has a risk team that's very relevant to the DevSecOps concept Joe. So if you're looking for the secured automations it's a challenging to find a balance of what is the level of security we have to do. Like if you have infinite time and budget we can do so much security testing things we can do so much a great systems and we can make the resilient system we can spend a lot of time, we can write the best code in the world but we don't have time, we don't have infinite budget either so we have to go fast when we go fast and every organization should comply with their compliances and with their risk team. So RMF is the risk manager frame from the NIST which is giving advice and how you align the organization risk into the software delivery process and what needs to be done with a secret perspective and throughout the life cycle. So which is kind of like a giving an advice. Select the specific security controllers that organization or domain needs to establish and every domain is different like if in the healthcare industry is different and then financial sectors are different. So it depends on which sector you're in or depends on what type of application we are building. Select the appropriate security controls in it or select appropriate the security feature needs to be implemented that's basically aligned with the threat modeling as well. When we do threat modeling we typically look at the what are the typical threats that we are facing and what are the mitigation strategies are. So think about that concept and if I know the threat which is threat is basically based on our language, based on our architectures and based on the tools that we used it, based on the dependencies, what we have and then put the mitigation into the threat and see how we are doing well and putting a mitigation into our code and risk management framework. Also the complementing says select my controllers that needs to be implemented and implement in the life cycle. Like lifecycle means an introvert development phase have an implementation. Did we really implement a write controllers in it which is SQL injection is another example or the cross site scripting or eliminating buffer which is an implementation. Then after that assess the environment did we the code, did we really implement it? And based on the test results we can get a static analysis result and then feedback to our team that we said we can implement the cross site scripting and we implement it. Now we test it out and test says yes, we have done it, then our security team will authorize the code that has been tested properly, has been implemented that will go to our production environment. In the production environment that we are monitoring in terms of any new threats or maybe new vector attack vectors. We discover that and go back again, address the new vector attack vectors and add the new controllers and new security requirements into our project so we can go continuously. So basically it's a continuous security throughout the life cycle. Start from what to do with respect to security, then implement as how to do and verify what is the outcome of the pipeline. Get the test result and then approve it based on the fact. Not based on yes I did it based on the fact has been generated from the systems and monitor it. That's overall process actually. So as I said, almost every organization does typically work now Hasan, this is.

Such a vast topic. You have a whole semester around DevOps. Is there any resources someone can go to for self study maybe that you recommend reading to your students or anything like that.

So I typically advise my students definitely like the SCI website CMU website is available for the resources and then go take a look and also advise the any kind of conference such as all day DevOps conference and we follow that type of conference conferences and actually we have a spring edition is coming up April 17th next Friday and also be follow up such as the DevOps.com or DZone or info queues and sharing and remember that the DevOps is about the sharing the our journey, sharing our practices. So I really advise everybody and if you want to learn what it is and look at what others does and ask the question for yourself what I am doing it, what I'm doing it? How can I learn from others to make it better for our work environment? Not a copycat what they did, but what about learning from others and try to implement what you need in organization. So I really advise looking for the crowdsourcing Internet. Also follow up the SCI or other websites or conferences to increase the knowledge.

Okay Hasan, before we go, is there one piece of actionable advice you can give to someone to help them with their security testing efforts and what's the best way to find or contact you?

So they can contact me through my CMU Asia HGSRME EDU or they can connect me through LinkedIn. I'm willing to talk anytime for an answer to anybody's questions. This is a community effort and definitely reach out to me on LinkedIn.

Great. And Hasan, do you have any like one piece of actual advice? You gave a bunch today, but anything they could put into place right away, right?

One advice I can give Pick up the phone, talk with your peers, understand what they do, put yourself in a situation other team members not I did my job. Ask the question how can I help my other team members including Security Fox.

Joe thank you Hassan for your security testing awesomeness. The links to everything of value we covered in this episode. Head on over to test guild.coms19. And while you're there, make sure to click on the Try Today link under the exclusive Sponsor section to learn all about Micro Focus Fortify, which is the recognized market leader in application security. So that's it for this episode of the Test Test Guild Security Podcast. I'm Joe and my mission is to help you succeed with creating end to end full stack testing awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Security Podcast. Head on over to testguild.com for full show notes, amazing blog articles and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.