A Warrior's Journey: Building a Global AppSec Program - OWASP Global AppSec 2020

"Adapt what is useful, reject what is useless, and add what is specifically your own." -Bruce Lee

This talk covers critical foundations for building a scalable Application Security Program.

Drawing on warrior-tested strategies and assurance frameworks such as OWASP SAMM and BSIMM, this session gives actionable guidance on building and advancing a global application security program.

Whether you are starting a fledgling security journey or managing a mature Secure Software Development Lifecycle (SSDLC), these foundational elements are core for achieving continuous security at scale.

Brian Levine is Senior Director of Product and Cloud Security for Axway, an enterprise software company, delivering product solutions and cloud services to global Fortune 500 enterprises and government customers.

Transcription:

[SLIDE 1-3]

Hello, I am Brian Levine and welcome to this session, "A Warrior’s Journey, Building a Global Application Security Program". Before we get started just some quick highlights on my background. These days I'm Senior Director of Product & Cloud Security for Axway software. If you haven't heard of Axway, Axway is an enterprise cloud services provider to Fortune 500 enterprises and global government customers. My team and I support a global engineering organization with roughly 500 developers in 7 R&D locations and over a hundred unique products in our catalogue. My university education is in Industrial Engineering and before getting into security full time, I held a number of Systems Engineering positions delivering security, content management, and data protection solutions to Federal, Aerospace and Manufacturing businesses. For fun, I also enjoy practicing Jiu Jitsu, and I don't back down from a challenge even if it's from my niece, the Karate Master pictured here.

So, this journey began when I was interviewing for a security product management role at a SAAS startup. During the process, I asked if I could speak to their Chief Information Security Officer and Head of Security Engineering, and after a few awkward stares, the effective response was, well we don’t have those yet either, and we’ll need you to handle that for us too. That’s where my story begins, so let’s have some fun with it.

[SLIDE 4] – Where would you begin?

I’ll start off asking you the question, where would you begin? There’s an Alphabet Soup of standards and frameworks to choose from, you have ISO, OWASP, NIST, BSIMM, PCI, SOC 2 and on and on. If you were tasked with building a security program, if it were day 1 for you in a new role, which playbook would you use? Is there a script you could follow, which set of frameworks would you use to get started in the right direction?

[SLIDE 5]

My talk today is going to draw on this quote and the wisdom of the martial arts master and philosopher Bruce Lee.

“Adapt what is useful, reject what is useless, and add what is specifically your own.”

In that spirit I’m going to draw on my own experience with some of these frameworks and guidelines and cover the core foundational components that I feel have led to my success and I hope will help you get started.

[SLIDE 6] – Outline & Agenda

What I’m hoping you’ll get out of this talk are some strategies and tactics that you can use to develop and improve your program. What we’re going to cover in these three core areas. We’ll focus on establishing a security Culture, we’ll look at developing and scaling security Processes and we’ll look at Governance for ensuring visibility and executive accountability. I really look forward to hearing your feedback, questions, and maybe some different opinions other ideas that maybe you found to be successful.

[SLIDE 7] - Culture

So, we’ll start with culture. Security is everyone’s responsibility. You’ve probably heard this declaration in some shape or form in your career but how do we really implement that. Everyone can’t do security all of the time, developers, managers, operations teams, all have day jobs to do outside of worrying about security. So, let’s unpack this a little bit and talk about some of the key role players and what their responsibilities are in delivering a security program.

[SLIDE 8] – Centralized Application Security Group

The first role we’re going to take a look at is the Central Applications’ Security Group. This group goes by many names, you may recognize some of these if you’re looking at the OWASP SAMM, they call it the Secure Software Center of Excellence or the SSCE, it’s a mouthful. The BSIMM call this team the Software Security Group. In my company, Axway, we call our application security program the Product Security Group.

[SLIDE 9] – OWASP SAMM Secure Software Center of Excellence (SSCE)

So, there’s many names and you may recognize a few but whatever you choose to call it, let’s look at some of the guidance from OWASP SAMM and BSIMM and others on what the roles and responsibilities are for your Central Security Team. So, highlighted in red here, a couple of the ones that I resonate with and have worked well for me, number one, defining the charter. Make sure the charter is defined, written down and that you have executive management buy-in and executive sponsorship on the charter. The security group needs to publish SDLC standards and guidelines for App Security and the products champions are typically the ones that are going to be required and responsible to perform the security activities in the development lifecycle.

[SLIDE 10] – BSIMM – Software Security Group (SSG)

I also have some very interesting data and pointers from BSIMM. If you look at the BSIMM study and what they recommend for the SSG, you can read the slide here but according to their observations, the very first step is to form an SSG, without an SSG, success is very unlikely. So, they looked at a hundred and thirty firms in the latest study and noted that every single one of them has a security software group. So, basically, the message is that, don’t try this at home if you don’t have an SSG. Also, some interesting data from this study, if you look at the first red arrow there, the average size is two percent of SSG members to developers. So in other words for every one hundred developers, you should have two members on your SSG team and then with firms with over five hundred developers or more, in that group the largest size of the SSG was three percent. So, there’s a benchmark somewhere around one to three percent of your SSG team to the number of developers seems like the right goal to shoot for. And, of course, it all depends on the size of your organization, your mission, the business you’re in and the information and assets you’re trying to protect.

[SLIDE 11] – Establishing & Scaling your central security group

Now some advice and notes on building your Software Security Center of Excellence. Number one, getting started, so, securing an executive sponsorship is critical, you have to get executive buy-in and executive skin in the game. Believe me, you’ll need this when it comes time to prioritize security alongside with feature development, product roadmap and other stakeholders and other challenges to the business, everyone’s got a say in what the development teams should work on next. So, getting visibility and sponsorship for security, at the top down is critical for your success. Additionally, the charter and scope should be written, documented and shared within the organization on a wiki or the corporate intranet, make sure this is documented, shared, and everyone has access to it and visibly understands it. Your goals and objectives need to be defined, achievable and measurable.

Assuming you’ve done the first three, you want to make sure you’re aligned with R&D. So, you make sure that they are aligned in agreement with your scope, your charter, the goals of your SSDLC, if not, iterate, go back again until you have that alignment. Additionally, when you’re working with engineering, make sure you understand the procedures and tools they use today, are they using AGILE, are they doing SCRUM, are they doing continuous integration. What’s their product lifecycle look like? What does their roadmap look like? Make sure your security group understands the environment and has that business awareness of the product development lifecycle before you start, and then start with internal evangelism. Again, the blogs, internal blogs, internal wikis, corporate intranet, slack channels, teams, establishing a forum where you can put out your guidance, put out your guidelines, get discussion going, ask for feedback, these are all important goals to start building a culture and start evangelizing security.

And then also, the SSCE is responsible to look at and see what platforms are we using, what languages, what type of vulnerabilities are being discovered and identify tools to address and identify those types of issues. So, those are some points on getting started. Now, let’s look at if you already have the software security group and you’re looking to level up and take things further in improvement program.

Number one, stay focused on the customer, R&D is your customer and as you progress you need to remember that their needs come first in terms of helping them achieve security in their workflow. You have to align and make it so that they can accomplish their goal of shipping software and you’re serving their need and helping them achieve security in their software delivery. We go from the starting phase and now looking at leveling up we’re enforcing and publishing SSDLC standards, procedures and best practices. We’re no longer just sharing ideas but we’re enforcing standards and then as you grow and as the team continues to grow and scale you want to start identifying your promising members of the development organization that have a knack for security, promote them or designate them as security champions or maybe they might be interested in joining the Security Software Center of Excellence coming on to the core team. And as you scale, external evangelism becomes a bigger part of the program so, not only are you selling internally but now you’re publicly talking about your program going to trade shows and giving talks, going to conferences, blog posts, engaging with customers and doing a lot more outbound public marketing of your security program and the security of your products that you’re delivering, and as you scale up, of course, DevSecOps automation, really leveling up requires automation and continuous delivery at DevOps speed, we’ll talk about some of that here today and then data-driven program management. Again, as you scale to a hundreds and potentially thousands of applications in your catalogue, you need to measure the activities, measure what your team is doing, measure the security issues that are being discovered in the products that you’re working with. Mine that data to improve your product, process to prove your team’s workflow, improve your team’s charter.

So, I’ll give you an example, in our company we looked at defects that we were being reported on certain category of products. We discovered across all of these products, twenty five percent of the reported issues were third party component related so we initiated a program and a procedure to improve our software composition analysis and select some new tools and some better procedures to reduce that specific type of vulnerability.

[SLIDE 12] – Security Champions

So, the next role I’ll talk about is the Security Champion. Here you can see what SAMM, their definition for Security Champion is, and you can also see some statistics from BSIMM, the important thing here is that the Security Champion is defined for each development team and forty two percent of the firms in the BSIMM study, forty two percent of them have a Security Champions program and sixty five percent of the firms have a program if they’ve been part of the BSIMM study more than one time.

[SLIDE 13] - Security Champions Program

Now we’ll look at some specifics on building that Security Champions Program and leveling up the Security Champions Program. You can also see some companies call these Security Champions, others call it Security Point of Contact or the SPOC. So you can choose a creative name that works for you but the idea here is to identify individuals with a passion and an interest for security.

Volunteers work a lot better than voluntold, so it’s always better to find the people that are already coming to you asking about security topics, asking about privacy, raising security issues, coming to you with challenges and things to work on already, those are the people you want to identify and name as your Security Champions. What the Security Champions’ main responsibility is to execute the security development life cycle procedures; the testing, the threat modeling, the security scanning inside of the development project that they work on. They’re kind of the point of contact for the SSG as well as the individual on the team who’s responsible, hands on the keyboard for setting up and executing the scans, they’ll also work with DevOps team and others on the team to instrument the tooling as well as some other activities, we’ll talk about in the next slide. But triaging the findings that come back from the tooling, getting that into the backlog. In terms of scaling the program, once you’re up and running and scaling, now, you’re looking at multiple full-time champions per project. Some teams and some development teams will have multiple SPOCs. Some will have multiple full-time Security Champions working on one development project or a set of micro-services for one product. Again, as you scale up and mature your security champions are likely going to be pushing the curve, they’re going to be identifying improvements, identifying new procedures, new security tools that you can then leverage back within the SSG and also leverage to other development projects using these tools and advancements. And then also, sometimes the Security Champions, as you evolve these team members will eventually make their way on to your Central Security Software Team or your Software Security Group and become core members of the SSG.

[SLIDE 14] Security Champions Program – Anti-patterns

Some gotchas and some anti-patterns to look out for with Security Champions, number one is, if your SPOC is the only member of the team responsible for security then, and all security tasks and questions are assigned to the SPOC. Now, the SPOC should be working with DevOps and build managers to prioritize and automate security testing into the pipeline. It’s not only the SPOC’s responsibility to create the pipeline or create the DevOps tooling to run the scans. The SPOC’s responsible to help triage and review the findings and to put them into the backlog but then the entire development team is responsible to work on that backlog fixing security issues.

The second anti-pattern is, the SPOC is responsible to prioritize security into the product delivery cycle and the third anti-pattern is if there is an adversarial or subordinate relationship between the SSG and the security champion. So, for the last two bullets, in terms of responsibilities for priorities and, it’s the engineering manager and the product manager and the executive’s responsibility to prioritize security. It’s not the SPOC’s role to prioritize security, you don’t want to make the SPOC’s job harder than it is already by also making them now convince their boss that security is priority. The priority has to be top down driven from the executive level. So, the executives are responsible to prioritize security, the engineering managers, the product managers are responsible to be aware and have visibility on the security of their product and they’re accountable to make sure security is accomplished within their roadmap, within their delivery cycle, it’s not the SPOC’s responsibility to prioritize. And as far as the third bullet, again, making sure that your SSG keeps in mind that R&D is the customer, the SPOC is the customer, the SSG is serving the Security Champion, it’s not a blaming culture. If the product is failing security, it’s a business, organizational product management issue, it’s not the SPOC’s fault. You want to execute in a blameless culture and a blameless environment.

[SLIDE 15-16] Education & Awareness

So, now that we’ve established who the role players are, how do we get teams to know and apply what we need them to do?

I like this Bruce Lee quote here, "Knowing is not enough, we must apply. Willing is not enough, we must do."

So, I’ll talk about three core components for the education awareness part of culture. Number one is structured training programs, so we’ve got Mandatory Developer Security Training, Advanced, Role-Specific and Platform-Specific Training, which is a little more hands on. The mandatory training is for all developers when they hire on, they are required to take a training and then either annually or biannually after that take a refresher. The advanced blue belt classes, these are for the more role-specific, platform-specific, language-specific where we want to get more hands on into fixing code and looking at specific problems in the language that they work every day and do hands-on exercises, hands-on coding challenges related to their day-to-day responsibility. The black belt is a more behavioral achievements and certifications so this can be hosting a security meet up, writing a security blog post, publishing something internally or externally. These are all behaviors that contribute to someone’s achievement of a black belt and it may take several years to build out a training program and to see a cohort go from white belt to blue belt to black belt.

If you’re just starting out, start with the mandatory developer security training, which will include the top ten OWASP vulnerabilities, common security mistakes developers make and spreading that awareness around common challenges. The second group of activities here on the slide is Security Events, these are kind of Security Days where we do our workshop day, we’ll have someone from the SSG go into the site, R&D site or within the team and do some immersive activities, usually we’ll do some structured training and then do some tournament or challenges. We also use capture the flags, this is a great way to again, bring people together in a fun way, it’s hands-on, it’s fun, it gets people interacting and really kind of seeing some real world challenges either on sort of the capture the flag type of targets or even setting up challenges where they get to attack and hack into their own products. Just to mention here, we use OWASP Security Shepherd is one of the tools we use for capture the flags and there’s a lot of other great tools out there, open source and commercial that you can use here. Then the third part is the Recognitions and Rewards, you really want to reward and praise people for taking the initiative and for taking advantage of the training available to them and going the extra mile as needed in their products and across these challenges. Publicly praising them in slack, teams, company intranet. You can launch a security stars program, where on a periodic basis, either quarterly or monthly or just ad-hoc, you name who the Security Stars are for that time period and announce them publicly. And also of course, a little swag, and some inexpensive gift cards can go a long way, those are some fun engaging ways to breed awareness and brand your application security program, have t-shirts made, come up with a logo for your team and share those as rewards and also of course, hit-up your vendors. They’re great resource for hoodies, stickers and other fun giveaways.

[SLIDE 17] - Process

Now we’re going to move into the second area, we talked about and highlight some of the processes and technologies needed at the foundation level, and also what’s needed to advance your program.

[SLIDE 18] – Define Security Gates and Passing Criteria

So, the first step is to define your security gates and passing criteria. I just highlighted this is from the Microsoft SDL but I’m highlighting here kind of three areas we’re going to focus on at this point. In establishing the security requirements, the idea here is this is the optimal time to define the requirements upfront during the initial planning phases of the product and we want to integrate security in a way that it doesn’t disrupt delivery and the lifecycle of the product. So, letting the team know upfront as far in advance when they start developing this project, what are the security requirements, what are the security bars, what does the development team need to do in order to achieve a passing grade on your security requirement. So, it’s really important to document this, it’s really important to meet with the teams and go over the security requirements so that everyone knows what they are, to build trust in the program, to improve understanding, the goal is to reduce friction and frustration at the end of the process. Nobody likes surprises, engineering will be frustrated and lose goodwill if requirements are constantly moving or not well defined. So, here in a typical SDL we would define our requirements during the requirements phase, the development team executes the procedures and then we get to the release phase then we’ll have a final security review or FSR where we check and verify that the team has hit the requirements that were agreed to at the start of the project.

[SLIDE 19] – Security Gates (Pro-tip)

This pro-tip draws some inspiration from these words, from Bruce Lee again,

“Be shapeless, formless, like water. Water can flow or water can crash. Be water.”

What does that mean with respect to our security program? Well, it means you want to gently merge your security practices into the existing development cycle. First you identify the gates and collect the results but don’t enforce them yet, you want to flow into the process like water. You’re not here to cause a team, a development crash at this phase. You want to show up, observe the security posture of the product, observe what the team is doing and adapt your security requirements to the tools and instruments that are in place and already in use by the developers. And so, both of these statements come from BSIMM SM1.4. Defining checks but without enforcing them is also a tip from BSIMM. Again, what this means is that, we’re going to define the security metrics, the security criteria we want to look at. We decide which points in the process we should be doing these security requirements, but we will not start enforcing them straight out of the gate, we’ll just observe and then when the team gets comfortable with the tools, when the team gets comfortable with the metrics, then we’ll choose and agree upon timeline to then start enforcing them and making them mandatory. But first, just observe and measure.

[SLIDE 20] – Example Security Gate / Security Bar

I’m just going to give an example here, here’s what one security gate might look like for one control, such as, software component analysis. For software component analysis, for ISR and FSR, the project is scanned using the approved SCA tools, all results are reviewed by the development team and then assuming we’re doing enforcement now, all critical high and medium issues are resolved prior to the release. It doesn’t have to be extremely verbose, just something that’s clear, concrete, measurable, you can put it in writing to the development team. They can see exactly what the requirement is, they know how to measure it and they know how to know if they’ve achieved and passed this security bar. In practice, you’ll follow kind of a pattern like this on the other security gates or security bars you need for your environment. For example, you may have security gates for threat modeling, secure design review, static application testing, container analysis, attack surface analysis and dynamic scanning. Each of these controls will have a security bar definition, which says, how do you test this, how do you measure it and what’s the passing criteria for that bar.

[SLIDE 21]

In the process section here, so far, we’ve covered the foundational aspects to defining your SDLC process. Now, we’ll take a look at what this looks like at scale in a DevOps world. Keeping pace with continuous delivery, continuous deployment and continuous security. I refer to this quote from Bruce Lee here,

“I fear not who has practiced 10,000 kicks once, but I fear who has practiced one kick 10,000 times.”

That 10,000 times is what we’re talking about here. Secure SDLC is great but how to do you do that 10,000 times? How do you do a hundred releases per week or per day if needed and still meet your security requirements?

[SLIDE 22] – Continuous Security & DevSecOps

Now we’re going to move into more of an advanced SDLC at scale, I’ll be talking about continuous security and DevSecOps. So, we’ll go through all of the details on this slide but the point here is that in today’s DevOps lifecycle where we have frequent small changes, your MVP, hot fixes, minor changes, etc… all going out, multiple deployments sometimes per day, there isn’t enough time to perform a hundred percent of the manual security tests and sometimes the automated security tests on every single change and release to production. So, how do we overcome that and scale security requirements to DevOps speed and scale. There’s no longer a shift left, as you can see there is no left, it’s a shift everywhere.

[SLIDE 23] – Continuous Security Pipeline (Example)

So now, security is ingrained in every step of the DevOps lifecycle. Here’s a notional example of what a continuous security pipeline at scale kind of look likes. Obviously this requires a high level of automation and orchestration and again, this is just an example, there’s some vendor names and examples here on this slide. Again, these are just notional examples, they’re not meant to endorse any of these vendors, just some examples of some that we’ve seen. What we’re illustrating is that security is integrated into the CI pipeline, then security monitoring is integrated into operations, telemetry and procedures, defect management is centralized, backlog management is centralized and it’s all integrated and we want to use the same systems that the engineers use every single day to do their work, that’s where we want the security information and security tasks, security defects, all entered into that same information that the engineers are looking at everyday, security can’t be something that happens on the side or at the end and dropping off a report to someone to now go look through a document to figure out what needs to be done. The security tooling is integrated directly into the engineering and development pipeline.

So, we’ll kind of just quickly go through this example here, we won’t spend too much time on it. Again, if you want more details about this or the previous slide, ping me, I’ll be happy to share more depth but in this example you can see that security tests are done before deploying to staging. Once we deploy to stage, we run the dynamic testing and surface analysis testing if it passes those tests then we can deploy to production and for out of band tests that don’t happen during the pipeline, we check via APIs to see the latest result from those tools, for example, in threat modelling we can check and see has this product or feature had a threat model and if so, are there any open or outstanding issues that were raised from that threat model that haven’t been addressed yet.

[35:52 SLIDE 24] – Security in CICD example

Another quick example before we move on. Highlighted on that last slide what a kind of end-to-end security automation framework looks like. Now, we’re going to take a deeper look into an actual CICD pipeline with security built-in. The point and example here is developers want pipelines that run fast, they want fast build-time and they want immediate feedback. The problem with that is some security tests cannot be done on every single build. So, what a CI pipeline like this would do is on the test, that can be run directly in the pipeline, for example, dependency check for composition analysis and twist lock for container scan, they will be run inline, but other security requirements that happen either maybe on a weekly basis or is a manual activity, they will just go and fetch the latest result from that job and then as long as everything passes then the build passes and the pipeline passes and they can move on to the next stage.

[SLIDE 25] - Governance

So, now we’ve come to the third and final stop on our tour and I appreciate your patience and attention so far, I’m going to cover two core components on governance and then we’ll wrap up. So, the first concept we’re going to talk about is establishing meaningful metrics and KPIs and in the second point, we’ll talk about is handling security exceptions and risk acceptance as part of your SDLC procedure.

[SLIDE 26] – KPIs & Dashboards

So, in this example, I’m sharing some KPI metrics and dashboards. This example I’m showing, we have an aggregated risk score, and this risk level is determined by aggregation of all the components included in this product. You can see, by product but then also by component, what the score is and then this score is made up of the individual results of the various security scans and security requirements for that component. So, one of the really important point I’m trying to illustrate here is that, number one, you want to aggregate and radiate this information, you want to aggregate the metrics and use these to communicate risk to the business, you want to share this at the executive level, so that they see trends and they’re aware of the current security posture. You also want to use this, share it across all of R&D so every team can see how they’re doing relative to other areas of the business. You’re going to measure those trends and use those trends to improve your program.

[SLIDE 27] – Security Exceptions & Risk Approval

The next point related to governance is around accountability. We talked about the SDLC process and how the product and an engineering project have to achieve a pass in the security bar to move forward. Earlier, we touched on what happens if the product doesn’t make the security gate and that’s what we’re talking about here. Remember, in the end your goal is to release software and get your valuable products out into the customers’ hands. We don’t live in an ideal world, where every engineering project is going to meet every requirement on every release, whether we’re talking about quality, feature development, security tests. Occasionally, at least once in a while, you may have a scenario where a product is trying to hit a release target date and needs to move forward. So, this is why we have an exception procedure and this is a critical part of the business so that a hot fix or important release is not caught in a loop and getting blocked due to a security finding that may not be a high risk. What does a security exception and risk approval process look like? Well, first requires a mitigation plan and a timeline and the timeline and plan should be relative to the risk and the severity and requires approval, documented in a system of record, so the official approval is documented and depending on the issue and level of severity escalate up to the executive level to get that approval. This way, it’s the line of business and the executive that owns the risk, the development team is not owning the risk. The security champion doesn’t own the risk, the business managers and the executives own the risk and the most important point here is that we capture these approvals and mitigation plans in our system of record, in our engineering tooling. So, in this case, using JIRA to make sure that the approval is captured but also that we enforce it with our automation system and orchestration. So that if we agree to fix something by December and the engineering team is now trying to release in December and they haven’t fixed that issue yet. Then that will automatically block the build that will block the release and they will have to go back and resolve and meet their mitigation plan or the project is blocked. And so that again, is necessary when really, we want to scale the risk acceptance program and achieve this at scale across a hundred and fifty products with continuous deployment. You need to make sure that decision is enforced in an automated fashion.

[SLIDE 28] - Summary

I’m going to wrap up here just to recap, we explored the foundations and three key areas, culture, process and governance and I gave examples that you can take, number one, if you’re just getting started in building a program but also hopefully if you’re looking to improve your App Sec program and improve your App Sec program to the next stage, some takeaways and actions you can take to go to the next level. Some parting advice and takeaway, begin where you are, focus on these foundations, set yourself up for success and some questions to ask as you go along, what are the critical risks and objectives for your business? Start small, use iteration and minimum viable product and that mindset to get things moving, establish some success and build momentum from there.

[SLIDE 29] – Where to find me

I hope you found something useful, and I really, would love to hear your feedback, please let me know your thoughts. Where you can find me on social media, shoot me a private message or mention me in a comment. I greatly appreciate any thoughts, comments, feedback, disagreements, complaints, arguments, anything, just engage me and I’ll be happy to discuss more and share additional details.

Thank you very much.