RUGGEDDevOps:

10 WAYS TO STARTEMBEDDING SECURITY

INTO DEVOPSPATTERNS

BY ERICKA CHICKOWSKI

Security industry conferences and articles in the trade press are fond of harping on the broken IT security model. They point to growing breach statistics, embarrassing configuration and patch management practices, and mounting vulnerability exploits as ample evidence of the problems plaguing infosec.

It’s easy to point out problems, but what about solutions?

Interestingly, as scary as DevOps may be to the typical security pro, its fast iterations and continuous delivery patterns may be just the solution to fixing what’s broken in today’s security model.

“Security pros need to understand that this is the future of IT and of IT security,” says Rich Mogull, analyst and CEO at security analyst firm Securosis. “I see DevOps doing nothing but improving security when done right. Not only is software going to be more secure, but if you learn from these techniques it can help you get your day-to-day security more efficient.”

But the devil is in the details. What exactly does it take to do it right? How can security operate within the DevOps context in order to ensure a more sustainable and less risky continuous delivery value chain? The complete answers to those questions are still being written. But the innovators in the security and DevOps communities are finding early success with a number of key practices that contribute to what many security pundits call a Rugged DevOps experience.

Attackers are already using their own form of continuous delivery to overwhelm the good guys. The reason security teams can’t keep up is because the bad guys have already figured out how to use automation and cloud-style technologies to scale up their attacks, says Tim Prendergast, CEO of Evident.io and a long-time DevOps and security practitioner. Predergast is best known for his former role, leading up cloud architecture and security for Adobe.

“But the defenders are still relying on this model where you’ve got a person in front of a console,” says Prendergast. “They’re just outgunned and outnumbered, so they have to move into this automation philosophy. They have to make time for renovating their approach and they have to be open to new ideas.”

This means getting unstuck from security patterns they’ve settled into for the past 15 or 20 years, says Prendergast.

“DevOps is bringing a whole new

approach to security to the table that’s actually way better than what we used to be doing,” he says, explaining that security teams will be able to pivot more quickly with attack trends if they learn how to deploy platforms and protection mechanisms in lean, iterative cycles rather than relying on “big-bang releases.”

And rather than trying to be a ‘gating factor’ at the tail end of

enterprise development, security should

be baked into operational

practices and automated tooling throughout the development pipeline.

“A DevOps team will deploy

code and when a bug pops up,

they’re very well instrumented to

gather telemetry on the bug, get it into repair and turn

a fix around the same day and redeploy,” he says. This approach is in contrast to the old model of rolling back the entire deployment on the hunt for a pristine deployment, delaying necessary features for the sake of a whole list of bugs that may take days or weeks to take care of.

UnderstandThat AttackersAlready Deliver

Continuously

What is Rugged Software?

“Rugged” describes software development organizations which have a culture of rapidly evolving their ability to create available, survivable, defensible, secure, and resilient software.

—From www.ruggedsoftware.org

The Rugged Manifesto I am rugged and, more importantly, my code is rugged.

I recognize that software has become a foundation of our modern world.

I recognize the awesome responsibility that comes with this foundational role.

I recognize that my code will be used in ways I cannot anticipate, in ways it was not designed, and for longer than it was ever intended.

I recognize that my code will be attacked by talented and persistent adversaries who threaten our physical, economic and national security.

I recognize these things – and I choose to be rugged.

I am rugged because I refuse to be a source of vulnerability or weakness.

I am rugged because I assure my code will support its mission.

I am rugged because my code can face these challenges and persist in spite of them.

I am rugged, not because it is easy, but because it is necessary and I am up for the challenge.

-From www.ruggedsoftware.org

DevOps cycle times can seem impossibly, dangerously fast to security veterans, who may hold the belief that code can’t possibly be secured without days-long static and dynamic analysis.

But rule number one of achieving a more rugged DevOps process is not taking the ball away and going home when you don’t get your way.

“You don’t throw up your hands and say ‘If they’re doing a daily build and it takes four days for feedback, I give up,’” says Josh Corman, CTO at Sonatype. “You ask if there’s some sniff test or some subset of tests we can do that conforms to the compressed cycle time. And maybe once a week, once a month or once a quarter we do a deeper thing.”

Rather than entrenching themselves against new waves of continuous deployment and automation, security professionals need to “get clever about

how to adapt,” says Corman. Organizations are finding as much as a

10x improvement in development

speed through continuous delivery. They’re bringing new features to customers more quickly, freeing up

resources to develop new lines of business

fueled by digital innovation and clearing

up their application backlogs. Security should be participating.

“If you’re a speed bump, everyone will go around you,” warns Mogull. “Security leaders need to figure out the best way to get into the rugged DevOps process.”

Don’t ThrowUp YourHands

SECURITY was named as the number one DevOps obstacle

by 28 percent of enterprises

Before security pros even start to think about automated testing, new workflows or updated technology, they should begin by exercising their empathy muscles, says Corman.

One way to build that empathy is to start thinking about what the common ground is between DevOps-minded practitioners and security professionals.

“We both hate complexity. We want to fail fast, early and often so we can iterate,“ says Corman. “And we like to instrument everything—we all have an almost maniacal desire to get a ton of actionable telemetry.”

Just as dev and ops have had to bridge some serious cultural gulfs to start collaborating better, security needs to meet the developers and operations teams half way.

A longtime practitioner in both operations and security, as well as the primary developer of the open source automated security test framework Gauntlt, James Wickett recalls an eye-opening interaction he once had with a superior in an enterprise setting.

“My boss at the time said, ‘You’re a security guy, so you probably just

want all the machines turned off and unplugged,’” says Wickett, who is currently a senior engineer at Signal Sciences, a stealth-mode security firm working to build products that work in

DevOps environments. “I was sort of offended by that, because I’m

not an idiot and I’m not a nihilist.”

Wickett, like many other security folks, understood that the business was trying to move forward with its goals. But the lesson was

that there may be a reason for some of

that misapprehension. As he puts it, a lot of

security professionals will mutter something akin to ‘those

stupid developers’ as they put out fires. If security pros want to move the needle on risk management, they need to help everyone come to a mutual understanding of the time pressures and restrictions on organizational resources that each group faces, Wickett says.

“In particular, the security people need to know and learn and understand how development works,” agrees Mogull. “Obviously, there’s the operational side, which they tend to be more familiar with. But they also need to try to understand developer’s terms and processes, and use that to bring security into it. You don’t need to be a developer, but you need to know how they work.”

Start With Mutual

Understanding and Respect

Coming to a mutual understanding could start with semantics. Cooperation between security, developers and operations staff could be improved by overcoming some language barriers, Corman says.

Take the Heartbleed vulnerability as an example. A security person might see the problem as a potential breach, a security risk or an incident response exercise. Meanwhile, an ops person would probably refer to it as a break-fix or a service interruption.

“ [Ops doesn’t] care about security or risk, they’re concerned with being measured on the five nines of availability. They had an outage and they

had to create a remedy ticket and bring all hands on deck to create a war room,” Corman says.

From the perspective of developers or enterprise architects, however,

something like the Heartbleed response

would be referred to as unplanned, unscheduled work that probably caused them to be late on their sprint or their epic,

leaving business frustrated.

Enlightened security professionals

should understand how security problems that they

identify eventually turn into break-fix nightmares or unscheduled work that holds up current development work.

BecomeEffective

Translators

AWS Security is NOT self-Evident!It’s no surprise that DevOps teams love the flexibility, agility, and cost efficiency of AWS. Security and compliance teams, however, struggle to adapt traditional tools, processes, and frameworks to public cloud infrastructure and the rapid change created by the DevOps continuous delivery model.

Built by a team that has secured some of the world’s largest public cloud infrastructures, the Evident Security Platform provides security and compliance automation for AWS infrastructure. ESP bridges the gaps between DevOps & IT Sec, and makes security & compliance an integrated, automated part of your DevOps.

The Security and Compliance IT Requires… at the Speed DevOps Needs!

Secure your AWS Cloud Today!For a free 14 day trial, visit us at www.evident.io or call us at +1 (855) 933-1337 to schedule a demo with one of our cloud security experts. AWS Security at the Speed of DevOps

Risk Analytics & Threat VisibilityContinuous monitoring and risk-based threat analysis of all AWS Accounts, Services, and Regions.

Continuous ComplianceAdaptively manage compliance & automate policy enforcement across your entire AWS infrastructure.

DevOps ReadyIntegrates with the tools your DevOps teams love. Open and extensible design with RESTful API and SDK support.

Automation is absolutely the lynchpin to the whole strategy.

“At the speed people want to move, reliance on manual testing just doesn’t cut it,” explained Damon Edwards, host of the DevOps Café podcast and long-time DevOps evangelist, last year in a talk at OWASP AppSecUSA. “The external inspection by a third party who is going to manually test something is not going to be enough to catch all of our problems.”

This does not necessarily mean that penetration tests go away altogether, but they may shift in importance as automated tests become the tip of the spear and manual tests become a backstop. At minimum, security teams may want to start thinking of ways to automatically integrate manual testing results back into the pipeline.

“Even if you don’t do anything else, whenever you shell out your $50,000 or whatever for your PCI compliance

testing, tell the penetration testers that at the end, you won’t read a PDF,” Wickett says. “Tell them the only output you will accept is integration tests written in Cucumber, Gauntlt, Behave or whatever you want your stack to be, even if it’s just Python scripts that test different parts of my application to verify whatever it is that needs verifying. Tell them you want

code delivered at the end of the engagement.”

All of this is, of course, easier said than done. As Mogull explains, security teams will need to vet their tools for DevOps speeds and processes, weeding out

some, retuning others and bringing in new tools that are

purpose built for the new bar set by the continuous delivery model.

“If you’re getting a lot of false positives using a static analysis tool and that’s impeding dev velocity, that’s a problem,” Mogull says. “Some tools aren’t well suited for working in this environment.”

On the tools side, embedding security into DevOps for a truly Rugged DevOps experience requires that security teams find ways to test early and often by automating as much testing as they can.

As Wickett explains, current security testing cycles are really geared toward waterfall approaches, even as the rest of the IT industry has drifted into Agile and DevOps. And so things like risk assessments and analysis, plus PCI compliance checks tend to be

completely out of sync with DevOps cycles.

“We need better security testing earlier on in the development

process and inside the build pipeline,” Wickett says.

“Right after you commit code, you should have unit tests that run, functional tests that run, integration tests that

run and, at that point, you should also have a suite of

security tests that run. Right now, those tests usually are run on

an annual basis or as the product is just about to ship.”

Test Earlyand Often

Automate,Automate,Automate

AT THE SPEED PEOPLE WANT

TO MOVE, RELIANCE

ON MANUAL TESTING JUST

DOESN’T CUT IT

But Rugged DevOps isn’t simply about automating security testing. As Corman explains, it’s about fundamentally changing the design process to account for security from the start. This means thinking about how to insert security into requirements and architecture discussions before any code is written and instituting threat modeling in the earliest stages of development.

“DevOps is an amplifier—it will amplify the good and the bad,” Corman says. “It’s incumbent upon us to measure twice and cut once. You want to be doing threat modeling and making resilient defensible infrastructure choices.”

As Corman puts it, for too long the app sec community has been used to being “the caboose at the end of the process.” While engaging early in the architecture and requirements phase won’t eliminate after-the-fact testing, security is going to get a higher yield this way.

“That’s what security people are

there for,” Mogull says. “They might not be great programmers, but there are places where they could really help with some threat modeling to identify if something could be really bad in the big scheme of things.”

Getting involved early, and also taking advantage of feature flag and

keyword alerts early in the development process can

help the security team home in on the more

sensitive parts of the code or the architecture that may require a closer look than what automated tests will provide.

“For example, you’ve got to pay

close attention to user authentication and, in

microservices architectures, dealing with component

authentication and authorization,” he says. “So the entire identity and access management model, plus anything dealing with API sanitization and crypto are a huge deal and are probably the top three things that’d keep me up at night when I’m building stuff.”

Engaging atArchitecture and

Requirements

Improve application quality and performance 42%

34%

29%

28%

27%

25%

19%

14%

Improve end customer experience

Increasing use of mobile devices

Pressures to release applications more quickly

Increase collaboration between development and operations teams

Need to develop/deploy cloud-based applications

Increasingly complex IT infrastructure

Greater need for simultaneous deployment across different platforms

DEV

OP

S D

EMA

ND

DR

IVER

S

5%Need to reduce IT costs

As security tries to find its place in the DevOps workflow, one of the best things CISOs can do is try to get security staff embedded within dev or ops teams, or more formally involved in a DevOps pilot project.

“What you see with the DevOps people is that they’re boundary

spanners and they assume that their teammates are going to

bring something to the mix that they can’t do on their own, so they’re looking for your complimentary contribution,” Corman says.

As these security representatives approach their peers

across the organization, Corman suggests they

come at it with the hearts of ambassadors. When you

visit their land you should be observing more than you talk, trying

to understand their culture and speaking their language, while also offering some understanding about your own land. Ultimately the goal should be to look for ways to change the security dynamic to

Participate inCross-Functional

Teams

help speak in the universal language of Rugged DevOps: speed of delivery.

“There should be a really big shift for security people, moving from an approver and requester relationship with DevOps guys to more of a facilitative and partnership relationship,” says Prendergast, “where developers and operations are helping with security and learning along the way, and security is able to mentor and continue to pass responsibility to the DevOps guys, because automation is making it possible to remove the human element from lot of processes.”

As Edwards explained in his OWASP talk last year, security would do well to learn from many QA professionals who have embraced their role as toolsmiths and mentors within the DevOps value chain. As toolsmiths, they do their best to develop a toolset that enables as

much self-service help as possible. And as mentors, security should be helping to identify security champions who can lift some of the everyday security mitigation workload off the security team’s shoulders, allowing them to focus on more strategic work.

Wickett related a story about how cheaply a security empire building can be won. He explained that Ken Johnson, while at Living Social, ran an internal capture-the-flag, bug-finding competition. The prize was a specially designed, very limited-edition t-shirt.The program was simple but effective.

“That’s a unique way to get people interested in security,” he says. “It helps you identify people in your organization who you can tap in the future and whenever the winners wear the shirt, people see it and are more aware of security.”

In its role as toolsmith, the security team can get some of its biggest wins by finding ways to expose operational security tooling to the rest of the organization, Wickett says.

“Security folks have a ton of really great technical knowledge that other technical people just don’t have,” Wickett says. “It’s a shame that it is sort of locked away in that part of the organization, and it’s not exposed or helping their coworkers with self help.”

Finding ways to embed security tooling into the general ops toolkit can

amplify the positive influence of small security teams that otherwise can’t extend their operational reach across

large dev or ops organizations. Giving DevOps teammates

insight from security tools allows them to

get more work done on the infosec team’s behalf.

“You’re putting a broader workforce with a great set

of knowledge and operational capabilities

on the front line of resolving security issues,”

Prendergast says. “Then if they need to escalate up, they’d do that through security, just like they’d escalate operational issues through engineering.”

Open Up Your Toolset to the Rest of the

Organization

Often the biggest obstacles thrown up by security pros about DevOps is what they perceive as continuous delivery’s inability to support separation of duties and audit-worthiness. But as Mogull explains, that’s not necessarily true. To security pros, it might look like developers are pushing code into production, but there’s a whole lot of automated testing and approval processes built into the commit cycles. And everything along the way is typically tracked and measured, not necessarily for security’s sake, but to improve performance along the way.

“If you’re really doing this well, you should have individual user accounts and granular logs of any changes made. That’s essential for a good development process in the first place,” Mogull says. “But it can really help from the Rugged DevOps standpoint.”

Not only will these audit trails help backtrack when security issues pop up, but they’ll also keep the auditors at bay.

“What this has accidentally created is an audit log of who did what, when, where, how and why,” Corman says. He explains that all security has to do is find ways to fine tune this data collection to satisfy specific regulatory compliance demands. “If you’re going to instrument everything, why don’t you instrument things in such a way that they can create evidence at a moment’s notice for any auditor?”

Take Advantageof “Accidental”

Audit Trails

http://www.devops.com https://twitter.com/devopsdotcom https://www.facebook.com/devopscom https://instagram.com/devopsdotcom https://plus.google.com/+Devopsdotcom/posts