Faster Secure Software Development with Continuous Deployment

Nick Galbreath IPONWEB nickg@iponweb.net

PHDays May 24, 2013 Moscow, Russia

Follow Along

Latest version posted online:  

http://slidesha.re/199wo7Qhttp://client9.com/20130524

Nick Galbreath (@ngalbreath)

! Spoken at Black Hat, DEFCON, OWASP ! Book on cryptography ! but really... ! Engineering Management and Software

Development for high growth startups. ! Personal site http://www.client9.com/

libinjection

! Author of libinjection ! A very different way of doing SQLi detection ! Right now in another room, Vladimir

Vorontsov is showing how to bypass it (to be fixed shortly)

! Check it out and file bugs on github http://libinjection.client9.com/

! Found a bypass? What database and query is needed to exploit it?

RU ♥ SQLi

IPONWEB

! customized online advertising infrastructure and exchanges

! engineering offices in Moscow, with business offices in London, New York and Tokyo.

! YEAH IN MOSCOW ! YEAH WE ARE HIRING ! Send email to nickg@iponweb.net

Well that's a bold statement...

Fixing Security byFixing DevelopmentUsing Continuous Deployment

and here's another

For web applications, our release-based software development lifecycle is still based on a pre-Internet model and is harmful to organizations andparticularly harmful for security.

What needs fixing?

! SQLi dropped from #8 to #14 in the latest White Hat "The State of Web Security" report. Good news, right?

! This means SQLi is only 7% of websites. That's 1 in 15.And this is the #14 vulnerability!

! And time to fix was on average 196 days. That's embarrassing.

Veracode claims 32% of incoming web applications have SQLihttps://info.veracode.com/state-of-software-security-report-volume5.html

https://reg.whitehatsec.com/WPstats0513

Even worse...

! Number 1 driver to fix security problems...compliance.

! Number 1 reason to not fix security is...compliance.

! Not.. ! keeping our employees and customers safe ! protecting corporate interests. ! improving quality ! being good at what we do.

Security Products #1 .. in security bugs VeraCode: State of Software Security, V4 December 2011

Security Product 74% Fail Rate

Let's Just Give Up

! “You could spend all your resources chasing such things as this,” William Ribich, the former president of Technology Solutions Group [ QinteliQ ], said in an interview in January. Ribich, who retired in November 2009, shortly after the discovery of a major data theft, said he needed to balance the uncertain risk that the hackers could use what they stole against a growing shopping list of security products and consulting fees.

! "You finally have to reach a point where you say ’let’s move on,’” he said.

http://www.bloomberg.com/news/2013-05-01/china-cyberspies-outwit-u-s-stealing-military-secrets.html

^^^ The Russian and Chinese hackers did not move on ^^^^.

I would call that broken

! But preventing SQLi isn't a technically hard problem.

! And most security patches are very small. ! How did we get here?

Software Product Model

! Code flows between functional groups ! Product Managers spec code ! Engineers write code ! QA engineers tests code ! Release engineers package code ! Operations runs code ! ... and Security does something too.

High Distribution Cost

! The Software Product Model is designed for software where the cost of distribution is high. "High" might be financial, risk, time, resources, customer annoyance. ! Retail, physical product, CD/DVD ! Embedded of Exotic Hardware ! Safety, Medical or Defense Systems ! Operating Systems (desk or phone) ! Homework (1-time deploy)

Software Product Lifecycle

Change to Production

Web Applications Year 2000

! Mostly followed Software Product Model since that's all we knew.

! High barrier to entry ! Specialized Hardware, Software and

People needed to get started. ! Lots of engineering needed to keep

things running. ! (side note: CERT/CVE started in

1999)

True Story #1

! "Can't push out the spelling error fix – it's too risky"

! "That code as already been through QA, it's locked down."

! "Product has to prioritize that change, else we aren't touching it."

True Story #2

! We'll do an iteration, where we try to fix as many things as possible.

! This won't be a scheduled iteration,it will be done because so many things are piled up.

! So the spelling error will get fixed... uhh, who knows when.

Web Applications 2013

! Almost no barrier to entry ! Commodity hardware ! Programming not that hard ! Scaling problems can

be mostly outsourced (mostly)

Cost of Distribution 2013

! Frequently no compile step or it's very fast.

! Moving to production a few kilobytes or megabytes of code over 1Gbps, 10Gbps link.

! In other words... free

Failure is very different however

! Most web applications are data-driven. ! Frequently have social features, APIs,

user-generated content. ! Failures might be due to algorithmic

problems... but... ! Most likely to due to user input, bad data

in database or operational load. ! this means data in past can cause

problems in the future.

Releases and Problems

! When a web-release goes out, and has problems....

! Next week is spent tracking down who changed what, where.

! Re-QA ! Re-Push ! meanwhile new code is piling up.

When SPM meets Web Apps

! A long time between code being written and code being released.

! Might be weeks or months ! Feedback loop between code-in-dev

and code-in-production is broken ! When security or bug reports come in,

the author is likely on a different project.

Hypothesis

! It is impossible to simulate the production environment in development, either due to operational differences or data differences.

! No amount of QA or Security Testing can prove you don't have bugs, vulnerabilities, or cause severe operational problems.

! You have bugs and vulnerabilities,right now, in your application.

Impedance Mismatch

! Easy to write code, + ! Long release cycles + ! Security as an end-of-line or out

of band process == ! no one cares

! Something is going to break, and most people don't care.

Security Problems Will Happen

MTTF ≪ ∞

So the Answer is...

! Going slower? I'm sure your boss will love that suggestion

! More steps and process? In other words, slower.

! Asking for more people? Sure but good luck hiring them. Doesn't scale.

! Asking for more products? Since the others have worked so well.

Continuous Deployment

! Also known as Continuous Delivery. ! A System of Software Production

Characterized by Numerous Small Changes the Production Environment, initiated by the author of the change.

You change it, you push it to prod.

Deployment != Feature Release

time

chan

ge

New code goes out all the time. New features get turned on in a separate process.

"Writing Software"

! Software Developers think their job is writing software.

! And so, they love to make things perfect before anyone else sees it.

! Impolite: "data hiding" ! code is hiding on developer's computer ! or on some branch ! in other words invisible until it's ready.

Actually

! The software engineer's job is actually writing running software, that works well.

! This idea is so alien, that companies have to remind the engineers of this.

Rackspace Haiku

writing code is hardif you cannot deploy it does not matter

@paulvx from DevOpsDays Austin 2013

Facebook's Analog Labs Poster

"Move Fast and Break Things....

Except "Push" (deployment system)

via http://mitadmissions.org/blogs/entry/move_fast_and_break_things

Etsy Mantra Just Ship

Today's goal

! but for today the goal is getting the developer to care about their code in production.

! If you don't have that, I don't think you can really solve security problems.

How does this work?

! Really? Developers push their own code out?

! How is this not a disaster. ! How is this not a security disaster?

The Deploy Button

! What is you had a button that said "DEPLOY" ! That pushed to production, whatever

is current in your source control system. ! And took about a minute ! The change and who pressed the

button is logged, but that's it.

Part 1: Fear

! No one is going to push it ;-)

! Meanwhile code is piling up

Real example: A new hire I had at Etsy was afraid of deploying an HTML change that they made. "But I don't want to break the site!"

Part 2: First Push

! Someone brave will press the button ! And very likely the site will explode,

and a rollback will need to be done. ! They'll know since someone else will

have told them.

Part 3: With Graphs

! Let's get all those operational graphs out in the open. And put them right next to the button.

http://codeascraft.com/2011/02/15/measure-anything-measure-everything/

Part 4: Push #2

! Repush ! Site might still explode ! But the developer is aware and

can rollback.

Take 5: Isolation

! Hmmm, the developer notices that in the change set, a million things are going out.

! Maybe just pushing out a smaller change will help isolate the issue.

Take 6: Success!

! Yes, the developer just pushed out some code and made the site better.

! The secret about continuous deployment is small changes that can be easily understood.

Take 7: Dark Pushes

! Now we got some bugs fixed, let's push a feature.

! First let's push out all the supporting files. Since they aren't being called, they do nothing and are safe to push out.

! Now everyone can see them

Take 8: Getting the feature live

! Instead of "all at once", we slowly ramp up a feature.

! if (user_id % 20 == 0): do new feature;

! we change change the percentage easily with another code push.

! or turn it down. Much nicer change log. ! While the site didn't explode, it's hard to see if

the feature is being used or not.

Take 9: Application Level Graphs

! Allow developers to instrument their code so they can see what is happening in production. ! Enter StatsD and other

UDP-based tools ! Enter centralized logging and in-

application method to make it easy to log problems.

Take 10: Communication

! So far good for one developer. ! To scale up, you'll need a system to allow

developers. ! IRC-like tools work well (e.g. "the push

channel") – skype, jabber, hangouts, etc

Along the way

! Expose production logs to developers ! Add in a staging-step where the code

goes to faction of the cluster, so developers can test with real traffic

! Try to make development closer to prod. ! Make "smoke tests" to catch basic errors ! Add syntax checkers to eliminate obvious

issues. ! Use static analysis to find bugs

Mistakes will happen

! Do postmortem analysis ! Everyone thought they were doing

the right thing at the time. ! "How can the environment be

changed to prevent this" and build tools to enforce it. ! (Rarely can you truly change people)

Sounds good, but what about...

That guy who pushes at 3am

! Courtesy and convention will converge very quickly when the site goes down at 3am and the developer starts getting calls ;-) ! Of hours pushes of course can

happen, when they notify operations.

What About Code Reviews?

! Yes, please do them. ! Nothing here prevents code reviews. ! In fact code reviews are easier since ! they are small ! they are in mainline not some branch

What about Security Reviews

! Please do them. ! Nothing here eliminates

architectural planning or review. ! This actually doesn't change the

SDLC very much.

What about Agile Methods

! (everyone seems to have a different idea of what Agile is but..)

! Agile methodologies typically work to improve the business spec / development cycle. (are you building what the customer wants)

! But doesn't address code deployment. ! They are complimentary practices.

What about Customer Service?

! "Don't they freak out with all the changes?"

! Remember: deployment != feature release ! Most deployments do very little from the

customer point of view ! Feature releases (frequently controlled by

ramp-ups or flags) always needs to be coordinated with product and customer service.

What about Compliance? PCI?

! Let me tell you about compliance... ! mechanism not policy ! compliance is a lot easier when it's done

every day instead of a once-a-year audit.

Back to Security

Obvious Benefit to Security

! Security patches can go out quickly ! You know this since they are now

just part of a normal development cycle and code goes out regularly. ! Why not clear out those low-priority

security problems?

More Importantly

! That Engineer who previously did not push code is now sensitized that their code has consequences and are responsive and empowered to fix it.

! It’s amazing how interested engineers become in security when you find problems with their code when they are able to fix it quickly themselves.

New Security Math

! Instead of focusing only on increasing MTTF, which will never be infinite ! more firewalls, more process, more magic

! You can focus on how fast can you detect faults, and how fast can you fix them.

! How low can you go? ! MTTD - Mean time to Detect ! MTTR - Mean time to Repair

Hack The Stack

! A side effect of this you now have tools to repurpose for security and monitoring of production ! Note that most changes are not

security problems.

Logging

! Due to allow developers to see application logging, it's now very easy to instrument the application to log security events. ! Or add logs to times when you are

under attack.

Graphing

! Make dashboards of ! SQLi and XSS attacks ! Every type of log-in failure ! Core Dumps ! Database Syntax Errors

Static Analysis

! You now have a place to insert them. ! Work with QA group to add more code

quality tests.

Post-Commit Checks

! Alert on when sensitive areas of the code are changed (auth, login)

! Alert on crypto usage (why is developer using MD5.. hmmm)

! Alert on your programming language's "dangerous functions"

This allows you to engage the developer at the start of the cycle.

Faster is Better

! You could do most of this in a normal release-cycle software lifecycle. ! The difference is you are finding

problems at the start instead of 10m before the launch and telling everyone to stop. ! The feedback loop works.

New Roles, Less Silos

! Developers: works with operations ! QA: works on building systems for

testing, to empower others to write better tests ! Release Engineering: tools to enable

code to flow faster ! Security: in-house consultancy,

secure-by-default architecture, monitoring

Getting Started

Goal: 50% reduction in deploy time

! Whatever your state of deployment is, no matter how many people are involved, no matter how long it currently takes, make a goal of cutting it in half. ! This is an easy sell to management

just on cost basis. ! Everything else flows from this.

Mechanism not Policy

! Strive for the fastest deployment mechanism for possible

! But you define the "continuous" in Continuous Deployment

! Yes, Etsy was 60+ deploys per day, with each having multiple authors.

! Current gig? we have rules of no more than 3 per week since our customer have asked for that, and only deployed at "low-tide"

In Other Contexts

In other contexts: Operations

! How fast can you deploy OS changes to you production environment? ! How fast can you deploy router

changes? ! How fast can you deploy patches to

the desktop

You probably don't do it that often since it's really painful and time consuming! That's exactly the problem.

In other contexts: software product

! here "production" might be getting code into the main branch and running automated build / test.

! It's the flow of code: little changes vs big.

In other contexts: silicon

! Continuous deployment already done for silicon! wut?

! Only small changes, with tests are allowed to be committed!

! Big changes are rejected.Learned the hard way that big changes are completely unmanageable

One more thing...

Your Attackers Use Continuous Deployment

Continuous Deployment Start with 50% Reduction Build the Deploy Button

Thanks nickg@client9.com

http://client9.com/20130524