Securing deployment pipeline

NICTA Copyright 2014

Securing the

“Last Mile” of

the Software

Supply Chain

Len Bass, An Binh Tran, Paul

Rimba, Ralph Holz, Liming Zhu


Suppliers from all over

the world

The Software Supply Chain has a

Great Deal of Diversity

2

Operational

System

Open Source

In House

Proprietary

3rd party


Deployment pipeline is the “last mile”

of the supply chain

3

Pre-commit tests

Xpromote to normal

production

Build Image and Perform Integration

tests

UAT / staging / performance

tests

Deploy to production

Commit

...

...

Pre-commit tests

Commit

Developers

Suppliers from

all over the

world Operational

System

• The term “Last Mile” comes from telco and logistics.

• It refers to the difficulties in getting calls, goods, software to the

consumer from a distribution center.

• See http://blogs.atlassian.com/2014/04/practical-continuous-

deployment/ for a discussion of the “last mile” as it pertains to

software

Deployment pipeline

http://blogs.atlassian.com/2014/04/practical-continuous-deployment/







What are the security requirements and

threats to those requirements?

4

• The security requirement we will discuss in this

talk: the image deployed into operation is a valid

image.

• Threats:

– The integrity of the specification of the image has

been compromised

– The image built is not the image specified.

– The image deployed is not the image built.


Steps of our process for hardening the

deployment pipeline

5

• Identify security requirement to be satisfied

• Model steps in deployment pipeline that pertain to that

requirement.

• Identify components that implement the steps.

• Repeat until no vulnerabilities found

– Identify trusted and untrusted components

– Formally analyze model to identify vulnerabilities

– Refine and modify the model through adding trusted

components to resist vulnerabilities found,

• Implement necessary modifications to components


Security Requirements for the

Deployment Pipeline

6

Create an image of

the operational

system

Place an image into

production

The deployed

image is the image

that was created

earlier.

The created

image is the

desired image.

Security Requirements:

Pre-commit tests

Xpromote to normal

production

Build Image and Perform Integration

tests

UAT / staging / performance

tests

Deploy to production

Commit

...

...

Pre-commit tests

Commit

Developers


Trusted and untrusted components

7

• Divide components that are implied in the

deployment pipeline into

– Trusted

• Ideally these components are small enough to be verified via

code verification techniques.

• More likely they are trusted because they are widely used

and problems get reported and patched.

– Untrusted

• All other components

• The trusted components prevent the untrusted

components from violating a security

requirement.


Are there vulnerabilities?

8

Has the

specification been

compromised?

Is this the desired

image?

Image is built by Jenkins & Docker.

Do we trust Jenkins & Docker?

Both Jenkins and Docker are large

open source projects. We do not trust

them.

We need to refine the activity of “build

image” and decompose Jenkins/Docker

to identify trusted and untrusted

portions


Refine Step of “Build Image”

9

Jenkins builds a Docker image

Docker, in turn loads additional portions of

the image

Jenkins then deploys the image into an

environment.

What do we see?

• Docker protects user’s credentials

• Formal analysis identifies additional

files to be protected:

• Application artifacts

• Dockerfile (specification for

building image)

We do not trust Jenkins and Docker


Add image verification step

10

Dockerfile is encrypted.

New Step – Compute Docker

image checksum

Can Docker be trusted with this

step?

No. There are several reported

vulnerabilities.

We intend to create an integrity

checker solely to calculate the

checksum. This should be

formally verifiable.


Second security question

11

Deploy to Production

Is this the image

that was created

earlier?

Deployment is managed by

Jenkins

The questions are

“Do we trust Jenkins?”

“Do we trust AWS

security mechanisms.?”

Answers: no and yes

We must refine model and

Jenkins


Refinement of deployment step

12

• Calculate checksum of created image.

• Verify that deployed image has same

checksum

• These operations are performed by

Jenkins.

• Do we trust Jenkins?

• No – must do more refinement


Summary

13

• Our contribution is the creation of an

engineering process to evaluate/modify the

design of a deployment pipeline

• Our process is based on

– identifying trusted components,

– Analyzing a process using formal methods

– Patching vulnerabilities by creating small trusted

components

– Refining until no vulnerabilities remain

• The specifics of what we have done depends on

the technologies we use but the process will

work for any collection of technologies