Debian Cloud - building the Debian AMIs

Debian-Cloud: EC2 AMIs

James Bromberger

<jeb@debian.org>

Agenda

• What is Debian

• What is AWS EC2

• A meander through block storage for EC2 instances

• Types of images

• Generating & distributing Debian’s AMIs

• Debuab Image lifecycle and security

• If there is time: Debian via Cloudfront CDN

WHAT IS DEBIAN

What is Debian

• Computer Operating System

– 14 CPU/kernel architectures

– 37,500 packages of software

– Translated into a bunch of languages

What is Debian

• Primarily of free and open-source software

– GNU General Public License and many other licenses

What is Debian

• Started 1993

– 21 years old now

• Democratic, volunteer organisation - ~1,000 people (please join!)

– Zero payed employees

WHAT IS AWS EC2

What is AWS and EC2

• AWS = Amazon Web Services

• EC2 = Elastic Compute Cloud– Virtual servers running Linux, Windows, BSD

• Started 2006

• Now with 11 Regions and 52 Edge Locations

• Compute, storage, platform, infrastructure – as-a-service– typically billed by the hour or by the month

Amazon EC2

What is EC2

• Compute requires:

– CPU, Memory (RAM)

– Block Storage (disk)

– Network

– Automation & bootstrapping

– Self-service

instance

Amazon EBS

Amazon VPC

What is EC2

• Amount of CPU & Memory is combined into “instance type”:

– Small

– Medium

– Large

– ...

instance

instance

instance

What is EC2

• Several instance types are grouped into an “instance family”:

– General Purpose (balanced memory:cpu)

– Memory Optimised (more memory:cpu)

– CPU Optimised (more cpu:memory)

– Storage Optimised (more ‘ephemerial’ storage)

– GPU (CUDA, OpenCL)

– Cluster Nodes (10 GB/sec networking and more)

What is EC2

• EC2 instance run on real servers!

instanceinstanceinstanceinstance

Total number of (hyperthread) CPU cores, each dedicated* to an instance

Disk inside the physical server is deemed ‘ephemeral’. Not raid, but is local to CPU and Memory. Different amounts of storage depending on instance type

RAM is dedicated to each instance

Each instance can send a certain number of packets per second

A MEANDER THROUGH STORAGE

Ephemeral (instance) Storage

instanceinstanceinstanceinstance

Persistent (EBS) Storage

instanceinstanceinstanceinstance

Amazon EBS

Persistent (EBS) Storage

instanceinstanceinstanceinstance

Amazon EBS

Persistent (EBS) Storage

Amazon EBS

Mechanical disk

General Purpose SSD (GP2)

Provisioned IOPS (SSD)

Amazon S3

Persistent (EBS) Storage

Amazon EBS

Mechanical disk

General Purpose SSD (GP2)

Provisioned IOPS (SSD)

Amazon S3

AFR of a typical standard HDDDesigned for 99.999% availability (5.26 min/yr)Single instance attach only (currently)1GB..1TB (currently)Your choice of file-systemOptional transparent encryption by AWSNetwork attached to your instance back in the EC2 environment

99.999999999% durabilityReplicated multiple times within the same RegionCheck-summed and re-check-summed periodicallyDesigned for 99.99% availability (SLA at 99.9%)Can be shared with other customers (specific, or all) unless AWS-encryptedCan be used to create a new EBS volumeEBS snapshots cannot be seen in your S3 buckets

Persistent (EBS) Storage

Amazon EBS

Mechanical disk

General Purpose SSD (GP2)

Provisioned IOPS (SSD)

Amazon S3

AFR of a typical standard HDDDesigned for 99.999% availability (5.26 min/yr)Single instance attach only (currently)1GB..1TB (currently)Your choice of file-systemOptional transparent encryption by AWSNetwork attached to your instance back in the EC2 environment

99.999999999% durabilityReplicated multiple times within the same RegionCheck-summed and re-check-summed periodicallyDesigned for 99.99% availability (SLA at 99.9%)Can be shared with other customers (specific, or all) unless AWS-encryptedCan be used to create a new EBS volumeEBS snapshots cannot be seen in your S3 buckets

Persistent (EBS) Storage

instanceinstanceinstanceinstance

Amazon EBS

Instance stop w/EBS

instanceinstanceinstance

Amazon EBS

Instance restart w/EBS

instanceinstanceinstance

Amazon EBS

instanceinstanceinstanceinstance

EBS volume(s) reattached, ephemeral volume(s) blank

TYPES OF MACHINE IMAGES

Amazon Machine Images

• AMI is “golden master”

• Start as many instances as you like*

AMI

instance

instance

instance

instance instance instance

Ephemeral and EBS

• Why is the Ephemeral and EBS storage options important in AMIs?

Your root volume

/ -> persistent (EBS)/ -> transitory (Ephemeral)

Ephemeral and EBS

• Why is the Ephemeral and EBS storage options important in AMIs?

Your root volume

1,000 systems for 24 hours, 8 GB EBS each in SYD: ~$30.85

Ephemeral and EBS

• Why is the Ephemeral and EBS storage options important in AMIs?

Your root volume

1,000 systems for 24 hours, Ephemeral in SYD: $0

Ephemeral and EBS

• Why is the Ephemeral and EBS storage options important in AMIs?

S3 backed AMIAmazon S3

snapshotEBS backed AMI

CPU Architectures

• EC2 currently supports 2 architectures:

S3 backed AMIEBS backed AMI S3 backed AMIEBS backed AMI

Virtualisation Types

• EC2 uses (highly customised) Xen, and supports two virtualisation types:

Para-Virtualization

(threads)

HardwareVirtualization(emulation)

S3 backed AMIEBS backed AMI S3 backed AMIEBS backed AMI S3 backed AMIEBS backed AMI S3 backed AMIEBS backed AMI

Each Region is independentPara-

Virtualization(threads)

HardwareVirtualization(emulation)

S3 backed AMIEBS backed AMI S3 backed AMIEBS backed AMI S3 backed AMIEBS backed AMI S3 backed AMIEBS backed AMI

S3 backed AMIEBS backed AMI S3 backed AMIEBS backed AMI S3 backed AMIEBS backed AMI S3 backed AMIEBS backed AMI

S3 backed AMIEBS backed AMI S3 backed AMIEBS backed AMI S3 backed AMIEBS backed AMI S3 backed AMIEBS backed AMI

US

East

1U

S W

est

1A

P...

Now multiply that by:

• Wheezy

• Jessie

• Sarge

• ...

• 2 architectures

• 2 virtualisation types

• 2 root volume types

• 11 Regions

• 3 Debian releases

= 198 images

(Plus images currently being end-of-lifed, experimented with, and used for other purposes)

Current Debian AMIs: Squeeze (6)

Architecture EBS Backed S3 Backed

32 bit PVM Yes

64 bit PVM Yes

32 bit HVM

64 bit HVM

Current Debian AMIs: Wheezy (7)

Architecture EBS Backed S3 Backed

32 bit PVM Yes

64 bit PVM Yes Yes

32 bit HVM

64 bit HVM Yes (experimental)

Future Debian AMIs: Jessie (8)

Architecture EBS Backed S3 Backed

32 bit PVM

64 bit PVM Yes

32 bit HVM

64 bit HVM Yes Yes*

Two ways of creating AMIs

Start from scratch

• Uses a fresh, blank volume, install as a debootstrap

Update existing

• Start existing instance, customise, create new image

EBS Backed AMI overview

instance

volume

/

volume

/target

snapshot

EC2 API

Endpoint

AMI

Let’s create a Jessie image

• Fire up an existing instance (easiest is to use an existing Debian AMI)

• Install git, debootstrap, python-boto, python-jsonschema, and some other python bits

– Configure your AWS IAM credentials for boto

• Grab bootstrap-vz from Github

DEMO

Distributing images globally

Each region has separate copies of AMIs

Distributing images

Three “groups” of Regions:

• GovCloud

• Beijing

• Everywhere else*

Debian AWS Accounts

Region AWS Account ID

Beijing 673060587306*

Gov Cloud 256493402735**

Standard Regions 379101102735

Community Shared AMIs

• Un-vetted by AWS

– Trojan horses

– Left over SSH keys in other accounts

– Cron jobs that go bump in the night

• Anyone can share any AMI under their control (provided they have access within their AWS account to do so – IAM Policy)

– Caveat emptor

Pushing images to Marketplace

Vendor AWS

Account ID

Vendor Display Name

Product ID Version ID ASIN SKU Software by

Title Version Title

Release Notes

Short Description

Description Highlight1

IMAGE LIFECYCLE AND SECURITY

AMI Lifecycle

Our aim is to keep the final point release AMI available for each Debian major release, starting from Squeeze:

• 6.0.10

• 7.7

AMI Lifecycle

Wheezy 7.4

Wheezy 7.5

Wheezy 7.6

Wheezy 7.6.aws.

1

Wheezy 7.6.aws.2

Wheezy 7.7

Try to keep a 2 – 5 week overlap for point releases, then un-share for a period, then delete

Time

Occasionally security releases that are urgent in BASE images (AMIs) force additional version numbers out of step with Debian. This was shellshock,

Security in base images

• EC2 instances may be deployed such that they don’t have direct access to fetch updates

• Administrators may chose not to install updates unattended

Debian AMIs in US East 1

Workflow overview

1. Generate AMIs in US East 12. Tag AMIs and Snapshot3. Test image in US East 14. Copy to all Standard Regions (python script)5. Mark AMI and Snapshot as Public (python script)6. Generate in Beijing and Gov Cloud, tag, mark public7. Generate signed message to the Debian-cloud mailing list, update wiki8. Wait a few days (for bugs to surface), then push to AWS Marketplace9. Announce deprecation of previous versions (typically 3 – 5 weeks notice)

in signed email to Debian-cloud ML10. After elapsed period, remove public sharing from AMI and Snapshots

(python script)11. A day or so later, deregister the AMI and delete the snapshot (python

script)

What’s new in Jessie EC2 images

• Single Root IO Virtualisation (Enhanced Networking)

• Multiple Network Interfaces (ENI)

• Multiple sub-interfaces

• AWS CLI and python-boto installed in base image

• Cloud-init (since Wheezy 7.4)

Cloud-init

• Insert this as “User Data”

• Can be embedded into CloudFormationtemplates

#cloud-config

package_update: true

package_upgrade: true

package_reboot_if_required: true

packages:

- pwgen

- less

locale: fr_FR.UTF-8

ssh_authorized_keys:

- ssh-rsa AAAAB3Nz....89dGp5 me@mykey1

- ssh-rsa AAAAB3Nz....89dGp5 me@mykey2

final_message: "The system is finally up,

after $UPTIME seconds"

DEBIAN ON CLOUDFRONT CDN

Debian Archive via CDN

• Default apt sources.listfor EC2 images uses cloudfront.debian.net

• Primarily for EC2 instances, but is active in all 52 Cloudfrontlocations world-wide

CloudFront

Cloudfront.debian.net

• Each edge location is independent of all others

edge location

edge location

edge location

traditional server

Cloudfront.debian.net

• However, Debian HTTP servers don’t put any cache advisory headers on how long objects (files) may be cached for; some of these are quite volatile, and some are very stable

edge location

edge location

edge location

traditional server

Cloudfront.debian.net

• Luickly, Cloudfrontsupports “Cache behaviours”, mapping different URL paths to alternate origin servers

edge location

edge location

edge location

traditional server

Cloudfront.debian.net

• Default: => S3 bucket

• /debian/ => ftp.us.debian.org

• /debian/dists => my proxy server edge location

edge location

edge location

http://ftp.us.debian.org

bucket

instancesElastic Load

Balancing

Jessie on Ice(ie, Jessie is frozen)

Debian 9:Stretch

Debian 10: Buster

1.1 (‘96)

1.2

1.3

2

2.1

2.2

3 (2002)

3.1

4

5

6

7

8 (2015)

James BrombergerE: jeb@debian.orgT: @JamesBrombergerL: https://www.linkedin.com/in/jamesbrombergerPerth, Australia

GPG: 4096R/9D85C53C 2011-11-29Key fingerprint = 8591 20FE 0D9F A6A5 B054 C775 AEC8 2874 9D85 C53C

https://github.com/JamesBromberger/bootstrap-vz

https://wiki.debian.org/Cloud/AmazonEC2Image

https://aws.amazon.com/marketplace/pp/B00AA27RK4

https://lists.debian.org/debian-cloud/