38

(BAC307) The Cold Data Playbook: Building the Ultimate Archive Solution in Amazon Glacier | AWS re:Invent 2014

Embed Size (px)

Citation preview

Page 1: (BAC307) The Cold Data Playbook: Building the Ultimate Archive Solution in Amazon Glacier | AWS re:Invent 2014
Page 2: (BAC307) The Cold Data Playbook: Building the Ultimate Archive Solution in Amazon Glacier | AWS re:Invent 2014
Page 3: (BAC307) The Cold Data Playbook: Building the Ultimate Archive Solution in Amazon Glacier | AWS re:Invent 2014
Page 4: (BAC307) The Cold Data Playbook: Building the Ultimate Archive Solution in Amazon Glacier | AWS re:Invent 2014

Archive:

Data retained for the

long term – for

compliance or potential

future reference

Backup:

Data retained to

support near-term

business continuity

Page 5: (BAC307) The Cold Data Playbook: Building the Ultimate Archive Solution in Amazon Glacier | AWS re:Invent 2014

Archive:

Data retained for the

long term – for

compliance or potential

future reference

Page 6: (BAC307) The Cold Data Playbook: Building the Ultimate Archive Solution in Amazon Glacier | AWS re:Invent 2014

Archive:

Data retained for the

long term – for

compliance or potential

future reference

Page 7: (BAC307) The Cold Data Playbook: Building the Ultimate Archive Solution in Amazon Glacier | AWS re:Invent 2014

What is archival?: Traditional archival approaches

Page 8: (BAC307) The Cold Data Playbook: Building the Ultimate Archive Solution in Amazon Glacier | AWS re:Invent 2014
Page 9: (BAC307) The Cold Data Playbook: Building the Ultimate Archive Solution in Amazon Glacier | AWS re:Invent 2014

Metered usage:

Pay as you go

No capital investment

No commitment

No risky capacity planning

Avoid opex and risks of

physical media

handling

Control your

geographic locality for

performance and

compliance

Page 10: (BAC307) The Cold Data Playbook: Building the Ultimate Archive Solution in Amazon Glacier | AWS re:Invent 2014
Page 11: (BAC307) The Cold Data Playbook: Building the Ultimate Archive Solution in Amazon Glacier | AWS re:Invent 2014

Amazon

Glacier

Application

HTTP(S) /

REST APIs

Data collection Local index

Compression

Raw data

Page 12: (BAC307) The Cold Data Playbook: Building the Ultimate Archive Solution in Amazon Glacier | AWS re:Invent 2014

Amazon

Glacier

Lifecycle policies

Amazon

S3

HTTP(S) /

REST APIs

Raw data

Page 13: (BAC307) The Cold Data Playbook: Building the Ultimate Archive Solution in Amazon Glacier | AWS re:Invent 2014
Page 14: (BAC307) The Cold Data Playbook: Building the Ultimate Archive Solution in Amazon Glacier | AWS re:Invent 2014
Page 15: (BAC307) The Cold Data Playbook: Building the Ultimate Archive Solution in Amazon Glacier | AWS re:Invent 2014
Page 16: (BAC307) The Cold Data Playbook: Building the Ultimate Archive Solution in Amazon Glacier | AWS re:Invent 2014

Create vault (films)

1

Configure access policies

2

ArchiveApp user policy

Effect:Allow

Resource:

arn:aws:glacier:<accountId>:vaults/Films

Action: glacier:UploadArchive

3

Upload archivesUploadArchive(data) ->

Archive ID

Page 17: (BAC307) The Cold Data Playbook: Building the Ultimate Archive Solution in Amazon Glacier | AWS re:Invent 2014

Initiate job

ArchiveId: AE99F…

Vault: Films -> Job ID

1

3-5 hours for job completion

2

3

Job completion

notification

4

Download output

Page 18: (BAC307) The Cold Data Playbook: Building the Ultimate Archive Solution in Amazon Glacier | AWS re:Invent 2014
Page 19: (BAC307) The Cold Data Playbook: Building the Ultimate Archive Solution in Amazon Glacier | AWS re:Invent 2014

1. Primary storage hardware (primary / remote site)

2. DR / remote site storage hardware

3. Raw to utilized storage (both primary and DR)

4. Storage growth (cost of upgrades)

5. Storage management software and 3rd party tools

6. Professional services

7. Hardware maintenance

8. Software maintenance

9. Backup software

10.Backup hardware (primary / remote site)

11.Offsite tape storage / vault

12.Archive software

13.Archive hardware

14.Power

15.Cooling

16.Space

17.Labor

18.Cost of capital

19.Training

20.Asset depreciation

21.Migration

22.Decommission / remove

23.Recycle

Page 20: (BAC307) The Cold Data Playbook: Building the Ultimate Archive Solution in Amazon Glacier | AWS re:Invent 2014
Page 21: (BAC307) The Cold Data Playbook: Building the Ultimate Archive Solution in Amazon Glacier | AWS re:Invent 2014
Page 22: (BAC307) The Cold Data Playbook: Building the Ultimate Archive Solution in Amazon Glacier | AWS re:Invent 2014

Checksum 2

Checksum 1

File 2

Checksum 3

. . .

Local index

File 1 offset

File 1

File 2 offset

File 3 offset

Index/directory

Checksum & metadata

Checksum & metadata

Checksum & metadata

Archive

Page 23: (BAC307) The Cold Data Playbook: Building the Ultimate Archive Solution in Amazon Glacier | AWS re:Invent 2014

AWS Direct

ConnectDedicated bandwidth between

your site and AWS

InternetTransfer data in a secure SSL tunnel over

the public Internet

AWS Import/ExportPhysical transfer of media into and

out of AWS

Page 24: (BAC307) The Cold Data Playbook: Building the Ultimate Archive Solution in Amazon Glacier | AWS re:Invent 2014

Arc

hiv

e

Parallel Uploads

Parts

Page 25: (BAC307) The Cold Data Playbook: Building the Ultimate Archive Solution in Amazon Glacier | AWS re:Invent 2014
Page 26: (BAC307) The Cold Data Playbook: Building the Ultimate Archive Solution in Amazon Glacier | AWS re:Invent 2014

Media Cloud ServicesOn-demand cloud-based solutions designed to capture,

collect, create, manage, share, produce, and archive high-

value, high-resolution content.

What is Sony MCS?

Page 27: (BAC307) The Cold Data Playbook: Building the Ultimate Archive Solution in Amazon Glacier | AWS re:Invent 2014

Your professional, all-in-one media production

workspace.Instantly connect your team to your media files, hardware,

and applications for streamlined communication, faster

turnaround times, and reduced production costs.

Ci

Page 28: (BAC307) The Cold Data Playbook: Building the Ultimate Archive Solution in Amazon Glacier | AWS re:Invent 2014

TELEVISION EDITORIAL & LEGAL REVIEW

MAJOR MOTION PICTURE DAILIES PREVIEWING

SMALL BUDGET PRODUCTIONS & ORIGINAL CONTENT

SHARE CONTENT BETWEEN STATIONS

ARCHIVECONTENT

STOCK FOOTAGE STORAGE & OPERATIONS

THOUSANDS OF ASSETS

DISTRIBUTED USERS

TRACKING/ MONITORING

Page 29: (BAC307) The Cold Data Playbook: Building the Ultimate Archive Solution in Amazon Glacier | AWS re:Invent 2014
Page 30: (BAC307) The Cold Data Playbook: Building the Ultimate Archive Solution in Amazon Glacier | AWS re:Invent 2014

Same challengeCREATIVE DRIVERS BUSINESS DRIVERS

Better Image QualityHigh dynamic range

High resolution

More Raw FootageHigher shooting ratios

High frame rate

3D / 4K

More ContentGreater diversity

More collateral marketing

Greater Distribution NeedsMulti-screen expectation

Physical, satellite, Internet

Page 31: (BAC307) The Cold Data Playbook: Building the Ultimate Archive Solution in Amazon Glacier | AWS re:Invent 2014
Page 32: (BAC307) The Cold Data Playbook: Building the Ultimate Archive Solution in Amazon Glacier | AWS re:Invent 2014
Page 33: (BAC307) The Cold Data Playbook: Building the Ultimate Archive Solution in Amazon Glacier | AWS re:Invent 2014

0

2

4

6

8

10

12

14

FY13 FY14 FY15 FY16 FY17 FY18

Storage (PB)

Storage (PB)

Growth projections

Page 34: (BAC307) The Cold Data Playbook: Building the Ultimate Archive Solution in Amazon Glacier | AWS re:Invent 2014

Cost comparison

$-

$1,000,000.00

$2,000,000.00

$3,000,000.00

$4,000,000.00

$5,000,000.00

$6,000,000.00

$7,000,000.00

FY13 FY14 FY15 FY16 FY17 FY18

On Prem

Ci/AWS

Page 35: (BAC307) The Cold Data Playbook: Building the Ultimate Archive Solution in Amazon Glacier | AWS re:Invent 2014

Transfer to

S3

Copy to

archive

bucket

Ingest

pipeline

Register in

Ci

Worker

updates Ci

status

Page 36: (BAC307) The Cold Data Playbook: Building the Ultimate Archive Solution in Amazon Glacier | AWS re:Invent 2014

Custom

app calls

Ci API

Ci calls S3

API

Worker

monitors

Ci status

updated

Update Ci

status

Page 37: (BAC307) The Cold Data Playbook: Building the Ultimate Archive Solution in Amazon Glacier | AWS re:Invent 2014
Page 38: (BAC307) The Cold Data Playbook: Building the Ultimate Archive Solution in Amazon Glacier | AWS re:Invent 2014

Please give us your feedback on this session.

Complete session evaluations and earn re:Invent swag.

http://bit.ly/awsevals