Embed Size (px)
Citation preview
An Overview of the Open Cloud Consortium
Robert GrossmanOpen Cloud Consortium
OMG Cloud Computing Interoperability Workshop
July 13, 2009
www.opencloudconsortium.org
This talk represents my personal opinions and does not necessarily represent a consensus position of the Open Cloud Consortium or positions of any of its members.
2
Part 1.
Introduction
3
Two Types of Clouds
On-demand resources & services over a network at the scale of a data center
On-demand computing instances (IaaS)– IaaS: Amazon EC2, S3, etc.; Eucalyptus– supports many Web 2.0 applications/users
On-demand cloud services for large data cloud applications (Large Data PaaS)– GFS/MapReduce/Bigtable, Hadoop, Sector, …– Manage and compute with large data (say 100+ TB)
4
Delivering On-Demand Computing Instances
For both architectures, you are interested in filling a data center.
5
Cloud Storage Services
Cloud Compute Services (MapReduce, Variants & Generalizations)
Table-based Data (BigTable) Services
Relational Data Services (Pig, etc.)
App App App App App
App App
App App
App App App…
Activities Currently Focused Around Five Use Cases
1. Moving an existing cloud application from Cloud 1 to Cloud 2 without changing the application.
2. Providing surge capacity for an application on Cloud 1 using any of the Clouds 2, 3, … (without changing the application).
Cloud 1 Cloud 2
1. Migrate / port2. Surge / burst
Large Data Cloud Use Cases3. Moving a large data cloud application from
one large data cloud storage service to another.
4. Moving a large data cloud application from one large data cloud compute service to another.
Large Data Cloud Storage Services
Large Data Cloud Compute Services
App 1 App 2
Inter-Cloud Use Case
5. Inter-cloud communication between two HIPAA compliant clouds.
HIPAA Cloud 1
HIPPA Cloud 2
Part 2.
The Open Cloud Consortium (OCC)
www.opencloudconsortium.org
9
www.opencloudconsortium.org
OCC is a member driven organization that:1. Supports the development of standards.2. Supports the development of benchmarks.3. Supports reference implementations for
cloud computing, preferably open source. 4. Sponsors workshops and other events related
to cloud computing.
10
OCC Members
Current Members– Aerospace, Cisco, MIT Lincoln Labs, Northwestern
University, Open Data Group, Sector Project, University of Chicago, University of Illinois at Chicago and Yahoo.
Current Contributing Members & Partners– CalIT2, Johns Hopkins University, National Lambda
Rail
OCC Structure OCC is a consortium operated by a not-for-profit. OCC is member-driven. OCC is oriented to companies and organizations,
not individuals. Governance structure is available from web site.
12
OCC Working Groups
Members set up working groups.1. Large Data Clouds Working Group2. Open Cloud Testbed Working Group.3. Standard Cloud Performance Measurement
(SCPM)- Working collaboratively with CCIF
4. Information Sharing and Security Working Group.
Part 3. Large Data Clouds Benchmarks and Interoperability Working Group
14
*OCC is not affiliated with the Hadoop Project.
*
Cloud Benchmarks
To help inform purchase decisions. To help inform the design of cloud
applications. To help inform cloud architecture decisions.
Benchmarks for Large Data Clouds
Until recently, the only benchmark used was Terasort (sorting 10 billion 100 byte records)
Replaced by Gray Sort and Minute Sort Gray Sort tries to maximize TB / min sorted on
100 TB or more of data. Hadoop holds the current Gray Sort and
Minute Sort records. Problem: sort is just one of the types of work
load for analytic applications
MalStone
MalGen – generates synthetic data with realistic distributions.
MalStone A & B – “stylized” computations that can be used as benchmarks for architectures, software and systems for large data clouds.
Open source and available at code.google.com/p/malgen
17
MalStone B
time18
dk-2 dk-1 dk
sites entities
MalStone B Benchmark
MalStone BHadoop v0.18.3 799 minHadoop Streaming v0.18.3 142 minSector v1.19 44 min# Nodes 20 nodes# Records 10 BillionSize of Dataset 1 TB
19
ThriftStore
Each cloud application can access any of the large data cloud storage services without changing any of its APIs.
Based upon the open source Thrift project.
Hadoop DFS KFS Sector DFS
ThriftStore
Cloud App 1 Cloud App 2 App n…
Part 4.
Standard Cloud Performance Measurement (SCPM) Working Group
21
Establish Benchmarks Appropriate for Four Use Cases
1. Moving an application between two clouds.2. Obtaining burst instances from multiple cloud
service providers for a private/public hybrid application.
3. Moving a large data cloud application to another large data cloud storage service.
4. Moving a large data cloud application to another large data cloud compute service.
Hierarchy of Benchmarks
Benchmarks for basic operations Kernel Benchmarks Basic functions and routines Stripped down versions of programs Full applications
Goal is to develop small suite of these benchmarks for four use cases.
Please Get Involved…
The Standard Cloud Performance Measurement (SCPM) Working
Group is just starting, please get involved.
Part 5. OCC Sponsored Events
25
OCC/CCIF Jointly Sponsored Events
International Summit on Governmental Cloud Computing– Ottawa's Westin Hotel on October 6, 2009
Cloud Computing and Its Applications 2009– Chicago on October 20-21, 2009
We Welcome New Members
Companies and organizations are welcome to join the Open Cloud Consortium (OCC)www.opencloudconsortium.org/membership.html
Join one of our working groups– Large Data Clouds Working Group– Standard Cloud Performance Measurement
(SCPM) Working Group– Information Sharing & Security Working Group
