On the Varieties of Clouds for Data Intensive Computing 董耀文 1098308101 碩資工一甲 @...

On the Varieties of Clouds for Data Intensive Computing

董耀文1098308101 碩資工一甲 @ Antslab

Robert L. GrossmanUniversity of Illinois at

ChicagoAnd Open Data Group

Yunhong GuUniversity of Illinois at Chicago

Outline

What is Cloud ? Types of Clouds Clouds provide on-demand computing

capacity Experimental Studies Research Questions

What is Cloud?

What is Cloud?

An infrastructure. At scale and reliability of a data center. Provides resources or services.

Google Hadoop Amazon’s EC2

Types of Clouds

Types of Clouds

Architecural Model Loosely coupled commodity computers. Computing instances on demand.

Amazon’s EC2 US $0.10 /1hr 1.0~1.2GHz 2007 Opteron or Xeon processor 1.7 GB memory 160GB disk Moderate I/O performance

Types of Clouds

Tightly Coupled Loosely coupled

Types of Clouds

Architecural Model Computing capacity on demand

Google’s MapReduce TeraSort use1800 machines.

2GHz Xeon process 4GB memory 2 * 160GB IDE disks

TeraSort Sort 100-byte records. ( 1TB data ) Required ~= 891s

1010

Types of Clouds

Architecural Model Open source

Eucalyptus (Elastic Utility Computing Architecture for Linking Your Programs To Useful Systems)

University of California, Santa Barbara Linux & Xen

Hadoop Hadoop MapReduce HDFS HBase

Types of Clouds

Programming Model On-demand computing support any computing

model compatible with loosely coupled clusters. Amazon EC2 MapReduce : < key, value >

Map : map each < key, value >pair into a new pair of < key,value >

Reduce : merges values with the same key Sector/Sphere : User Defined Function(UDF)

Types of Clouds

Management Model Internal vs Hosted Private vs Shared Combinations(Hybrid)

Types of Clouds

Payment Model Pay as you go. Buy. make arrangements with a third party to pay

for the exclusive use of cloud resources for a specified period of time.

Types of Clouds

What’s New? New scale.

Hadoop Google

New simplicity clouds provide. Amazon’s EC2,S3

AMI(Amazon Machine Image)

Clouds provide on-demand computing capacity

Google Cloud GFS (Google File System) MapReduce BigTable

Clouds provide on-demand computing capacity

MapReduce

<key,value> m(Bear)

m(River)

Clouds provide on-demand computing capacity

Clouds provide on-demand computing capacity

UDT(UDP-based Data Transfer) Designed for extremely high speed networks. Concurrent UDT flows share the available

bandwidth fairly. Resides completely at the application level. User defined congestion control algorithms. Easier to traverse the firewall.

Experimental Studies

Experimental Studies

10 Gb/s networks. 30 DELL 1435 computer

4G memory 1TB disk 2.0GHz dual-core ADM Opteron 2212 1Gb/s NIC

Experimental Studies

hadoop sector

Experimental Studies

CreditStone credit card transactions. flags some of the transactions.

Research Questions

Quite easy to use. Develop appropriate network protocols,

architectures and middleware for wide area clouds.

How different clouds can interoperate. Develop standards and standards based

architectures for cloud services. Alternate storage Compute Table services

THANKS