AHM 9-11 Nov 2009Bologna Engineering Ingegneria Informatica S.p.A.
INFSO-RI-223782
Andrea Turli e Daniele Rufini
9-11 nov 2009
AWS Submitter Analysis and Design
2
Bologna, 9-1 November 2009
ETICS SubmissionManager and AWS
Power of Cloud ArchitecturesFrom “GrepTheWeb-AWS white paper”• Processing Pipelines• Batch processing systems
o Nightly builds, automated unit/deployment testing • Websites
Amazon AWS • Amazon EC2 for running a large distributed processing
cluster on-demand• Amazon S3 for retrieving/storing persistent datasets• Amazon SQS for buffering requests - loosely coupled• Amazon SimpleDB for storing data - status, log
3
Bologna, 9-1 November 2009
Architectural Abstraction : Level 1
AWS as a Grid
Input System• sendJob : send ETICS jobs to Amazon Cloud • getStatus : receive information about the status of ETICS
job.Output System• GetOutput : result of processing ETICS jobs.
4
Bologna, 9-1 November 2009
Architectural Abstraction : Level 2
sendJob
getSatus getOutput
5
Bologna, 9-1 November 2009
Architectural Abstraction : Level 2
The ETICS Submitter• send a message in to the SQS queue for each request
The controller (EC2 instance):• takes a message from the SQS queue • use SimpleDB to store Job information• launches EC2 instances (worker nodes) to run the job• monitor the job status• Store the output on S3
6
Bologna, 9-1 November 2009
Architectural Abstraction : Level 3
7
Bologna, 9-1 November 2009
Architectural Abstraction : Level 3
On application start• SQS queue creation and start all the controller
Launch Controller• pick up message from launch queue, populated by ETICS• send the job request to EC2 cluster• update status in SimpleDB (ready)• delete the message in the launch queue• insert a message in monitor queue
Monitor Controller• pick up message from monitor queue• ping EC2 cluster to check the job status• take/update the status in SimpleDB (running)• insert a message in delete queueShutdown Controller• shutdown EC2 instances• pick up message from shutdown queue• update the status in SimpleDB• delete a message after processing