1

Accelerating Research 10-minute Vignette for 2012 Amazon re:Invent Conference

chris@bioteam.net1

2

I’m Chris.

I’m an infrastructure/bio geek.

I work for the BioTeam.

Twitstuff: @chris_dag2

Who, what & whyBioTeam - Not your standard AWS user

‣ Independent consulting shop‣ Run by scientists forced to

learn IT, SW & HPC to get our own genetics research done

‣ 12+ years bridging the “gap” between science, IT & high performance computing

‣ www.bioteam.net

33

4

Meta-IssuesWhat you don’t know WILL hurt you.

4

A common theme across all our biotech, pharma, EDU and research clients ...

5

Lets Be Blunt

‣ It’s almost 2013‣ AWS is comprehendible in minute:hours; pilot projects

and proof-of-concepts can be done in hours:days. Really.‣ The TECHNICAL bits are easy or at least manageable‣ The POLICY & PROCEDURE bits will destroy your soul‣ If I could only give 1 piece of advice to researchers ...

5

For anyone doing research HPC or “science” on the cloud

6

One Piece of Advice ...

‣ The technical issues surrounding cloud-based scientific research are easy or at least surmountable

‣ The hard part is sorting out the internal policies and procedures that govern cloud use

‣ Ignore this at your peril

6

7

Five Minute Case StudyComputer Aided Engineering (CAE) on AWS

7

Next Generation Nuclear Magnetic Resonance

8

NMR Probehead Simulation on AWS

‣ CAE Simulation Project‣ via www.hpcexperiment.com‣ Software: CST Studio 2012‣ My role: Volunteer HPC Mentor

8

Simulating next-generation NMR probeheads

9

Why this was an interesting project

‣ Frontend interface is graphics heavy and requires Windows

‣ Studio ‘solvers’ run Linux or Windows; support GPUs and MPI task distribution

‣ Simultaneous use of local and cloud-based solvers actually works

‣ flexLM license server involved

‣ Non-trivial security and geo-location requirements

9

Initial Architecture1010

At experiment end it looked like this ...1111

12

Amazon FTWAWS makes the easy stuff fun and the hard stuff possible

12

Why this work was ‘easy’ on Amazon AWS ...

13

Nightmare on any other cloud

‣ Tick Tock ...‣ Don’t have the time to get into the CAE simulation details‣ Lets discuss why this simulation workload would be

much, much harder to do on some other cloud platform ...

13

Why this work was ‘easy’ on Amazon AWS ...

14

Nightmare on any other cloud

1. Virtual Servers2. Block Storage3. Object Storage4. ... and maybe some other

stuff if I’m lucky

‣ EC2, S3, EBS, RDS, SNS, SQS, SWS, GPUs, SSDs, CloudFormation, VPC, ENIs, SecurityGroups, 10GbE DirectConnect, Reserved Instances, ImportExport, Spot Market

‣ And ~25 other products and service features with more added monthly

‘Brand X’ Cloud AWS

14

Easy on AWS; much harder elsewhereOne very specific example

15

‣ The widely used FLEXlm license server uses NIC MAC addresses when generating license keys

‣ Different MAC? Science stops. Screwed.

‣ VPC ENIs allow separation of MAC address from Network Interface. Badass.

15

Why this work was ‘easy’ on Amazon AWS ...A few other examples ...

16

VPC

Spot Market

cc* & cg* ec2 instance

types

Incredibly powerful. Actually useful.

Approachable even if you are not an IPSEC or BGP routing god.

Compelling economics. Once you start you’ll likely never run anywhere else.

The competition can’t compete.

Fat nodes with bidirectional 10GbE bandwidth.

And don’t get me started on SSD or Provisioned-performance EBS volumes.

16

Wrapping up ...Science on AWS

17

‣ It’s not rocket science; yeah we can do this just about anywhere

‣ It’s just easier and faster on AWS because they have more building blocks; I can spend more time on actual science problems You mean I get paid for this?

17

Wrapping up ...Science on AWS, cont.

18

‣ There is no one-size-fits-all research design pattern ...

‣ Very few of us have a single pipeline or workflow that we can devote endless engineering effort to

‣ We are not going to toss out hundreds of legacy codes and rewrite everything for GPUs or MapReduce

‣ For research HPC it’s all about the building blocks { and how we can effectively use/deploy them }

I love my job

18

One last plug

‣ Want more info or details?‣ Watch http://bioteam.net for upcoming posts that

document in detail the CST Studio on AWS effort‣ And by the way ...‣ hpcexperiment.com round #2 is starting soon; They need

volunteers, resource providers and new use cass

1919

20

Thanks! Slides online at: http://slideshare.net/chrisdag/

Twitter: @chris_dag

20