Amazon Web Services for scientific computing

Alexander Novikov

June 18, 2015

Alexander Novikov AWS for scientific computing June 18, 2015 1 / 21

1 Overview

2 Ipython Notebook example

3 Other stuff

4 Cleaning up

5 Troubleshooting

Alexander Novikov AWS for scientific computing June 18, 2015 2 / 21

Summary

1 Start 1 or 1000 configured servers in 5 minutes;

2 Run anything you want (you control OS and software);

3 Vary the size of your server (from micro instance to 32 cores 244 Gbof RAM, to 4 GPUs onboard);

4 Pay per hour;

5 1 server for 100 hours cost the same as 100 servers for 1 hour.

Alexander Novikov AWS for scientific computing June 18, 2015 3 / 21

Amazon sell books don’t they?

Alexander Novikov AWS for scientific computing June 18, 2015 4 / 21

Amazon Web Services – AWS

We focus on Amazon Elastic Compute Cloud – EC2.

Alexander Novikov AWS for scientific computing June 18, 2015 5 / 21

Ipython in a cloud

You will run Ipython Notebook on AWS server in Frankfurt.

If you don’t have an AWS account but want to follow the tutorial, go tothe spreadsheet from my email and claim one of the servers.

Alexander Novikov AWS for scientific computing June 18, 2015 6 / 21

Running a server

1 Log into the AWS console;2 Click on EC2;3 Choose a region (Frankfurt is the closest; N. Virginia and Oregon are

cost effective);4 Click Launch Instance (server);5 Choose Ubuntu 14.04 (HVM);6 Choose t2.micro (it’s free);7 Next;8 Choose the size of the root drive;9 Name your instance;10 Click Review and Launch;11 Now review and click Launch.

Alexander Novikov AWS for scientific computing June 18, 2015 7 / 21

SSH keys

AWS use ssh keys to protect your connection to the server.

Name your key, click download, and save it in a safe location.

On Linux, you will need to restrict the access to the file:

$ cd Downloads/$ chmod 400 key_name.pem

Alexander Novikov AWS for scientific computing June 18, 2015 8 / 21

Connect & update

On Linux and Max, connect with ssh tool

$ ssh -i tutorial.pem ubuntu@52.28.135.168

On Windows, convert your key to ppk and connect with PuTTY (username is ubuntu, ssh key file is key_name.ppk).

Update the server

$ sudo apt-get update$ sudo apt-get upgrade$ sudo reboot

Alexander Novikov AWS for scientific computing June 18, 2015 9 / 21

Install miniconda

Download the installer

$ wget https://repo.continuum.io/miniconda/...

Make it runnable and install (use tab for autocompletion)

$ chmod +x ./Miniconda-latest-Linux-x86_64.sh$ ./Miniconda-latest-Linux-x86_64.sh

Now reconnect to the server (ctr+ d to disconnect) and install condapackages

$ conda install ipython ipython-notebook

Alexander Novikov AWS for scientific computing June 18, 2015 10 / 21

Networking

Generate an HTTPS certificate (you can leave the defaults for all thequestions)

$ ipython profile create nbserver$ openssl req -x509 -nodes -days 365 -newkey rsa:1024 \-keyout mycert.pem -out mycert.pem

Hash your password

$ ipythonfrom IPython.lib import passwdpasswd()

Alexander Novikov AWS for scientific computing June 18, 2015 11 / 21

Configure Ipython Notebook

Let’s find and edit the configuration file

$ find ./ -name "*notebook_config*"$ nano \.ipython/profile_nbserver/ipython_notebook_config.py

Add the following lines after c = get_config()

c.NotebookApp.certfile = u'/home/ubuntu/mycert.pem'c.NotebookApp.ip = '*'c.NotebookApp.open_browser = Falsec.NotebookApp.password = u'sha1:207eb1f4671f:92af6...'c.NotebookApp.port = 8888

Alexander Novikov AWS for scientific computing June 18, 2015 12 / 21

Nano

Nano is a command line editor.

Essential commands (like ctr+ o for saving and ctr+ x to exit) are listedon the bottom of the interface.

Alexander Novikov AWS for scientific computing June 18, 2015 13 / 21

Almost there

Run the notebook

$ ipython notebook --profile=nbserver

Find the public URL of your server in AWS console (e.g. https://ec2-52-28-135-168.eu-central-1.compute.amazonaws.com:8888)and open it in a browser.

Aaand it doesn’t work.

Alexander Novikov AWS for scientific computing June 18, 2015 14 / 21

Security groups

Go to the Security Groups tab and open incoming traffic on the 8888 port.

Alexander Novikov AWS for scientific computing June 18, 2015 15 / 21

Tmux

If you close the terminal now, the program will be killed.

Use tmux to run it in the background and be able to see the output.

$ tmux # Create new terminal.$ ctr + b d # Detach from the current terminal.$ tmux attach # Attach to an existing terminal.

Alexander Novikov AWS for scientific computing June 18, 2015 16 / 21

Terminating a server

When you terminate your server it’s hard drive is erased.

1 Next time you have to configure the server from scratch;

2 You cannot store any data (experiment results).

Alexander Novikov AWS for scientific computing June 18, 2015 17 / 21

Amazon Machine Image – AMI

Template from AWS instances are launched;

OS + software;

Launch different instance types from the same AMI;

You customize (install custom software, services, config, etc) andmake your own AMI;

AWS Marketplace makes it easy to launch a number of popularapplications (e.g, MongoDB, Cafee, OpenCV, etc.).

Alexander Novikov AWS for scientific computing June 18, 2015 18 / 21

Elastic Block Storage – EBS

Create a permanent magnetic or SSD volume of any size;

Connect (disconnect) the volume to any server in its data center(availability group).

Alexander Novikov AWS for scientific computing June 18, 2015 19 / 21

Spot instances

Bid $X per hour;If current price > bid, instance terminates;You pay the market price, not the bids.

Alexander Novikov AWS for scientific computing June 18, 2015 20 / 21

Workflow

Create a small instance, set up all the software, download all the data;

Make an AMI;

Start a powerful spot instance (or a dozen) from your AMI;

Create a permanent volume and connect it to the instance;

Process the data (in tmux!) and write the results into the volume;

Terminate the instance.

Alexander Novikov AWS for scientific computing June 18, 2015 21 / 21

Other stuff

Run a program on a server and see its GUI on your laptop (ssh -X);

Use scp to download data to and from your server;

Use df -h to see all the disk mounted to the server;

Use htop to see running processes;

Set up a monitoring so when your server stops working you get anemail;

Explore AWS bill https://console.aws.amazon.com/billing;

Apply for an AWS academic grant.

Alexander Novikov AWS for scientific computing June 18, 2015 22 / 21

Cleaning up

Got to the console and terminate your server;

Go to the volume tab and remove all your volumes;

Alexander Novikov AWS for scientific computing June 18, 2015 23 / 21

Unprotected private key

If you see this message when trying to connect

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ WARNING: UNPROTECTED PRIVATE KEY FILE! @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@Permissions 0660 for ’tutorial.pem’ are too open.It is required that your private key files are NOT

accessible by others.This private key will be ignored.bad permissions: ignore key: tutorial.pemPermission denied (publickey).

Just set more restrictive permissions for your key file

$ chmod 400 key_name.pem

Alexander Novikov AWS for scientific computing June 18, 2015 22 / 21

Full volume

Your program says «Cannot write a file»?

Check your volumes, they are probably full. To list all mounted volumesand their sizes use

$ df -h

Alexander Novikov AWS for scientific computing June 18, 2015 21 / 21

Slow down

Everything start beeing slow?

You probably burnt all your volumes I/O capacity. Wait so it can restoreor order a bigger volume (see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSVolumeTypes.html for details).

Alexander Novikov AWS for scientific computing June 18, 2015 21 / 21