Your Results From Disk to Device(Preview)

Amit ChourasiaMona Wong-BarnumMichael L. Norman

San Diego Supercomputer Center, UCSD

Presented at: XSEDE 13, San Diego, CA, Jul 23, 2013

Stream Encode Explore Disseminate My Experiments

Computation Cycle

Compute/IngestPost ProcessAnalyze & Save

Monitor?

Validate?

Abstract?

Derived Products

Monitoring

ValidationAbstraction

Direct Access Direct Sharing

Not All Members Have HPC

Access

Security Implications

On HPC

Derived Products

Concise derived products• Plots & Text Documents• Image Sequences & Videos

Share with the research group1. Download derived products2. Share with collaborators

a. Email snippets b. Set up a webpage

– Add content– Email

Derived Visual Content Survey

Derived content for a single research group in respective domain

Pitfalls in sharing derived content

Download Upload

Round Trip

Video Encoding

Complexity Web Setup Process

Duplication Scalability

Direct AccessNot All Members Have HPC Access

Direct SharingSecurity

Implications On HPC

Gaps

Download Upload

Video Encoding Web Setup Process

Duplication Scalability

Direct Transfer Easy Sharing Reusable Ubiquitous

SeedMe.org

Why not use existing tools?Features SeedMe YouTube & Vimeo FlickrSharing Public, Group, Private Public, Private Public

Upload Method Command Line, API, Web API, Web API, Web

Meta Data Rich Limited Limited

Download Support Yes Vimeo only No

Copyright Unencumbered Encumbered Encumbered

Still Images Yes No Yes

Periodic Image Update Yes No No

Image Sequence to Video Yes No No

Frame Rate Arbitrary 29.97 fps Unknown

Video Resolution Arbitrary Upto 4K Upto HD

Desirable Limiting Crippling

SeedMe Architecture

Ingestion (Atomic or Recurring)

Instant Processing

(Text, Images)

NotificationResponsive

Content Access

Batch Processing

(Videos)

SeedMe Hardware

Ingest Temp

Storage (10TB) Encode Cluster (16 nodes) :

Apple Compressor + ffmpeg

Web Cluster (2-8 nodes) : Drupal

Mirrored Storage (50 TB) : ZFS

Desktops

Phones

Tablets

20 gbe

20 gbe

20 gbe

Ingestion (Atomic or Recurring)

Instant Processing

(Text, Images)

NotificationResponsive

Content Access

Batch Processing

(Videos)

Compressor Encoding Benchmark

SeedMe Collection

Ticker* Plots Image Sequences* Videos

* Recurring: periodic update

Meta Data

Access: Public, Group, Private

Collaborators

Create New CollectionSet Access Control (Public, Group, Private)

Specify collaborators* and meta data

Upload/Update contenthttp/https, ftp, Globus Online

Interactively on Web, Command Line or REST Services

View Collection on Web

SeedMe: Interaction Process

SeedMe: Interaction Modality

• Command Line Interface (using curl)• Web Interface (WYSIWYG)• RESTful API (integrate your tool chain)– JSON

Content

Videos

Permissions

Plot Files

Doc Files

Authentication

Text Ticker

Image Sequences

Meta Data

{  "email": ”jdoe@sdsc.edu",  "api_key": "XXXXXXXXXXX",   ”view_permission": "public",  ”viewer_emails": ["a1@a.com", "b1@b.com"],   "title": "simulation XYZ",  "description": "input conditions",  "credits": "Visservices group, SDSC",  "key_values": { “key1”: "v1”, "key2": "v2”},  "expire_date": "2013-05-01",  "tags": [ "x", "y", "z"],

  "ticker": ["time step 100", "time step 200"],   "files":{      ”docs" : ["a.pdf", ”b.txt"],

     "plots" : ["a.png", "b.jpg", "c.tiff"],      "image_sequences" : {                 "ash": ["ash1.jpg", "ash2.jpg", "ash3.jpg"],        "air": ["air1.jpg", "air2.jpg", "air3.jpg"]      },      "movies": ["m1.mov", "m2.mov »]   }}

Application Scenarios

• Monitor progress with text tickers and images• Encode and stream videos from image

sequences• Share & collect visualization results• Use the collections for education and

outreach

Content accessible on most devices

Work in progress

How it Works Examples #1 & #2

Periodic update at simulation run time from a Magneto Hydrodynamics SimulationCourtesy of Mahidhar Tatineni and Homa Karimabadi, UCSD

Future Work

• Production SeedMe Service• Example Scripts & Documentation• Globus integration with SeedMe

Acknowledgements

National Science FoundationThis material is based upon work supported by the National Science

Foundation under Grant No. OCI-1235505

Andrew Ferbert, Michael Dwyer, Kristen Levy & Doug Weimer, SDSC : Hardware/software setup

Mahidhar Tatineni, SDSC & Homa Karimabadi, UCSD: in-situ testsApple Inc. : Provided test hardware/software on loan during project

conceptualization phase

Sign up InvitationFor notification and early access

(we won’t spam)

Contact: amit @ sdsc.edu

Persistence?

Results

Move to own

resource

Move to XSEDE

resource

Move to YouTube/Flickr, Vimeo

Move to Dropbox/C

loud

Freemium

PURGE