Upload
kareem-livingston
View
43
Download
0
Tags:
Embed Size (px)
DESCRIPTION
TeraGrid Science Gateways. Nancy Wilkins-Diehr TeraGrid Area Director for Science Gateways [email protected]. TeraGrid is one of the largest investments in shared CI from NSF’s Office of Cyberinfrastructure. Dedicated high-speed, cross—country network. Staff & Advanced Support. - PowerPoint PPT Presentation
Citation preview
TeraGrid Science Gateways
Nancy Wilkins-DiehrTeraGrid Area Director for
Science [email protected]
MURPA lecture, March 12, 2010
TeraGrid is one of the largest investments in shared CI from NSF’s Office of Cyberinfrastructure
MURPA lecture, March 12, 2010
TeraGrid resources today include:•Tightly Coupled Distributed Memory Systems, 2 systems in the top 10 at top500.org
– Kraken (NICS): Cray XT5, 99,072 cores, 1.03 Pflop– Ranger (TACC): Sun Constellation, 62,976 cores, 579 Tflop, 123 TB RAM
•Shared Memory Systems– Cobalt (NCSA): Altix, 8 Tflop, 3 TB shared memory– Pople (PSC): Altix, 5 Tflop, 1.5 TB shared memory
•Clusters with Infiniband– Abe (NCSA): 90 Tflops– Lonestar (TACC): 61 Tflops– QueenBee (LONI): 51 Tflops
•Condor Pool (Loosely Coupled)– Purdue- up to 22,000 cpus
•Gateway hosting– Quarry (IU): virtual machine support
•Visualization Resources– TeraDRE (Purdue): 48 node nVIDIA GPUs– Spur (TACC): 32 nVIDIA GPUs
•Storage Resources– GPFS-WAN (SDSC)– Lustre-WAN (IU)– Various archival resources
MURPA lecture, March 12, 2010 Source: Dan Katz, U Chicago
New syste
ms:
•Data Analysis and Vis s
ystems
•Longhorn (TACC): D
ell/NVID
IA, C
PU
and GPU
•Nautilus (N
ICS): S
GI UltraViolet, 1
024
cores, 4TB global sh
ared memory
•Data-Intensiv
e Computing
•Dash (SDSC): I
ntel Nehalem, 544
processors, 4
TB flash memory
•FutureGrid
•Experimental computin
g grid and cloud
test-bed to tackle research challenges in
computer science
•Keeneland
•Experimental, h
igh-performance
computing sy
stem with
NVIDIA
Tesla
accelerators
So how did the Gateway program develop?A natural result of the impact of the internet on worldwide
communication and information retrieval
MURPA lecture, March 12, 2010
Only 18 years since the release of Mosaic!
Access to supercomputers hasn’t changed much in 20 years
But the world around them sure has!
MURPA lecture, March 12, 2010
Why are gateways worth the effort?
•Increasing range of expertise needed to tackle the most challenging scientific problems–How many details do you
want each individual scientist to need to know?•PBS, RSL, Condor•Coupling multi-scale codes•Assembling data from multiple sources
•Collaboration frameworks
MURPA lecture, March 12, 2010
#! /bin/sh#PBS -q dque#PBS -l nodes=1:ppn=2 #PBS -l walltime=00:02:00#PBS -o pbs.out#PBS -e pbs.err#PBS -Vcd /users/wilkinsn/tutorial/exercise_3../bin/mcell nmj_recon.main.mdl
+( &(resourceManagerContact="tg-login1.sdsc.teragrid.org/jobmanager-pbs") (executable="/users/birnbaum/tutorial/bin/mcell") (arguments=nmj_recon.main.mdl) (count=128) (hostCount=10) (maxtime=2) (directory="/users/birnbaum/tutorial/exercise_3") (stdout="/users/birnbaum/tutorial/exercise_3/globus.out") (stderr="/users/birnbaum/tutorial/exercise_3/globus.err"))
=======# Full path to executableexecutable=/users/wilkinsn/tutorial/bin/mcell
# Working directory, where Condor-G will write # its output and error files on the local machine.initialdir=/users/wilkinsn/tutorial/exercise_3
# To set the working directory of the remote job, we# specify it in this globus RSL, which will be appended# to the RSL that Condor-G generatesglobusrsl=(directory='/users/wilkinsn/tutorial/exercise_3')
# Arguments to pass to executable.arguments=nmj_recon.main.mdl
# Condor-G can stage the executabletransfer_executable=false
# Specify the globus resource to execute the jobglobusscheduler=tg-login1.sdsc.teragrid.org/jobmanager-pbs
# Condor has multiple universes, but Condor-G always uses globusuniverse=globus
# Files to receive sdout and stderr.output=condor.outerror=condor.err
# Specify the number of copies of the job to submit to the condor queue.queue 1
Gateways democratize access to high end resources
•Almost anyone can investigate scientific questions using high end resources–Not just those in the research groups of those who request
allocations–Gateways allow anyone with a web browser to explore
•Opportunities can be uncovered via google–My then 11-year-old son discovered nanoHUB.org when his science class was studying Bucky Balls
•Foster new ideas, cross-disciplinary approaches–Encourage students to experiment
•But used in production too–Significant number of papers resulting from gateways including
GridChem, nanoHUB–Scientists can focus on challenging science problems rather
than challenging infrastructure problemsMURPA lecture, March 12, 2010
NSF understands the impact technology can have on science
•“Virtual environments have the potential to enhance collaboration, education, and experimentation in ways that we are just beginning to explore.”
•“In every discipline, we need new techniques that can help scientists and engineers uncover fresh knowledge from vast amounts of data generated by sensors, telescopes, satellites, or even the media and the Internet.”
MURPA lecture, March 12, 2010
Gateways are a terrific example of interfaces that can support
transformative science
Not just ease of useWhat can scientists do that they
couldn’t do previously?•Linked Environments for Atmospheric Discovery (LEAD) - access to radar data•National Virtual Observatory (NVO) – access to sky surveys•Ocean Observing Initiative (OOI) – access to sensor data•PolarGrid – access to polar ice sheet data•SIDGrid – expensive datasets, analysis tools•GridChem –coupling multiscale codes
•How would this have been done before gateways?MURPA lecture, March 12, 2010
What makes a gateway a TeraGrid gateway?
•TeraGrid gateways are gateways that use TeraGrid resources•Are they all developed by TeraGrid?
– No, we don’t make gateways the gateways you use, we make the gateways you use better
– The strength of the program lies in the development of end user interfaces by those in the community
•TeraGrid does provide staff to assist with gateway use of the resources– Anyone can request support via the same peer review process used to
request CPU hours or a data allocation•Assistance from a staff member dedicated to your project, multi-month time investment to really understand and help solve complex problems
– Must have commitment from PIs•Want to make sure work is incorporated into production codes and gateways
– Good candidates for targeted support include:•Large, high impact projects•Ability to influence new communities•Happy for feedback from directorates on important projects
– Lessons learned move into training and documentation
MURPA lecture, March 12, 2010
3 steps to connect a gateway to TeraGrid
•Request an allocation–Only a 1 paragraph abstract
required for up to 200k CPU hours
•Register your gateway–Visibility on public TeraGrid page
•Request a community account–Run jobs for others via your
portal
•Staff support is available!•www.teragrid.org/gateways
MURPA lecture, March 12, 2010
Tremendous Opportunities Using the Largest Shared Resources -
Challenges too!•What’s different when the resource doesn’t belong just to me?–Resource discovery–Accounting–Security–Proposal-based requests for resources (peer-reviewed access)
•Code scaling and performance numbers•Justification of resources•Gateway citations
•Tremendous benefits at the high end, but even more work for the developers•Potential impact on science is huge
–Small number of developers can impact thousands of scientists
–But need a way to train and fund those developersMURPA lecture, March 12, 2010
When is a gateway appropriate?
•Researchers using defined sets of tools in different ways–Same executables, different input
•GridChem, CHARMM
–Creating multi-scale workflows–Datasets
•Common data formats–National Virtual Observatory–Earth System Grid–Some groups have invested significant efforts here
•caBIG, extensive discussions to develop common terminology and formats
•BIRN, extensive data sharing agreements
•Difficult to access data/advanced workflows–Sensor/radar input
•LEAD, GEON
MURPA lecture, March 12, 2010
How to get started?
•Conduct a needs assessment–Should I build a gateway?–Can I use an existing gateway?–What problems am I trying to solve?
•All gateways don’t need high end computing
•Decide on a software approach–Recommended software at www.teragrid.org
•Targeted effort by a few can benefit many–Could a pool of developers design gateways for different
domain areas? Yes!
•TeraGrid staff assistance
MURPA lecture, March 12, 2010
Gateways in the marketplaceKids control telescopes and share images
•“In seconds my computer screen was transformed into a live telescopic view”–“Slooh's users include
newbies and professional astronomers in 70 countries”
•Observatories in the Canary Islands and Chile, Australia coming soon •5000 images/month since 2003•Increases public support for investment in these facilities
MURPA lecture, March 12, 2010
Linked Environments for Atmospheric DiscoveryLEAD
•Providing tools that are needed to make accurate predictions of tornados and hurricanes•Data exploration and Grid workflow
MURPA lecture, March 12, 2010
Highlights: LEAD Inspires StudentsAdvanced capabilities regardless of location
•A student gets excited about what he was able to do with LEAD•“Dr. Sikora:Attached is a display of 2-m T and wind depicting the WRF's interpretation of the coastal front on 14 February 2007. It's interesting that I found an example using IDV that parallels our discussion of mesoscale boundaries in class. It illustrates very nicely the transition to a coastal low and the strong baroclinic zone with a location very similar to Markowski's depiction. I created this image in IDV after running a 5-km WRF run (initialized with NAM output) via the LEAD Portal. This simple 1-level plot is just a precursor of the many capabilities IDV will eventually offer to visualize high-res WRF output. Enjoy!• Eric” (email, March 2007)
MURPA lecture, March 12, 2010
PolarGrid brings CI to polar ice sheet measurement
MURPA lecture, March 12, 2010
•Cyberinfrastructure Center for Polar Science (CICPS)–Experts in polar science,
remote sensing and cyberinfrastructure
– Indiana, ECSU, CReSIS
•Satellite observations show disintegration of ice shelves in West Antarctica and speed-up of several glaciers in southern Greenland–Most existing ice sheet
models, including those used by IPCC cannot explain the rapid changes
•Components of PolarGrid– Expedition grid consisting of ruggedized laptops in a field grid linked to
a low power multi-core base camp cluster– Prototype and two production expedition grids feed into a 17 Teraflops
"lower 48" system at Indiana University and Elizabeth City State (ECSU) split between research, education and training.
– Gives ECSU (a minority serving institution) a top-ranked 5 Teraflop high performance computing system
•Access to expensive data•TeraGrid resources for analysis
– Large level 0 and level 1 data sets require once and done processing and storage
– Filters applied to level 1 data by users in real time via the web
•Student involvement
MURPA lecture, March 12, 2010Source: Geoffrey Fox
Social Informatics Data GridCollaborative access to large, complex datasets
•SIDGrid is unique among social science data archive projects–Streaming data which
change over time•Voice, video, images (e.g. fMRI), text, numerical (e.g. heart rate, eye movement)
– Investigate multiple datasets, collected at different time scales, simultaneously•Large data requirements•Sophisticated analysis tools
MURPA lecture, March 12, 2010
http://www.ci.uchicago.edu/research/files/sidgrid.mov
Viewing multimodal data like a symphony conductor
•“Music-score” display and synchronized playback of video and audio files– Pitch tracks– Text– Head nods, pause, gesture
references
•Central archive of multi-modal data, annotations, and analyses– Distributed annotation efforts by
multiple researchers working on a common data set•History of updates
•Computational tools– Distributed acoustic analysis using
Praat– Statistical analysis using R– Matrix computations using Matlab
and Octave
MURPA lecture, March 12, 2010
Source: Studying Discourse and Dialog with SIDGrid, Levow, 2008
SCEC Gateway used to produce realistic hazard map
•Probabilistic Seismic Hazard Analysis (PSHA) map for California– Created from Earthquake
Rupture Forecasts (ERC)•~7000 ruptures can have 415,000 variations
•Warm colors indicate regions with a high probability of experiencing strong ground motion in the next 50 years•Ground motion calculated using full 3-D waveform modeling for improved accuracy– Results in significant CPU use
MURPA lecture, March 12, 2010
SCEC: Why a gateway?
•Calculations need to be done for each of the hundreds of thousands of rupture variations–SCEC has developed the “CyberShake computational
platform”•Hardware, software and people which combine to produce a useful scientific result
–For each site of interest - two large-scale MPI calculations and hundreds of thousands of independent post-processing jobs with significant data generation»Jobs aggregated to appear as a single job to the TeraGrid»Workflow throughput optimizations and use of SCEC’s
gateway “platform” reduced time to solution by a factor of three
–Computationally-intensive tasks, plus the need for reduced time to solution is a priority make TeraGrid a good fit
MURPA lecture, March 12, 2010
Source: S. Callahan et.al. “Reducing Time-to-Solution Using Distributed High-Throughput Mega-Workflows – Experiences from SCEC CyberShake”.
GridChem
•Understanding molecular structure and function increasingly important in many fields– Materials for electronics, biotechnology, medical devices,
pharmaceutical design
•GridChem provides reliable infrastructure for computational chemists– NSF Middleware Initiative (NMI) project
•Requested and received advanced support from TeraGrid– Addressing issues which benefit all gateways, support team led by IU
•Common user environments for domain software access•Standardized licensing•Application performance characteristics•Incorporation of additional data handling tools and data resources•Fault tolerant workflows•Scheduling policies for community users•Remote visualization
MURPA lecture, March 12, 2010
GridChem: Why a gateway?
•Integrates high end resources in a desktop environment– Client-server approach allows
work to continue while disconnected (plane flights)
– Ability to monitor jobs across sites
– Access to individual allocations– In the future, linkage of multi-
scale packages
•Focus on chemistry research without learning the intricacies of each system– Time limits, nodes, processors,
memory, disk space, etc
MURPA lecture, March 12, 2010
Robetta GatewayProtein structure prediction with an award-winning code
•Protein structure prediction is among many important problems in bioinformatics. •The Rosetta code, from the David Baker laboratory, has performed very well at CASP (Critical Assessment of Techniques for Protein Structure Prediction) competitions–Available for use by any academic scientist via the Robetta server
•Robetta developers able to use TeraGrid’s existing gateway infrastructure, including community accounts and Globus–This very successful group needed no additional TeraGrid
assistance to incorporate TeraGrid resources into the Robetta gateway
•Google scholar reports 601 references to the Robetta gateway, including many PubMed publications
MURPA lecture, March 12, 2010
Robetta: Why a gateway?
•Bioinformatics has long history of web-based services–NCBI Blast server from the 1990s
•Easy input from the web•Access to top modeling code for all researchers
MURPA lecture, March 12, 2010
Using the TeraGrid to Understand the Stars and Search for Planets:
Problem: The oscillations modes of stars give clues to the stellar interior.•Accurate modeling of these oscillations can tell us the size, composition and age of stars, and in turn help interpret the data returned by NASA’s Kepler Mission in the hunt for Earth-like extrasolar planets.•Solution: NCAR is working with the international science community to develop a TeraGrid science gateway to fit oscillation data with stellar models to determine these parameters automatically for large numbers of stars.•Key Elements: international science impact, gateway technology.
The turbulent solar atmosphere captured with NASA’s TRACE
satellite is overlaid with a model of
one of the millions of oscillation modes that allow scientists to
deduce the Sun’s hidden internal structure and dynamics. Similar measurements are now being
made for other solar-type stars.MURPA lecture, March 12, 2010Source: Dr. Rich Loft, NCAR
Analytical UltracentrifugationEmerging computational tool for the study of proteins
•The Center for Analytical Ultracentrifugation of Macromolecular Assemblies, UT Health Sciences – Major advances in the
characterization of proteins and protein complexes as a result of new instrumentation and powerful software
– Monitoring the sedimentation of macromolecules in real time in the centrifugal field allows their hydrodynamic and thermodynamic characterization in solution
– Observations are electronically digitized and stored for further mathematical analysis
– http://uslims.uthscsa.edu/MURPA lecture, March 12, 2010
Source: Modern analytical ultracentrifugation in protein science: A tutorial review, Wikipedia
UltraScan provides a comprehensive data analysis environment
•Management of analytical ultracentrifugation data for single users or entire facilities•Support for storage, editing, sharing and analysis of data
–HPC facilities used for 2-D spectrum analysis and genetic algorithm analysis•TeraGrid (~2M CPU hours used)•Technische University of Munich•Juelich Supercomputing Center
•Portable graphical user interface•MySQL database backend for data management•Over 30 active institutions•TeraGrid advanced support
–Fault tolerance, workflows, use of multiple TG resources, community account implementation
MURPA lecture, March 12, 2010
Future Technical Areas•Web technologies change fast
– Must be able to adapt quickly
•Gateways and gadgets– Gateway components
incorporated into any social networking page
– 75% of 18 to 24 year-olds have social networking websites
•iPhone apps?•Web 3.0
– Beyond social networking and sharing content
– Standards and querying interfaces to programmatically share data across sites
•Resource Description Framework (RDF), SPARQL
MURPA lecture, March 12, 2010
Gateways can further investments in other projects
•Increase access–To instruments, expensive data collections
•Increase capabilities–To analyze data
•Improve workforce development–Can prepare students to function in today’s cross-disciplinary
world
•Increase outreach•Increase public awareness
–Public sees value in investments in large facilities–Pew 2006 study indicates that half of all internet users have been
to a site specializing in science–Those who seek out science information on the internet are more
likely to believe that scientific pursuits have a positive impact on society
MURPA lecture, March 12, 2010
But, sustained funding is a problemGateway white paper provides recommendations
•Gateways can be used for the most challenging problems, but–Scientists won’t rely on something that they are not confident
will be around for the duration•We see this with software, but even more so with gateway infrastructure
•A sustained gateway program can–Reduce duplication of effort
•Sporadic development with many small programs
– Increase diversity of end users– Increase skill set diversity of developers–Bring together teams to address the toughest problems
MURPA lecture, March 12, 2010
Recommend 10-year programwith interim reviews
•Characteristics of 5-year or less cycles– Build exciting prototypes with input
from scientists– Work with early adopters to extend
capabilities– Tools are publicized, more scientists
interested– Funding ends– Scientists who invested their time to
use new tools are disillusioned•Less likely to try something new again
– Start again on new short-term project
•Need to break this cycle•EAGER grant to look at characteristics of successful gateways and domain areas where a gateway could have a big impact
– www.sciencegateways.org
MURPA lecture, March 12, 2010
Tremendous Potential for Gateways•In only 18 years, the Web has fundamentally changed human communication•Science Gateways can leverage this amazingly powerful tool to:– Transform the way scientists
collaborate– Streamline conduct of science– Influence the public’s perception of
science
•Reliability, trust, continuity are fundamental to truly change the conduct of science through the use of gateways– Study ongoing to identify
characteristics of successful gateways
•The future is very exciting!– High end resources can have a
profound impact
MURPA lecture, March 12, 2010
MURPA lecture, March 12, 2010
Thank you for your attention!Questions?
Nancy Wilkins-Diehr, [email protected]