2004-09-13NVO Summer School, Aspen Center for Physics1 National Virtual Observatory Applications Software Development Summer School: Introduction and Overview

2004-09-13NVO Summer School, Aspen Center for Physics 1

National Virtual ObservatoryApplications Software Development

Summer School:Introduction and Overview

Bob HanischProject Manager, US NVO ProjectSpace Telescope Science Institute

THE US NATIONAL VIRTUAL OBSERVATORY


Motivation

• NVO development now three years along…software environment is sufficiently well-developed that the community can start to build VO-enabled applications

• Want community to start to see the VO as an important toolkit for doing astronomical research and public outreach/education

• Want feedback from the community on how to improve and extend VO tools

• Want you to be our agents, our collaborators


Faculty

• The “faculty” is drawn from the NVO Project development team

• Programmers and programmer/scientists with experience in development of web-based applications and services

• Introductions…


Faculty

Tamas Budavari, The Johns Hopkins University, Baltimore


Faculty

Dave De Young, National Optical Astronomy Observatories, Tucson


Faculty

Matthew Graham, Caltech, Pasadena


Faculty

Gretchen Greene, Space Telescope Science Institute, Baltimore


Faculty

Bob Hanisch, Space Telescope Science Institute, Baltimore


Faculty

Tom McGlynn, NASA Goddard Space Flight Center, Greenbelt, MD


Faculty

Maria Nieto-Santisteban, The Johns Hopkins University, Baltimore


Faculty

Wil O’Mullane, The Johns Hopkins University, Baltimore


Faculty

Ray Plante, National Center for Supercomputing Applications, Urbana, IL


Faculty

Doug Tody, National Radio Astronomy Observatory, Socorro, NM


Faculty

Roy Williams, Caltech, Pasadena

(Roy)


Thanks to…

• NSF and NASA for their financial support• Aspen Center for Physics, president Dave De Young, for

agreeing to provide the facility and Jane Kelly for helping with all contractual and logistical matters

• Shelly Meyett, Ranpal Gill, and Sadie Lingham for registration, catering, travel, and all other matters of logistical support


Logistical Matters

• Breakfast and lunch to be provided at ACP, except no lunch on Wednesday (free afternoon).

• Summer School dinner Wednesday evening at the Mezzaluna Restaurant (see city maps)

• Wireless network available here at ACP and at Aspen Meadows• ACP bicycles are available• See also http://chart.stsci.edu/twiki/bin/view/Main/

LogisticalInformation, or ask Ranpal or Jane

http://chart.stsci.edu/twiki/bin/view/Main/LogisticalInformation

http://chart.stsci.edu/twiki/bin/view/Main/LogisticalInformation


About the Program

• Day 1: Overview and tour of science capabilities (Magical Mystery Tour)

• Day 2: How the VO works—standards, protocols, web services, clients and servers (A Hard Day’s Night)

• Day 3: The registry (how to publish and find things) and the Grid (how to Compute), plus free time (Strawberry Fields Forever)

• Day 4: A look inside some existing VO science applications, followed by participants forming teams and working on their own projects (With a Little Help from My Friends)

• Day 5: Participants show their work (Ticket to Ride)


Caveats

• The VO is a work in progress. Standards and interfaces are in development and changing. Future versions will be similar to current ones, but code may break and have to be updated.

• Software libraries and tools have not yet been used by many people outside of the development projects—you will probably uncover bugs and be able to make things fail. Please tell us what works and what doesn’t.

• This is our first Summer School. We don’t know if we have the right balance of topics and time. We will try to adjust as we go along. We’ll take a few minutes for feedback at the end of each day.

• Participants have diverse interests and backgrounds. Material may be advanced for some, simple for others. When we get to doing hands-on work, we ask that the more experienced software people lend a hand.


Your Project

• Start thinking NOW about a project you might like to do at the end of the week. We will form teams of ~4 people each to work together on these projects. Some suggestions are posted on the SS web.

• On Friday the faculty will review the presentations, and select what they consider to be the best project. One member of that project team (chosen by the group) will get an expense-paid trip to the San Diego AAS meeting (January 2005) and will be invited to speak in the special session on VO Applications.

http://www.sandiego.org/visitorcenter.asp


NVO: History and Motivation


History

• 1990s: NASA establishes wavelength-oriented science archive centers; multiple large ground-based digital sky survey projects initiated

• April 1999, Decadal Survey Panel on Theory, Computation, and Data Discovery met in Los Alamos

– Szalay, Prince, and Alcock coin the name “National Virtual Observatory”

• November 1999, NVO organizational workshop at JHU• February 2000, 2nd NVO workshop at NOAO-Tucson• June 2000, conference held at Caltech, “Towards a Virtual

Observatory”• June 2000, ad hoc steering committee formed• February 2001, AASC/NAS report “Astronomy and Astrophysics in the

New Millennium” released• April 2001, proposal submitted to NSF ITR program, 17 collaborating

organizations, led by A. Szalay (JHU)• September 2001, NSF announces proposal selection• January 2003, first NVO science prototypes shown at Seattle AAS


Decadal Survey Recommendation

• National Academy of Sciences Decadal Survey recommended NVO as highest priority small (<$100M) project “ Several small initiatives recommended by the committee

span both ground and space. The first among them—the National Virtual Observatory (NVO)—is the committee’s top priority among the small initiatives. The NVO will provide a “virtual sky” based on the enormous data sets being created now and the even larger ones proposed for the future. It will enable a new mode of research for professional astronomers and will provide to the public an unparalleled opportunity for education and discovery.”

—Astronomy and Astrophysics in the New Millennium, p. 14


What the Virtual Observatory is…

• A suite of international standards for the discovery, exchange, intercomparison, and analysis of network-accessible astronomical data

• A data access and analysis environment that exploits the emerging computation/software/data Grid

• A framework for data processing that enables and encourages the re-use of algorithms

• A tool for science planning: Identify gaps in coverage of parameter space. Which new missions, instruments, experiments will have largest impact?

• A catalyst for world-wide access to astronomical archives• A routinely used tool of the research astronomer• A vehicle for education and public outreach


What the Virtual Observatory is not…

• A replacement for building new telescopes and instruments• A centralized repository for data• A data quality enforcement organization


Astronomy is Facing a Data Avalanche

Multi-Terabyte (soon: multi-

Petabyte) sky surveys and

archives over a broad range of

wavelengths

Billions of detected sources, hundreds of measured attributes per source

1 nanoSky (HDF-S)

1 microSky (DPOSS)


The Changing Face of Observational Astronomy

• Large digital sky surveys are becoming the dominant source of data in astronomy: > 100 TB, growing rapidly– Current examples: SDSS, 2MASS, DPOSS, GSC, FIRST, NVSS,

RASS, IRAS; CMBR experiments; Microlensing experiments; NEAT, LONEOS, and other searches for Solar system objects …

– Digital libraries: ADS, astro-ph, NED, CDS, NSSDC– Observatory archives: HST, CXO, space and ground-based– Future: QUEST2, LSST, and other synoptic surveys; GALEX,

SIRTF, astrometric missions, GW detectors

• Data sets orders of magnitude larger and more complex than in the past

• Roughly 1 TB/Sky/band/epoch (1 arcsec pixels, 1 byte per pixel)– Human Genome is < 1 GB, Library of Congress ~ 20 TB


Science of a Qualitatively Different Nature

• Statistical astronomy done right – Precision cosmology, Galactic structure, stellar astrophysics– Discovery of significant patterns and multivariate correlations– Poissonian errors unimportant

• Systematic exploration of the observable parameter spaces– Searches for rare or unknown types of objects and phenomena– Low surface brightness universe, the time domain– Confronting massive numerical simulations with massive data

sets


Precision Cosmologyand a better marriage of theory and observations

DPOSS clusters Numerical simulation


Multi-Wavelength Data paint a morecomplete (and more complex!) picture of the universe

Infrared emission from interstellar dust

Smoothed galaxy density map


A PanchromaticApproach to the Universe…

…revealsa more completephysical picture

The resultingcomplexity of

data translatesinto increased

demands fordata analysis,

visualization, andunderstanding


The Time Domain

Megaflares on normal main

sequence stars (DPOSS)


Faint, Fast Transients

The Time Domain


NVO: Technology and Infrastructure


RegistriesNVO Resource

Discovery

Computational Services

Virtual DataConeSear c

hSIA

P, SSA

P VO

Table

F ITS, G

I F, …

Catalogs, Archives, Collections, Models

AD

QL ,

OSQ

NVO Data Access Layer

Queri

es

Resp

onse

s

Portals, User Interfaces, ToolsVOPlot Mirage Topcat conVOT DIS H

TTP, W

eb

, & G

rid S

erv

ices

NVO Architecture

Data

Mod

els, U

CD

s , M

eta

data

Day 1



Discovery



hSIA

P, SSA

P VO

Table

F ITS, G

I F, …


AD

QL ,

OSQ


Queri

es

Resp

onse

s


TTP, W

eb

, & G

rid S

erv

ices

NVO Architecture

Data

Mod

els, U

CD

s , M

eta

data

Day 2



Discovery



hSIA

P, SSA

P VO

Table

F ITS, G

I F, …


AD

QL ,

OSQ


Queri

es

Resp

onse

s


TTP, W

eb

, & G

rid S

erv

ices

NVO Architecture

Data

Mod

els, U

CD

s , M

eta

data

Day 3


VOTable

• Reached international agreement on VOTable V1.0 specification in April 2002

• XML-based standard with in-line data or links to external data• Utilized for basic catalog and image access protocols• Merges “AstroRes” heritage with XML flexibility• Complements FITS• Multiple I/O libraries available (Java, Perl, C++, C#)


Data Models

• Data modeling effort aimed at defining basic data types and relationships among them

• High-level entities: image, spectrum, time series, catalog• Low-level entries: quantity, resolution, time of observation• Interfaces and protocols for other VO services derived from

DM relationships


Data Access Layer

• Data Access Layer is mediator between NVO data requests and data delivery

• Defined “Cone Search” protocol and have ~100 implementations

• Defined Simple Image Access Protocol (SIAP) and have 20+ implementations

• OpenSkyQuery provides database interface to OpenSkyNodes with extended SQL-like VO Query Language

• Specification for Simple Spectral Access Protocol in development


Resource Metadata and Registry

• Resource Metadata describes NVO data collections, services; this metadata is collected into a Registry

• Resource Identifiers are component of resource metadata; have agreed on syntax

• Using Open Archive Initiative protocols for metadata harvesting

• Now focusing on query mechanisms and general updating/synching options

• Prototype registry utilized in science demonstration, Data Inventory Service


Unified Content Descriptors

• UCDs provide common data dictionary for describing contents of catalogs

• CDS initiative, now broadened to international VO discussion• Current discussion focusing on structure and extensibility


VO Query Language

• Working on minimal extensions to SQL to support astronomical queries (e.g., spatial proximity) Astronomical Data Query Language

• Defining standard query service based on SDSS SkyQuery: OpenSkyNode and OpenSkyQuery

• Investigating higher-level query languages; natural language• Xquery


Grid and Web Services

• Increasing number of web services (cone search and SIAP wrappers, for example)

• Registry services will be implemented as web services• Prototyped use of Grid in galaxy morphology science

demonstration• ROME (Remote Object Management Environment) project

provides stateful user interface to Grid-based or other compute-intensive applications

• Working closely with Grid community to understand progress on Grid services, e.g., OGSA, and to determine best time to adopt


NVO-Enabled Science


Science Prototypes

• Science prototypes guide and validate technical initiatives• NVO Year 1

– Brown dwarf candidate search– Gamma-ray burst follow-up– Galaxy morphology measurement (utilizing computational grid)

• Year 1.5– Data Inventory Service

• Year 2– Data Inventory Service with registry-based resource selection– Access to theoretical simulation (globular cluster) and virtual

observations• European VO project

– Type 2 (obscured) quasars: 40 new candidates found– Galactic star formation regions


Data Inventory Service: NVO DataScope

Positions of HST and Chandra observations for GRB010222

Scientific Motivation: Rapid collection of multi-wavelength imaging, catalog and observation data following an interesting transient event is essential. This service can also be used as a general tool to quickly access all data available on any patch of sky for any science use. Data Resources: Multi-wavelength data from any number of sites (currently 13 different sites) sampling energies from X-ray to radio, and including images, object lists, and catalogues of observations. What the VO Brings: Integration and organization of a variety of data sources into an easily comprehensible information set. Scalability to an arbitrary number of data providers. Integrates data with multiple data visualization services. Enabling Technologies:

Standard protocols to remote services such as Cone Search and Simple Image Access, standardized VOTables for data retrieval transformation, and standardized semantics encoded as Uniform Content Descriptors (UCDs). Resource registry. Future Prospects: Customization and quality control of resources searched; more sophisticated use of metadata.


Data Inventory Service






Brown Dwarf Candidate Search

Scientific Motivation: The search for brown dwarfs has been revolutionized by the latest deep sky surveys. A key attribute to discovering brown dwarfs is the federation of many surveys over different wavelengths. Such matching of catalogs is currently laborious and time consuming. This matching problem is generic to many areas of astrophysics.

Data Resources: Sloan Digital Sky Survey (SDSS) Early Data Release (15 million objects) 2-Micron All Sky Survey (2MASS) 2nd Incremental Point Source Catalog (162 million objects)

What the VO Brings: Today, doing datasets is user-intensive and is replicated by many different users. Also, the correlation of these two datasets can take years of CPU time if not done correctly. The NVO brings two key aspects to this problem. First, it removes the need for the user to download large data to their machine, making direct use of distributed data. Second, the matching algorithm used here is computa-tionally efficient and designed to give answers in minutes rather than hours; results can be returned to the user in real-time.

Sloan z magnitudes vs. 2MASS J magnitudes, with brown dwarf candidates in red. Data are from the SDSS Early Data Release and 2MASS 2nd Incremental Release.

Future Prospects: Catalog matching of large datasets is a generic problem in astrophysics. Therefore, making the matching facility available to any user for use on any dataset will greatly enhance the productivity of scientists. Standard I/O formats allow developers to create tools to use the matched data and easily integrate with existing visualization and analysis tools (anomaly detector). Bringing these data together on remote machines with enough CPU to perform analysis (Grid technology) will allow cross-comparisons of unprecedented scale.


As a T dwarf becomes cooler (i.e., methane and water absorptions increase) or more distant…– SDSS detects it only at z’ band– 2MASS detects it only at J band


Demo Leads to Discovery!

• New brown dwarf candidate confirmed spectroscopically with Keck Observatory


Galaxy Morphology in Clusters

Scientific Motivation: Investigate the dynamical state of galaxy clusters and galaxy evolution within the context of large-scale structure. Use galaxy morphology as a probe of dynamical history by calculating, for each galaxy in a cluster:

• Surface brightness • Concentration index• Asymmetry indexThese parameters are analyzed with other indicators such as magnitude, color, peculiar

velocity, position in cluster, and cluster large-scale structure. Data Resources: Computing Resources:Chandra X-ray image (SAO/CXC) USC/ISIROSAT image (GSFC/HEASARC) UW-Madison/NCSADSS image (STScI/MAST) FermilabGalaxy cluster catalogs (NED)CNOC1 cluster images and catalogs (CADC)

What the VO Brings: Distributed data access and Grid-based computing make possible for the first time effective integration of multiple datasets and real-time computing. Integration of data from diverse sources is enabled by standardized data objects and standardized remote computing services. Flexibility of access means that further NVO-compliant images and catalogs can be added easily. Users can select their visualization portal (Aladin, OASIS, DS9).

Enabling Technologies: VOTable, NVO-compliant catalog and image access, standard semantics, Grid computing infrastructure.Future Prospects: Dynamic discovery and selection of image, catalog, and computing resources. User-selection of analysis tools and ability to publish data to the NVO framework.








Globular Cluster Simulations


Globular Cluster Simulations


Spectral Database Browser










NVO: Broader Context and Vision


International VO Alliance



• The IVOA brings together the astronomers, developers, and managers of the VO initiatives world-wide– Agreements on standards for data access (VOTable, catalog

queries, image retrieval, resource descriptions, etc.)– Coordination of development activities– Sharing of software– Sharing of experience

• 15 participating organizations: Astrogrid, AVO, US-NVO, VO-Australia, VO-Canada, VO-China, VO-France, VO-Germany (GAVO), VO-Spain, VO-Hungary, VO-India, VO-Italy (DRACO), VO-Japan, VO-Korea, VO-Russia

• http:www.ivoa.net




The VO Vision

• The VO is the “semantic web” for astronomy (Tim Berners-Lee)

• The VO democratizes astronomical research• The VO brings the universe to your desktop

– The professional astronomer– Graduate students– Undergraduates– K-12– Amateurs– The public

http://us-vo.org


On with the tour…


On with the tour…

Documents

2004-09-13NVO Summer School, Aspen Center for Physics1 National Virtual Observatory Applications Software Development Summer School: Introduction and Overview