Virtual organizations in astronomy and beyond

Prof. Giuseppe LongoChair of Astrophysics - Department of Physical Sciences University of Napoli Federico II – ItalyNational Institute of Astrophysics – Napoli Unitlongo@na.infn.it http://people.na.infn.it/~longo/

Tblisi, March 28-30 2007

The Exponential Growth of Information in Astronomy

Total area of 3m+ telescopes in the world in m2, total number of CCD pixels in Megapix, as a function of time. Growth over 25 years is a factor of 30 in glass, 3000 in pixels.

•Gigapixel arrays are a reality,hence optical and near infrared surveys are becoming common

• Space missions archives are being federated

• Old datasets (space and ground based instruments) are being federated

• Estimated 1 TB per day in 2008

Massive numerical simulationsDistributed computing (PB per simulation)

Astronomy, more than other sciences is facing a Major Data Avalanche ( … a true tsunami…)

Large survey projects from ground and from space

(past, ongoing, future)

Distributed data repositoriesData are not where the users are PetaBytes of data / week

GRID INFRASTRUCTURE

Data federation of MDS

Adoption of standards and common onthologies

Data analysis and interpretation

Need for a new generation of tools (A.I. based) capable to work in a distributed environment

International Virtual Observatory Alliance

The distributed environment

• Once the VO’a will come operationals, there will be no need to have locally powerful computing facilities,

• Federation of existing and new databases through adoption of common standards Network access to the databases

• To provide the user with user friendly access to all federated data• To allow the user to access distributed computing facilities and to exploit all

available data withouth moving the data but the codes (… data remain at data centers where the expertise is)

• To open entirely new paths to discovery process in astronomy (but not only!)

What are some of the goals of VO’s

• VO are the most democratic tool ever implemented by any scientific community.

• Data repositories are mostly public (either immediately or after proprietary period of observers)

• Data analysis and data mining tools are available to the international community through a distributed computing environment

• Every one can contribute (either with new data or with new SW-tools)

• Once the VO be implemented, new – top level science will be at the “fingers” of any competent scientist who has minimal computing facilities and a good access to the WWW

What is being done in Napoli 1 – The surveys

VLT Survey Telescope(Napoli,ESO)P.I: Prof. M. Capaccioli

2.5 m diameter - OPTICAL1x1 sq deg f.o.v.16 k x 16k CCD mosaic (optical)

New technologyAdaptive optics

0.2 arcsec psf

Operational end 2007

100 GB raw data/night

Nobel laureate R. Giacconi visiting VST factory

VLT site, cerro Paranal (Chile)

Omegacam

French – Netherlands – Italy consortium

16 k x 16 k array CCd mosaic

Ready

Data processing pipeline

European FP6 network ASTROWISE

Real time storage and processing of the VST data

What is being done in Napoli 2 – The detector

What is being done in Napoli 3 – The computingCAMPUS GRID

Dipartimento di Chimica

CDSGARRArmadio

telematicoinfrastrutturale

di Centro stella

Dipartimento diMatematica e Applicazioni

Dipartimento di Scienze FisicheLocale 1G01

“Sala dell’infrastruttura GRID principale”

Campus GRID

512 +15 + 24 + 16 + 128 nodes

150 TB storage

(IBM, DEC - Alpha, etc.)

16 GBaud optical fibers backbone

Recently evolved intoPON - SCOPE

3.6 M€ (8.2 M€ total) for Hardware (512 boards with 4 CPU’s)

Financed by Italian Government

Operational end 2007

What is being done in Napoli 4 – The Data mining

Draco Projectbuilding the GRID infrastructure for the Italian VO400 k€ - MIUR

Euro – VO, VO-Tech European Virtual Observatory Technological Infrastructures

European Infrastructures for VO (UK, D, I, F, etc.) 6.6 M€ - EU

VO- Neural (Napoli lead)

Building Data Mining and Visualization for Massive Data Sets in a Distributed Environment

Cost- Action 283 EU

Parameter space of incredibly

high dimensionality (N>>100)

Complex parameter space

Example 1: panchromatic view of the universe

X

Opt.

radio

IR.

Crab Nebula: SN 1054 a.C.

Example 2: a new way to do conventional astronomy

Selection of quasar candidates from a 3 band photometric survey

Example: exploring a 3D Parameter Space

Given an arbitrary parameter space:

• Data Clusters• Points between Data

Clusters• Isolated Data Clusters• Isolated Data Groups• Holes in Data Clusters• Isolated Points

Nichol et al. 2001

Slide courtesy of Robert Brunner @ CalTech.

Example: 21-D parameter space

VO- NeuralProbabilistic Principal Surfaces Negative ENtropy Clustering + Dendrogram

• Multiwavelenght – multiepoch – multinstrument data (federation of databases) hence there is a strong need for a new generation of data processing, data visualization and data-mining tools

• These tools must be largely based on Artificial Intelligence

• Interoparibility is a must (Plastic is a standard)

Many probles to be solved:

• Missing data (bew data models are needed)

• Parallelization of existing codes

• Sensibilization of the community through selected scientific cases (astrophysics, bioinformatic, marketing, etc.)

THESE TOOLS ARE OF WIDE APPLICATION: bioinformatics, geophysics (environment, stratigraphy, etc.), business (stock market, marketing strategies, etc.). Therefore interdisciplinarity is a must!

We (UK, F, I, D, USA, India) intend to pursue the above tasks using the following instruments:

National funds and private companies

EU funds through new COST Action and ITN

Eventually through RI

US funds through NSF

Conferences and Schools for young students (dissemination is CRUCIAL)

NEW POTENTIAL PARTNERS ARE ENCOURAGED TO CONTACT ME: longo@na.infn.it

Plate or digital archives of astronomical dataOther types of scientific dataAdvanced programming and mathematical know-how’s