Hel

ioph

ysic

s In

tegr

ated

Obs

erva

tory

(H

ELI

O)

Building a VxO for Heliophysics

Bob Bentley (UCL-MSSL)

21 June 2007IHY General Assembly, Torino, Italy

2

Hel

ioph

ysic

s In

tegr

ated

Obs

erva

tory

(H

ELI

O)

Heliophysics

Heliophysics explores the Sun-Solar System ConnectionAn generalization of the study of space weather (SWx++)

Heliophysics sits in the boundary between two communities

Astrophysicso Study of solar phenomena helps understanding of stellar

observations

Planetary sciences (including Earth sciences)o Solar activity can influence the environment on/around the

planets

The discipline must be aware of the need to support the interests of both communities

A virtual observatory that supports Heliophysics must facilitate access to data from a number of communities

As such, it is in essence a next generation VxO

Proposal submitted under EC’s FP7Heliophysics Integrated Observatory (HELIO)

3

Hel

ioph

ysic

s In

tegr

ated

Obs

erva

tory

(H

ELI

O)

Heliophysics

The communities that constitute heliophysics have evolved independently over decades and even centuries

Communities include solar, heliospheric, magnetospheric and (possibly) ionospheric physics

o Sharing ionospheric physics with planetary sciences gives overlap

Little or no coordination of how each has evolved Each has very different ways of describing, storing and exploiting the data from their observations, varying use of standards

The desire to solve science problems that span disciplinary boundaries is driving the need to provide combined access to these data To do this, we need to find ways to:

Tie the data together through searches across all the domainsPresent any results in a form that does not require a detailed understanding of each discipline

This requires the re-evaluation of the capabilities provided within each community and some corrective action

4

Hel

ioph

ysic

s In

tegr

ated

Obs

erva

tory

(H

ELI

O)

Types of metadata

To facilitate a VxO for heliophysics we need to examine the metadata that is required.

There are many ways metadata can be grouped, one is:Search metadata

o Metadata used to identify time intervals and sets of data of interest

Observational metadatao Metadata used to describes the observations, e.g. FITS headers

Storage metadatao Metadata that describes how the data are stored and accessed

Administrative metadatao Metadata that allows the system to exploit the available

resources

The rest of the talk will discuss issues related to these

5

Hel

ioph

ysic

s In

tegr

ated

Obs

erva

tory

(H

ELI

O)

Searches

In heliophysics, we are interested in how an event on or near the solar surface can propagate through the heliosphere and affect a planetary environment

May also want to work backwards and look for the cause of an effect – what solar event caused this ionospheric activity…

Searches should identify interesting time intervals based on a combination of event, features, etc. metadata

Light curves and images my also be used to augment the search

Each community of some combination of these metadataThere are concerns about the quality and integrity of these metadata and whether it is adequate to support the searches we would like to undertake

6

Hel

ioph

ysic

s In

tegr

ated

Obs

erva

tory

(H

ELI

O)

Solar search metadata

Searches in Heliophysics are mainly event drivenPhenomena occur on or near the solar surface

Event data gives time and location of phenomenaFeature data provides details of location and size of structures that may be relevant

Time information can be expressed in many waysEssentially these are the same, with simple transformations

Spatial information can be expressed in terms of:Coordinates in the observing frame – e.g. arcsecs from disc centreCoordinates on the rotating body of the Sun – Carrington coords.

The position of the observer was ignored for the most part

Helio-seismology is an exception

In the bigger picture of heliophysics, also need to include the viewing perspective (c.f. STEREO)

7

Hel

ioph

ysic

s In

tegr

ated

Obs

erva

tory

(H

ELI

O)

Other search data

Observation of phenomena in the heliosphere and near/on planets are more complex

For in-situ observations in the heliosphereTime is when a phenomena affected (passed) the observerPosition of the observer relative to the Sun is key to understanding

When the in-situ observations are made on/near a planetPosition of the observer relative to the planet is also important

Relating events that are defined from in-situ data to those on/near the Sun requires an understanding of how events propagate

Details of the velocity structure of CMEs and the solar wind are not easy to determine…HELIO plans to develop a tool that will use even/feature data to refine a model to trace effects forwards from causes, etc.

8

Hel

ioph

ysic

s In

tegr

ated

Obs

erva

tory

(H

ELI

O)

Simple, but not so simple

In principle this all seems fairly obvious, but lets look in detail at some common solar event data

On 20 January 2005 there was an X7.1 flare that was intensely geo-effective. The flare was associated with particle event and a CME; it was also observed by ground-level neutron monitors – a GLE.

Many superlatives were used to describe the evento "The solar energetic particle event of January 20 2005 has been called,

by some measures, the most intense in 15 years..." (Mewaldt et al., 2005)

o ”The fastest rising SEP event of current cycle [cycle 23]" (Rawat et al., 2006)

o ”The most spectacular [solar event] of the Space Age" (Tylka et al., 2006)

o ”The largest GLE [GLE 69] in half a century" (Bartol Research Institute)

But event is absent from the NOAA SEC list of "Solar Proton Events Affecting the Earth Environment"

When you look at the data and how lists are created, you realize that the lists are deficient in several ways

Humans and SmFCACs can understand what happened, but It is harder for machines...

9

Hel

ioph

ysic

s In

tegr

ated

Obs

erva

tory

(H

ELI

O)

X7.1 of 20 Jan 2005

The event was one of several from AR 10720Two other X class flares and several M class flares occurred in previous 3 days; others before this

10

Hel

ioph

ysic

s In

tegr

ated

Obs

erva

tory

(H

ELI

O)

X7.1 of 20 Jan 2005

11

Hel

ioph

ysic

s In

tegr

ated

Obs

erva

tory

(H

ELI

O)

X7.1 of 20 Jan 2005

At the time of the event, the proton levels had not returned to normal after previous events

The criteria fails to recognize a new event

o NOAA lists event on 16 Jan

The X-ray data also suffers from problems

The end of an event is defined by when the counts drop to 50%

o New events can “interrupt” existing events

The shape and true duration of the decay phase are lost

o NOAA gives start 0636; end 0726

Not all locations are tagged!!Significant brightenings seen on images not declared as flares

12

Hel

ioph

ysic

s In

tegr

ated

Obs

erva

tory

(H

ELI

O)

Some of the problems

That major events can be “missed” is worrying and makes automated searches difficultA search for long duration events would yield spurious resultsSince the locations of all flares are not known, it is impossible to know if they will be geo-effectiveInstrument flare lists have gaps – nights, off times, etc. – but the reason for a null result is not included

13

Hel

ioph

ysic

s In

tegr

ated

Obs

erva

tory

(H

ELI

O)

Improving event data

Existing flare lists can give a distorted picture what has occurred

Such deficiencies make it difficult for non-experts to use themThe community “knowledge” is not written down

Need to re-evaluate and regenerate the event data in all domains with the idea that they will be used in a joint search across the domains

Ensure events more accurately describedInclude information that might explain null results

14

Hel

ioph

ysic

s In

tegr

ated

Obs

erva

tory

(H

ELI

O)

Observation metadata

Metadata related to observations gives information about how the observation was made, etc.

Often quality issues related to the metadata that is providedParameters sometimes missing, or wrongInconsistent use of information, “synonyms” for keywords

In solar data, space-based observations much better described than their ground-based counterparts

Ground-based observations are only source in some wavelengths and need access to as many observatories as possibleIf ionospheric included, good coverage of ground stations key

Researchers often used to deficiencies in their own domainDifficult for machines to handle if it is not quantified properly

VxOs can sometimes develop ways of patching the gapsWhat do we do with this information? How is it shared?

15

Hel

ioph

ysic

s In

tegr

ated

Obs

erva

tory

(H

ELI

O)

Observational metadata

In solar physics we have developed suggested standards

SOHO made good start with their keyword documentEGSO enhanced concepts with its data model

Situation better than it was but adoption still not universal

Even some problems within SOHO…

At the Virtual Observatories in Geoscience (VOiG) Conference we discussed whether a standard for observational metadata that span wide set of domains is possible

Core part that is required – (time, location, etc.)Additional information that may be specific to a domainOther information that the instrument team wants to add

Not practical to have metadata same for all domains

HELIO proposes to use semantics to develop a data model that bridges the data models of the domains

16

Hel

ioph

ysic

s In

tegr

ated

Obs

erva

tory

(H

ELI

O)

Storage Metadata

How data are stored in a data source can make a lot of difference to their accessibility

EGSO has the concept of resource-rich and resource-poor providers

Resource-rich providers – e.g. SDAC – should provide what is needed in response to simple queryResource-poor providers may only be able to made data accessible over Internet

At VOiG we discussed whether we should provide guidelines on ways that data could be organized for resource-poor providers

Existing VxOs have opinions on what is easy to useProviding data following simple naming conventions in an ordered directory structure would make them simpler to accessSimple textual catalogue might provide additional information

17

Hel

ioph

ysic

s In

tegr

ated

Obs

erva

tory

(H

ELI

O)

Sources available through EGSO

18

Hel

ioph

ysic

s In

tegr

ated

Obs

erva

tory

(H

ELI

O) Ground based observatories

need to make their data more available and in more useful formats (e.g. FITS) and better metadata

19

Hel

ioph

ysic

s In

tegr

ated

Obs

erva

tory

(H

ELI

O)

Standards

Need to extend the use of standards to all domains so that all future data are more compatible with the VxOs

The situation has changed with enhancements to technology; providers need to ensure they are more compliantVxO will need to handle problems with the older data; providers cannot be expected to do it

Standards need to be developed in collaboration with the community and funding agencies

In developing standards need to draw on the experience gained within the general (VO) community and adopt the best ideas available

eGY is a good forum to engender discussion on these topics!!

20

Hel

ioph

ysic

s In

tegr

ated

Obs

erva

tory

(H

ELI

O)

Conclusions

Developing a virtual observatory to support heliophysics will not be simple, but we have the technologyCooperation of the community essential if we are to succeed

Some of the possible problems have been highlightedTo address them, we need to engage the communities in all the domains that constitute heliophysics and develop standards that will facilitate the process

In principle no reason why all ground-based observatories should not be included – if they want to be…

21

Hel

ioph

ysic

s In

tegr

ated

Obs

erva

tory

(H

ELI

O)

22

Hel

ioph

ysic

s In

tegr

ated

Obs

erva

tory

(H

ELI

O)

Outline

Examine some of the items needed to facilitate a virtual observatory for heliophysicsIdentify deficiencies and suggest solutions

Talk evolved as I was preparing it…Started looking at proton events and GLEs because of space weather effects related to aviation

Found problems with the metadata