DAL (Data Access Layer) protocols

Workshop on How to Publish Data in VO

ESAC, June 25-June 29 2007

DAL (Data Access Layer) DAL (Data Access Layer) protocolsprotocols

Jesus SalgadoJesus SalgadoESAVO-ESACESAVO-ESAC

[email protected]@sciops.esa.int

Jesus SalgadoJesus Salgado Plenary Session, Page Plenary Session, Page 22

Why do we need protocols?Why do we need protocols?


Why do we need protocols? (II)Why do we need protocols? (II)


Simple ProtocolsSimple Protocols

SIAP (Images), SSAP (Spectra), SIAP (Images), SSAP (Spectra), SLAP(Spectral Lines)SLAP(Spectral Lines)

HTTP basedHTTP based VOTable responseVOTable response Similar interfaces; different protocols for Similar interfaces; different protocols for

different types of products (images, spectra, different types of products (images, spectra, lines,…) to converge in a latter statelines,…) to converge in a latter state

Not difficult to be developedNot difficult to be developed


SIAPSIAPSimple Image Access ProtocolSimple Image Access Protocol

(Tody et al.)(Tody et al.)


SIAP: IVOA standardSIAP: IVOA standard […] […] standard for retrieving image data from a variety of standard for retrieving image data from a variety of

astronomical image repositories through a uniform astronomical image repositories through a uniform interfaceinterface […][…]

[…] […] based primarily on two documentsbased primarily on two documents […] […]"Simple Image Retrieval: Interface Concepts and Issues""Simple Image Retrieval: Interface Concepts and Issues",, […] […] "Simple Cone Search specification""Simple Cone Search specification" [...] [...]

Originally [...] primarily as an "image on demand" service, with Originally [...] primarily as an "image on demand" service, with images created on-the-fly [...]images created on-the-fly [...]

[…][…]Required query parameters: POS, SIZE, FORMATRequired query parameters: POS, SIZE, FORMAT[…][…] […][…]Required response columns: Image_TitleRequired response columns: Image_Title, POS_EQ_RA_MAIN, , POS_EQ_RA_MAIN,

POS_EQ_DEC_MAIN , Image_Format […]POS_EQ_DEC_MAIN , Image_Format […]

http://bill.cacr.caltech.edu/cfdocs/usvo-pubs/files/vo-simage.pdf

http://us-vo.org/metadata/conesearch/


SIAP BasicsSIAP Basics

SIAP Servers service URLs

Query Response:VOTable with list of

matches

Data Response:Image

Coordinates/Target matches?

SIAP Servers available?

Give ACCREF data (real Data)

Client application REGISTRY


SIAP: factsSIAP: facts

More than 80 SIAP services registeredMore than 80 SIAP services registered The most successful IVOA protocol up to The most successful IVOA protocol up to

nownow Easy access to images from different Easy access to images from different

spectral ranges in the same VO application spectral ranges in the same VO application (e.g. Aladin)(e.g. Aladin)

Some examples:Some examples:– HST, 2MASS, XMM, SDSS, ISO, Integral, HST, 2MASS, XMM, SDSS, ISO, Integral,

ROSAT, FIRST, IRAS,… and many moreROSAT, FIRST, IRAS,… and many more


SSAPSSAPSimple Spectral Access ProtocolSimple Spectral Access Protocol

(Tody et al.) (Tody et al.)


SSAP: A summarySSAP: A summary […][…]To define a uniform interface to spectral dataTo define a uniform interface to spectral data[…][…]

[…][…]The term “Simple” in Simple Spectral Access Protocol The term “Simple” in Simple Spectral Access Protocol refers to the design goal of simplicity in both implementing refers to the design goal of simplicity in both implementing spectral data services and in retrieving spectroscopic data spectral data services and in retrieving spectroscopic data from distributed data collectionsfrom distributed data collections[…][…]

[…][…]Required query parameters: POS, SIZE, FORMATRequired query parameters: POS, SIZE, FORMAT[…][…]

[…][…]Required response columns: FORMAT, ACREF, SED Required response columns: FORMAT, ACREF, SED Object, Data Source, Coverage MetadataObject, Data Source, Coverage Metadata…]…]


SSAP BasicsSSAP Basics

SSA Servers service URLs

Query Response:VOTable with list of

matches

Data Response:Spectrum

( fits, VOTable, …)

Coordinates/Target matches?

SSA Servers available?

Give ACCREF data (real Data)

Client application REGISTRY


SSAP 1.01 - Basic UsageSSAP 1.01 - Basic Usage Simple querySimple query

– POS, SIZE - like cone searchPOS, SIZE - like cone search– Possibly refined by spectral or time bandpass, etc.Possibly refined by spectral or time bandpass, etc.– Most metadata in query response is optionalMost metadata in query response is optional

Data retrievalData retrieval– URL-basedURL-based– Get back a dataset (normally VOTable or FITS)Get back a dataset (normally VOTable or FITS)– In simplest case could be wavelength, flux as text (for In simplest case could be wavelength, flux as text (for

Spectrum)Spectrum)– Or external data pass-throughOr external data pass-through


SSAP 1.01 - Query InterfaceSSAP 1.01 - Query Interface Mandatory query parametersMandatory query parameters

– POSPOS RA, DEC (ICRS)RA, DEC (ICRS)– SIZESIZE diameter (decimal degrees)diameter (decimal degrees)– FORMATFORMAT VOTable, fits, xml, text, graphics, VOTable, fits, xml, text, graphics,

html, external html, external

Should other parameters be mandatory?Should other parameters be mandatory?– e.g., BANDPASS, TIME, SPECRES, APERTUREe.g., BANDPASS, TIME, SPECRES, APERTURE


SSAP 1.01 - Query ResponseSSAP 1.01 - Query Response Dataset MetadataDataset Metadata

– Dataset.Type Dataset.Type Spectrum, TimeSeries, SED, etc.Spectrum, TimeSeries, SED, etc.– Dataset.DataModel Dataset.DataModel DM name, e.g., "SSA-V1.0"DM name, e.g., "SSA-V1.0"– DataID.Title DataID.Title Brief descriptive title of datasetBrief descriptive title of dataset– Dataset..Length Dataset..Length Number of points in Number of points in

datasetdataset– Dataset.SSA.SpectralAxis Dataset.SSA.SpectralAxis SpectralCoord axis (external data)SpectralCoord axis (external data)– Dataset.SSA.FluxAxis Dataset.SSA.FluxAxis Flux axis (external data)Flux axis (external data)– Dataset.SpectralSIDataset.SpectralSI SI factor and dimensionsSI factor and dimensions– Dataset.FluxSIDataset.FluxSI SI factor and dimensionsSI factor and dimensions– Dataset.CreationType Dataset.CreationType atlas, pointed, cutout, resampledatlas, pointed, cutout, resampled– DataID.DataSourceDataID.DataSource Original source of data Original source of data


SSAP 1.1– SSAP 1.1– Some specialized response metadataSome specialized response metadata

Characterization - Reference FramesCharacterization - Reference Frames– CoordSys.SpaceFrame.NameCoordSys.SpaceFrame.Name Spatial Coordinate frame (default ICRS)Spatial Coordinate frame (default ICRS)– CoordSys.SpaceFrame.Equinox CoordSys.SpaceFrame.Equinox Equinox (2000)Equinox (2000)– CoordSys.TimeFrame.Name CoordSys.TimeFrame.Name Timescale (TT)Timescale (TT)

Characterization - CoverageCharacterization - Coverage– Char.SpatialAxis.Coverage.Location.ValueChar.SpatialAxis.Coverage.Location.Value

Observed position, e.g., RA DECObserved position, e.g., RA DEC– Char.SpatialAxis.Coverage.Bounds.Extent Char.SpatialAxis.Coverage.Bounds.Extent

Aperture angular diameterAperture angular diameter– Char.SpectralAxis.Coverage.Location.Value Char.SpectralAxis.Coverage.Location.Value

Midpoint of Spectral coord. rangeMidpoint of Spectral coord. range– Char.SpectralAxis.Coverage.Bound.Extent Char.SpectralAxis.Coverage.Bound.Extent

Width of spectrumWidth of spectrum


SSAP implementationsSSAP implementations Implement service able to provide SSAP-like accessImplement service able to provide SSAP-like access

– ESA’s Infrared Space Observatory (ISO) spectra available in ESA’s Infrared Space Observatory (ISO) spectra available in SSAP-like form since December 2003. First ever SSAP-like server.SSAP-like form since December 2003. First ever SSAP-like server.

– Other SSAP-like services that followed suite (in chronological Other SSAP-like services that followed suite (in chronological order): IUE, HST, SDSS, FUSE, HYPERLEDA….up to 18 services order): IUE, HST, SDSS, FUSE, HYPERLEDA….up to 18 services

(all available from (all available from VOSpecVOSpec tool) tool) Implementation of latest protocol specification not ready for Implementation of latest protocol specification not ready for

most of themmost of them Design and implement client able to consume SSAP Design and implement client able to consume SSAP

spectra servicesspectra services– ESA’s initiative response called VOSpec: a tool for handling SSAP-ESA’s initiative response called VOSpec: a tool for handling SSAP-

like spectra in VO contextlike spectra in VO context

http://esavo.esa.int/vospec/


TSAP (Theoretical spectra)TSAP (Theoretical spectra)

Same protocol (input/output) re-used to give Same protocol (input/output) re-used to give support theoretical spectral serverssupport theoretical spectral servers

Both coverage input parameters and output Both coverage input parameters and output characterization coverage metadata non-characterization coverage metadata non-compulsorycompulsory

As it will be seen in next section, discovery As it will be seen in next section, discovery of server capabilities is crucialof server capabilities is crucial

9 services already in place9 services already in place


SLAPSLAPSimple Line Access ProtocolSimple Line Access Protocol

(Salgado et al.) (Salgado et al.)


SLAP (Simple Line Access Protocol)SLAP (Simple Line Access Protocol)

VOTable, VOTable, one spectral line per rowone spectral line per row

A SIAP-like URL service, would allow A SIAP-like URL service, would allow an easy implementation for all spectral an easy implementation for all spectral line providersline providers


SLAP input parametersSLAP input parameters If the only compulsory parameters are the minimum and maximum If the only compulsory parameters are the minimum and maximum

wavelength, the number of lines in the output result, in particular for wavelength, the number of lines in the output result, in particular for theoretical spectral line databases, could be HUGE.theoretical spectral line databases, could be HUGE.– Some non-compulsory parameters added to the protocol to filter or Some non-compulsory parameters added to the protocol to filter or

score the output linesscore the output lines– CHEMICAL_ELEMENT:CHEMICAL_ELEMENT: This parameter would constraint the This parameter would constraint the

search to the chemical element selected (syntax problems, but search to the chemical element selected (syntax problems, but some standards found)some standards found)

– LEVEL_ENERGY_START & LEVEL_ENERGY_END: LEVEL_ENERGY_START & LEVEL_ENERGY_END: Parameters Parameters to specify the minimum and maximum energy for the START & to specify the minimum and maximum energy for the START & FINAL level of the transition. Filter at CLIENT sideFINAL level of the transition. Filter at CLIENT side

– TEMPERATURE:TEMPERATURE: This parameter would be used (in particular for This parameter would be used (in particular for theoretical spectral line databases) to score the lines in the output theoretical spectral line databases) to score the lines in the output using physical models. Score at SERVER sideusing physical models. Score at SERVER side


SLAP extra parametersSLAP extra parameters The servers could have extra parameters that cannot be identified The servers could have extra parameters that cannot be identified

easily and included in the document.easily and included in the document. For Theoretical Spectral Line services, the parameters could be used For Theoretical Spectral Line services, the parameters could be used

to filter/score the result due to physical modelsto filter/score the result due to physical models For Observational Spectral Line services, the parameters could be For Observational Spectral Line services, the parameters could be

project dependentproject dependent Solution adopted:Solution adopted:

– Use same approach than proposed for TSAP. All the input parameters will Use same approach than proposed for TSAP. All the input parameters will be DESCRIBED in a FORMAT=METADATA query.be DESCRIBED in a FORMAT=METADATA query.

– The description should include not only the parameter names but a The description should include not only the parameter names but a description, UCDs (to identify the physical meaning of the parameters), description, UCDs (to identify the physical meaning of the parameters), utypes whenever possible (to Line Data Model or another IVOA data utypes whenever possible (to Line Data Model or another IVOA data model)model)

– This FORMAT=METADATA could be requested by the registry services or This FORMAT=METADATA could be requested by the registry services or by the VO clientsby the VO clients


FORMAT=METADATAFORMAT=METADATA <RESOURCE type="results"> <DESCRIPTION>IASD Simple Line Access Service</DESCRIPTION> <INFO name="QUERY_STATUS" value="OK"/> <PARAM name="INPUT:WAVELENGTH" ucd="em.wl;stat.min" utype=”ldm:Line.wavelength” value="30"> <DESCRIPTION> Specify the wavelength spectral range. To be specified in micrometers. This wavelength will be interpreted as the wavelength in the vacuum of the transition originating the line </DESCRIPTION> </PARAM> <PARAM name="INPUT:CHEMICAL_ELEMENT" ucd=“phys.atmol.element" utype=”ldm:Line.initialElement.name” value="190"> <DESCRIPTION> Specify the chemical element that originates the line </DESCRIPTION> </PARAM> <PARAM name="INPUT:OBSNO" ucd="obs.id"> <DESCRIPTION> Specify the ISO observation number where this line has been identified </DESCRIPTION> </PARAM> ……………………………………..


Output “MUST” Compulsory fields– wavelength (ucd="em.wl”; utype=”ldm:Line.wavelength”)

wavelength in the vacuum of the transition originating the line in micrometers

– title (ucd=“em.line”;utype=”ldm:Line.title”) contains a small description identifying the line.

Output “SHOULD” non-compulsory field. – chemicalelement_name

(ucd=“phys.atmol.element”;utype=”ldm:ChemicalElement.name”) contains a name of the chemical element source of this line

– initial_level & final_level (ucd=“phys.atmol.[initial/final];phys.atmol.level”; utype=”ldm:Line.[initial/final]Level) contain a full description of the initial & final levels of the transition originating the line

SLAP output parameters (I)SLAP output parameters (I)


SLAP output parameters (II)SLAP output parameters (II) Summary of output COULD optional fields

– observed_wavelength (ucd="em.wl“; utype=”ldm:Line.observedWavelength”) contains the observed wavelength in the vacuum of the transition originating the line in micrometers, as it was observed. This could be used by observational spectral line databases

– score (utype=”Query.Score”) similar concept to SSAP 0.9. Easier to be implemented for physical models

– level_energy_start & level_energy_end (ucd="phys.energy; phys.atmol.[initial/final];phys.atmol.level”; utype=”ldm:Level.energy.[start/end]”) describe the energy for the starting & final levels of the transition which originates this line.


URL-like fields– publication_link (ucd="meta.bib”) specifies a http link that contains

a bibliographic reference related to the spectral line – Other fields could have ucd=“meta.ref.url" specifying URLs that

contains extra information related to the spectral line Non-Standard output fields

– In many occasions, extra scientifically interesting parameters could be added to the output. Implementors are encouraged to add descriptions and UCDs to the return fields to clarify the meaning of this information and utypes to the Line data Model or other existing Data Model, whenever possible.

4 services already in place; IASD, Cielo (observational), NIST, Lerma (laboratory). Plans to include ALMA in a near future

SLAP output parameters (III)SLAP output parameters (III)


Documents

DAL (Data Access Layer) protocols