Studying the Milky Way using the WFAU Archives

Nicholas Cross, Rob Blake, Ross Collins, Mark Holliman, Mike Read, Eckhard Sutorius, Nigel Hambly,

Andy Lawrence, Bob Mann, Keith NoddleWide Field Astronomy Unit, Institute for Astronomy,

University of Edinburgh, UK

MW Astrophysics from WF Surveys 2

WFAU Science Archives

30th March 2015

EARLIER PROCESSING AT CASU

WFAU

MW Astrophysics from WF Surveys 3

Roles of the Science ArchiveRepository of (e.g. VISTA) data from reduced images

to complex, catalogue products.Interface for survey teams and community to

explore survey products to do science.Ability to work with data from project and other data

Interface for survey teams to check data for quality control.

Requires both:a dynamic main-DB which is updated with new data,

better calibration, reprocessing, quality control, higher order products.

Static, well documented release-DBs that can be referred to in publications.

30th March 2015

Common DesignWSA, VSA, OSA developed through VDFS – VISTA Data

Flow SystemSimilar input image and catalogue FITS files generated

by CASU MEF format, header info, table columns etc

Common software architecture, data pipeline, user interfaces, hardware architecture. Instrument dependent functionality (as well as

programme)GES is significantly different: spectroscopic, wider

range of data from multiple groups, different interfacesBUT uses a lot of the underlying philosophy, software

architecture, hardware designs.

30th March 2015 MW Astrophysics from WF Surveys 4

VST- ATLASGaia-ESO SpectroscopicVISTA VVV, VHS, [VMC]UKIDSS GCS, GPS, LAS; UHS

Milky Way Surveys

30th March 2015 MW Astrophysics from WF Surveys 5

MW Astrophysics from WF Surveys 6

Archives: http://surveys.roe.ac.uk/

30th March 2015

MW Astrophysics from WF Surveys 7

Imaging Surveys PipelineIngest nightly processed image and catalogue data from

CASUProvenance - link related imagesQuality Control: Automated + input from teams.Link tile and pawprint data (VISTA)Process data for semester - done per programme :

Produce and ingest deep stacks/tiles/mosaics + cataloguesMerge pass-band catalogues to create source tablesCreate neighbour tables to link external cataloguesLink multi-epoch data and calculate variability statistics

Release a documented, static data product to users Create useful interface tools for users to query specific

data, view and analyse itConvert data to ESO format and copy to ESO (VISTA, VST)

30th March 2015

GES ArchiveSpectroscopic survey of ~100,000 stars, covering all

major components of MW using VLT-FLAMES Archive includes:

list of targets with photometry used in the target selection

spectra processed by CASUatomic and molecular line listsRecommended astrophysical parameters derived by

various working groups: Teff, log(g), abundances …

Neighbour tables to other surveys: SuperCOSMOS, VST-ATLAS, 2MASS, VISTA-VHS

No public release yet. 3rd internal data release. Public releases in a few months time.

30th March 2015 MW Astrophysics from WF Surveys 8

Value Added DataData created by science teams:VMC :

PSF cataloguesVariable types (Cepheid, Eclipsing Binary)

VVV:PSF catalogues (1 epoch) [Not yet delivered]Proper motion [Not yet delivered]

GES:Wide range of parameters from multiple working

groups.Work to improve connections of team generated data

with CASU/WFAU generated to maximise usefulness.

30th March 2015 MW Astrophysics from WF Surveys 9

3D Extinction MapsIncluded

VVV/Glimpse extinction maps of bulge (Chen et al. 2013)

Join to vvvSource table.

Filter conversions for VISTA, UKIRT, VST, Spitzer and WISE filters

Easily produce extinction corrected SEDs, CMDs.

Would like to add in more (IPHAS, GES …)

30th March 2015 MW Astrophysics from WF Surveys 10

Survey Interfaces (see DEMO)Coverage MapsArchive ListingGetImage / Multi-GetImageRegionCrossIDFreeform SQL

Enhanced versionADQL (OSA only so far)

30th March 2015 MW Astrophysics from WF Surveys 11

MW Astrophysics from WF Surveys 12

Data VolumesImages:

WFCAM 1720 raw image frames a day (4 2kx2k)VISTA 580 raw image frames a day (16 2kx2k)VST 100 raw image frames a day (32 4kx2k) ~1PB of image data

Catalogues: VVV completely dominates: combination of high

source density, short exposure time.Next VVV release will be 40TB55 billion detections. Largest astronomical SQL database!

30th March 2015

MW Astrophysics from WF Surveys 13

Surveys are not independent entities

Importance of multi-wavelength astronomyWide range in temperatures, dust extinctionRare objects (e.g. high-z QSOs, BDs, odd

transients)Common survey fields

VST-ATLAS and VHSUKIDSS LAS/UHS and SDSSVVV, GPS, IPHAS, VPHAS+

Need for data integration Cross-neighbours tables, publishing to VOMatched aperture photometry

30th March 2015

MW Astrophysics from WF Surveys 14

WFAU VDFS Publications

Hambly et al. 2008, MNRAS, 384, 637 (WSA)

Cross et al. 2009, MNRAS, 399, 1730 (Multi-Epoch processing)

Cross et al. 2012, A&A, 548A, 119 (VSA)

Various other technical documents:http://surveys.roe.ac.uk/vsa/publications

30th March 2015

Future workMatched Aperture Pipeline:

Basic pipeline complete, BUT bugs and many additions necessary

Between surveys, inter survey, user runVariables – see talk by Carlos Ferreira-Lopes:

Add in improved indices for variable selectionAdd in period-finding

Proper Motions.Flagging of bright stars.Interface improvementsCross-programme selection (for public available

data)Better connections to Value-Added products.

30th March 2015 MW Astrophysics from WF Surveys 15

Please speak to me …Current users:

What could be improved?What is missing?

PIs of new VISTA/VST surveys or small programmes: If you want the pipeline outputs / interfaces

Producers of VA products (e.g. 3D extinction maps) We can host them linked to useful data

PIs of new imaging / spectroscopic surveys who need data management.We have the experience

30th March 2015 MW Astrophysics from WF Surveys 16