OmegaCAM: The 16k x 16k Survey Camera for the VST

OmegaCAM: The 16k x 16k Survey Camera for the VST

OmegaCAM: The 16k x 16k Survey Camera for the VST

Calibration, Data Analysis Strategy and Software

Calibration, Data Analysis Strategy and Software

Erik R. DeulKonrad KuijkenEdwin A. Valentijn

Erik R. DeulKonrad KuijkenEdwin A. Valentijn

28/08/2002 SPIE Conference [4836] Survey and Other Telescopes Technologies and Discoveries

2

People involvedPeople involved

• The Netherlands Kapteyn Institute: J-W. Pel, K. Begeman, D.R. Boxhoorn, E. Valentijn, K.

KuijkenSterrewacht Leiden: R. Rengelink, E.R. Deul

• Germany Universitäts-Sternwarte München: R. Bender, L. Greggio, R. Häfner, U. Hopp,

H. Kravkar, W. Mitsch, B. Muschielok, M. Neeser, R. SagliaUniversitäts-Sternwarte Göttingen: R. Harke, H. Nicklas, W. Wellem Sternwarte der Universität Bonn: K. Reif

• ItalyAstronomical Observatory of Capodimonte - Napoli: E. CasconeOsservatorio Astronomico di Padova: A. Baruffolo, E. Cappellaro, E. V. Held,

H. Nazaryan, G. Piotto, H. Navarsadyan, L. Rizzi• ESO

D. Baade, A. Balestra, J-L. Beckers, C. Cavadore, C. Cumani, F. Christen, S. D'Odorico, S. Deiries, N. Devillard, C. Geimer, N. Haddad, G. Hess, J. Hess, O. Iwert, H. Kotzlowski, J-L Lizon, A. Longinotti, W. Nees, A. Renzini, J. Reyes Moreno, G. Sikkema, M. Tarenghi

• The Netherlands Kapteyn Institute: J-W. Pel, K. Begeman, D.R. Boxhoorn, E. Valentijn, K.

KuijkenSterrewacht Leiden: R. Rengelink, E.R. Deul

• Germany Universitäts-Sternwarte München: R. Bender, L. Greggio, R. Häfner, U. Hopp,

H. Kravkar, W. Mitsch, B. Muschielok, M. Neeser, R. SagliaUniversitäts-Sternwarte Göttingen: R. Harke, H. Nicklas, W. Wellem Sternwarte der Universität Bonn: K. Reif

• ItalyAstronomical Observatory of Capodimonte - Napoli: E. CasconeOsservatorio Astronomico di Padova: A. Baruffolo, E. Cappellaro, E. V. Held,

H. Nazaryan, G. Piotto, H. Navarsadyan, L. Rizzi• ESO

D. Baade, A. Balestra, J-L. Beckers, C. Cavadore, C. Cumani, F. Christen, S. D'Odorico, S. Deiries, N. Devillard, C. Geimer, N. Haddad, G. Hess, J. Hess, O. Iwert, H. Kotzlowski, J-L Lizon, A. Longinotti, W. Nees, A. Renzini, J. Reyes Moreno, G. Sikkema, M. Tarenghi


3

DetectorsDetectors

• Science array 1 x 1 degree, 32 CCDs– 15 m pixels – 0.21 arcsec/pixel– Marconi (former EEV) 2k x 4k– 16k x 16k pixels

• Auxiliary CCD’s – 4 CCDs– For guiding– Image analysis

• Science array 1 x 1 degree, 32 CCDs– 15 m pixels – 0.21 arcsec/pixel– Marconi (former EEV) 2k x 4k– 16k x 16k pixels

• Auxiliary CCD’s – 4 CCDs– For guiding– Image analysis


4

FiltersFilters

• Primary set– Sloan u’, g’, r’, i’, z’– Johnson B, V– Narrow-band e.g. H up to 8000 km/s– Composite u’,B,V,i’ in four quadrants

• User filter

• Primary set– Sloan u’, g’, r’, i’, z’– Johnson B, V– Narrow-band e.g. H up to 8000 km/s– Composite u’,B,V,i’ in four quadrants

• User filter

More details see Harald Nicklas [4836-34]

More details see Harald Nicklas [4836-34]

28/08/2002 5

VST constructionsee [4836-09]

Mancini

Details instrument control see [4848-10] Baruffolo

VST constructionsee [4836-09]

Mancini

Details instrument control see [4848-10] Baruffolo


6

Wide Field Imaging ScienceWide Field Imaging Science

• Provide targets for VLT• 2/3 of time through ESO’s OPC• Individual programs

– Supernovae, Lensing, Kuiper belt objects, Gamma ray, bursts, Microlensing, Brown dwarfs, High proper motion objects, Galactic halo objects, Quasars, AGNs

• Sky Surveys• Long term archival research (10 yr mission)

• Science Cases– Finding exceptional single, rare objects– Statistics on large samples of objects

• Provide targets for VLT• 2/3 of time through ESO’s OPC• Individual programs

– Supernovae, Lensing, Kuiper belt objects, Gamma ray, bursts, Microlensing, Brown dwarfs, High proper motion objects, Galactic halo objects, Quasars, AGNs

• Sky Surveys• Long term archival research (10 yr mission)

• Science Cases– Finding exceptional single, rare objects– Statistics on large samples of objects


7

Large Data VolumeLarge Data Volume

• Wide-field imaging instruments, vast amounts of data– E.g.: VST = Southern sky (30 min exp, 300 nights/y) in

3 years. Large amount of data! 100 Tbyte

• Wide-field imaging instruments, vast amounts of data– E.g.: VST = Southern sky (30 min exp, 300 nights/y) in

3 years. Large amount of data! 100 Tbyte

• Science can only be archive-based• Science can only be archive-based

• Handling of the data is non-trivial– Pipeline data reduction– Calibration and re-calibration– Image comparisons and combinations– Working with source lists– Visualization

• Handling of the data is non-trivial– Pipeline data reduction– Calibration and re-calibration– Image comparisons and combinations– Working with source lists– Visualization

ESOcompliantESOcompliant}}


8

Concepts for solutionConcepts for solution

• Environment that provides systematic and controlled– Access to all raw and calibration data– Execution and modification reduction/calibration pipelines– Execution of source extraction algorithms– Archiving reduced data and source lists, or regenerates these

dynamically– Can be federated to link different data centers

• Environment that provides systematic and controlled– Access to all raw and calibration data– Execution and modification reduction/calibration pipelines– Execution of source extraction algorithms– Archiving reduced data and source lists, or regenerates these

dynamically– Can be federated to link different data centers

• Dynamical archive continuously grows, can be used for – small or large science projects– generating and checking calibration data– exchanging methods, scripts and configuration

• Dynamical archive continuously grows, can be used for – small or large science projects– generating and checking calibration data– exchanging methods, scripts and configuration

• Key functionality– Link back from source data to the original raw pixel data and

calibration files

• Key functionality– Link back from source data to the original raw pixel data and

calibration files


9

How to use thisHow to use this

• Deep multi-color fields– No need to take all data in one campaign– Combine data of particular quality, assess results– Select sources, visualize interesting ones, …

• 1-in-1,000,000 events spurious or not?

• Deep multi-color fields– No need to take all data in one campaign– Combine data of particular quality, assess results– Select sources, visualize interesting ones, …

• 1-in-1,000,000 events spurious or not?

• Large homogeneous surveys– E.g. weak lensing maps, cluster searches, star counts

• Large homogeneous surveys– E.g. weak lensing maps, cluster searches, star counts

• Variability (source list - or pixel based) – Proper motions (asteroids, nearby stars)– Flux variations

• Variability (source list - or pixel based) – Proper motions (asteroids, nearby stars)– Flux variations

• Monitor instrument (calibration files)• Monitor instrument (calibration files)

• Planning observations– View quality of existing data– Build on what already exists, add more filters, more

exposure time, better seeing, …

• Planning observations– View quality of existing data– Build on what already exists, add more filters, more

exposure time, better seeing, …


10

SolutionSolution

• Procedurizing– Data taking at telescope for both science and

calibration data– Full integration with data reduction– Design – Data model (classes) defined for data reduction and

calibration– View pipeline as an administrative problem

• Procedurizing– Data taking at telescope for both science and

calibration data– Full integration with data reduction– Design – Data model (classes) defined for data reduction and

calibration– View pipeline as an administrative problem


11

Observing ModesObserving Modes

• Dither matching max. gap between arrays ~400 pixels– N pointings (N=5 is standard) – nearly cover all gaps in focal plane and maximizes sky coverage– the context map will be very complex – couple the photometry among individual CCDs.

• Dither matching max. gap between arrays ~400 pixels– N pointings (N=5 is standard) – nearly cover all gaps in focal plane and maximizes sky coverage– the context map will be very complex – couple the photometry among individual CCDs.

• Jitter matching the smallest gaps in CCDs ~5 pixels– optimizes for maximum homogeneity of the context map – observations for which the wide CCD gaps are not critical– all data from single sky pixel originates from single chip

• Jitter matching the smallest gaps in CCDs ~5 pixels– optimizes for maximum homogeneity of the context map – observations for which the wide CCD gaps are not critical– all data from single sky pixel originates from single chip

• Stare reobserving fixed pointing positions multiple times– main workhorse monitoring instrument and optical

transients.

• Stare reobserving fixed pointing positions multiple times– main workhorse monitoring instrument and optical

transients.

• SSO observing Solar System objects– non-siderial tracking and the auto guiding switched off.

• SSO observing Solar System objects– non-siderial tracking and the auto guiding switched off.


12

Observing StrategiesObserving Strategies

• Standard– Single observations (one observing block)

• Deep– Long, multiple integrations– Selected atmospheric conditions– Several nights

• Frequent– Monitors same field– Timescales from minutes to months (overriding)

• Mosaïc– Maps areas of sky > 1o

• Standard– Single observations (one observing block)

• Deep– Long, multiple integrations– Selected atmospheric conditions– Several nights

• Frequent– Monitors same field– Timescales from minutes to months (overriding)

• Mosaïc– Maps areas of sky > 1o


13

Calibration proceduresCalibration procedures

Sanity checksSanity checks

Quality controlQuality controlCalibration proceduresCalibration procedures

Image pipelineImage pipeline

Source pipelineSource pipeline


14

Science ObservationsScience Observations

Photometric pipelinePhotometric pipeline

Bias pipeline

Flatfield pipeline

Image pipeline

Source pipeline


15

Monitoring Photometric CalibrationMonitoring Photometric Calibration


16

Share the loadShare the load

• Processing– Hardware

• Beowulf processors – 32 (most cases)• Multi Terabyte disks (10 – 100)

– Data reduction• Derive calibration• Run image pipeline (1 Mpx/s)

• Processing– Hardware

• Beowulf processors – 32 (most cases)• Multi Terabyte disks (10 – 100)

– Data reduction• Derive calibration• Run image pipeline (1 Mpx/s)

• Archiving– Storage

• Images (100’s Tbyte), Calibration files (10 Tbyte)• Source parameters (1-10 Tbyte)

– Federate (network speed)• 5 Mb/s (24 hours/day) full replication • 200 Mb/s no replication, on-the-fly retrieval


17

Contents of federationContents of federation

• Raw data– Observed images– Ancillary information

• Calibration results– Calibration files time stamped

• Reduced images– Single observation– Coadded images

• Software– Methods (pipelines) for processing calibration– Configuration files

• Source lists – catalogues– Extracted source information– Associated among different data objects

• Raw data– Observed images– Ancillary information

• Calibration results– Calibration files time stamped

• Reduced images– Single observation– Coadded images

• Software– Methods (pipelines) for processing calibration– Configuration files

• Source lists – catalogues– Extracted source information– Associated among different data objects


18

Concepts of federationConcepts of federation

• Federation maintained by a single database• Full history tracking

– of all input that went into result – providing on-the fly reprocessing

• Dynamical archive - Context as object attributes– Project: Calibration, Science, Survey, Personal– Owner: Pipeline, Developer, User– Strategy: Standard, Deep, Freq (monitoring), Mosaïc– Mode: Stare, Jitter, Dither, SSO– Time: Time stamping

• Software standards– Classes/data model/procedures– 00 – inheritance/ persistency– Python scripts/ c-libraries

• Federation maintained by a single database• Full history tracking

– of all input that went into result – providing on-the fly reprocessing

• Dynamical archive - Context as object attributes– Project: Calibration, Science, Survey, Personal– Owner: Pipeline, Developer, User– Strategy: Standard, Deep, Freq (monitoring), Mosaïc– Mode: Stare, Jitter, Dither, SSO– Time: Time stamping

• Software standards– Classes/data model/procedures– 00 – inheritance/ persistency– Python scripts/ c-libraries


19

ScheduleSchedule

• Hardware– Dome/Telescope erected at location– Camera on telescope Q1 2004– First run: Jan 2004– Second run: Mar 2004

• Software– Design – review Q2 2002- Done– Basic operations – Q4 2003– Evaluate and prepare for mass production 2004– Qualify and populate 2005– Deliver survey system – satellites

• Hardware– Dome/Telescope erected at location– Camera on telescope Q1 2004– First run: Jan 2004– Second run: Mar 2004

• Software– Design – review Q2 2002- Done– Basic operations – Q4 2003– Evaluate and prepare for mass production 2004– Qualify and populate 2005– Deliver survey system – satellites

Documents

OmegaCAM: The 16k x 16k Survey Camera for the VST