Table of Contents€¦ · Web viewDocument # Date Effective Status

Document # Date Effective Status

LSE-61 6/28/2011Version 1Author(s)

Gregory Dubois-Felsmann

Data Management Requirements

Subsystem/Office

Data ManagementDocument Title

Data Management Subsystem Requirements

The Large Synoptic Survey Telescope

(LSST)



Change History Log

Revision Effective Date Description of Changes

2023-05-22 LSE-61 Page 2 of 59


Table of Contents

Data Management Requirements ..................................................................................81 Introduction ...........................................................................................................82 Application Layer ...................................................................................................9

2.1 Data Products Overview ..................................................................................92.1.1 Level 1, 2, and 3 Data Products .................................................................9

2.1.1.1 General Data Products Requirements ...............................................101.1.1.1.1 Public Access to Science Data .....................................................101.1.1.1.2 Timely Access To Transient Data .................................................101.1.1.1.3 Create and Maintain Science Data Archive ..................................11

1.1.1.1.3.1 Nightly Data Accessible Within 24 hrs ....................................111.1.1.1.3.2 Produce Data Releases ..........................................................11

1.1.1.1.3.2.1 Data Releases Produced Yearly .......................................112.1.2 Overview of Pipeline Processing ..............................................................11

2.1.2.1 General Production and Pipeline Requirements ................................111.1.1.1.4 Pipeline Infrastructure ...................................................................12

1.1.1.1.4.1 drProductionInfrastructure ......................................................121.1.1.1.5 Pipeline Availability .......................................................................12

1.1.1.1.5.1 productionAvailability ..............................................................131.1.1.1.6 Simulated, pre-cursor, and real data ............................................13

2.1.2.2 Alert Production ..................................................................................132.1.2.2.1 Alert Production Requirements .....................................................14

2.1.2.2.1.1 Exposure Data Acquisition and Archiving ...............................141.1.1.1.6.0.1 Raw Data Acquisition ........................................................141.1.1.1.6.0.2 Raw Data Archiving ...........................................................151.1.1.1.6.0.3 Wavefront Sensor Data Acquisition ...................................151.1.1.1.6.0.4 Wavefront Sensor Data Archiving .....................................151.1.1.1.6.0.5 Cross-talk Corrected Data Acquisition ..............................15

2.1.2.2.1.2 Instrument Signature Removal Pipeline (ISR) Requirements . 161.1.1.1.6.0.6 Provide ISR Pipeline .........................................................161.1.1.1.6.0.7 Provide Image Assembly Software ...................................161.1.1.1.6.0.8 Provide Linearization Software ..........................................161.1.1.1.6.0.9 Provide Artifact Masking Software ....................................171.1.1.1.6.0.10 Provide PSF Determination Software ..............................171.1.1.1.6.0.11 Provide Image Data Quality Assessment Software .........17

2.1.2.2.1.3 WCS Pipeline Requirements ..................................................171.1.1.1.6.0.12 Generate Photometric Zeropoint for Exposure ................171.1.1.1.6.0.13 Generate WCS ................................................................18

1.1.1.1.6.0.13.1 astrometricQuality ......................................................181.1.1.1.6.0.14 Provide Astrometric Calibration Software ........................18

2.1.2.2.1.4 Image Difference Pipeline Requirements ................................181.1.1.1.6.0.15 Perform Image Differencing ............................................18

2023-05-22 LSE-61 Page 3 of 59


2.1.2.2.1.5 Detection Pipeline (DP) Requirements ...................................191.1.1.1.6.0.16 Provide Detection Pipeline ..............................................19

2.1.2.2.1.6 Association Pipeline (AP) Requirements ................................191.1.1.1.6.0.17 Provide Association Pipeline ...........................................19

2.1.2.2.1.7 Moving Object Pipeline (MOP) Requirements ........................201.1.1.1.6.0.18 Provide Moving Object Pipeline Software .......................201.1.1.1.6.0.19 Moving Object Pipeline Accuracy ....................................20

2.1.2.3 Data Release Production ...................................................................202.1.2.3.1 Data Release Production Requirements ......................................21

2.1.2.3.1.1 General Data Release Requirements .....................................211.1.1.1.6.0.20 Calibrated Photometry of Difference Sources .................211.1.1.1.6.0.21 Detect Faint Objects ........................................................211.1.1.1.6.0.22 Detect Time Varying Point Source Superimposed on Extended Object ........................................................................................221.1.1.1.6.0.23 Enable BAO Analysis ......................................................221.1.1.1.6.0.24 Measure Intrinsic Ellipticities of Small Galaxies ..............221.1.1.1.6.0.25 Provide Astrometric Model ..............................................221.1.1.1.6.0.26 Provide Calibrated Photometry .......................................22

1.1.1.1.6.0.26.1 Provide Calibrated Photometry corrected for Supernova SED ........................................................................................................23

1.1.1.1.6.0.27 Provide PSF For an Exposure .........................................231.1.1.1.6.0.28 Provide Photometric Redshift of Supernovae .................231.1.1.1.6.0.29 Provide Photometric Redshifts of Galaxies .....................23

2.1.2.3.1.2 CoAdd Images Pipeline Requirements ...................................231.1.1.1.6.0.30 General CoAdd Images Pipeline Requirements ..............23

2.1.2.3.1.3 Deep Detection Pipeline (DDP) Requirements .......................241.1.1.1.6.0.31 Provide Image Addition Software ....................................241.1.1.1.6.0.32 Provide Deep Detection Software ...................................241.1.1.1.6.0.33 Provide Shape Measurement Software ...........................24

1.1.1.1.6.0.33.1 Enable a Range of Shape Measurement Approaches 252.1.2.3.1.4 Astrometric Calibration Pipeline ..............................................25

1.1.1.1.6.0.34 General Astrometric Calibration Pipeline Requirements . 252.1.2.3.1.5 Photometric Calibration Pipeline .............................................25

1.1.1.1.6.0.35 Provide Photometric Calibration Software .......................252.1.2.4 Calibration Products Production .........................................................25

2.1.2.4.1 Calibration Products Requirements ..............................................261.1.1.1.6.1 Correct for Camera Bias Structure ..........................................261.1.1.1.6.2 Correct for Camera Crosstalk .................................................261.1.1.1.6.3 Correct for Detector Fringing ..................................................261.1.1.1.6.4 Correct for Instrument Sensitivity Variation .............................261.1.1.1.6.5 Produce Bad Pixel Map ..........................................................261.1.1.1.6.6 Produce Bias Frame ...............................................................261.1.1.1.6.7 Produce Crosstalk Correction Matrix ......................................271.1.1.1.6.8 Produce Illumination Correction Frame ...................................271.1.1.1.6.9 Produce Monochromatic Flatfield Data Cube .........................271.1.1.1.6.10 Astrometric Quality ................................................................27

2023-05-22 LSE-61 Page 4 of 59


1.1.1.1.6.11 Photometric Quality ...............................................................272.1.3 Contents of Principal Database Tables ....................................................27

2.1.3.1 Object Table .......................................................................................282.1.3.2 MovingObject Table ...........................................................................282.1.3.3 Source Table ......................................................................................282.1.3.4 DIASource Table ................................................................................282.1.3.5 FaintSource Table ..............................................................................282.1.3.6 Exposure Table ..................................................................................28

2.1.4 Common Data Objects .............................................................................292.1.4.1 MaskedImages ...................................................................................292.1.4.2 Exposures ..........................................................................................292.1.4.3 Co-Added Exposures .........................................................................292.1.4.4 Alerts ..................................................................................................29

2.2 Level 1 Data Products ...................................................................................302.2.1 Exposures ................................................................................................30

2.2.1.1 Exposures Requirements ...................................................................302.2.1.1.1 General Exposure Requirements .................................................30

1.1.1.1.6.12 Exposure Archive Publicly Accessible ..................................301.1.1.1.6.13 Exposure Formats .................................................................311.1.1.1.6.14 Keep Exposure Archive ........................................................31

2.2.1.1.2 Raw Science Exposure ................................................................311.1.1.1.6.15 Raw Science Images Available within 24 hours ....................311.1.1.1.6.16 Raw Science Image Attributes ..............................................31

2.2.1.1.3 Processed Science Exposure .......................................................321.1.1.1.6.17 Produce Processed Science Exposures ...............................32

1.1.1.1.6.17.1 Determine Accurate PSF for an Exposure ......................321.1.1.1.6.17.2 Calibrated Science Exposures Available witihin 24 hours 321.1.1.1.6.17.3 Processed Science Exposure Attributes .........................32

2.2.1.1.4 Difference Exposure .....................................................................331.1.1.1.6.18 Difference Exposures Available within 24 hours ...................331.1.1.1.6.19 Difference Exposure Attributes .............................................33

2.2.2 Catalogs ...................................................................................................332.2.2.1 Catalogs Requirements ......................................................................33

2.2.2.1.1 General Catalog Requirements ....................................................331.1.1.1.6.20 Queryable .............................................................................341.1.1.1.6.21 Keep Science Data Archive ..................................................341.1.1.1.6.22 Maintain Archive Publicly Accessible ....................................341.1.1.1.6.23 External Formats ...................................................................34

2.2.2.1.2 Difference Source Catalog ............................................................341.1.1.1.6.24 Difference Source Attributes .................................................341.1.1.1.6.25 Difference Sources Available within 24 hours .......................35

2.2.2.1.3 Object Catalog ..............................................................................351.1.1.1.6.26 Produce Object Catalog ........................................................351.1.1.1.6.27 Source List Updates ..............................................................351.1.1.1.6.28 Level 1 Object Attributes .......................................................351.1.1.1.6.29 Summary Metadata Attributes ..............................................36

2023-05-22 LSE-61 Page 5 of 59


1.1.1.1.6.30 Summary Metadata Updates ................................................362.2.2.1.4 Orbit Catalog ................................................................................36

1.1.1.1.6.31 Produce Orbit Catalog ..........................................................361.1.1.1.6.32 Source Lists Update Frequency ............................................361.1.1.1.6.33 Moving Object Attributes .......................................................361.1.1.1.6.34 Moving Objects Available within 24 hours .............................37

2.2.3 Alerts ........................................................................................................372.2.3.1 Alerts Requirements ...........................................................................37

1.1.1.1.7 Generate Alerts ............................................................................371.1.1.1.7.1 Detect in Two Exposures in Visit ............................................371.1.1.1.7.2 Alert Attributes ........................................................................37

1.1.1.1.8 Public Distribution of Alerts ...........................................................381.1.1.1.9 Keep Historical Alert Archive ........................................................381.1.1.1.10 Archive Alerts within 24 hours ....................................................38

2.2.4 Nightly Summary Products ......................................................................382.2.4.1 Nightly Summary Products Requirements .........................................38

1.1.1.1.11 Generate Data Quality Report within 4 hours .............................381.1.1.1.11.1 Data Quality Report Contents ...............................................39

1.1.1.1.12 Generate DMS Performance Report within 4 hours ...................391.1.1.1.12.1 Performance Report Contents ..............................................39

1.1.1.1.13 Generate Calibration Report within 4 hours ................................391.1.1.1.13.1 Calibration Report Contents ..................................................40

2.2.5 Engineering and Facility Database Archive .............................................402.2.5.1 E&F Database Requirements .............................................................40

1.1.1.1.14 Provide E&F Database Archive ..................................................402.3 Level 2 Data Products ...................................................................................40

2.3.1 Co-Added Exposures ...............................................................................402.3.1.1 Co-Added Exposures Requirements ..................................................41

1.1.1.1.15 Produce Images for EPO ............................................................411.1.1.1.16 Produce Co-Added Exposures ...................................................411.1.1.1.17 Co-Added Images Updated at Least Every 6 months ................411.1.1.1.18 Co-Added Image Atrributes ........................................................42

2.3.2 Catalogs ...................................................................................................422.3.2.1 Object Catalog – Level 2 Data ...........................................................42

1.1.1.1.19 Astrometric Model .......................................................................421.1.1.1.20 Calibrated Light Curve ................................................................421.1.1.1.21 Calibrated Point Source Photometry ..........................................421.1.1.1.22 Calibrated Position Curve ...........................................................431.1.1.1.23 Extended Object Photometry ......................................................431.1.1.1.24 Level 2 Object Attributes .............................................................431.1.1.1.25 Object Classification ...................................................................431.1.1.1.26 Photomeric Redshift PDF ...........................................................431.1.1.1.27 Photometric Redshifts ................................................................441.1.1.1.28 Extended Object Shape Parameters ..........................................44

2.3.2.2 Source Catalogs - Level 2 Data .........................................................441.1.1.1.29 Low SNR Source Attributes ........................................................44

2023-05-22 LSE-61 Page 6 of 59


1.1.1.1.30 Source Attributes ........................................................................442.4 Level 3 Data Products ...................................................................................45

2.4.1 General Level 3 Data Product Requirements ..........................................461.1.1.2 DAC resource allocation for Level 3 processing .................................461.1.1.3 Level 3 Data Product Self Consistency ..............................................461.1.1.4 Provenance for Level 3 processing at DACs ......................................46

2.4.2 Level 3 Catalog Processing Requirements ..............................................471.1.1.5 Access to catalogs for external Level 3 processing ............................471.1.1.6 Access to input catalogs for DAC-based Level 3 processing .............471.1.1.7 Federation with external catalogs .......................................................471.1.1.8 Software framework for Level 3 catalog processing ...........................47

2.4.3 Level 3 Image Processing Requirements ................................................481.1.1.9 Access to images for external Level 3 processing .............................481.1.1.10 Access to input images for DAC-based Level 3 processing .............481.1.1.11 Software framework for Level 3 image processing ...........................48

2.5 Calibration Data Products ..............................................................................482.5.1 Calibration Products Requirements .........................................................49

1.1.1.12 Crosstalk Correction Matrix Attributes...............................................491.1.1.13 Calibration Data Products ................................................................491.1.1.14 Calibration Images Available 1 Hour Before Observing ...................491.1.1.15 Calibration Image Attributes .............................................................49

2.6 Detection and Measurement of Objects ........................................................502.6.1 Deep Detection Processing .....................................................................502.6.2 Difference Exposure Processing ..............................................................502.6.3 Measuring the properties of Objects ........................................................51

2.6.3.1 Slowly Moving Point Source Model ....................................................522.6.3.2 Small Object Model ............................................................................522.6.3.3 Large Object Model ............................................................................522.6.3.4 Solar System Model ...........................................................................53

2.6.4 The Multifit Algorithm ...............................................................................532.6.5 Model Residuals ......................................................................................542.6.6 Limitations and Possible Extensions ........................................................56

2.6.6.1 Faint Transients ..................................................................................562.6.6.2 Faint Solar System Objects ................................................................572.6.6.3 Possible Multifit Improvements ...........................................................57

2.7 Inserting Synthetic Objects ............................................................................572.8 Data Access ..................................................................................................58

2.8.1 Computing Resources .............................................................................582.8.2 Data Access Tools ...................................................................................58

2.9 Incorporation of User Analysis Modules ........................................................593 Middleware Layer ................................................................................................60

3.1 Middleware Layer Requirements ...................................................................611.1.2 Provide Data Access Services .................................................................611.1.3 Provide Pipeline Execution Services ........................................................611.1.4 Provide Management and Control Services .............................................611.1.5 Provide Pipeline Construction Services ...................................................62

2023-05-22 LSE-61 Page 7 of 59


1.1.6 Provide Data/Catalog Construction Services ...........................................621.1.7 Provide User Interface Services ...............................................................62

4 Infrastructure Layer .............................................................................................634.1 Infrastructure Layer Requirements ................................................................63

4.1.1 General Infrastructure Requirements .......................................................631.1.7.1 Optimization of Cost, Reliability and Availability in Order ...................631.1.7.2 Pipeline throughput ............................................................................631.1.7.3 Re-processing Capacity .....................................................................631.1.7.4 Temporary Storage for Communications Links ..................................631.1.7.5 Infrastructure Sizing for "catching up" ................................................641.1.7.6 Incorporate Fault-Tolerance ...............................................................641.1.7.7 Incorporate Autonomics .....................................................................64

4.1.2 Mountaintop Site Requirements ...............................................................641.1.7.8 Mountaintop Site Data Communications ............................................641.1.7.9 Mountaintop Site Temporary Storage ................................................641.1.7.10 Prefer Computing and Storage Down ..............................................65

4.1.3 Mountain to Base Network Requirements ................................................651.1.7.11 Mountain to Base Network ...............................................................651.1.7.12 Mountain to Base Network Availability .............................................651.1.7.13 Mountain to Base Network Reliability ...............................................651.1.7.14 Mountain to Base Network Secondary Link ......................................651.1.7.15 Mountain to Base Network Ownership and Operation .....................66

4.1.4 Base Facility Requirements .....................................................................661.1.7.16 Base Facility Infrastructure ...............................................................661.1.7.17 Base Facility Temporary Storage .....................................................661.1.7.18 Base Facility Co-Location with Existing Facility ................................661.1.7.19 Base Facility Data Quality Assessment ............................................66

4.1.5 Base to Archive Network Requirements ..................................................671.1.7.20 Base to Archive Network ..................................................................671.1.7.21 Base to Archive Network Availability ................................................671.1.7.22 Base to Archive Network Reliability ..................................................671.1.7.23 Base to Archive Network Secondary Link ........................................671.1.7.24 Base to Archive Network Lease .......................................................67

4.1.6 Archive Center Requirements ..................................................................681.1.7.25 Archive Center ..................................................................................681.1.7.26 Archive Center Disaster Recovery ...................................................681.1.7.27 Archive Center Co-Location with Existing Facility ............................68

4.1.7 Archive to Data Access Center Network Requirements ...........................681.1.7.28 Archive to Data Access Center Network ..........................................681.1.7.29 Archive to Data Access Center Network Availability ........................681.1.7.30 Archive to Data Access Center Network Reliability ..........................691.1.7.31 Archive to Data Access Center Network Secondary Link .................691.1.7.32 Archive to Data Access Center Network Lease ................................69

4.1.8 Data Access Center Requirements ..........................................................691.1.7.33 Data Access Centers ........................................................................691.1.7.34 Data Access Center Simultaneous Connections ..............................69

2023-05-22 LSE-61 Page 8 of 59


1.1.7.35 Data Access Center VO Standards ..................................................701.1.7.36 Data Access Center Geographical Distribution ................................701.1.7.37 No Limit on Data Access Centers ....................................................70

2023-05-22 LSE-61 Page 9 of 59


Data Management Requirements This package contains the subsystem level requirements for the LSST Data Management System (DMS).

1 Introduction The LSST Science Requirements Document specifies a set of science goals to be achieved from the LSST observing program. To enable the achievement of these goals, the LSST Data Management System (DMS) is required to generate, or enable the generation of, a set of data products, and to make them available to scientists and the public. To carry out this mission the DMS performs the following major functions:· Processes the incoming stream of images generated by the camera system during

observing to produce transient alerts and to archive the raw images.· Roughly once per year, creates and archives a Data Release (DR), which is a static

self-consistent collection of data products generated from all survey data taken from the date of survey initiation to the cutoff date for the Data Release. The data products include optimal measurements of the properties (shapes, positions, fluxes, motions) of all objects, including those below the single visit sensitivity limit, astrometric and photometric calibration of the full survey object catalog, and limited classification of objects based on both their static properties and time-dependent behavior. Deep coadded images of the full survey area are produced as well.

· Periodically creates new calibration data products, such as bias frames and flat fields, that will be used by the other processing functions.

· Makes all LSST data available through an interface that utilizes, to the maximum possible extent, community-based standards such as those being developed by the Virtual Observatory (VO), and facilitates user data analysis and the production of user-defined data products at Data Access Centers (DAC) and at external sites.

The data management system begins at the data acquisition interface between the camera and telescope subsystems and flows through to the data products accessed by end users. On the way, it moves through three types of managed facilities supporting data management, as well as end user sites that may conduct science using LSST data or pipeline resources on their own computing infrastructure.The Data Management facilities are a Mountain Summit/Base Facility, a central Archive Center, multiple Data Access Centers, and a System Operations Center. The data will be transported over existing high-speed optical fiber links from the mountain summit/base facility in South America to the archive center in the U.S. Data will also flow from the mountain summit/base facility and the archive center to the data access centers over existing fiber optic links.· The Mountain Summit/Base Facility is composed of the mountaintop telescope site,

where data acquisition must interface to the other LSST subsystems, and the Base Facility, where rapid-turnaround processing will occur for data quality assessment and near real-time alerts.

· The Archive Center is a super-computing class data center with high reliability and availability. This is where the data will undergo complete processing and re-processing and permanent storage. It is also the main repository feeding the distribution of LSST data to the community.

2023-05-22 LSE-61 Page 10 of 59


· Data Access Centers for broad user access are envisioned, according to a tiered access model where the tiers define the capacity and response available. There are two project funded Data Access Centers co-located with the Base Facility and the Archive Center. These centers provide replication of all of the LSST data to ensure that disaster recovery is possible. They provide Virtual Observatory interfaces to the LSST data products. LSST is encouraging non-US/non-Chilean funding and partners to host additional Data Access Centers to increase end user access bandwidth and help amortize observatory operations costs.

· The System Operations Center (SOC) provides a control room and large-screen display for supervisory monitoring and control of the DM System. Network and facility status are available as well as the capability to "drill down" to individual facilities. DM support to observatory science operations and an end user help desk are also at the SOC.

2 Application Layer The Data Products and the Productions and Pipelines that produce those products comprise the DMS Application Layer.

2.1 Data Products Overview

2.1.1 Level 1, 2, and 3 Data Products · Level 1 products are generated by pipeline processing the stream of data from the

camera system during normal observing. Level 1 data products are therefore continuously generated and / or updated every observing night. This process is of necessity highly automated, and must proceed with absolutely minimal human interaction. In addition to science data products, a number of Level 1 SDQA data products are generated to assess quality and to provide feedback to the Observatory Control System.

· Level 2 products are generated as part of a Data Release, which is required to be performed at least yearly, and will be performed more frequently during the first year of the survey. Level 2 products use Level 1 products as input, and include data products for which extensive computation is required, often because they combine information from many exposures. Although the steps that generate Level 2 products may be automated, significant human interaction will be required at key points to ensure the quality of the data.

· Level 3 data products are derived from Level 1 and / or Level 2 data products to support particular science goals, often requiring the combination of LSST data across significant areas on the sky. The DMS is required to facilitate the creation of Level 3 data products, for example by providing suitable APIs and computing infrastructure, but is not itself required to create any Level 3 data product. Instead these data products are created externally to the DMS, using software written by, e.g., science collaborations. Once created, Level 3 data products may be associated

2023-05-22 LSE-61 Page 11 of 59


with Level 1 and Level 2 data products through database federation (Wikipedia 09). In rare cases, the LSST Project, with the agreement of the Level 3 creators, may decide to incorporate Level 3 data products into the DMS production flow, thereby promoting them to Level 2 data products.

Level 1 and Level 2 data products that have passed quality control tests are required to be accessible to the public without restriction. Additionally, the source code used to generate them will be made available, and LSST will provide support for builds on selected platforms. The access policies for Level 3 data products will be product- and source-specific, and in some cases will be proprietary.

2.1.1.1 General Data Products Requirements These requirements apply to all Data Products.

1.1.1.1.1 Public Access to Science Data Id: DM-APP-DP-GEN-5Version: 1.0 Modified: 6/16/2011 Status: Proposed

1.1.1.1.2 Timely Access To Transient Data Id: DM-APP-DP-GEN-6Version: 1.0 Modified: 6/16/2011 Status: Proposed

Transient alerts shall be published to community alert distribution networks using community-standard protocols, to be determined during the LSST construction phase as community standards evolve.

1.1.1.1.3 Create and Maintain Science Data Archive Id: DM-APP-DP-GEN-1Version: 1.0 Modified: 6/16/2011 Status: Proposed

1.1.1.1.3.1 Nightly Data Accessible Within 24 hrs Id: DM-APP-DP-GEN-3Version: 1.0 Modified: 6/16/2011 Status: Proposed

1.1.1.1.3.2 Produce Data Releases Id: DM-APP-DP-GEN-4Version: 1.0 Modified: 6/16/2011 Status: Proposed

2023-05-22 LSE-61 Page 12 of 59


1.1.1.1.3.2.1 Data Releases Produced Yearly Id: DM-APP-DP-GEN-2Version: 1.0 Modified: 6/16/2011 Status: Proposed

2.1.2 Overview of Pipeline Processing

2.1.2.1 General Production and Pipeline Requirements · The Alert Production runs at the Base Facility, and processes the images streaming

from the LSST Camera to produce real time transient Alerts· The Data Release Production runs at the Archive Facility, and is responsible for

creating a new Data Release, starting from Raw Science Exposures.· The Calibration Products Production runs at the Archive Facility, and services both

the Alert and Data Release Productions. It is responsible for creating the calibration data products, such as flats and biases, that are required to remove the instrumental signature from the Raw Science Exposures.

1.1.1.1.4 Pipeline Infrastructure Id: DM-APP-PL-1Version: 1.0 Modified: 6/16/2011 Status: Proposed

Discussion: For now, for compatibility with earlier releases of the FRS, this time remains set at one year, but this can only be correct if it includes time for development and release validation between production cycles. Most likely the actual production period will need to be significantly shorter than one year in order to meet the higher-level requirement of one year for the interval between releases.

1.1.1.1.4.1 drProductionInfrastructure isEncapsulated:Version: 1.0 Modified: 9/19/2010 Status: Proposed

Description Value Unit Attribute1.0 Years drProcessi

ngPeriod

1.1.1.1.5 Pipeline Availability Id: DM-APP-PL-2

2023-05-22 LSE-61 Page 13 of 59


Version: 1.0 Modified: 6/16/2011 Status: Proposed

Discussion: This applies to active pipelines only. It is allowed for pipelines to be down for longer periods during observatory scheduled maintenance, for Alert Production, and during development/validation periods between productions, for Data Release Production.

1.1.1.1.5.1 productionAvailability isEncapsulated:Version: 1.0 Modified: 9/19/2010 Status: Proposed

Description Value Unit AttributeMaximum allowable outage of active DM production. 24 Hour production

MaxDowntime

1.1.1.1.6 Simulated, pre-cursor, and real data Id: DM-APP-PL-3Version: 1.0 Modified: 6/16/2011 Status: Proposed

2.1.2.2 Alert Production · Acquire the raw science images from the Camera, and move them to the Archive

Center for permanent storage.· Acquire the raw wavefront sensor images from the Camera, and move them to the

Archive Center for permanent storage and access by the Telescope and Site group.· Process the crosstalk-corrected images from the Camera to detect transient events

within 60 seconds of shutter closure for the second exposure in a Visit.· Package information about detected transients as Alerts, and distribute them to the

community as VOEvents.· Continuously assess the data quality of the data stream.The major steps in the processing flow are:· Image processing of the Raw Exposures to remove the instrumental signature.· Determination of the WCS, PSF, and rough photometric zeropoint. This produces

Processed Exposures. WCS solutions are also reported to the Observatory control system for use by the TCS.

· Subtraction of a registered Template Exposure from the Processed Exposure, producing a Difference Exposure. The Template Exposure is a coadd created by the Data Release Production.

· Detection of sources of either polarity in the Difference Exposure, producing DIASources.

· Visit processing logic, which compares the DIASources from the two Exposures in the Visit to discriminate against cosmic rays, and to flag very rapidly moving Solar System objects.

· FaintSources, abbreviated measurements of low SNR detections, are produced for

2023-05-22 LSE-61 Page 14 of 59


Objects of particular interest, e.g. predicted positions for Solar System objects, or objects that have previously produced Alerts.

· Comparison of positive flux DIASources with predictions from the Moving Object System for already known Solar System objects, as contained in the MovingObject table.

· The Association Pipeline is run to match DIASources to already known astronomical objects, as contained in the Object table.

· Generation of Alerts. DIASources that are detected in both Exposures of a Visit, and are not matched to a known Solar System object, will produce an Alert.

· SDQA is performed at every pipeline stage, stored in database tables, and fed to the OCS as required.

· The Moving Object Pipeline (MOPS) is run during the day to interpret each new detection of a moving object as a new measurement of a Solar System object already in the MovingObject table, or as a previously unknown object, which will be added to the MovingObject table. All orbits are refined based on the new measurements from the night.

As the raw images arrive at the Archive Center, the same processing flow is performed there, with the consistency of the databases at the Base and Archive Centers being periodically checked. The duplication of processing is carried out to reduce the data bandwidth required between the Base and Archive Centers.

2.1.2.2.1 Alert Production Requirements

2.1.2.2.1.1 Exposure Data Acquisition and Archiving

1.1.1.1.6.1.1 Raw Data Acquisition id:text:Version: 1.0

Modified: 6/16/2011 Status: Proposed

Discussion: The manner of data acquisition is a matter for the DM-Camera ICD in this area.

1.1.1.1.6.1.2 Raw Data Archiving id:text:Version: 1.0


Discussion: This will involve assembly of the acquired data at the Base Center and its transfer to the Archive Center.

2023-05-22 LSE-61 Page 15 of 59


1.1.1.1.6.1.3 Wavefront Sensor Data Acquisition id:text:Version: 1.0


Discussion: The manner of data acquisition is a matter for the DM-Camera ICD in this area. It should be identical, except for the selection of the specific data sources, to that for the raw science sensor data.

There is no currently established requirement for the acquisition or archiving of any raw guider sensor data.

1.1.1.1.6.1.4 Wavefront Sensor Data Archiving id:text:Version: 1.0


Discussion: This will involve assembly of the acquired data at the Base Center and its transfer to the Archive Center.

1.1.1.1.6.1.5 Cross-talk Corrected Data Acquisition id:text:Version: 1.0


Discussion: The manner of data acquisition is a matter for the DM-Camera ICD in this area. It may be different for that for the raw science sensor data, as it is subject to quite different latency and reliability requirements.

This data is the input to the main Alert Production pipelines. It is not planned to be archived.

2.1.2.2.1.2 Instrument Signature Removal Pipeline (ISR) Requirements · Level 1 Science Reduction Mode· Level 2 Science Reduction Mode· Calibration Mode.These modes produce the associated Image Data Products. The modes are implemented by sequencing the execution of a mode-specific subset of a common set of image processing tasks.

1.1.1.1.6.1.6 Provide ISR Pipeline Id:

2023-05-22 LSE-61 Page 16 of 59



1.1.1.1.6.1.7 Provide Image Assembly Software Id: DM-APP-PL-IP-1Version: 1.0 Modified: 6/16/2011 Status: Proposed

1.1.1.1.6.1.8 Provide Linearization Software Id: DM-APP-PL-IP-2Version: 1.0 Modified: 6/16/2011 Status: Proposed

[ Want to be limited by systematics, rather than random noise. Req will be different for each product

]

[Is "linearization" really the right community-standard name for this?]

1.1.1.1.6.1.9 Provide Artifact Masking Software Id: DM-APP-PL-IP-3Version: 1.0 Modified: 6/16/2011 Status: Proposed

1.1.1.1.6.1.10 Provide PSF Determination Software Id: DM-APP-PL-IP-5Version: 1.0 Modified: 6/16/2011 Status: Proposed

TBD metric/accuracy requirement?

[ The PSF determined here is mostly used for image DQA, possibly for increasing the astrometric and/or photometric precision. It will not be good enough to use for WL analysis. It's very unclear how to set a requirement on the PSF accuracy for the latter purposes]

1.1.1.1.6.1.11 Provide Image Data Quality Assessment Software Id: DM-APP-PL-IP-7Version: 1.0 Modified: 6/16/2011 Status: Proposed

[ What remains to be provided here is a list of metrics.]

2023-05-22 LSE-61 Page 17 of 59


2.1.2.2.1.3 WCS Pipeline Requirements

1.1.1.1.6.1.12 Generate Photometric Zeropoint for Exposure Id:Version: 1.0 Modified: 6/16/2011 Status: Proposed

1.1.1.1.6.1.13 Generate WCS Id:Version: 1.0 Modified: 6/16/2011 Status: Proposed

1.1.1.1.6.1.13.1 astrometricQuality isEncapsulated:Version: 1.0 Modified: 6/16/2011 Status: Proposed

Description Value Unit AttributeThis value corresponds to about one-quarter of a pixel. 50 mili-Arcsec astrometric

Accuracy5 int astrometric

MinStandards

1.1.1.1.6.1.14 Provide Astrometric Calibration Software Id: DM-APP-PL-IP-4Version: 1.0 Modified: 6/16/2011 Status: Proposed

2.1.2.2.1.4 Image Difference Pipeline Requirements

1.1.1.1.6.1.15 Perform Image Differencing id:text:Version: 1.0


2.1.2.2.1.5 Detection Pipeline (DP) Requirements · Calibrated Science Image· Subtracted Science Image

2023-05-22 LSE-61 Page 18 of 59


· Co-added Science ImageThese modes produce the associated Source Catalog Data Product.

[ Not quite right - needs work ]

1.1.1.1.6.1.16 Provide Detection Pipeline Id: DM-APP-PL-DP-1Version: 1.0 Modified: 6/16/2011 Status: Proposed

2.1.2.2.1.6 Association Pipeline (AP) Requirements · Proper motions and parallaxes of stars· Classification of variable stars· Phased light curves for periodic variable stars· Orbital elements for solar system objects· Recognition that an alertable transient event has occurred.

1.1.1.1.6.1.17 Provide Association Pipeline Id: DM-APP-PL-AP-1Version: 1.0 Modified: 6/16/2011 Status: Proposed

TBD metric/accuracy requirement?

[ possibly require, for each source, the probability that the source is correctly associated with the object]

2.1.2.2.1.7 Moving Object Pipeline (MOP) Requirements

1.1.1.1.6.1.18 Provide Moving Object Pipeline Software Id: DM-APP-PL-MO-1Version: 1.0 Modified: 6/16/2011 Status: Proposed

Discussion: The magnitude and angular velocity limits for this identification are TBD. These limits may be driven more by computational resource constraints than by the raw reach of the collected data. The software may well be capable of exceeding the required limits, but at an unacceptable cost.

2023-05-22 LSE-61 Page 19 of 59


1.1.1.1.6.1.19 Moving Object Pipeline Accuracy Id: DM-APP-PL-MO-2Version: 1.0 Modified: 6/16/2011 Status: Proposed

2.1.2.3 Data Release Production · As in the Alert Production, all Raw Exposures from the camera are processed to

remove the instrumental signature, and to determine the WCS and PSF, producing Processed Exposures. This is done with the best available calibration products, which in general will not be those available when the processing was initially done.

· The survey region is tessellated into a set of sky patches, and several Coadded Exposures are produced for each patch from the Processed Exposures. These are a per-band Template Coadd used for image subtraction; a Detection Coadd used in the Deep Detection Pipeline, possibly per-band; and a RGB Coadd used for visualization.

· The Deep Detection Pipeline is run, populating the Object, Source, and FaintSource tables. Deep Detection uses the Multifit algorithm to fit object models simultaneously to the entire stack of Exposures which contain the Object. This results in a set of optimal measurements of the Object attributes over the full time span of the survey, including astrometric parameters such as proper motion and parallax.

· The Image Subtraction Pipeline is run, as in the Alert Production, yielding DIASources and FaintSources, for transient objects

· The Moving Object Pipeline is run on DIASources, to yield a complete set of orbits for Solar System Objects in the MovingObject table.

· The Photometric Calibration Pipeline is run on the full set of measurements in the Source, DIASource, and FaintSource catalogs, incorporating measurements from the Auxiliary Telescope and other sources of data about the atmosphere to perform a global photometric calibration of the survey. In addition to accurate photometry for every measurement, this yields an atmosphere model for every Exposure.

2.1.2.3.1 Data Release Production Requirements

2.1.2.3.1.1 General Data Release Requirements

1.1.1.1.6.1.20 Calibrated Photometry of Difference Sources Id: DM-APP-PL-DRP-2Version: 1.0 Modified: 6/16/2011 Status: Proposed

Discussion: SRD: 1 - 2% accuracy required for Solar System Object taxonomy based on

2023-05-22 LSE-61 Page 20 of 59


photometry

1.1.1.1.6.1.21 Detect Faint Objects Id: DM-APP-PL-DRP-3Version: 1.0 Modified: 6/16/2011 Status: Proposed

1.1.1.1.6.1.22 Detect Time Varying Point Source Superimposed on Extended Object Id: DM-APP-PL-DRP-4Version: 1.0 Modified: 6/16/2011 Status: Proposed

Discussion: Supernovae detected by LSST will typically appear superimposed on a resolved galaxy.

1.1.1.1.6.1.23 Enable BAO Analysis Id: DM-APP-PL-DRP-5Version: 1.0 Modified: 6/16/2011 Status: Proposed

1.1.1.1.6.1.24 Measure Intrinsic Ellipticities of Small Galaxies Id: DM-APP-PL-DRP-1Id: DM-APP-

PL-DRP-1Version: 1.0


The measurement error goal for this is TBD.

1.1.1.1.6.1.25 Provide Astrometric Model Id: DM-APP-PL-DRP-6Version: 1.0 Modified: 6/16/2011 Status: Proposed

1.1.1.1.6.1.26 Provide Calibrated Photometry Id: DM-APP-PL-DRP-7Version: 1.0 Modified: 6/16/2011 Status: Proposed

1.1.1.1.6.1.26.1 Provide Calibrated Photometry corrected for Supernova SED Id: DM-APP-PL-DRP-8

2023-05-22 LSE-61 Page 21 of 59



Discussion: The Spectral Energy Distribution (SED) of supernovae is very different from that of stars.

1.1.1.1.6.1.27 Provide PSF For an Exposure Id: DM-APP-PL-DRP-9Version: 1.0 Modified: 6/16/2011 Status: Proposed

1.1.1.1.6.1.28 Provide Photometric Redshift of Supernovae Id: DM-APP-PL-DRP-10Version: 1.0 Modified: 6/16/2011 Status: Proposed

1.1.1.1.6.1.29 Provide Photometric Redshifts of Galaxies Id: DM-APP-PL-DRP-11Version: 1.0 Modified: 6/16/2011 Status: Proposed

2.1.2.3.1.2 CoAdd Images Pipeline Requirements

1.1.1.1.6.1.30 General CoAdd Images Pipeline Requirements Id:Version: 1.0 Modified: 6/16/2011 Status: Proposed

The software shall support the generation of monochromatic, panchromatic, and polychromatic coadds.

Each Exposure contributing to a Co-Added Exposure shall be projected onto a common reference frame using a projection selected from a set of options by a Policy.

2.1.2.3.1.3 Deep Detection Pipeline (DDP) Requirements

1.1.1.1.6.1.31 Provide Image Addition Software Id: DM-APP-PL-DD-1Version: 1.0 Modified: 6/16/2011 Status: Proposed

2023-05-22 LSE-61 Page 22 of 59


1.1.1.1.6.1.32 Provide Deep Detection Software Id: DM-APP-PL-DD-2Version: 1.0 Modified: 6/16/2011 Status: Proposed

1.1.1.1.6.1.33 Provide Shape Measurement Software Id: DM-APP-PL-DD-3Version: 1.0 Modified: 6/16/2011 Status: Proposed

Discussion: This does not specify the algorithms or architecture to be used, and in particular does not specify which of the approaches described in the narrative section "Detection and Measurement of Objects" below may be used.

1.1.1.1.6.1.33.1 Enable a Range of Shape Measurement Approaches id:text:Version: 1.0


2.1.2.3.1.4 Astrometric Calibration Pipeline

1.1.1.1.6.1.34 General Astrometric Calibration Pipeline Requirements Id:Version: 1.0 Modified: 6/16/2011 Status: Proposed

2.1.2.3.1.5 Photometric Calibration Pipeline

1.1.1.1.6.1.35 Provide Photometric Calibration Software Id: DM-APP-PL-IP-6Version: 1.0 Modified: 6/16/2011 Status: Proposed

Discussion: Need to resolve why the word "Preliminary" appeared in the original text here, and what the relationship with the global photometric calibration is. gpdf: I think the latter may not be adequately framed in the existing requirements.

2.1.2.4 Calibration Products Production

2023-05-22 LSE-61 Page 23 of 59


2.1.2.4.1 Calibration Products Requirements (Note: the requirements below remain to be fleshed out pending the current review of the Calibration Planby the new Calibration Scientist.)

1.1.1.1.6.2 Correct for Camera Bias Structure Id: DM-APP-PL-CAP-1Version: 1.0 Modified: 6/22/2009 Status: Proposed

TBD

1.1.1.1.6.3 Correct for Camera Crosstalk Id: DM-APP-PL-CAP-2Version: 1.0 Modified: 6/22/2009 Status: Proposed

TBD

1.1.1.1.6.4 Correct for Detector Fringing Id: DM-APP-PL-CAP-4Version: 1.0 Modified: 6/22/2009 Status: Proposed

TBD

1.1.1.1.6.5 Correct for Instrument Sensitivity Variation Id: DM-APP-PL-CAP-3Version: 1.0 Modified: 6/22/2009 Status: Proposed

TBD

1.1.1.1.6.6 Produce Bad Pixel Map Id: DM-APP-PL-CAP-7Version: 1.0 Modified: 6/22/2009 Status: Proposed

TBD

1.1.1.1.6.7 Produce Bias Frame Id: DM-APP-PL-CAP-5Version: 1.0 Modified: 6/22/2009 Status: Proposed

TBD

2023-05-22 LSE-61 Page 24 of 59


1.1.1.1.6.8 Produce Crosstalk Correction Matrix Id: DM-APP-PL-CAP-6Version: 1.0 Modified: 6/22/2009 Status: Proposed

TBD

1.1.1.1.6.9 Produce Illumination Correction Frame Id: DM-APP-PL-CAP-8Version: 1.0 Modified: 6/22/2009 Status: Proposed

TBD

1.1.1.1.6.10 Produce Monochromatic Flatfield Data Cube Id: DM-APP-PL-CAP-9Version: 1.0 Modified: 6/22/2009 Status: Proposed

TBD

1.1.1.1.6.11 Astrometric Quality Version: 1.0 Modified: 6/22/2009 Status: Proposed

1.1.1.1.6.12 Photometric Quality Version: 1.0 Modified: 6/22/2009 Status: Proposed

2.1.3 Contents of Principal Database Tables Two important definitions that underly the design of the database are those for "Object" and "Source". An Object (or, when confusion is possible, an "AstroObject") is a representation of an astrophysical object: a star; a galaxy; a Solar System object. A "Source" is the representation of a measurement of an Object's properties from a single image that contains its footprint on the sky. As we discuss in the remainder of this section, both Objects and Sources come in a few different flavors that are specialized for particular situations that frequently arise.The highest level tables are Object and MovingObject, which together organize on a per-astrophysical-object basis all information from the survey. An entry in either of these tables links to all the individual measurements of the Object, contained in the DIASource, FaintSource, and Source tables. Each of these measurements, in turn, links to the metadata for each telescope exposure in the Exposure table.

2.1.3.1 Object Table

2023-05-22 LSE-61 Page 25 of 59


The Object Table has a row for every non-Solar-System astrophysical object found in the LSST images. Each Object Table row has a set of columns which together summarize what has been measured for the object over the history of the survey.

2.1.3.2 MovingObject Table Note that there will be occasions during nightly alert processing in which an Object with only a single associated measurement is subsequently found to be a measurement of a MovingObject. In these relatively rare cases, the original Object will be deleted.

2.1.3.3 Source Table An entry in the Source Table is made in conjunction with Multifit measurement of an Object, and represents a measurement at high SNR from a single Exposure.

2.1.3.4 DIASource Table An entry in the DIASource Table is made as a result of a high SNR measurement of an Object in a difference Exposure.

2.1.3.5 FaintSource Table An entry in the FaintSource Table is made in conjunction with a low SNR measurement of an Object either in Deep Detection and Measurement (e.g., with Multifit) or in a difference Exposure.

2.1.3.6 Exposure Table

2.1.4 Common Data Objects

2.1.4.1 MaskedImages · The Science image, with the pixel data type being either 16-bit integers or 32-bit

floats.· The Mask image, typically 16 bits deep, with each plane representing a particular

logical statement about the corresponding science image pixel, e.g. "this pixel is saturated".

· The Variance image, which represents the expected variance in the corresponding science pixel values.

2.1.4.2 Exposures

2023-05-22 LSE-61 Page 26 of 59


2.1.4.3 Co-Added Exposures

2.1.4.4 Alerts The data bundled with the Alert includes:· The rows from the Exposure table for the two Exposures in the Visit.· The rows from the DIASource table for the two relevant DIASources.· The row from the Object table for the associated Object.· A postage stamp image centered on the DIASource from each of the two Difference

Exposures, and from the Template Exposure.Note that no explicit classification of an Alert is provided, but users can readily construct classifiers and filters based on information in the Science Database. This includes past time dependence, colors, and shape information for the associated Object. Additionally, database queries can readily be formulated which will identify Exposures that have generated anomalously large numbers of Alerts, presumably due to image artifacts or processing problems.

2.2 Level 1 Data Products Level 1 data products are divided into Exposures, Catalogs, Alerts, Nightly Summary Products, and the Engineering and Facility Database Archive.

2.2.1 Exposures · Raw Exposures· Processed Exposures (trimmed, debiased, flattened, etc.)· Difference Exposures.Note that the Raw Exposures are sent to the Archive Center, where they are immediately made available for retrieval. Processed and Difference Exposures are recreated on-the-fly at the Archive Center as needed.

2.2.1.1 Exposures Requirements

2.2.1.1.1 General Exposure Requirements The requirements in this section apply to all Exposures.

1.1.1.1.6.13 Exposure Archive Publicly Accessible Id: DM-APP-DP-IM-2Version: 1.0 Modified: 1/23/2009 Status: Proposed

All releases of the DMS exposure archive shall be maintained and preserved in a publicly accessible state for the entire operational life of the LSST observatory.

2023-05-22 LSE-61 Page 27 of 59


1.1.1.1.6.14 Exposure Formats Id: DM-APP-DP-IM-3Version: 1.0 Modified: 1/23/2009 Status: Proposed

Additionally, full exposures shall be made available in LSST Pipeline Input Format, for processing by LSST pipelines or by external pipelines.

1.1.1.1.6.15 Keep Exposure Archive Id: DM-APP-DP-IM-1Version: 1.0 Modified: 6/16/2011 Status: Proposed

The Archive shall be designed to be capable of managing the data for the full duration of the survey and associated data processing.

2.2.1.1.2 Raw Science Exposure

1.1.1.1.6.16 Raw Science Images Available within 24 hours Id: DM-APP-DP-IM-4Version: 1.0 Modified: 7/12/2007 Status: Proposed

The raw science images and associated metadata shall be available for public access inthe DMS image archive within 24 hours of their read-out from the Camera.

1.1.1.1.6.17 Raw Science Image Attributes Id: DM-APP-DP-IM-5Version: 1.0 Modified: 8/8/2007 Status: Proposed

* Time of exposure start and end, referenced to TAI* Site metadata (site seeing, transparency, weather)* Telescope metadata (telescope pointing, active optics state, environmental state)* Camera metadata (shutter trajectory, wavefront sensors, environmental state)

2.2.1.1.3 Processed Science Exposure

1.1.1.1.6.18 Produce Processed Science Exposures Id:Version: 1.0 Modified: 1/27/2009 Status: Proposed

2023-05-22 LSE-61 Page 28 of 59


1.1.1.1.6.18.1 Determine Accurate PSF for an Exposure Id:Version: 1.0 Modified: 6/16/2011 Status: Proposed

Intrinsic Object shapes can be measured only if the PSF is accurately known at all focal plane positions within the Exposure

1.1.1.1.6.18.2 Calibrated Science Exposures Available witihin 24 hours Id: DM-APP-DP-IM-6Version: 1.0 Modified: 6/16/2011 Status: Proposed

Calibrated science exposures shall be available for public access in the DMS Exposure archive within 24 hours of their generation by the DMS.

1.1.1.1.6.18.3 Processed Science Exposure Attributes Id: DM-APP-DP-IM-7Version: 1.0 Modified: 6/16/2011 Status: Proposed

* List of readouts of a single ccd that have been combined into a single image structure* Associated pixel mask, identifying anomalous sensor pixels, saturated pixels, and pixels affected by artifacts such as satellite trails, cosmic rays, or stray light from nearby bright sources* Astrometric calibration in the form of a WCS, ie. a mapping from pixel coordinates to ICRS sky coordinates* Photometric calibration, i.e a mapping from a source's intensity in data units to its AB magnitude at the top of the atmosphere for the filter used.* A representation of the spatially varying point spread function (PSF)* Data quality assessment

2.2.1.1.4 Difference Exposure

1.1.1.1.6.19 Difference Exposures Available within 24 hours Id: DM-APP-DP-IM-8Version: 1.0 Modified: 1/23/2009 Status: Proposed

Difference Exposures shall be available for public access in the DMS exposure archive within 24 hours of their generation by the DMS.

1.1.1.1.6.20 Difference Exposure Attributes

2023-05-22 LSE-61 Page 29 of 59


Id: DM-APP-DP-IM-9Version: 1.0 Modified: 1/23/2009 Status: Proposed

* The ratio of the RMS level of the image noise in areas far from bright sources to the sky noise in the input images, and the spatial variation of that ratio.* The ratio of the RMS level of the image noise in areas centered on bright sources compared the source level, and the spatial variation of that ratio.* Representation of the PSF matching kernel used in the differencing.

2.2.2 Catalogs · Exposure· Object· MovingObject· DIASource· FaintSource.

2.2.2.1 Catalogs Requirements

2.2.2.1.1 General Catalog Requirements The requirements in this section apply to all Catalogs.

1.1.1.1.6.21 Queryable Id:Version: 1.0 Modified: 1/23/2009 Status: Proposed

The catalogs shall be queryable in SQL

1.1.1.1.6.22 Keep Science Data Archive Id: DM-APP-DP-CA-1Version: 1.0 Modified: 6/19/2007 Status: Proposed

The DMS shall provide a science data archive containing catalogs of all detected sources and objects and associated metadata that meet data quality thresholds for retention.

1.1.1.1.6.23 Maintain Archive Publicly Accessible Id: DM-APP-DP-CA-2Version: 1.0 Modified: 6/19/2007 Status: Proposed

All releases of the DMS catalog archive shall be maintained and preserved in a publicly

2023-05-22 LSE-61 Page 30 of 59


accessible state for the entire operational life of the LSST observatory.

1.1.1.1.6.24 External Formats Id: DM-APP-DP-CA-3Version: 1.0 Modified: 6/19/2007 Status: Proposed

* Comma-separated ASCII text* eXtensible Markup Language (XML) format, including VOTable (http://www.ivoa.net/twiki/bin/view/IVOA/IvoaVOTable)

2.2.2.1.2 Difference Source Catalog

1.1.1.1.6.25 Difference Source Attributes Id:Version: 1.0 Modified: 2/6/2009 Status: Proposed

* Focal plane position centroid (pixels)* Expected focal plane position centroid error (pixels)* Instrumental flux (ADU)* Expected instrumental flux error (ADU)* Sky background level (ADU)* Expected error in sky background (ADU)* Source extraction flags, which identify any problems encountered by the DP in processing the Difference Source* Shape characterization, including at least an extendedness parameter* ICRS sky position derived from the focal plane position (radec)* Expected error in ICRS sky position (radec)

1.1.1.1.6.26 Difference Sources Available within 24 hours Id: DM-APP-DP-CA-4Version: 1.0 Modified: 1/30/2009 Status: Proposed

Detected sources and associated metadata shall be available for public access in the DMS science data archive within 24 hours of their generation by the DMS.

2.2.2.1.3 Object Catalog

1.1.1.1.6.27 Produce Object Catalog Id: DM-APP-DP-CA-6Version: 1.0 Modified: 1/23/2009 Status: Proposed

2023-05-22 LSE-61 Page 31 of 59


The LSST DMS shall produce a catalog of all objects detected by the Detection Pipelinein at least two Difference Exposures from a single Visit.

1.1.1.1.6.28 Source List Updates Id: DM-APP-DP-CA-7Version: 1.0 Modified: 1/24/2009 Status: Proposed

The Difference Source lists shall be updated following each exposure with a latency of no more than 24 hrs.

1.1.1.1.6.29 Level 1 Object Attributes Id: DM-APP-DP-CA-8Version: 1.0 Modified: 1/24/2009 Status: Proposed

* A unique Object ID* All Difference Sources, Sources, and Low SNR Sources associated with the Object* A set of Object properties. A restricted set of properties is generated by the Alert Production, while the full set is generated by the Data Release Production.

1.1.1.1.6.30 Summary Metadata Attributes Id: DM-APP-DP-CA-9Version: 1.0 Modified: 6/19/2007 Status: Proposed

* Mean ICRS sky position* Best estimate (eg median) for AB top-of-atmosphere magnitude in each filter band* Variability measures (eg Welch-Stetson) for each filter band* Proper motion (None if not measured)* Parallax (None if not measured)* Object classification

1.1.1.1.6.31 Summary Metadata Updates Id: DM-APP-DP-CA-11Version: 1.0 Modified: 1/24/2009 Status: Proposed

The object summary metadata attributes shall be updated no less frequently than every 6 months.

2.2.2.1.4 Orbit Catalog

1.1.1.1.6.32 Produce Orbit Catalog Id: DM-APP-DP-CA-12Version: 1.0 Modified: 8/9/2007 Status: Proposed

2023-05-22 LSE-61 Page 32 of 59


The LSST DMS shall produce a catalog of all moving objects detected by the Moving Object Pipeline (MOP)

1.1.1.1.6.33 Source Lists Update Frequency Id: DM-APP-DP-CA-13Version: 1.0 Modified: 8/8/2007 Status: Proposed

The source lists shall be updated following each exposure with a latency of no more than 24 hrs.

1.1.1.1.6.34 Moving Object Attributes Id: DM-APP-DP-CA-14Version: 1.0 Modified: 8/14/2007 Status: Proposed

* Orbital elements* Probable errors in orbital elements* Summary photometric information, as for other objects in Object Catalog* The Sources associated with this Moving Object, as for other Objects in Object Catalog

1.1.1.1.6.35 Moving Objects Available within 24 hours Id:Id: DM-APP-

DP-CA-15Version: 1.0


Detected moving objects and associated metadata shall be available for public access in the DMS science data archive within 24 hours of their generation by the DMS.

2.2.3 Alerts Alerts are distributed as VOEvents, and archived at the Archive Center.

2.2.3.1 Alerts Requirements

1.1.1.1.7 Generate Alerts Id:Version: 1.0 Modified: 6/22/2009 Status: Proposed

2023-05-22 LSE-61 Page 33 of 59


1.1.1.1.7.1 Detect in Two Exposures in Visit Id: DM-APP-DP-AL-1Version: 1.0 Modified: 6/16/2011 Status: Proposed

1.1.1.1.7.2 Alert Attributes Id: DM-APP-DP-AL-3Version: 1.0 Modified: 6/16/2011 Status: Proposed

* Class of event detected* Confidence of detection* All associated information from the Object Catalog (includes Sources associated with this Object)* Postage stamp images centered on the Object from both difference exposures of the visit which triggered the alert, and from the template image used in image substraction.

1.1.1.1.8 Public Distribution of Alerts Id: DM-APP-DP-AL-2Version: 1.0 Modified: 6/22/2009 Status: Proposed

1.1.1.1.9 Keep Historical Alert Archive Id: DM-APP-DP-AL-4Version: 1.0 Modified: 6/22/2009 Status: Proposed

The DMS shall preserve and keep in an accessible state an alert archive with all issued alerts for a historical record and for false alert analysis.

1.1.1.1.10 Archive Alerts within 24 hours Id: DM-APP-DP-AL-5Version: 1.0 Modified: 6/22/2009 Status: Proposed

Alerts shall be available for public access in the DMS alert archive within 24 hours of their initial issue.

2.2.4 Nightly Summary Products A variety of reports will be generated every night to summarize the performance of the DMS and the SDQA metrics on the Level 1 data. The contents of these is TBD.

2.2.4.1 Nightly Summary Products Requirements

2023-05-22 LSE-61 Page 34 of 59


1.1.1.1.11 Generate Data Quality Report within 4 hours Id: DM-APP-DP-SR-1Version: 1.0 Modified: 6/22/2009 Status: Proposed

The DMS shall generate a Data Quality Report in both human-readable and machine-readable forms within 4 hours of the completion of each observing night.

1.1.1.1.11.1 Data Quality Report Contents Id: DM-APP-DP-SR-2Version: 1.0 Modified: 6/16/2011 Status: Proposed

* Photometric zero point vs. time for each utilized filter* Sky brightness vs. time for each utilized filter* Seeing vs. time for each utilized filter - this is a broad brush measure of image quality* PSF parameters vs. time for each utilized filter - this is a lot more information, since it includes asymmetries and field dependence* Detection efficiency for point sources vs. mag for each utilized filter.

1.1.1.1.12 Generate DMS Performance Report within 4 hours Id: DM-APP-DP-SR-3Version: 1.0 Modified: 6/22/2009 Status: Proposed

The DMS shall generate a DMS Performance Report in both human-readable and machine-readable forms within 4 hours of the completion of each observing night.

1.1.1.1.12.1 Performance Report Contents Id: DM-APP-DP-SR-4Version: 1.0 Modified: 6/16/2011 Status: Proposed

* Number of observations successfully processed through each pipeline* Number of observations for each pipeline that had recoverable failures - group by failure type (e.g. hardware or cpu / software failure cleared by automated intervention)* Number of observations for each pipeline that had unrecoverable failures - similar grouping* Number of observations archived at base camp* Number of observations archived at archive center* Number of observations satisfying the science criteria for each active science program

1.1.1.1.13 Generate Calibration Report within 4 hours Id: DM-APP-DP-SR-5Version: 1.0 Modified: 6/22/2009 Status: Proposed

The DMS shall generate a Calibration Report in both human-readable and machine-readable forms within 4 hours of the completion of each observing night.

2023-05-22 LSE-61 Page 35 of 59


1.1.1.1.13.1 Calibration Report Contents Id: DM-APP-DP-SR-6Version: 1.0 Modified: 6/16/2011 Status: Proposed

* A super-flat from all observations on the sky, with filters applied to avoid using cloudy exposures and implementing quality thresholds * A combined bias from all biases.

2.2.5 Engineering and Facility Database Archive Every 24 hrs, the DMS synchronizes the Engineering and Facility Database, which is generated by the Observatory Control System (OCS), with the copy at the Archive Center. This Level 1 product contains comprehensive metadata for the entire LSST observatory, and can be queried together with the Science Database.

2.2.5.1 E&F Database Requirements

1.1.1.1.14 Provide E&F Database Archive Id: DM-APP-DP-EF-1Version: 1.0 Modified: 6/22/2009 Status: Proposed

2.3 Level 2 Data Products Level 2 data products include Coadded Science Exposures, Calibration Products, and Catalogs.

2.3.1 Co-Added Exposures · Template Co-Add for creating Difference Exposures. This Co-Add is optimized for a

narrow PSF, so that during image differencing the Template is never narrowed by the matching kernel, even in the best seeing of the survey. A Template is specific to a filter band.

· Detection Co-Add for object detection. This Co-Add is optimized so that the peaks from faint objects have the highest signal-to-noise. The Detection Co-Add may combine information from multiple filter bands.

· RGB Co-Add for visualization. This Co-Add will be formed by combining the Template Co-Adds for the individual filter bands to give a visually pleasing and useful color image.

2.3.1.1 Co-Added Exposures Requirements

1.1.1.1.15 Produce Images for EPO

2023-05-22 LSE-61 Page 36 of 59


Id:Version: 1.0 Modified: 6/16/2011 Status: Proposed

Discussion: This is expected to include polychromatic (e.g., RGB JPEG) images for casual users.

1.1.1.1.16 Produce Co-Added Exposures Id: DM-APP-DP-CO-1Version: 1.0 Modified: 6/16/2011 Status: Proposed

Each Exposure contributing to a Co-Added Exposure shall be projected onto a common reference frame using a projection selected from a set of options by a Policy.

[ACTION: harmonize with co-add pipeline requirement, enumerate the types of coadds required]

1.1.1.1.17 Co-Added Images Updated at Least Every 6 months Id: DM-APP-DP-CO-2Version: 1.0 Modified: 6/22/2009 Status: Proposed

Co-Added Images shall be created from the latest image archive no less frequently thanevery 6 months and for every Deep Detection processing run.

1.1.1.1.18 Co-Added Image Atrributes Id: DM-APP-DP-CO-3Version: 1.0 Modified: 6/22/2009 Status: Proposed

* registration information* PSF convolution information

2.3.2 Catalogs · Exposure· Object· MovingObject· Source· DIASource· FaintSource.Exposure is updated with the latest derived metadata, such as WCS and PSF.

2.3.2.1 Object Catalog – Level 2 Data

2023-05-22 LSE-61 Page 37 of 59


1.1.1.1.19 Astrometric Model Id:Version: 1.0 Modified: 1/25/2009 Status: Proposed

1.1.1.1.20 Calibrated Light Curve Id:Version: 1.0 Modified: 1/25/2009 Status: Proposed

1.1.1.1.21 Calibrated Point Source Photometry Id:Version: 1.0 Modified: 1/25/2009 Status: Proposed

1.1.1.1.22 Calibrated Position Curve Id:Version: 1.0 Modified: 1/25/2009 Status: Proposed

1.1.1.1.23 Extended Object Photometry Id:Version: 1.0 Modified: 6/16/2011 Status: Proposed

TBD

1.1.1.1.24 Level 2 Object Attributes Id: DM-APP-DP-DO-1Version: 1.0 Modified: 1/25/2009 Status: Proposed

* extended object shape parameters* extended object photometry* photometric redshifts* calibrated point source photometry* fully sampled photometric time history* astrometric model

1.1.1.1.25 Object Classification Id:Version: 1.0 Modified: 6/16/2011 Status: Proposed

2023-05-22 LSE-61 Page 38 of 59


TBD

1.1.1.1.26 Photomeric Redshift PDF Id:Version: 1.0 Modified: 1/30/2009 Status: Proposed

1.1.1.1.27 Photometric Redshifts Id:Version: 1.0 Modified: 1/30/2009 Status: Proposed

Photometric redshifts shall be calculated for all extended objects which have calibrated magnitudes in all filters.

1.1.1.1.28 Extended Object Shape Parameters Id: DM-APP-DP-CA-10Version: 1.0 Modified: 1/25/2009 Status: Proposed

Weak lensing studies require the image moments (or equivalent) through N=2.

For weak lensing extended objects of size greater than TBD (10 arcsec?) do not require shape parameters to be measured

2.3.2.2 Source Catalogs - Level 2 Data

1.1.1.1.29 Low SNR Source Attributes Id:Version: 1.0 Modified: 6/16/2011 Status: Proposed

* Instrumental flux (ADU)* Expected error in instrumental flux (ADU)* Sky background level (ADU)* Expected error in sky background (ADU)* Source extraction flags, which identify any problems encountered by the DP in processing the Source

1.1.1.1.30 Source Attributes Id: DM-APP-DP-CA-5Version: 1.0 Modified: 1/23/2009 Status: Proposed

2023-05-22 LSE-61 Page 39 of 59


* Focal plane position centroid (pixels)* Expected focal plane position centroid error (pixels)* Instrumental flux (ADU)* Expected instrumental flux error (ADU)* Sky background level (ADU)* Expected error in sky background (ADU)* Source extraction flags, which identify any problems encountered by the DP in processing the Source* Shape characterization, including at least an extendedness parameter* ICRS sky position derived from the focal plane position (radec)* Expected error in ICRS sky position (radec)

2.4 Level 3 Data Products · Phase-folded light curves for periodic variables \item catalogs of specific subsets of

objects· Maps of derived properties such as lensing shear· Catalogs of clusters· Catalogs of morphologically analyzed galaxies.An essential feature of Level 3 data products is that their creation is not a responsibility of the DMS. They will arise from user analysis projects. The generation of Level 3 data products may thus involve the use of code and query definitions from outside the LSST project, and that is not part of the project's open-source code base. When Level 3 data products are created as the output of analysis of LSST data, whether with the basic interactive user tools or via custom pipelines, tools will be provided so that they may be federated with the Level 1 and Level 2 datasets, so that joins may easily be made between data-release and user-provided tables.It is anticipated that certain of the Level 3 data products will be archived by the DMS, using project resources provided for this purpose. The allocation of these resources to archiving, and the duration of storage, will be determined by the LSST Project based on the value of the data to the community and the cost of recomputing the data versus persisting it. The DMS is further required to facilitate the archiving of Level 3 data products using external resources, by providing data import and export tools, and tools to assist external users in maintaining the consistency of large multi-file datasets.It is anticipated that some, but not all, Level 3 data products will be generated using the physical resources of the Data Access Centers (DACs) provided as part of the LSST project; others will be generated using external resources. These may be in the form of additional DACs externally funded, but configured and managed much like the internal ones, or may be truly external computing facilities brought to bear on specific LSST data analysis challenges. The DMS is thus required to facilitate the production of Level 3 data products by clients external to the DMS, both at LSST-controlled DACs and at external sites. This will be done by providing stable and well-documented APIs and libraries based on open-source software, freely downloadable to external sites, and by providing a modest level of user support from the project.

2023-05-22 LSE-61 Page 40 of 59


2.4.1 General Level 3 Data Product Requirements

1.1.1.2 DAC resource allocation for Level 3 processing Id: DM-APP-DP-LTG-1Version: 1.0 Modified: 2/6/2009 Status: Proposed

1.1.1.3 Level 3 Data Product Self Consistency Id: DM-APP-DP-LTG-2Version: 1.0 Modified: 2/6/2009 Status: Proposed

The DMS shall provide a means for ensuring that users' Level 3 processing tasks are carried out on self-consistent inputs - i.e., catalogs, images, metadata, calibrations, camera configuration data, etc., that match each other and all arise from consistent Level 1 and Level 2 processings.

1.1.1.4 Provenance for Level 3 processing at DACs Id: DM-APP-DP-LTG-3Version: 1.0 Modified: 2/6/2009 Status: Proposed

The DMS should also provide an optional means for Level 3 processing users at DACs to maintain basic provenance information on their own inputs to a processing task, such as code or additional calibration data.

Rationale: the DMS should facilitate Level 3 processing users in being able to carry out their work in a reproducible way.

2.4.2 Level 3 Catalog Processing Requirements

1.1.1.5 Access to catalogs for external Level 3 processing Id: DM-APP-DP-LTC-2Version: 1.0 Modified: 2/6/2009 Status: Proposed

1.1.1.6 Access to input catalogs for DAC-based Level 3 processing Id: DM-APP-DP-LTC-1Version: 1.0 Modified: 2/6/2009 Status: Proposed

The DMS shall provide access to all Level 1 and Level 2 catalog products through the

2023-05-22 LSE-61 Page 41 of 59


LSST project's Data Access Centers, and any others that have been established and funded, for Level 3 processing that takes place at the DACs.

1.1.1.7 Federation with external catalogs Id: DM-APP-DP-LTC-4Version: 1.0 Modified: 2/6/2009 Status: Proposed

1.1.1.8 Software framework for Level 3 catalog processing Id: DM-APP-DP-LTC-3Version: 1.0 Modified: 2/6/2009 Status: Proposed

2.4.3 Level 3 Image Processing Requirements This section contains requirements on the Level 3 processing of images, separated from those for catalogs because they may in time diverge.

1.1.1.9 Access to images for external Level 3 processing Id: DM-APP-DP-LTI-2Version: 1.0 Modified: 2/6/2009 Status: Proposed

1.1.1.10 Access to input images for DAC-based Level 3 processing Id: DM-APP-DP-LTI-1Version: 1.0 Modified: 2/6/2009 Status: Proposed

The DMS shall provide access to all Level 1 and Level 2 image products through the LSST project's Data Access Centers, and any others that have been established and funded, for Level 3 processing that takes place at the DACs.

1.1.1.11 Software framework for Level 3 image processing Id: DM-APP-DP-LTI-3Version: 1.0 Modified: 2/6/2009 Status: Proposed

2.5 Calibration Data Products · Bias Exposures· Monochromatic Dome Flats· Broadband Dome Flats· Pupil Ghost Image · Crosstalk Correction Matrix

2023-05-22 LSE-61 Page 42 of 59


· Fringe Images· Illumination Correction· A variety of products required for the Auxiliary Telescope (TBD).

2.5.1 Calibration Products Requirements

Crosstalk Correction Matrix Attributes Id:Version: 1.0 Modified: 6/22/2009 Status: Proposed

The crosstalk correction matrix will supply the coupling coefficient due to electronic crosstalk between the pixel value read out in each channel of the camera and the value read out in every other channel.

1.1.1.12 Calibration Data Products Id: DM-APP-DP-CI-1Version: 1.0 Modified: 6/22/2009 Status: Proposed

* Monochromatic dome flats over a representative wavelength range* Sky flat* Bias frame* Dark frame* Fringe frame* Illumination correction frame* Crosstalk correction matrix

1.1.1.13 Calibration Images Available 1 Hour Before Observing Id: DM-APP-DP-CI-2Version: 1.0 Modified: 6/22/2009 Status: Proposed

Calibration Images shall be available for nightly observing in the DMS image archive at least 1 hour prior to the start of observing.

1.1.1.14 Calibration Image Attributes Id: DM-APP-DP-CI-3Version: 1.0 Modified: 6/22/2009 Status: Proposed

* The calibration image type (eg dome flat, superflat, bias, etc)* The filter to which it applies (not relevant for bias)* The range of raw image exposure dates for which the calibration image is valid

2023-05-22 LSE-61 Page 43 of 59


2.6 Detection and Measurement of Objects

2.6.1 Deep Detection Processing

2.6.2 Difference Exposure Processing Note that this process cannot be perfect, since measuring the extendedness of objects near the PSF size will always be uncertain. Consequently, there will be cases where flux from a supernova or AGN point source will be incorrectly added to the underlying galaxy rather than to a new point source. Between successive Data Releases, however, these errors will decrease in number: as the survey goes deeper, and accumulates images in better seeing, extendedness will be better measured by the Multifit procedure.

2.6.3 Measuring the properties of Objects Our choice of models for Multifit is driven by astrophysics, by characteristics of the LSST system, and by computing practicalities, as follows:· Within the context of the LSST survey, clearly extended objects do not have

measurable proper motion or parallax. Comets are a clear exception, but they are processed only within difference exposures, not through Multifit. Supernova light echoes might also be an exception, and they deserve further thought.

· While co-added exposures largely erase the effects of the gaps between the individual CCDs, individual exposures are strongly affected by them. Processing of objects in individual exposures that extend across CCD gaps creates a significant overhead of programming complexity and computational cost, and will not initially be implemented in the DMS. This capability can be incorporated in later versions of the DMS if it is warranted by the science gain.

· Given the above constraint, Multifit is only useful if the object being fit is wholly contained within a single CCD field for the majority of the exposures in which it appears. Note that this does not mean that an object needs to appear in the same CCD in different exposures! This will rarely occur, given the survey's dithering pattern. If we want the object containment probability to be at least 0.8, the object size, d can be no larger than 0.1 D, where D is the angular size of the CCD on the sky, 13 arcmin in the case of LSST. This sets a natural upper limit on object size to be processed by Multifit of approximately 1 arcmin. We note that this size comfortably encompasses the objects required for the SRD's science drivers.

2.6.3.1 Slowly Moving Point Source Model The SMPS Model will be fit only to objects which are leaf nodes in the segmentation tree.

2023-05-22 LSE-61 Page 44 of 59


2.6.3.2 Small Object Model The SO Model will be fit only to objects which are leaf nodes in the segmentation tree.

2.6.3.3 Large Object Model A typical large object will be segmented into smaller component objects. The leaf objects in the resulting segmentation tree will be measured as described above, providing they qualify as ``small''. The large object itself, the root of the segmentation tree, will be represented only by its ``Footprint'', with the following attributes:· Ellipse equivalent to Object footprint (same moments through second order)· Footprint bounding box· Flux within footprint.Parameters resulting from model fitting, and from analysis of time dependent properties, will be absent. Fits of morphological models (eg bulge/disk) to large objects must be created as Level 3 data products.Objects larger than "large" will not be represented in the DMS.

2.6.3.4 Solar System Model The details of the FaintSource measurement process are not yet well defined. In particular, it is unclear if the measurements should be at a position entirely fixed by the orbit prediction, or should be allowed to compensate for prediction error by "peaking up" within some error bound around the prediction.

2.6.4 The Multifit Algorithm The library of model types is described in the preceding subsections. An initial model will be fit to the co-add, to provide a good starting point for the fit to the full dataset. Multifit will then read in all the pixels from the $n$ exposures and perform a maximum likelihood fit for the model which, when convolved with the $n$ PSFs, best matches the n observations. This naturally incorporates the effects of varying seeing, as the contribution of the better-seeing images to the likelihood will be sharper. This approach also facilitates proper accounting for masked areas, cosmic rays, etc. The best-fit model parameters and their uncertainties will be recorded in an Object table row.Slightly simplified versions of this fitting approach have already been used in major surveys. 2MASS used it to fit point sources to their multiple-epoch observations which had nontrivial seeing variations. SDSS uses it to fit a model simultaneously to the ugriz images. The extension here is to fit simultaneously across wavelengths and across a large number of multiple-epoch observations with seeing variations.

2.6.5 Model Residuals 1. An Example: Supernova in a Visible GalaxySuppose that a supernova explodes in a ``small'' galaxy that is clearly resolved in LSST imagery, and is already listed in the Object table from a previous Data Release.

2023-05-22 LSE-61 Page 45 of 59


Suppose further that the supernova is bright enough that it will be above the detection threshold in at least one difference exposure. The following sequence of events will occur during the nightly Alert Production processing:· The first time the supernova is detected above threshold in both difference

exposures from a Visit, the Association Pipeline (AP) will attempt to match the resulting DIASources to an Object in the Object table. In this case, it will find that each DIASource is contained within the footprint of its host galaxy, but based on the fact that the galaxy is extended, and the DIASources are not, the AP will create a new Object (SN) at the position of the supernova.

· An Alert will be issued for the supernova.· On subsequent Visits, as long as the supernova remains above the detection

threshold, new DIASources will be created from each Exposure, and the AP will associate them to the SN object. A query to the science database will readily retrieve all difference image photometry for SN from the DIASources linked to the SN Object entry.

· A likely policy for the survey to follow will be to add every Object that results in an Alert to a list of Objects to be force photometered in Difference Exposures. If this is done, the SN will result in FaintSource entries even after it has dropped below the detection threshold above which it would generate DIASources. This will allow a query to the science database to retrieve this photometry as well.

When it is time to create a new Data Release, a new, empty, science database is created. It is populated with the Raw Exposures from the survey extending from the inception of the survey up until the cutoff date for the DR, but nothing else. Note in particular that none of the other Level 1 data products created by the Alert Production are imported into the DR. Processing always begins from scratch. The following sequence of events involving the supernova will then occur during Data Release processing:· As with all sky patches, a subtraction template co-add for each filter is created for

the patch of sky containing the supernova by combining all the survey exposures that cover that patch. The combination algorithm strongly discriminates against transient flux, eg with a median. The template co-add will therefore contain an image of the galaxy uncontaminated by the supernova.

· A detection co-add is created for the patch of sky containing the supernova by combining all the survey exposures that cover that patch. The combination algorithm will discriminate against single visit transients, eg from solar system objects, but otherwise optimally combines the flux from all exposures, eg with the Kaiser algorithm. The detection co-add therefore contains the galaxy with the superimposed supernova. The brightness of the supernova in the co-add depends on the fraction of the total co-add duration where the supernova has significant flux (so it becomes progressively fainter in subsequent DRs).

· Object detection and deblending is run on the detection co-add. There are two cases to consider:

· The supernova is bright enough in the co-add to trigger deblending into a galaxy plus a point source (Case A). Three entries are made in the Object table: Gal+SN, Gal, and SN. The Gal+SN object is the root of the segmentation tree, while Gal and SN are the deblended leaves.

2023-05-22 LSE-61 Page 46 of 59


· The supernova is faint enough in the co-add that deblending is not triggered. A single object is detected, with the galaxy flux slightly distorted by the supernova flux (Case B). A single entry is made in the Object table: Gal*. Gal* is not deblended, so it is a leaf node in the segmentation tree.

· Multifit is run on the exposure stack for the Objects entered into the table that are leaf nodes in the segmentation tree, two objects for Case A, one for Case B. Given that Gal is ``small'', both the SO and SMPS models will be fit for these objects. Let us consider Case A and B separately:

· For Case A, Multifit is run separately on Gal and SN, fitting the SO and SMPS models for each. The model residuals should clearly favor the SO model for Gal and the SMPS for SN, but note that the model parameters for both Gal and SN will be distorted by the presence of the other. For example, the position of the SN/SMPS will appear to vary with time, since at low flux levels its profile will be significantly affected by the galaxy, but less so at high flux levels. The Gal/SO position is not allowed to vary with time, by definition of the SO model, but its flux will vary due to the presence of the SN. [This argues that it is desirable to run Multifit simultaneously on all models that overlap, or at least iteratively, subtracting models from the images. See Section 6.7] As a result of running Multifit, Source entries will be generated for Gal at each epoch in the stack. Source entries will be generated for SN at each epoch in the stack where its SNR exceeds some detection threshold. FaintSource entries will be created for Gal and for SN at each epoch in the stack from forced photometry with the model shape and position.

· For Case B, Multifit fits both the SO and SMPS models to Gal*. The model residuals for the individual exposures will likely vary greatly with time. If the Gal* was not deblended simply because the supernova occupied only a small fraction of the co-add time range, both SO and SMPS models will show large residuals, and in the case of SMPS, spurious motion as well. Both Source and FaintSource entries will be generated for Gal* at every epoch.

· With the Multifit procedure complete, the difference exposures are formed by psf-matching and subtracting the template exposure from each exposure in the exposure stack. By hypothesis, the supernova exceeds the detection threshold in at least some of these exposures. For each exceedance, a DIASource is created, and an attempt is then made to match it with an object already in the Object table just created by Multifit. Again, the outcome is different for Case A and Case B:

· For Case A, the DIASources will match the SN object, and will be given that Object Id.

· For Case B, the first DIASource will not match Gal* because the supernova is a point source, while Gal* (by hypothesis) is measurably extended (it is also unlikely to match in position, but it may). Therefore, a new Object is created for that DIASource, call it SN*. All subsequent DIASources will then be associated to SN*.

For both Case A and B, FaintSource entries will be created for every epoch in the stack by forced photometry on the difference exposures.When the processing of the Data Release is complete, the following information will be

2023-05-22 LSE-61 Page 47 of 59


available for the supernova and its underlying galaxy:· Entries in the Object table for both the supernova and the galaxy. For Case A, the

model parameters reported for the supernova and the galaxy may each be significantly biased by the presence of the other.

· A full lightcurve for the supernova measured in difference exposures and reported in the FaintSource entries associated with the supernova object.

· A full set of Sources for the galaxy, each of which quantifies the residual between the model and the exposure from which the Source was measured.

· For Case A, a full set of Sources for the supernova, each of which quantifies the residual between the model and the exposure from which the Source was measured.

· A partial set of DIASources for the supernova, limited to those epochs where it is brighter than the detection limit.

Note that the case of a time variable AGN embedded in a small galaxy is broadly parallel to the supernova case considered above. Unlike the supernova case, however, where the FaintSources from difference exposures give a nearly optimal extraction of the full light curve, in the AGN case, only the "AC" part of the AGN light curve will be so measured. Some "DC" part of the AGN lightcurve will be part of the flux associated with the galaxy.

2.6.6 Limitations and Possible Extensions

2.6.6.1 Faint Transients 1. The object is above the detection threshold in the co-add used for object detections

and/or2. The peak brightness is above the detection limit for difference images.We can imagine a case where condition 2 is not satisfied, but the luminosity peak is covered by enough LSST exposures that a co-add concentrated near the peak satisfies condition 1. If the peak occurs near the beginning of the survey, the transient will be measured in DR1, but conceivably missed in subsequent DR's, because the significance of the peak in the co-add will become progressively less as more exposures from the quiescent phase are added. This behavior is certainly undesirable, and will have an undetermined negative impact on transient science. It could be overcome by a strategy which created a variety of co-adds spanning different time intervals. This would clearly create significant additional computing costs, and is not currently planned. This decision can be revisited if there is a compelling science case.

2.6.6.2 Faint Solar System Objects

2.6.6.3 Possible Multifit Improvements

2023-05-22 LSE-61 Page 48 of 59


2.7 Inserting Synthetic Objects The strategy for this is still under discussion. An outline of the current strawman design:· LSST DM will provide a general facility to add synthetic objects to existing LSST

exposures, fully taking into account the observed PSF and sky background for each exposure.

· Science teams that wish to ascertain their detection efficiency for a particular class of objects will supply a model for the object class.

· One or more ``mini-DR''s will be created using the normal Data Release procedures, with appropriate synthetic objects added to the input exposures. Separate tables describing the synthetic objects will be maintained within the Science Database. A mini-DR will span a limited area of sky and/or time so that the computational requirements are tractable.

· The Science Database for each mini-DR will be maintained as a completely separate database from those of any other DR.

2.8 Data Access

2.8.1 Computing Resources

2.8.2 Data Access Tools Queries will be analyzed before their dispatch to DAC computing resources. Simple queries are expected to be able to be carried out with sub-minute latency. More complex or I/O intensive ones, up to and including ones requiring full table scans, will be bundled together for efficiency and scheduled for execution on a regular basis---hourly, in the present plan. The complexity and resource requirements of such queries have in the past been found to follow a power law. Based on the present analysis of expected queries, we expect to be able to support all but the most CPU-intensive queries on the project-provided DAC resources.The Data Management Functional Requirements Specification already anticipates the existence of a class of analyses and data products (Level 3) that will be carried out and obtained beyond the scope of these tools and of the required public data products and associated pipelines that are included in the project baseline (Level 1 and Level 2). Generally these will be analyses requiring bulk access to the data. Data Management is required to facilitate such custom pipeline analyses, including ones involving proprietary science algorithms. This will be done by providing a set of interfaces and tools that can be used at the DACs, with a modest level of user support provided by the project. These are intended to be the same tools that are used for the project-required Science Data Quality Analysis activity. These tools include both the APIs required to access image and catalog data within the LSST pipeline framework, and the ``steering'' tools required to ensure that such analyses may be carried out across the full dataset and to record the provenance information required to make such analyses reproducible.

2023-05-22 LSE-61 Page 49 of 59


As noted, it is possible that the volume of data analysis work will exceed the baselined capacity of the DACs, so that external facilities will be required to carry out the full science program. To facilitate this, the custom pipeline interfaces and tools will be open source and will also be made available for download to computing sites beyond the project-provided DACs. These tools will be designed according to industry standards, so as to be portable across a reasonable range of flavors of Unix, and are planned to have dependencies only on freely available, open-source packages.Data Management obviously cannot provide quality assurance over the scientific content of user analyses, but the DM-provided tools will allow users to maintain technical quality control of their analyses, e.g., ensuring that they have run on complete and self-consistent inputs and associated calibrations. Users will be able to provide quality control metrics from their analyses and will be able to aggregate them over all their processing of the full dataset to monitor the progress of their pipelines.

2.9 Incorporation of User Analysis Modules It would be beneficial to LSST and all its users to streamline the process of code contribution, with special attention given to low-level basic tools that might be used by many groups. Data Management will work to facilitate this by providing well-documented open-source interfaces, programming standards, and software quality metrics that external users can rely on in constructing their own pipelines and tools. Contributions which meet these standards will be eligible for inclusion in the project's code base. Data Management will provide a modest level of user support to assist users in meeting these requirements.There are some constraints, however: for instance, if contributed code brings in new external libraries or otherwise expands the dependencies of the code base, it may not be possible to include it. In addition, LSST may not be able to provide staff to support non-trivial external code contributions in the long term. We will encourage user groups to provide such support for their code, and will generally be willing to assist in this by making available the same support tools (such as for documentation and problem tracking) that we use in-house. User groups will be expected to produce documentation of the use and science data quality of contributed data products and code, and to take responsibility for its robustness when applied at LSST scales.A possible final stage of the incorporation of contributed code would be for it to become part of the standard LSST data production and used to generate public data products centrally. We expect that this will turn out to be desirable at some point in the life of the project for some subset of contributed code. There are important caveats: this will not be possible if it significantly increases the CPU, memory, or archival storage requirements of the system beyond its baseline, unless those costs are covered in some way at the time of the decision. It will also require an explicit agreement on the part of the contributors to support their code for the long term or find funding to increase the central support resources. Finally, any contributed code incorporated into production will have to be demonstrated to be highly reliable and must be integrated in the standard SDQA system, with the contributor responsible for defining and implementing the necessary metrics. For non-trivial contributions a peer review of the

2023-05-22 LSE-61 Page 50 of 59


quality assurance plan would be advisable.

3 Middleware Layer

3.1 Middleware Layer Requirements

1.1.2 Provide Data Access Services Id: DM-MW-1Version: 1.0 Modified: 6/22/2009 Status: Proposed

* Transparent access to pipeline input data regardless of execution and storage location* Transparent access to all output data products across LSST sites* Automated ingestion of data products * Data staging and transfer * Services for data access into files, structured objects, and database query results* Secure access control to all data* Services to download any of the released LSST data products, from raw to stacked, as desired, subject to the users’ computing capabilities (e.g., network bandwidth, storage)* Services to submit queries against all LSST catalogs and receive results in a research-ready form* Services to track the provenance of a data object, such as a transient detection in a catalog, back through the processing used and to the original raw data from which it was generated* Services to manage the replication of data across LSST facilities as needed.

1.1.3 Provide Pipeline Execution Services Id: DM-MW-2Version: 1.0 Modified: 6/22/2009 Status: Proposed

* Services for parallel processing on high-performance platforms, including distributed-memory and shared-memory architectures* Real-time monitoring of execution of pipelines* Automated detection of and recovery from faults.

1.1.4 Provide Management and Control Services Id: DM-MW-3Version: 1.0 Modified: 6/22/2009 Status: Proposed

* Services for re-configuring and redeploying pipelines, in particular, change where the particular processing takes place without manually rebuilding the pipeline.* Services to allow a pipeline to be paused in order to change module parameters.* Services to allow a pipeline to be restarted without extensive repetition of previously executed processing steps, as a new instance of processing with associated provenance.

2023-05-22 LSE-61 Page 51 of 59


* Services to allow portions of pipelines to be executed on selected sets of data without major reconfiguration.* Component interfaces that allow both local and secure, remote control of the data management system* Adaptive and prioritized process management/control/scheduling services* Workflow management* Monitoring, error handling, and failure recovery services* Secure access control to pipelines and data

1.1.5 Provide Pipeline Construction Services Id: DM-MW-4Version: 1.0 Modified: 6/22/2009 Status: Proposed

* Services to statically (i.e. not during execution) insert a new module in an existing process flow without recompilation of modules whose interface has not changed.* Services to perform both composition-time and run-time type checking of modules and data structures for compatibility.* Services to estimate the processing and storage workload represented by a given pipeline configuration and dataset to be processed.* Services to configure a data-driven strategy for use by pipeline execution management.* Services to integrate user-generated query or analysis modules into the LSST processing infrastructure at LSST facilities, subject to security restrictions and available LSST resources.* Services to perform TBD analysis operations remotely

1.1.6 Provide Data/Catalog Construction Services Id: DM-MW-5Version: 1.0 Modified: 6/22/2009 Status: Proposed

* Services to define new data types, including new science-related entities such as object classifications, new meta-data entities, and new data relationships.* Services to extend catalogs to include new data types and concepts* Services to integrate new types of storage hardware* Services to create queries and searches for new data access patterns* Services to define new data access control policies

1.1.7 Provide User Interface Services Id: DM-MW-6Version: 1.0 Modified: 6/22/2009 Status: Proposed

* Services and tools to browse LSST data products through TBD astronomical views or visualizations* Services to create and serve "best" images of selectable regions of the sky in a TBD form* Services to resample and re-project images to TBD* Services to co-add LSST with external images [ I don't think this is useful - TSA]

4 Infrastructure Layer

2023-05-22 LSE-61 Page 52 of 59


4.1 Infrastructure Layer Requirements

4.1.1 General Infrastructure Requirements The requirements in this section apply to all Infrastructure elements.

1.1.7.1 Optimization of Cost, Reliability and Availability in Order Id: DM-INF-1Version: 1.0 Modified: 7/3/2007 Status: Proposed

The allocation of processing and storage to facilities will be done to optimize (in priority order) cost, reliability, and availability of the total data management subsystem.

1.1.7.2 Pipeline throughput Id: DM-INF-2Version: 1.0 Modified: 7/3/2007 Status: Proposed

1.1.7.3 Re-processing Capacity Id: DM-INF-3Version: 1.0 Modified: 7/3/2007 Status: Proposed

The infrastructure will be sized such that complete re-processing of the entire raw imagedataset may occur once per year without interrupting observatory operations.

1.1.7.4 Temporary Storage for Communications Links Id: DM-INF-4Version: 1.0 Modified: 7/3/2007 Status: Proposed

The infrastructure will provide for temporary storage for a minimum of 200% of the mean time to repair of any communications network link at the source end of that link.

1.1.7.5 Infrastructure Sizing for "catching up" Id: DM-INF-5Version: 1.0 Modified: 9/16/2007 Status: Proposed

1.1.7.6 Incorporate Fault-Tolerance

2023-05-22 LSE-61 Page 53 of 59


Id: DM-INF-6Version: 1.0 Modified: 7/3/2007 Status: Proposed

The infrastructure will incorporate as fault-tolerance features to prevent loss of data in the event of hardware or software failure.

1.1.7.7 Incorporate Autonomics Id: DM-INF-7Version: 1.0 Modified: 7/3/2007 Status: Proposed

4.1.2 Mountaintop Site Requirements

1.1.7.8 Mountaintop Site Data Communications Id: DM-INF-MS-1Version: 1.0 Modified: 7/3/2007 Status: Proposed

The DMS shall provide data communications infrastructure to accept science and associated metadata read-outs and transfer to the nightly processing pipelines.

1.1.7.9 Mountaintop Site Temporary Storage Id: DM-INF-MS-2Version: 1.0 Modified: 7/3/2007 Status: Proposed

The DMS shall provide four nights (two nights, redundantly) of raw data storage on the Mountaintop Site in the event of Mountain to Base network outages.

1.1.7.10 Prefer Computing and Storage Down Id: DM-INF-MS-3Version: 1.0 Modified: 7/3/2007 Status: Proposed

The DMS computing and storage equipment will be preferentially located at Base Facility versus the Mountaintop Site due to lower support costs and fewer reliability issues (lower altitude), therefore any processing that can be done in either location will be allocated to the Base Facility.

4.1.3 Mountain to Base Network Requirements

1.1.7.11 Mountain to Base Network Id: DM-INF-MB-1

2023-05-22 LSE-61 Page 54 of 59



The DMS shall provide communications infrastructure between the Mountaintop Site and the Base Facility sufficient to carry scientific data and associated metadata for eachimage in <= 6 seconds.

1.1.7.12 Mountain to Base Network Availability Id: DM-INF-MB-2Version: 1.0 Modified: 7/3/2007 Status: Proposed

The Mountain to Base communications shall be highly available, with Mean Time Between Failures (MTBF) > 90 days, measured over a one-year period.

1.1.7.13 Mountain to Base Network Reliability Id: DM-INF-MB-3Version: 1.0 Modified: 7/3/2007 Status: Proposed

The Mountain to Base communications shall be highly reliable, with Mean Time to Repair (MTTR) < 24 hours, measured over a one-year period.

1.1.7.14 Mountain to Base Network Secondary Link Id: DM-INF-MB-4Version: 1.0 Modified: 7/13/2007 Status: Proposed

1.1.7.15 Mountain to Base Network Ownership and Operation Id: DM-INF-MB-5Version: 1.0 Modified: 7/3/2007 Status: Proposed

The Mountain to Base communications link shall be owned and operated by LSST and/or the operations entity to ensure responsiveness of support.

4.1.4 Base Facility Requirements

1.1.7.16 Base Facility Infrastructure Id: DM-INF-BF-1Version: 1.0 Modified: 8/9/2007 Status: Proposed

The Base Facility shall provide sufficient computing, storage, and network infrastructure to support nightly processing, including image processing, detection, association, and moving object pipelines, the generation of all time-critical data products, i.e. alerts, and

2023-05-22 LSE-61 Page 55 of 59


the assessment of nightly data quality.

1.1.7.17 Base Facility Temporary Storage Id: DM-INF-BF-2Version: 1.0 Modified: 7/3/2007 Status: Proposed

The Base Facility shall provide at least four nights (two nights, redundantly) of raw data storage in the event of Base to Archive Center network outage.

1.1.7.18 Base Facility Co-Location with Existing Facility Id: DM-INF-BF-3Version: 1.0 Modified: 7/3/2007 Status: Proposed

The Base Facility shall be co-located at an existing facility to leverage existing support and facility resources

1.1.7.19 Base Facility Data Quality Assessment Id: DM-INF-BF-4Version: 1.0 Modified: 9/16/2007 Status: Proposed

The Base Facility shall provide computing infrastructure sufficient to perform nightly data quality assessment on at least 10% of the nightly data, to ensure optimal observatory operations.

4.1.5 Base to Archive Network Requirements

1.1.7.20 Base to Archive Network Id: DM-INF-BA-1Version: 1.0 Modified: 7/3/2007 Status: Proposed

The DMS shall provide communications infrastructure between the Base Facility and the Archive Center sufficient to carry scientific data and associated metadata for a givennight before the start of observing preparation for the next night.

1.1.7.21 Base to Archive Network Availability Id: DM-INF-BA-2Version: 1.0 Modified: 7/3/2007 Status: Proposed

The Base to Archive communications shall be highly available, with MTBF > 180 days, measured over a one-year period.

2023-05-22 LSE-61 Page 56 of 59


1.1.7.22 Base to Archive Network Reliability Id: DM-INF-BA-3Version: 1.0 Modified: 7/3/2007 Status: Proposed

The Base to Archive communications shall be highly reliable, with MTTR < 48 hours, measured over a one-year period.

1.1.7.23 Base to Archive Network Secondary Link Id: DM-INF-BA-4Version: 1.0 Modified: 7/13/2007 Status: Proposed

1.1.7.24 Base to Archive Network Lease Id: DM-INF-BA-5Version: 1.0 Modified: 7/3/2007 Status: Proposed

The Base to Archive communications shall be leased from a public or commercial entity,due to high cost of deployment and support for long-haul links.

4.1.6 Archive Center Requirements

1.1.7.25 Archive Center Id: DM-INF-AC-1Version: 1.0 Modified: 7/3/2007 Status: Proposed

The Archive Center shall provide computing, storage, and network infrastructure to support pipeline processing and reprocessing, permanent storage for all data products (with provenance), and serve data for replication to data centers and end user sites.

1.1.7.26 Archive Center Disaster Recovery Id: DM-INF-AC-2Version: 1.0 Modified: 7/3/2007 Status: Proposed

1.1.7.27 Archive Center Co-Location with Existing Facility Id: DM-INF-AC-3Version: 1.0 Modified: 7/13/2007 Status: Proposed

The Archive Center shall be hosted at an existing NSF/DOE-funded supercomputing center.

2023-05-22 LSE-61 Page 57 of 59


4.1.7 Archive to Data Access Center Network Requirements

1.1.7.28 Archive to Data Access Center Network Id: DM-INF-AD-1Version: 1.0 Modified: 7/3/2007 Status: Proposed

The DMS shall provide communications infrastructure between the Archive Center and Data Access Centers sufficient to carry scientific data and associated metadata in support of community and EPO access. Aggregate bandwidth for data transfers from the Archive Center to Data Centers shall be at least 10 Gbps.

1.1.7.29 Archive to Data Access Center Network Availability Id: DM-INF-AD-2Version: 1.0 Modified: 7/3/2007 Status: Proposed

The Archive to Data Access Center communications shall be highly available, with MTBF > 180 days, measured over a one-year period.

1.1.7.30 Archive to Data Access Center Network Reliability Id: DM-INF-AD-3Version: 1.0 Modified: 7/3/2007 Status: Proposed

The Archive to Data Access Center communications shall be highly reliable, with MTTR < 48 hours, measured over a one-year period.

1.1.7.31 Archive to Data Access Center Network Secondary Link Id: DM-INF-AD-4Version: 1.0 Modified: 7/3/2007 Status: Proposed

1.1.7.32 Archive to Data Access Center Network Lease Id: DM-INF-AD-5Version: 1.0 Modified: 7/3/2007 Status: Proposed

The Archive to Data Access Center communications shall be leased from a publicly funded research network entity, due to high cost of deployment and support for long-haul links.

4.1.8 Data Access Center Requirements

2023-05-22 LSE-61 Page 58 of 59


1.1.7.33 Data Access Centers Id: DM-INF-DC-1Version: 1.0 Modified: 7/3/2007 Status: Proposed

The Data Access Centers shall provide computing, storage, and network infrastructure to support open access to LSST data products (with provenance) by end users.

1.1.7.34 Data Access Center Simultaneous Connections Id: DM-INF-DC-2Version: 1.0 Modified: 7/3/2007 Status: Proposed

At least 300 simultaneous connections shall be supported at each Data Access Center.

1.1.7.35 Data Access Center VO Standards Id: DM-INF-DC-3Version: 1.0 Modified: 7/3/2007 Status: Proposed

All Data Access Centers shall provide access to LSST data products via public interfaces and services in accordance with IVOA/NVO standards.

1.1.7.36 Data Access Center Geographical Distribution Id: DM-INF-DC-4Version: 1.0 Modified: 6/16/2011 Status: Proposed

The Data Access Centers will be hosted at facilities selected in at least the U.S. and theobservatory host country to permit widest possible access to LSST data with the fewest possible network hops.

1.1.7.37 No Limit on Data Access Centers Id: DM-INF-DC-5Version: 1.0 Modified: 6/16/2011 Status: Proposed

2023-05-22 LSE-61 Page 59 of 59

Documents

Table of Contents€¦ · Web viewDocument # Date Effective Status