DM INTEGRATION & COMMISSIONING

1

Milestones• PDAC (Nov 2016)

• OCS/CCS/DAQ/TCS/DM Start/End (Dec 2016)

• OCS/CCS/DAQ/TCS/DM Single Visit & Mini-Night (Jul 2017)

• OCS/CCS/DAQ/TCS/DM 24 Hours (Oct 2017)

• Aux Telescope On Sky (Aug 2018)

• Software Release 8.1 (Nov 2018)

• Base Center Integration (Mar 2019)

• ComCam On Summit (Apr 2019)

• 5% Archive Center/US DAC Integration (Sep 2019)

• ComCam On Sky (Nov 2019)

• Software Release 9.1 (Nov 2019)

• Main Camera On Summit (Apr 2020)

• DR1/2 Archive Center (and CC-IN2P3?) (Aug 2020)

• Science Verification (Mar 2021)

• DR1/2 Chilean and US DAC (May-Jul 2021)

• Operational Readiness Review (Aug 2021)

2

PDACNOV 2016

Available Components

• Qserv Database with shared scans, without data replication, resource management

• DAX Web Services with Butler back-end to images

• SUIT Server and web portal for SDSS Stripe 82 data

4

Initial Uses

• Observe scientist investigations and queries on scientifically-useful data

• Prove authentication and authorization systems, enable security auditing

• Locate and fix scalability bottlenecks

• Develop and practice deployment and upgrade procedures

• Evolve to add more data sets, features, and capabilities throughout Construction and Operations

5

OCS/CCS/DAQ/TCS/DMDEC 2016 - OCT 2017

Available Components

• Archiver device simulator and prototype

• Prompt Processing device simulator and prototype

• Telemetry Gateway simulator and prototype

• Calibration Processor simulator and prototype (2017)

7

Initial Uses

• Verify commanding protocols

• Test and evolve configuration protocols

• Verify pixel API, latency, and bandwidths with Camera DAQ Test Stand (2017)

• Develop and practice Service Transition (including deployment and upgrade) and Service Delivery (including user support) processes

8

DM RELEASES AND TESTING

DM Release Procedures• All "official" 6 month releases (will) undergo extensive testing, characterization, and

documentation: R8.0, R8.1, R9.0, R9.1, R10.0, R10.1.

• DM products (e.g. Science Pipelines, L1 control, qserv, SUIT) undergo continuous integration and moderate testing of all merges to the master branch. master is intended to be releasable at any time, although the feature set available may not be consistent or meaningful.

• Weekly releases (will) undergo more stringent testing.

• Bug fixes may be released as patches on an official release, as well as incorporated into master.

• Release processes are (being) automated to enable new distributions to be cut rapidly.

10

Service Deployment

• Service Transition processes will ensure that changes have addressed all areas of concern, including IT Operations, Security, Service Management, Facility Management, and Configuration Management, and that they have impact assessments, communication plans, and rollback plans.

• A Change Control Board will monitor and advise on all service changes.

11

Science Payload Deployment

• Payloads for controlled systems (e.g. Prompt Processing and Data Release Production) will undergo extensive testing prior to deployment, as they are changes to services.

• Payloads for OCS-controlled batch and other Batch Production services may be ad hoc.

12

Testing before System Integration

• All services and science pipelines are tested in integration environments (L1, Verification Cluster, PDAC) at NCSA on simulated and precursor data.

• Data Backbone and Batch Production Services tested with NCSA and CC-IN2P3 prior to Base Center Integration.

• During Operations, services and pipelines are tested on real data, including "parallel operation" of the L1 systems in the integration environment.

13

BASE CENTER INTEGRATIONMAR 2019

Hardware Components

• Commissioning Cluster

• Tape library system and Data Backbone

• All hardware installed and tested during May 2018-Mar 2019

• All network links operational at production speed

15

Software Release Plan Caveats

• All plans for future software releases are tentative at this point in the replanning.

• They are a "likely possible" estimate given current thinking but may change significantly before the end of the year.

• Requirements and prioritization would be much appreciated.

16

Service Components• Archiver service (Camera and Aux

Telescope Spectrograph)

• Prompt Processing service (in current baseline)

• Telemetry Gateway service

• OCS-Driven Batch service (Commissioning Cluster and NCSA verification cluster)

• Data Backbone service (with Observatory Operations endpoint at Base [or Summit])

• Evolved PDAC

17

Science Pipelines R8.1• Alert Production:

- Complete single frame processing

- Differencing

- Stack-integrated DayMOPS

• Data Release Production:

- Spectrophotometric pipeline for Aux Telescope spectrograph

- Prototype Calibration Products Production

- Prototype chromatic PSF across focal plane

18

Data Access R8.1• Qserv asynchronous queries

• Qserv non-partitioned tables, fault tolerance, result streaming, cost estimate

• Qserv data distribution and replication

• DRP to Qserv data loader

• Initial L1 database, provenance

• Deep drilling database

• Internal DRP database

19

QA Tools• NOTE: Release dates TBD

• SQuaSH QA harness

• Test execution harness

• Alert QA

• QA to Telescope Systems, Comfort Display, Science Pipelines, Workflow & Provenance

• Calibration QA

• Satellite Processing Comparison

• Data Release Characterization

20

SUIT R8.1

• Web portal and API for search/display images

• Visualize raft, CCD, amplifier images

• Interactive, user-driven commissioning data analysis

• Reference data catalogs (e.g. Gaia)

21

Initial Uses

• Develop and exercise US/Chile administration processes

• Test international network at full scale

22

AUX TELESCOPE ON SKYAUG 2018

First Time

• Real DAQ

• Real OCS devices

• Auxiliary TCS devices

24

Initial Uses

• Verify DAQ and OCS operations including calibration

• Verify spectrophotometry code

• Develop and exercise Service Transition and Service Delivery processes

25

COMCAM ON SUMMITAPR 2019

First Time

• Real TCS devices

27

Initial Uses

• Verify interfaces, including scripting

• Test camera calibration procedures and CPP

• Collect engineering data

• Collect calibration data

28

COMCAM ON SKYNOV 2019

First Time

• Real atmosphere, telescope optics, CCD effects

• Almost-real filters (Understood from test stands?)

• Almost-real PSF

• Real sky/transients/cadence (although some can be done through HSC)

30

Alert Production R9.1

• Ready for ComCam

• Spuriousness engine for difference sources

• Global photometric and astrometric calibration

• Complete alert distribution system

31

Data Release Production R9.1

• Ready for ComCam

• Advanced galaxy models

• Multi-coadd deblender

32

Data Access R9.1

• Database security

• Qserv L3 support

33

SUIT R9.1

• Visualize full focal plane

• Integration between portal, API, and workspace

• Advanced light curve visualization

• Alert subscription/notification

34

Initial Uses

• Test science algorithms at full depth

• Improve PSF modeling

• Develop and exercise data publication and release processes

35

MAIN CAMERA ON SUMMITAPR 2020

First Time

• Real Camera, corner rafts

• Real optics, filters, PSF

37

Initial Uses

• Take engineering and then science images

• Verify operation with real camera

• Verify science algorithms at full scale

• Improve PSF modeling on full focal plane

38

SCIENCE VERIFICATION STARTMAR 2021

Science Pipelines R10.1• Alert Production:

- Complete DayMOPS

• Data Release Production

- Complete MultiFit

- DRP + MOPS

40

SUIT R10.1

• Level 3 data support

• Scientific analysis tools

• Scalability

• User services

41

Required Uses• Verify DM and LSST requirements using real data

• Generate all-sky templates for difference imaging and photometric calibration

• Generate catalogs for bootstrapping

• Test sending of real alerts to external community

• Test DAC at full scale

• Exercise DAC Service Delivery processes

• Test EPO interfaces on real data at full scale

42

OTHER ISSUES

Performance Metrics

• KPMs are a subset of all metrics in LSR/OSS/DMSR that need to be verified, selected to indicate progress for DM management

• All required metrics will eventually be measured, but not at the same frequency

• Additional performance metrics will be added for internal purposes

44

Schedule Issues• Base Center integrated after Aux Telescope goes live

• 5% Archive Center integrated after ComCam arrives on Summit

• DR1/2 Archive Center integrated after Main Camera arrives on Summit

• Need to determine/confirm ramp-up of CC-IN2P3 capability

• Full DR1/2 DAC is not available during Main Camera Integration and most of Science Verification

• Full alert distribution/subscription capability not ready until Main Camera Integration

45