Upload
justina-osborne
View
218
Download
0
Embed Size (px)
Citation preview
Improved and automated QC process [calChecker] [HC monitor] [histo] April 27, 2009
Improved and automated QC process
Reinhard Hanuschik and the DPQC group
• fast data transfer: July 2008• full data stream VLT and VLTI (ex burst-mode data)• > 50% within 10 minutes
• new ways in data processing @ DPQC• impact on shared QC
• 2 main goals:• calibration completeness • calibration quality
Improved and automated QC process [calChecker] [HC monitor] [histo] April 27, 2009
Mission #1: calibration completeness (“save last night”)
• calibrations incomplete: science OBs to be repeated• focus on calibration plan:
– known, predictable, configured
– can be checked automatically
calChecker
1. Setting the stage
Improved and automated QC process [calChecker] [HC monitor] [histo] April 27, 2009
Mission #2: calibration product quality (“save next night”)
combining Health Check monitor and scoresscores:
quick-look HC monitor
Improved and automated QC process [calChecker] [HC monitor] [histo] April 27, 2009
data acq (Paranal)
data in Garching
opslog
• any problem not discovered on site: up to 10 days affected• strong need for 2 QC processes:
daytime astronomer Paranal (preliminary),QC scientist Garching (final)
• basic real-time QC with opslog data
(after ~ 10 days)
until 2008:
Improved and automated QC process [calChecker] [HC monitor] [histo] April 27, 2009
data acq (Paranal)
data in Garching
now (2009):• fast data delivery
• fast QC feedback
• improved QC feedback– measure quality (QC1 parameters) – compare quality (trending)– rate quality (scoringscoring) new!
Improved and automated QC process [calChecker] [HC monitor] [histo] April 27, 2009
• www.eso.org/qc/ALL/daily_cal.html• evaluation for 7 last days (columns) and all science setups (rows)• driven by science data• checks:
– all science setups for a given instrument
– checks calibration plan
– knows about validity and required number of files
life examples
2. calChecker
Improved and automated QC process [calChecker] [HC monitor] [histo] April 27, 2009
• evaluates each box in colors:– green (OK): all calibrations complete and within validity
(e.g.: 5 BIAS & 3 FLATs & 1 ARC & 1 STD within validity)
– yellow (NOK): calibrations complete but some outdated(e.g.: as above but 3 FLATs are 2 days old)
– red (MISS): calibrations incomplete, at least one missing(e.g.: 5 BIAS & 3 FLATs ok, 1 ARC & 1 STD missing)
• missing or outdated calibrations are indicated
Improved and automated QC process [calChecker] [HC monitor] [histo] April 27, 2009
• links to details– for each box: complete list of science files
– prog_id, obs_id etc. for easy reference
– OB grade and comment: useful for analysis
Improved and automated QC process [calChecker] [HC monitor] [histo] April 27, 2009
• ANALYSIS– QC scientist to provide analysis/comments,
but this requires interaction ( not always available!)
– may override tool NOK/MISS results
• BOCAL, EOCAL– signal about daytime calib queue
– calChecker displays “pending”, “ongoing”, “finished” for current date
Improved and automated QC process [calChecker] [HC monitor] [histo] April 27, 2009
• called: once per hour by cronjob– has update information, knows about its age
don’t hesitate to contact QC!
Improved and automated QC process [calChecker] [HC monitor] [histo] April 27, 2009
• documentation• association rules• contact
• planned: – include ticket mechanism (PSO scheme)
– memory function for long-term calibrations (once a month etc.) and HC calibrations
Improved and automated QC process [calChecker] [HC monitor] [histo] April 27, 2009
• general rule: calChecker follows calOBbuilder
• not planned:– short-term night calibrations (like standard stars within 2 hrs)
– complex dependencies (multiple standard stars a night; alternative calibrations; calibrations for calibrations)
• to be done on Paranal:– optimize calOBbuilder queue (input provided by QC)
Improved and automated QC process [calChecker] [HC monitor] [histo] April 27, 2009
• goes beyond completeness, includes quality• links to the QC processing page for that date (AB product monitorAB product monitor)
3. Links to product quality
calChecker HC monitor
products/AB monitor
Improved and automated QC process [calChecker] [HC monitor] [histo] April 27, 2009
AB product monitor
Improved and automated QC process [calChecker] [HC monitor] [histo] April 27, 2009
Scores
automatic evaluation scheme:
• 2 thresholds defined
• QC1 parameter evaluated
• total AB score: 0 / 1 / more than 1
• counts outliers
Improved and automated QC process [calChecker] [HC monitor] [histo] April 27, 2009
• links to other reports– data report
– nightlog (password protected, sciops)
Improved and automated QC process [calChecker] [HC monitor] [histo] April 27, 2009
• www.eso.org/HC
• final QC Garching data now display as quick as/quicker than opslog!(forget 10 day gap…)
4. Quick-look HC monitor
calChecker HC monitor
products/AB monitor
Improved and automated QC process [calChecker] [HC monitor] [histo] April 27, 2009
full version
score version
(Quick-look)
Improved and automated QC process [calChecker] [HC monitor] [histo] April 27, 2009
• Information on demand:
click on “more” if you want graphs, on “less” if you want scores only
Improved and automated QC process [calChecker] [HC monitor] [histo] April 27, 2009
• HC monitor: selected scores • selection: “relevant for instrument health”
• AB product monitor: all scores of a date
Improved and automated QC process [calChecker] [HC monitor] [histo] April 27, 2009
score propagation:– data scores for last week only
– plot: last data scores summed upplot score green if last data score is green (previous red scores ignored)
• report score: plot scores are aggregated in report score• report scores in group scores, and finally in instrument score
• all higher-level scores turn red if at least one lower-level score is red
Improved and automated QC process [calChecker] [HC monitor] [histo] April 27, 2009
• big advantage of score hierarchies:– instrument score green you know within a secondwithin a second: all is OK!
– red: the score overview can tell you more
• scheme intended to guide you to issues
Improved and automated QC process [calChecker] [HC monitor] [histo] April 27, 2009
• Health Check monitor– update pattern: refresh most important ones upon new opslog data
– less important ones at least once per 24 hour
• process health– monitored: opslog transfer, parser (1x per hour)
– coming soon: data transfer, ngas access
Improved and automated QC process [calChecker] [HC monitor] [histo] April 27, 2009
• planned: – defocus the OPSLOG data
(to become a fallback solution except for VIRCAM/OCAM)
– improve update pattern for HC reports (immediate after new data)
– product quality links (same as for calChecker)
– agree on mail alert mechanism
Improved and automated QC process [calChecker] [HC monitor] [histo] April 27, 2009
calChecker HC monitor
products/AB monitor
Improved and automated QC process [calChecker] [HC monitor] [histo] April 27, 2009
• the end …
Improved and automated QC process [calChecker] [HC monitor] [histo] April 27, 2009
Other QC services within this context
• Agora: FAQ page for PSO-QC related issues• refs: reference frames web page• QC: tutorial pages, documentation
Improved and automated QC process [calChecker] [HC monitor] [histo] April 27, 2009
• avoid being under-sensitive:– overlook that raw file is saturated or empty
– miss important properties = QC1 parameters
• avoid being too sensitive:– have too narrow thresholds
– have unstable thresholds (3SIG)
– use not fully understood parameters
– select data sets (setups) rarely acquired ( trending impossible or uncertain)
– use same QC1 parameter in various HC plots ( multiple alarms)
• rule of thumb: – every red score should have an explanation
– likely trigger an action
[Excursion: The art of scoring … ]
[E. Munch, Cry]
Improved and automated QC process [calChecker] [HC monitor] [histo] April 27, 2009
5. Operational impact
daytime astronomer: check completeness– take action for ylw/red boxes:
e.g. trigger calibrations
check Health Check monitor– check scores– pick red ones– analyze problem– trigger actions
check product quality (AB monitor)