ISOC Status Review

GLAST LAT Project ISOC

1

ISOC Status ReviewISOC Status Review

June 3, 2004


2

OverviewOverview

– We have developed an organization and staffing plan in concert with the SLAC management.

• ISOC buildup started, rapid ramp up over next year– We have completed initial work on an operations

architecture.– We have made good progress in addressing peer review

RFAs– Substantial work remains before CDR but we believe we

now understand the scope and will be ready by 7/15.


3

LAT/ISOC Organization Post LaunchLAT/ISOC Organization Post Launch

NASAGLAST Project

PI: P. MichelsonInst Sci: S.Ritz

Instrument Ops Advisory Board

H/W subsystem leads, key Technical Advisors from throughout collaboration

ISOC ManagerW. Craig (Acting)

4.1

4.1.B

Science Analysis Center4.1.B.4

LAT Ops Facility (LOF)4.1.B.1

Sci. Ops Group (SOG)4.1.B.2

Sci. Analysis SW (SAS)R. Dubois

4.1.B.3

Software

Operations

Calib

Optimization

Flt S/w & Testbed

Pipeline Config.

Pipeline

Anayl Tools

Collab

Computing

Resources

Only

SSAC

Science AnalysisCoordination

CommitteeSAC head, Analysis

leads, ISOC rep, SAS head


4

StaffingStaffing

• Rob Cameron has accepted the ISOC manager position, so there will (finally) be a permanent ISOC manager in place in August.

• Craig will be responsible for a successful CDR and will keep Cameron updated throughout. – Several month transition period planned

• Steve Culp has accepted S/W developer position and will start within a week. He will be responsible for fleshing out the architecture and first database implementations.


5

Staffing ProfilesStaffing Profiles

0

5

10

15

20

25

1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55

SOG

LOF

Manage

CY 2005 CY 2006 CY 2007 CY 2008CY 2004

Excludes SAS and SAC.


6

Staffing Profiles (with SAS/SAC)Staffing Profiles (with SAS/SAC)

0

5

10

15

20

25

30

35

40

45

1 5 9 13 17 21 25 29 33 37 41 45 49 53

SAC

SAS

SOG

LOF

Manage

CY 2005 CY 2006 CY 2007 CY 2008CY 2004

Does not include Stanford, UCSC, NRL, GSFC or collaboration members.


7

ArchitectureArchitecture

• Drivers– Minimize V&V burden and total cost– Maintain all science capabilities– Simplify interfaces and allow early testing

• Recognized that neither of the previously considered options were particularly attractive– ITOS/Commercial packages don’t accommodate

complexities of science data– Homegrown system doesn’t have heritage, not ready in

time to make project timelines. • Most of additional code needed duplicates that in

existing packages Studied hybrid solutions


8

ITOS/Astro RT TradeITOS/Astro RT Trade

• In favor of either– Both AstroRT and ITOS would provide basic instrument

health and safety functions• Telemetry display• EU conversion• Limit checking and monitoring• Trending• Command and telemetry database access

– Both products have learnable interfaces and scripting• AstroRT uses LabView for display and Perl scripts for

automation• ITOS displays are reportedly easy to create, uses STOL

for input


9


• Against either– Requires use of ITOS or Astro-RT specific interfaces and

scripting– Both have ITAR issues– Limitations are not fully understood

• Believe limitations will not affect monitoring and trending of housekeeping data – only science and instrument diagnostics


10


• In favor of AstroRT– LAT is using AstroRT for LAT flight software testing

• Against AstroRT– Does not handle character strings – not sure if that’s an

issue for us (it is with GBM)– Commercial product costing $$$ upfront and for support

throughout program life– Probably unable to alter AstroRT code

• In favor of ITOS– MOC and GBM will be using ITOS– May be able to alter ITOS code or have changes made

• Against ITOS– None that don’t also exist for AstroRT


11

Proposed ISOC S/W ArchitectureProposed ISOC S/W Architecture

ITOS

Cmd DB

SOH trending and display

All State of Health requirements

satisfied within ITOS

MOC/GSSC

LATTE Ops LATOPS

DataCmd

Science data/performance trending

Register load generation

Relational database interaction

Pipeline/SAS interactions

No req’ts on MOC that require

LATOPS layer


12

RFA responsesRFA responses

# RFA Summary Requestor Actionee Comment

1 a. Need ISOC Management Plan & Approachb. ISOC Documentation Set

R. Schweiss W. Craig Plan draft and list of ISOC documents on http://www-glast.slac.stanford.edu/ioc/

2 Need overall functional block diagram illustrating the functional capabilities and data flow during various phases

R. Schweiss L. Bator Draft response – slides attached

3 Risk Analysis R. Schweiss W. Craig Draft response – slides attached

4 Reschedule ISOC CDR M. RackleyC. Young

D. Lung Done. CDR scheduled for 8/4/04

5 Incomplete Level III requirements for LOF and SOG

M. Rackley L. Bator Drafts on http://www-glast.slac.stanford.edu/ioc/

6 Staffing plan and profile M. RackleyC. Young

W. CraigD. Lung

Staffing plan and profile presented, RFA response pending

7 Define the ISOC reports for internal use and external use

M. Rackley L. Bator Response complete – slides attached

8 The ISOC does not yet know what system it is using to process Observatory HSK data or perform the commanding

M. Rackley L. Bator Architecture presented, RFA response pending

9 Describe lesson learned & approach M. Rackley W. Craig Response complete – slides attached


13

RFA responses, cont’dRFA responses, cont’d

# RFA Summary Requestor Actionee Comment

10 ISOC verification does not involve early opportunities to validate/test using LAT instrument

M. Rackley

N. Johnson

L. Bator See RFA #2, also pending architecture approval

11 Verify LAT modes M. Davis L. Bator Draft response – slides attached

12 Understand the number of writes to EEPROM C. Young L. Bator Response submitted

13 ISOC detailed development schedule K. Lehtonen D. Lung Pending architecture approval

14 Enter a more formal agreement with SLAC management on required data storage and processing requirements

N. Johnson W. Craig Response completed – slides attached

15 ISOC organization structure & communications N. Johnson W. CraigD. Lung

Organization presented, RFA response pending

16 Define mechanism for ISOC requirements being placed on I&T and SAS

N. Johnson W. Craig Pending architecture approval

17 Define LOF/SOG tools R. Corbet L. BatorJ. Panetta

Draft response – slides attached

18 Specify plans and requirements for automation of Ops software

R. CorbetM. Rackley

L. BatorJ. Panetta

Draft response for 1st part, awaiting S. Culp for 2nd

19 Specify plans and requirements for Ops SW to be of sufficient robustness

R. Corbet L. BatorJ. Panetta

ECD 6/15/04 – S. Culp

20 Specify what other ground system elements will be involved in LAT operations

R. Corbet L. BatorD. Lung

ECD 7/5/04 - working group on contingency plans


14

RFA 2 – ISOC Functional Block DiagramRFA 2 – ISOC Functional Block Diagram

• RFA 2 Specific Request:– Need an overall functional block diagram illustrating the

functional capabilities and a data flow diagram showing the various data flows, with the differences among the I&T (pre-launch w/GSE) phase, L&EO phase, and nominal on-orbit phase configurations specified

– Diagrams for each phase might be needed


15

ISOC Dataflow During I&T Single Tower TestingISOC Dataflow During I&T Single Tower Testing

• Obtain data during I&T EM2 testing• Goal is to read houskeeping data off flat file produced by Online• Database development and maintenance is shared between I&T and

ISOC

ISOC

LOF Workstation· Telemetry

AEM/TEM

VME Chassis

-

• External Trigger

Electronics/OnlineSubsystem/simulator

Subsystem (or simulator)

Ethernet Routerand Firewall

Ethernet

Main Dataflow Ethernet

Back door debug RS-232

WA

N

Online Workstation

•Test Exec• Configuration• Commanding•Telemetry• Event Processing• Monitoring• Visualization• Data base• Message logger• Alerts• Electronic log

CPU

• Commanding

• Telemetry

• Monitoring

• Hardware Drivers

DC Power

External Hardware

I&T

CentralDatabase


16

ISOC Dataflow During I&T Multi-Tower TestingISOC Dataflow During I&T Multi-Tower Testing

• Obtain data during I&T testing

• Increase in ISOC functionality

I&T

T&DF System (LAT) EGSE

AEM

CPU

LCB

EGSE Crate

WA

N

GASU

CAL, TKRor

SensorSimulator

OnlineWorkstationTEMs

GEM

EBM

Fire

wal

l

28 V Supply

VME

ACDor

SensorSimulator

ISOC

LOF Workstation· Telemetry· Database· Log book

SOG Workstation· Visualization tools

CentralDatabase

SLAC


17

ISOC Dataflow with TestBed - Direct to SIUISOC Dataflow with TestBed - Direct to SIU

• Direct interface with SIU for CCSDS command and telemetry packets• Obtain testbed simulated data via SIU• Demonstration of ISOC capability increases as functionality is

developed

SIU

Testbed

CCSDS Telemetry Packets

ISOC

LOF Workstation· Telemetry display· Database mgmt· Log book· Command generation· Trending and analysis

SOG Workstation· Visualization tools· Calibration planning· Trending and analysis

Central Database

OPUS

WA

N

Instrument filesCCSDS Command Packets


18

ISOC Dataflow with TestBed - With SIISISOC Dataflow with TestBed - With SIIS

• Interface with SIIS/AstroRT for telemetry packets and commanding• Obtain testbed simulated data via SIU and SIIS• Demonstration of ISOC capability increases as functionality is

developed

SIIS

SIU

Testbed

PC Running AstroRT

High speed science

1553 HK and diagnostic

Instrument filesCommand requests

Telemetry packetsCommands

Telemetry packets

ISOC



Central Database

OPUS

WA

N


19

ISOC Dataflow During GRTs, L&EO and On-orbitISOC Dataflow During GRTs, L&EO and On-orbit

Internet

MOC

ITOS Workstation

GSSC

ISOC



Central Database

OPUS

WA

N

Level 1 data productsInstrument filesCommand requests

Level 0 dataOps products

Ops products

• Shows full ISOC capability for L&EO and On-orbit

• GRTs will test capabilities as they are available


20

RFA 3 - ISOC Risk AnalysisRFA 3 - ISOC Risk Analysis

• Process– Discussion with I&T personnel on risks– Internal discussion performed in concert with RFA’s from

peer review– Review and approval by ISOC stakeholders

• Follow-up– Entry into LAT risk management database by 06/01/04– Weekly tracking, updating by ISOC management


21

RFA 3 – ISOC Risk AnalysisRFA 3 – ISOC Risk Analysis

Number Date Rank Originator Description Mitigation

ISOC-0001 5/15/04 1 B. Craig ISOC lacks accepted architecture and plan for software implementation.

Trade study between possible front ends to be completed by 6/15/04. Hires into s/w architecture position.

ISOC-0002 5/15/04 3 B. Craig No response to PDR RFAs

Schedule and track RFA’s weekly.

ISOC-0003 5/17/04 2 B. Craig Inadequate staffing plan for ISOC.

Draft staffing plan in progress, to be released by 06/01 First req issued an offer out to highest priority position.

ISOC-0004 5/21/04 4 B. Craig No facility location identified for ISOC

Long-term solution identified, short term space to be requested from SLAC management.


22

RFA 3 – ISOC Risk AnalysisRFA 3 – ISOC Risk Analysis

Number Date Rank Originator Description Mitigation

ISOC-0005 5/21/04 2 B. Craig No requirements levied on I&T and Flt S/W subsystems

Mechanism in place with I&T, pending with Flt S/W. Implement these only after architecture is defined and accepted.

ISOC-0006 5/21/04 1 B. Craig ISOC will be unable to hold schedule due to staffing delays and unscoped work

Definition of work plan follows architecture development. If needed additional support will be requested from LAT management.


23

RFA 7 – ISOC ReportsRFA 7 – ISOC Reports

• Specific Request

• Define and document the types of reports that will be

generated by the ISOC for both internal use and for use by

external systems (like the MOC and GSSC)

• Response

• Reports will be documented in the Operations Product ICD

(external reports) and LAT Ops Plan (internal-only reports)


24


• LAT status and planning

• Reported daily (TBR)

• Summary of LAT health status

• Limit violations

• Alerts received

• Current LAT configuration

• Commanding and any other special activities that occurred

• Mission planning outlook for near term (time period TBD)

• Generated by LOF with automatic and manual inputs

• Published to web server


25


• LAT performance

• Reported daily (TBR)

• Quick look science data

• Performance metrics (details TBD)

• Generated by SOG

• Published to web server

• Level 0 data transmission report

• Data transmission metrics (details TBD)

• Automatically generated and sent to MOC following receipt

of Level 0 data


26


• Data Trending

• Housekeeping data

• Environmental data (temp, voltages, currents)

• Derived science quantities

• Trigger efficiency

• Total count rate

• Bright source monitoring

• Includes statistical analysis

• Generated automatically daily/weekly/monthly

• Published to the web


27

RFA 9 - ISOC Lessons LearnedRFA 9 - ISOC Lessons Learned

• Issue– No writeup on lessons learned from visits to other

instrument/mission operations center• Resolution

– Members of the ad hoc planning group for the definition of the LAT IOC (now ISOC) made visits to the operations centers for GP- B (launched April, 2004; Stanford Univ., Tom Langenstein & Brett Stroozas), RHESSI (launched 2002; Berkeley Space Sciences Lab., David Smith & Manfred Bester), and Chandra (launched in 1999; MIT & Harvard-Smithsonian Center for Astrophysics, Dan Schwartz & Paul Plucinsky)

– Each of these operations centers integrates mission operations with science (instrument) operations, and so they are not directly comparable to the ISOC in terms of complexity or staffing. (The operations center for RHESSI includes the ground station.)

• LAT ISOC can learn from others but there are no direct models.


28

RFA 9 - Lessons LearnedRFA 9 - Lessons Learned

– The science operations center for GP-B is co-located with the science team at Stanford. The GP-B data also will be distributed widely to collaborating institutions, but the co-location at Stanford was deliberate to maximize the interaction with the SOC on data issues.

• Colocation important to maximize science.– The staffing for RHESSI operations is especially spare. The

facility itself is also used to run operations for FAST and CHIPS and the routine operations, like scheduling of contacts and pipeline processing, are automated. Testbeds (simulators for the instrument computers) are maintained, and have been found vital for understanding anomalies as well as for testing flight software updates.

• Testbeds important for flight software updates.


29

RFA 9 - Lessons LearnedRFA 9 - Lessons Learned

– The Chandra Operations Control Center has a room with about 4 consoles for the ACIS instrument team to monitor and command the instruments. The ACIS team has developed an impressive, flexible facility for trend analysis. The importance of a flexible system that does not require deciding in advance what needs to be monitored routinely was stressed to us. The ground-based calibration data are still actively used, >4 years into the mission. Colocation of the operations (mission and instrument) and the ACIS instrument team has been important, at least in terms of increased efficiency. Instrument team members (like the PI) at Penn State can feel out of the loop or behind the times.

• Colocation important to keep all science members in the loop.


30

RFA 11 – LAT ModesRFA 11 – LAT Modes

• Specific Request:– LAT Operations Team and Spectrum Astro should work

together to verify if any interactions between LAT modes and spacecraft modes need to occur. For example, if a LAT mode change requires the spacecraft to change spacecraft mode and/or configuration

• Response:– SC modes are understood and accommodate the LAT

modes as designed


31

RFA 11 – LAT Modes, cont’dRFA 11 – LAT Modes, cont’d

Mission Modes SC Mode LAT Mode

Launch· S-Band rcvr/xmit· On battery power

Off

Early Orbit· Inertial capture· S-Band rcvr/xmit· Sun point with solar arrays tracking

Survival

Engineering

· Inertial point, zenith point, or maneuver

· Ku-Band xmit, S-Band rcvr/xmit· Solar arrays tracking

Engineering

Calibration

SAA

Sky Survey· Zenith point· Ku-Band xmit, S-Band rcvr/xmit· Solar arrays tracking

Science Mode

Pointed and Repointed

· Inertial point, maneuver· Ku-Band xmit, S-Band rcvr/xmit· Solar arrays tracking

Science Mode

Safemode· Inertial capture, sun point· S-Band rcvr/xmit· Solar arrays fixed

Hardware

Survival

Re-Entry· Cruise, delta-V· S-Band rcvr/xmit· Solar arrays tracking

Off


32

LAT Modes, cont’dLAT Modes, cont’d

Pointed &Repoint

Sky Survey

Observing Modes

Engineering

SAA

Standby Modes

Calibration

Survival

Hardware

Launch & Early

A = Autonomous

C = Commanded

C

C

C

C

Safing

Launch

Pointed and Repoint

Sky Survey

EngSAA

Standby ModesCal

Survival

Hardware Safe

Launch & Early Orbit

C

C

C

C

C

Safing

Launch

Observing Modes

A,C

A,C

A,CA,C

A,CC C

Valid science data

Science data may not be valid

No science data


33

RFA-12: Number of EEPROM WritesRFA-12: Number of EEPROM Writes

• Specific Request– Understand the number of writes to EEPROM on LAT from

all sources• Reason

– EEPROMs have a limited number of write cycles before they become unreliable

• Response– Not an issue due to use of TrueFlash File System overlay

(full description is on RFA response, available on ISOC web page)


34

RFA 14 - ISOC Data StorageRFA 14 - ISOC Data Storage

• Issue– No agreement with SLAC management on how data storage

and processing requirements will be funded.• Resolution

– Estimate of processing and data storage requirements performed for SAS by R. DuBois. Cost determined and built into ISOC outyear funding plan and accepted by SLAC Director of Research

– Database costs still being evaluated by database working group but now expected to be minimal or covered completely by SLAC central computing services due to small size (~ 1Tb) of database.


35

Monthly Compute Costs

0.0

20.0

40.0

60.0

80.0

100.0

120.0

May Nov May Nov May Nov May Nov May

k$

Processing

Storage Costs

RFA 14 - Monthly CostsRFA 14 - Monthly Costs

2005 2006 2007 2008


36

RFA 17 – Define LOF/SOG ToolsRFA 17 – Define LOF/SOG Tools

• Specific Request:– The tools needed to run the LOF/SOG need to be specified

• Which HK and science parameters will be monitored and in what way?

• What actions would be taken based on the results seen with these tools?

– How does the ISOC team know from a design perspective that the collection of the described I&T tools will function in the operations environment as an integrated system?

• Reason/Comment:– The overall requirements on the ISOC have been given– Detailed plans for which software components/libraries

such as Python will be used were given– However, lists of which software tools are required to

achieve the ISOC’s requirements are needed


37

RFA 17 - ResponseRFA 17 - Response

• Which HK and science parameters will be monitored and in what way?– HK parameters are defined in LAT-TD-02905– Routinely monitored science parameters are included

within the HK data as Low Rate Science• Use of high rate science data is being developed by

SVAC and will be further developed by SOG– Limits and use of HK data for monitoring are TBD

• What actions would be taken based on the results seen with these tools?– Calibration activities are in development in the SVAC– Contingency actions are TBD


38

RFA 17 - Response, cont’dRFA 17 - Response, cont’d

• How does the ISOC team know from a design perspective that the collection of the described I&T tools will function in the operations environment as an integrated system?– Development and testing of ISOC tools is in conjunction

with I&T• Lists of which software tools are required to achieve the

ISOC’s requirements are needed– The following slides detail the ISOC software tools


39

RFA 17 - ISOC Software ToolsRFA 17 - ISOC Software Tools

ISOC Tools

OP

US

ITO

S

Ex

ists

with

L

AT

TE

Ex

istin

g O

the

r

To

be

Writte

n

1 Data Transport and Management

1.1 File retrieval, transmission, and management (internet) X SLAC SCS and Fastcopy

1.2 Archive data files X SLAC SCS

1.3 Parse data into database X

1.3.1 Convert housekeeping data into Engineering Units X

1.4 Data integrity checks X

1.5 Science data reconstruction X

1.6 Calibration tracking X


40

RFA 17 - ISOC Software Tools, cont’dRFA 17 - ISOC Software Tools, cont’d

ISOC Tools

OP

US

ITO

S

Ex

ists

with

L

AT

TE

Ex

istin

g O

the

r

To

be

Writte

n

2 Operations Tools

2.1 Electronic logbook X

2.1.1 Reporting X X

2.1.2 Command history X X

2.2 Database management

2.2.1 Command and telemetry X

2.2.2 Science and calibration X

2.3 Archive management SLAC SCS


41


ISOC Tools

OP

US

ITO

S

Ex

ists

with

L

AT

TE

Ex

istin

g O

the

r

To

be

Writte

n

3 Instrument Health (LOF)

3.1 Real time housekeeping telemetry display X

3.2 Historical data trending display X X

3.3 Data monitoring and alarming systems X X

3.3.1 Autonomous reporting X

4 Instrument Diagnostic Tools

4.1 Diagnostic data display and analysis X

4.2 Memory dump parsing FSW

4.3 Testbed management and operation Elec


42


ISOC Tools

OP

US

ITO

S

Ex

ists

with

LA

TT

E

Ex

istin

g O

the

r

To

be

Writte

n

5 Instrument Performance (SOG)

5.1 Visualization tools X X

5.2 Offline calibration SVAC tools

X

5.3 Online calibration X


43


ISOC Tools

OP

US

ITO

S

Ex

ists

with

L

AT

TE

Ex

istin

g O

the

r

To

be

Writte

n

6 LAT Commanding Tools

6.1 Command procedure generation and management X

6.1.1 Instrument file generation FSW

6.1.1.1 File management X

6.1.1.2 File validation and verification X

6.1.1.3 File translation to ITOS Perl script

6.1.2 Telecommand generation X X

6.2 Procedure verification and validation on testbed X

6.3 Procedure transmission tools X

6.3.1 Command wrapper generation (for GSSC) X

6.3.2 Command load transfer to GSSC Fastcopy


44

RFA-18: ISOC Operations automationRFA-18: ISOC Operations automation

• Specific Request– Specify plans and requirements for automation of

operations software– Describe the software design for how the automation needs

will be met• Response

– Draft of the plans and requirements has been completed– Software design will commence when ISOC software

engineer is hired


45

RFA-18: ISOC Operations automationRFA-18: ISOC Operations automation

• Data retrieval from MOC• OPUS:

– Archiving raw data– Dispatch science data to SOG– Dispatch housekeeping to LOF

• LOF automated processing – Housekeeping: limit checks, warnings– Science data: raw data quality– Automated reporting of above (web/paging/email)

• Trending:– Weekly/monthly characterization of data

• Calibration tracking & computation• External agency alert retrieval (i.e., SEC, NIST)


46

Roadmap to CDRRoadmap to CDR

• Primary tasks– 1) Scenario definition

• Work with FSW and I&T for all operational modes (BC, LB, SC) July 1

• Detailed early orbit plans (BC,LB) July 15– 2) Contingency operations analysis

• Define possible actions by subsystem (BC,LB) July 7– 3) Draft Instrument Ops Section of Mission Plan (LB, SC,

who at GSFC?) July 15– 4) Update requirements documents to reflect architecture

(SC, LB) July 15


47

CDR Prep ScheduleCDR Prep Schedule

• July 8th Revisit roadmap• July 21st Laydown• July 26th Slides to GSFC• July 29th Dry Run • August 4th ISOC Peer Level CDR• August 18th CDR

Documents

ISOC Status Review