1

INTEGRAL Mission Control System (IMCS): an example of flexible integrated solutions supporting a complex and evolving Ground Segment

Gianpiero Di Girolamo

INTEGRAL MCS Project Manager E-mail: gianpiero.di.girolamo@esa.int

Tel +49-6151-902241 Fax +49-6151-903010

1 Abstract The INTEGRAL mission is aimed to the fine spectroscopy and fine imaging of celestial gamma-ray sources with concurrent source monitoring in the X-ray and optical energy ranges. The mission will be controlled from the INTEGRAL Mission Operation Centre (MOC) at ESOC in Darmstadt, based on scientific utilisation planning inputs provided by the INTEGRAL Science Operation Centre (ISOC) located at ESTEC (Nordwijk-NL). The satellite science data will be processed at INTEGRAL Science Data Centre (ISDC) in Versoix (Geneve-CH). This paper describes the IMCS interfaces in order to provide readers with an understanding of how the system is able to support a very complex Ground Segment. IMCS main functionality is to provide an interface to the monitoring, control and planning of the INTEGRAL on-board systems. Besides these basic functionality, IMCS provides a set of additional components which allows operational users to: • Telemetry and telecommand context configuration • Remote access via Web to the IMCS functionalities • Remote command capability • provide users with different views to the mission historical data in order to perform off-line analysis Integrated with the basic applications and services, other components have been developed using different technologies which provide users with their proper solution with the cost-effective. Among these components are: • off-line database system based on Microsoft Access database and editors • remote telemetry/telecommand monitoring and historical data retrieval using CORBA based

application • off-line performance evaluation system based retrieval data request coming from remote site and

served by a web-based application • remote commanding request from Science Ground Segment • infrastructure devoted to the auxiliary data exchange • real-time and off-line data interface with the Science Ground Segment (ISDC) • real-time data interface with Principal Investigators Clients at MOC during LEOP The result is a system fulfilling the user mission requirements providing a wide range of flexibility to local and remote users during operations. It will be shown ho the flexibility of the baseline design has allowed to accommodate additional system specification added on in the course of the Ground Segment Integration and Testing.

2

The article will complement the description of the interfaces with the Mission Control System operational scenario during the most critical routine phase (LEOP) in particular detailing the IMCS redundancy concept defined and proved during the Simulation Campaign. . Keywords: Mission Control System, interfaces, remote commanding, redundancy concept, failure recovery

2 IMCS Functional Architecture and External Interfaces Figure 1 presents the IMCS functional architecture together with its external interfaces. The core of the system is the Telemetry and Telecommand (TM/TC) processing based on the INTEGRAL operational database. Nominally all telecommands uplinked during a revolution (defined as the orbit period between two consecutive passages of the satellite through the perigee), are pre-planned in a timeline through the mission planning process involving the Mission Planning System (MPS) and the external interfaces with ISOC and with the Flight Dynamic System (FDS). Other sources of telecommands are: • The On-Board Software Maintenance (OBSM) subsystem which receives files containing on-board

images from the external interface with the OBSMS (On-Board Software Maintenance System) and converts such images into a set of telecommands to be transmitted to the satellite

• The FDS providing telecommands for specific orbit maintenance or anomaly recovery operations • The ISDC which, following the detection of a Gamma Ray Burst (GRB) in the telemetry of the on-

board imager, generates and transmit in near real-time a specific telecommand (OMC-TC) request to reconfigure the on-board optical camera for enhancement of the scientific observation of the phenomena.

• The manual stack provided to the operator to intervene in case of non-nominal operations All telecommands transmitted through the Network Controller Transport and Routing System (NCTRS) to the ground stations are archived together with their specific verification information (whether passed or failed) in the Mission Archive – Data Distribution System (MA-DDS). On the telemetry side, the raw telemetry received from the satellite via the ground station and the NCTRS, is provided to the TM/TC processing applications and in parallel to FDS and to MA-DDS which provides archiving and distribution to the external users as required. The different users are: • The ISDC receiving telemetry in near real-time together with the FDS Snapshot Attitude Information

for GRB detection and localization and for monitoring the on-board instruments performance via quick-look analysis of the received data

• The Principal Investigators I workstations installed in MOC during the commission phase of the mission for monitoring of the on-board instrument during the initial switch-on and electrical check-out operations.

In the MA-DDS also archiving of the IMCS processed telemetry and FDS generated orbit and attitude data is provided. From the MA-DDS periodical extraction of auxiliary files (Observation Log File, Predicted Attitude File, Orbit Files, Attitude History) is performed. Such files are transmitted to ISDC via the INTEGRAL File Transfer System (IFTS) which consist of an FTP based protocol providing automatic file transport mechanism. Data archived in MA-DDS covering predefined time intervals is regularly copied into CD-ROMs which are then provided to ISDC for off-line processing of the science telemetry and derivation of mission products. For the purpose of monitoring the proper execution of the pre-planned observation schedule, off-line access to telemetry history from MA-DDS is provided to ISOC. All data transmitted and received to/from users external to MOC (SGS and PI workstations) is exchanged

3

through a server outside the MOC firewall. The server and the firewall isolate the MOC preventing external users to access directly to the IMCS Operational Local Area Network (OPSLAN). The server is configured with a telemetry server application and IFTS instances. The telemetry server application is in charge of receiving telemetry from IMCS and distributing it to ISDC and PI workstation, while the IFTS instances are used to route files between the IMCS system and the external users.

NCTRS

IFTS ODMPS

IFTS

ISOC

FDS

IFTS

TCTM

SCOS-2000 TM, TC & EV

TM

TPF

OBSMS

Image Files

PI Workstation

Off-line TM History

IMCS

PSF POS ICP

TM

EPOS/APF

Timeline MPS

IFTS

TM/TC

MA-DDS

IFTS

ISDC

TM

CD ROM

User Access And performace Analysis System

GS Redu

GS Goldstone Simulator

AuxiliaryFiles

OMCTC

Ranging data

TPF

PAS-TDRS

History data

IMCS External I/F IMCS Internal I/F

OBSM

ASE (Option)

WEB-RM (Option)

TM ((NCTRS)

Figure 1 - IMCS Functional Architecture

3 IMCS Software configuration. In this section IMCS software components will be classified in terms of their RT criticality. We will identify here a set of IMCS functionality providing the basic and essential function to guarantee S/C control and Safety. These functions are defined as Kernel functions and comprise NCTRS TM/TC Interface, VC0 Telemetry Distribution, VC0 Telemetry Archiving (Short Term), VC0 Telemetry Retrieval, TM Spacon, Manual TC preparation, Manual Commanding, Automated Commanding, TC Spacon, IMCS Internal File Transfer System. Kernel functions above are those with most demanding availability requirement and with minimum turn around time in case of failures or malfunctions since used in most of the RT critical operations.

4

Moreover, and this is the crucial difference with respect to the second group identified below, this applications are required to run in a precise configuration which is dynamically evolving as long as the operations go on. In other words kernel applications when interrupted for any reason, in most of the cases require to be resumed from the precise operational contest valid at the time they have been interrupted. IMCS kernel functionality above are all hosted onto so called Main Server. A second group of functions, which can be named Auxiliary, comprises those subsystem used to carry off-line tasks (IOBSM, IMPS) or non RT critical task (Long Term archiving) and/or tasks not essential for the direct monitoring and control of the spacecraft. Within IMCS following tasks belongs to this group: Mission Planning (Timeline Validation and Generation), VC0 TM Long Term Archiving, MADDS-A (VC0&VC7 TM Frame Distribution), MADDS-B (VC0&VC7 TM Frame Consolidation), IOBSM, IMCS External File Transfer System, Telemetry Query and Retrieval server for Performance Analysis (PAS-TDRS) Application belonging to this group shall have less stringent requirements in terms of availability and turn-around time in case of failures. However the crucial aspect which makes easier to define a redundancy concept for these group of applications is the fact that, differently from Kernel applications, they do not require to be resumed from a dynamically evolving configuration.

A second family of servers, identified as Secondary or Auxiliary servers is used to host auxiliary applications described above. The diagram in Figure 2 shows the IMCS hardware and Software configuration during the Launch and Early Orbit Phase up to the end of the Commissioning Phase. In this phases the most crucial aspect of the Mission Operation to be guaranteed is to provide in case of any failure the immediate or almost immediate capability to give control of the Spacecraft to the FCT. Any other aspect with substantial value in the contest of the entire mission (example Data Completeness, Data Integrity, Data Delivery to External Interfaces) will not play in this phase a main role and might be overlooked in case of specific failure occurring. In a hot stand-by configuration it is foreseen to have a redundant node up and running for each server. In principle in the hot stand-by configuration only the Prime Server are needed to be fully backed-up by hot-standby redundancy. However we intend to guarantee hot redundancy also to IMCS Secondary Server described above.

In case of failures requiring having to stop and restart application (but not requiring to move operations onto the redundant chain) application running onto Main Server shall foresee warm restart capability, whilst application on Secondary server will require simple cold restart.

In case of failures requiring to move onto the redundant chain, Prime Server shall guarantee the highest level of operability within the minimum turn around time, whilst Secondary Server can have a reduced functionality and at the same time can cope up with longer turn around time.

These distinctions allows us to define different level of failure resiliency for each of the identified server.

5

Figure 2 - IMCS configuration during LEOP and Commissioning Phase

3.1 Prime Server (IMCA) IMCA Server hosts as shown in previous section all IMCS Kernel functionality. During LEOP and up to end of the Commissioning phase it is required to have the IMCS Server redunded in hot stand-by. By this term it is meant that it shall be possible to 'mirror' on the redundant node all the operations as performed onto IMCA, with the objective to resume immediately on-going operation when moving for any reason from the main onto the redundant node. IMCA redundant node is IMCB. IMCA and IMCB will be two completely separated nodes each with its

TMP 1.1

IMCSMAIN SERVER

PRIMEIMCA

Client 1sun120

Client 2sun121

Client Nsun xxx

Client 3sun122

TMP 1.2

NCTRS PRIME

INCTRA

NCTRS Redundant

INCTRB

TCE

IMCSMAIN SERVER

RedundantIMCB

IMCS generated informations

IMCSSecondary SERVER

PRIMEIDDA

IMCSSecondary Server

RedundantIDDB

F I R E W A L L

ISDSSecondary SERVER

PRIMEISDSA

ISDSSecondary SERVER

RedundantISDSB

File (IFTS)

Raw TM Frames(VC0&7)

6

own CPU and DISK set. No automatic disk shadowing (mirroring) is required. The highest level of duplication of the operational contest onto the redundant server at any one time will be guaranteed via the measures taken at operational and application level as described in this chapter. Following section will show which level of hot redundancy can be achieved on IMCS using features of the IMCS-S2K application complemented by appropriate operational measures.

3.1.1 Telemetry As concerning TM the maximum level of hot stand-by redundancy can be easily achieved as shown in diagram above. Same telemetry will be routed in parallel to the main and redundant systems (IMCA&IMCA) respectively via INCTRA&INCTRB. In case of server failure, it would be sufficient to switch clients from IMCA to IMCB and carry on with the operations. On IMCB all the previously received and processed telemetry will be available.

3.1.2 Telecommand Telecommand has a crucial operational limitation: only one system can be connected with the Spacecraft at any one time. This makes thinks easier. In case of node switch over (IMCA to IMCB), NCTRS operator shall close current TC connection and re-establish a new one via NCTRB. All IMCS clients shall switch onto the redundant server. Since the Manual Stacks are local to the clients, in most cases no information is lost in case we loose the main server. Operator will have to save their current Stacks before switching onto the redundant server.

3.1.3 Verification To guarantee an adequate level of hot stand by redundancy in the area of TC Verification: • the TC Verifier will send TC history packets to the backup server, in order that the TC history on the

backup is a mirror of that on the server • the TC Verifier will be able to issue messages due to the status consistency enable/disable to both

behaviour checkers on the prime and backup server.

3.1.4 Configuration To guarantee an adequate level of hot stand by redundancy in the area of operational configuration: • changes to dynamic configuration variables on the prime server will be mirrored on the backup • switching a client workstation to the backup will affect all connections between the client and the

server (User Manager, MISC dynamic server, PIF server, TC Multiplexer)

3.1.5 Ground Model consistency IMCS provides the capability to check whether the actual Time Tag Queue on Board is in line with the model as maintained on Ground and in case of differences it allows to update the model maintained on Ground in order to have it in line with the actual On-Board content. This would allow the FCT to have the capability to perform this check as soon as they are forced to move onto the redundant chain and to recreate on the redundant server the actual On-Board Time Tag Queue Model.

3.1.6 IFTS As introduced in Section 2 IFTS is a software component devoted to the data file exchange across the various component of the Ground Segment. Two instances will be active. A first one serving the INTEGRAL Ground Segment and allowing ISOC ISDC and MOC to exchange data files. This instance will be identified as IFTS Gateway. A second one running at MOC allowing IMCS to exchange data file with other MOC component (e.g. FDS, OBSMS). Such instance will be identified here as MOC Internal IFTS.

7

IFTS Gateway is hosted onto the Secondary Server IDDx. The MOC Internal Instance of the IFTS Application during the LEOP Phase will be active and running solely onto the currently Prime chain. This approach allows to isolate as much as possible the effects of a server failure. In fact in case of a secondary server failure, the file transfer the kernel functionality of the main server would not be impacted and similarely in case of a failure of a main server, the secondary server thus the interface to the Science Ground Segment would be less impacted. In case of IMCA failure IFTS Internal Instance shall be restarted onto IMCB. This would imply having to restart other MOC Internal IFTS instance (e.g. those hosted onto the Flight Dynamics node) but as soon as this is done.

3.2 Secondary Servers Since Secondary servers are hosting functions far less critical in term of S/C safety, availability requirement (turn-around time) are far less stringent. It is considered acceptable, in case of severe failure on the main server, to move onto the redundant one and perform a cold start of all relevant applications. It should however be noted that a certain level of hot redundancy is provided by IMCS also on the Secondary Server. It is in fact possible to have also IDDA and IDDB connected respectively via IMCA and IMCB to NCTRA&NCTRB and receiving independently in parallel the same telemetry. However a move onto the redundant secondary server might turn out in a reduction of the original functionality as foreseen in the nominal scenario or in bigger turn around time in order to bring the system in a nominal situation.

3.2.1 Data Distribution Server (IDDx) As shown in figure 2, the IDDA server is backed-up by IDDB. TM flow is duplicated on IDDB. In case of severe hardware failures requiring to move onto IDDB following concept is applied. Telemetry data flow is also duplicated and active on both Chain A (Prime) and B (Redundant). IDDB will be rebooted with the IDDA Identity and it will subsequently take over the operations. In order to do this the Disk set of IDDA, including the DATA DISK shall be restored onto IDDB. This might required longer turn-around time thus a temporary unavailability of the functions hosted onto the secondary server in subject. In other words this operation is estimated to require 1-hour outage of the IDDx functionality. When the reboot procedure is terminated the redundant node is ready to take over under IDDA identity. This implies that the outage will be longer but once the system is up again, the IDDA will provide the entire set of functionality. Data completeness is also affected (there will be a gap of data). Future data consolidation in order to fill those gaps is possible but not explicitly required assuming that the occurrence of these gaps in LTA will be limited and within the overall system requirements concerning Data Archiving this is considered compatible with the Operational Concept.

Advantages Data gap limited to the time needed to build IDDB with IDDA identity

Disadvantages

8

Interruption (outage) of IDDx functionality

However it is to be noted that the procedure drafted above might require some coordination with Project and other Ground Segment responsibles. It might be required to avoid the discontinuity of service in case the failure is occurring during particularly critical operation. In such cases we could take advantage of the high level of redundancy and we might decide to move the Science Ground Segment interface onto the redundant chain until the time the critical operation has been completed. IFTS reconfiguration would be required, but this only in case the file transfer capability would be considered essential.

3.2.2 Security Data Server (ISDSx) 1 ISDSB will be rebooted with the ISDSA Identity and it will subsequently take over the operations. In order to do this the Disk set of IDDA shall be restored onto IDDB. Since ISDSa does not store significant amount of data the turn-around time should be limited to half hour. The outage of ISDS functionality will imply ia discontinuity of IFTS services with the External Interfaces and TM Data Distribution to IFRD. When the reboot procedure is terminated the redundant node is ready to take over under ISDSA identity. This implies the outage will be paid back by not having to change any configuration both for the TM Distribution and for IFTS configuration.

Advantages Once the ISDSB machine has taken over with ISDSA identity, no reconfiguration is needed in order to carry on with operations (only application shall be started up)

Disadvantages Interruption (outage) of ISDSx functionality (IFTS and TM Distribution)

In case the Operation scenario in which the failure occurs makes unacceptable the outage required to perform the activity described above, then the Science Ground Segment interface would be moved temporarily onto ISDSB until the operation is completed.

4 Conclusions The INTEGRAL Ground Segment comprises a very complex architecture with high demanding requirements in term of real-time and (in same cases) time-critical interfaces. IMCS plays a key role in such architecture. The INTEGRAL Mission Control System has in fact been requested to provide a wide set of services complementing the traditional Control and Monitoring functionality: online data delivery to the Science Ground Segment, both via real-time delivery of the telemetry currently acquired from the Ground Station and via a web based service of Telemetry Query and Extraction functions, offline data delivery via CD-ROM consolidated at the MOC, capability for one of the PI to issue Command Request in case of detection of Gamma Ray Burst, Telecommand Execution echo via a regular periodic delivery of the Telecommand History File. However the modularity and the flexibility of the system has made possible to accommodate with relatively small extra efforts several new requirements raised even in deeply advanced phase (e.g. Transfer, Integration and Testing).

9

As concerning the redundancy concepts only partially reported in this article, it has required more substantial efforts. The move from centralised architecture as those previously in use at the ESA MOC to the highly distributed architecture based on the new SCOS-2000 infrastructure represented also a fundamental change in the definition of a redundancy concept. The consolidated concepts defined and consolidated over more than a decade onto the centralised SCOS-1 based MCS although had constituted a solid basement had to be reviewed and amended in order to fit with the new architecture and, in the particular case of the INTEGRAL mission, with the complexity of the System External interfaces. The big challenge IMCS has undertaken as first SCOS-2000 Client Mission is to define and pioneer a redundancy concept which will be of great benefit for the upcoming missions.