Status of Cluster Active Archive (CAA) and Cluster Final Archive (CFA)

2011 ILWS Science Workshop 28 Aug – 1 Sept 2011 1

Status of Cluster Active Archive (CAA) and Cluster Final Archive (CFA)

Harri Laakso, Philippe Escoubet, Arnaud Masson, and Matt TaylorEuropean Space Agency (ESA)

Science and Robotic Exploration DirectorateNoordwijk, The Netherlands

2011 ILWS Science Workshop 28 Aug – 1 Sept 2011


The CAA will contain all Cluster high-resolution data as well as all allied (supporting) data products necessary to the full interpretation of the full-resolution datasets

The entire high- and low-resolution datasets are fully calibrated (to the level achievable within the limited resources available)

The data are suitable for science use and publication by the world-wide scientific community (however, see quality flags and caveat files and extended documentation for interpretation)

All data are provided in a standard format (CEF - Cluster Exchange Format) and with a complete set of metadata in machine readable form; all data can be downloaded either in CEF or CDF format

The CAA aims to provide user friendly services for searching and accessing these data

What is CAA? http://caa.estec.esa.int/


Updates in CAA User Interface

Web interface has stayed quite stable and no major updates recently

Command-line interface has been updated Speed of requests has improved from minutes to seconds for regular requests Small requests (<100MB) are executed immediately if the resources permit,

otherwise it is scheduled (flag exists that user can make a scheduled request) If request is executed immediately, the requested data file is returned to the

user; if the request is scheduled the user gets a CAA_Info.log file as before

Streaming interface to the data is now available for beta-testing Data are not written on CAA disk, but are directly streamed to the user Usage: the data can be forwarded directly to an application or written in a file

(and zip if requested) wget “ hcp://caa.estec.esa.int/cgi--‐bin/stream_caa.cgi/?uname=[Uname]&

pwd=[pwd] & dataset_id=C1_CP_EDI_SPIN&time_range= 2004-01‐03T00:00:00Z/2004-01-04T00:00:00Z&gzip=1” -O OUTFILE.cef.gz

Limitations: No error reports produced; will be applied to CEF files only


CAA Cross-Calibration Workshops The Cross Calibration Workshops are a Critical Activity for the CAA and provides an active

forum for discussion of CAA issues Improves quality and standardisation of data The minutes of the meetings with presentations can be found at

ftp://ftp.rssd.esa.int/pub/Cluster/MoM/CAA/CrossCalibration/


Examples of CAA Cross-Calibration Plots: Density

CUSP MAGNETOSHEATH SOLAR WIND

N sounder

N e-

N i

N H+


Within the next 12 months, the CAA/BLIC project will determine the Cluster crossings of the bow shock, magnetopause, electron radiation belt and neutral sheet Three first events are needed to add quality flags to some observations, e.g. to

flag the instrument mode – region mismatch (BS and MP) or to flag the increased MCP noise (RB)

A Matlab code for semi-automated boundary recognition will be created for the work Manual verification and correction of the boundary crossing time, i.e. the first and

last crossings are verified for multiple boundary crossings A quality flag will be included to indicate the type crossing (including e.g. a flag for

quasi-parallel bow shock crossing or an unclear magnetopause crossing through the exterior cusp) and caveat remarks when necessary

The code will be made available for scientific use through CAA.

Cluster Boundary Crossings


B [n

T]

n CIS

[cm

-3]

bitm

ask

n WH

I

[cm

-3]

n EF

W

[cm

-3]

v CIS

[cm

-3]

EFW bitmask & B

nEFW & B

nWHI & B

vCIS & nCIS & B

Total

Cluster 1 March 9, 2001

Example: Bow Shock Crossings

Outward Inward


Double Star Raw Data Reprocessing Workshops

The original raw data was found to have a large number of data gaps, noisy intervals, wrong time stamps etc.

it was decided that it is better to regenerate all raw data New issues were found in regenerated data When the problems have been resolved, the final science datasets will be

generated and ingested on the CAA

The minutes of the meeting, with all presentations, can be found at ftp://ftp.rssd.esa.int/pub/Cluster/MoM/CAA/DoubleStar/


ECLAT provides context for the Cluster observations (EU-FP7)

How do Cluster observations relate to other observations of the magnetosphere, made from the ground or other space-borne observatories?

The Cluster Ground-based Working Group has provided contextual information for the duration of the Cluster mission to date on an ad hoc basis

This will be formalized, improved, and combined with a data-serving functionality within ECLAT

European Cluster Assimilation Technology (ECLAT)


ECLAT Data Products for the CAA A knowledge of the magnetospheric regions and boundaries Cluster is

encountering Region and boundary identification (OEAW)

An ability to know where Cluster is relative to other observatories in space or on the ground Magnetic field tracing (SPSU)

The ionospheric conditions at the mapped footprint of the Cluster spacecraft Miracle equivalent current mapping (FMI)

Knowledge of the large-scale magnetospheric state and behaviour SuperDARN ionospheric flow patterns and global auroral imagery (ULEIC)


CFA vs CAA The final Cluster data archive (CFA) will be located at ESAC (Madrid)

The CAA to CFA transition activities have started CFA will use standard ESAC archive architecture (in order to simplify the

maintenance in the future) Services and user interfaces of CFA should be similar to the those of CAA

Role of CAA the only interface to the data producers CAA will validate and ingest the new files and new datasets on its database Data will then be transferred to the CFA and be added in the CFA database Up to 1 year after the end of operations, CAA will stay active

The rough transition schedule is 2011: the CFA specification and a major part of implementation done 2012: both CAA and CFA user interfaces will coexist, and CFA has further

developments while the CAA GUIs are frozen 2013 onwards: CFA is the sole interface for the access of Cluster data


Delivery Schedule of 2010-12 Data All teams must deliver files more continuously than before (i.e. not once a year)

This helps the validation/ingestion tasks of the CAA Files of some datasets are needed by other teams in their processing, and

such files are needed even earlier

Delivery schedule for the last three years of spacecraft operations (assuming no further Cluster extension)

Measurements Delivery date 2010 January-June 1 Dec 2010 July-December 1 May 2011 2011 January-June 1 Dec 2011 July-December 1 May 2012 2012 January-June 1 Dec 2012 July-December 1 May 2013 Final calibrated datasets (2000-2012) 1 Nov 2013


Near-Term Activities (Next 12 mo.) Routine operations, data delivery and maintenance of the existing data access

and value added services System maintenance and upgrades

Production and ingestion of all dataset up to December 2011, and Ingestion of WBD CDF data and translation to CEF for years 2001-TBD Include the data mining tool to the CAA GUI

Calibration and cross-calibration of all measurements 14th Cross-Calibration Workshop, October 2011 15th Cross-Calibration Workshop, April 2012 Complete CAA cross-calibration analysis tools/plots

New data sources: Production of boundary crossing datasets Completion of Double Star reprocessing and production of datasets Start of the production of ECLAT datasets

CAA-CFA transition activities


Cluster Active Archive statisticsNow 1250 users


http://caa.estec.esa.int

Documents

Status of Cluster Active Archive (CAA) and Cluster Final Archive (CFA)