Slide 12011 ILWS Science Workshop 28 Aug – 1 Sept 2011 *  
Status of Cluster Active Archive (CAA) and Cluster Final Archive (CFA)
Harri Laakso, Philippe Escoubet, Arnaud Masson, and Matt Taylor
European Space Agency (ESA)
Noordwijk, The Netherlands
2011 ILWS Science Workshop 28 Aug – 1 Sept 2011
2011 ILWS Science Workshop 28 Aug – 1 Sept 2011 *  
The CAA will contain all Cluster high-resolution data as well as all allied (supporting) data products necessary to the full interpretation of the full-resolution datasets
The entire high- and low-resolution datasets are fully calibrated (to the level achievable within the limited resources available)
The data are suitable for science use and publication by the world-wide scientific community (however, see quality flags and caveat files and extended documentation for interpretation)
All data are provided in a standard format (CEF - Cluster Exchange Format) and with a complete set of metadata in machine readable form; all data can be downloaded either in CEF or CDF format
The CAA aims to provide user friendly services for searching and accessing these data
What is CAA?
Updates in CAA User Interface
Web interface has stayed quite stable and no major updates recently
Command-line interface has been updated
Speed of requests has improved from minutes to seconds for regular requests
Small requests (<100MB) are executed immediately if the resources permit, otherwise it is scheduled (flag exists that user can make a scheduled request)
If request is executed immediately, the requested data file is returned to the user; if the request is scheduled the user gets a CAA_Info.log file as before
Streaming interface to the data is now available for beta-testing
Data are not written on CAA disk, but are directly streamed to the user
Usage: the data can be forwarded directly to an application or written in a file (and zip if requested)
wget “ hcp://caa.estec.esa.int/cgi--bin/stream_caa.cgi/?uname=[Uname]& pwd=[pwd] & dataset_id=C1_CP_EDI_SPIN&time_range= 2004-0103T00:00:00Z/2004-01-04T00:00:00Z&gzip=1” -O OUTFILE.cef.gz
Limitations: No error reports produced; will be applied to CEF files only
2011 ILWS Science Workshop 28 Aug – 1 Sept 2011 *  
CAA Cross-Calibration Workshops
The Cross Calibration Workshops are a Critical Activity for the CAA and provides an active forum for discussion of CAA issues
Improves quality and standardisation of data
The minutes of the meetings with presentations can be found at
ftp://ftp.rssd.esa.int/pub/Cluster/MoM/CAA/CrossCalibration/
Examples of CAA Cross-Calibration Plots: Density
CUSP
MAGNETOSHEATH
2011 ILWS Science Workshop 28 Aug – 1 Sept 2011 *  
Within the next 12 months, the CAA/BLIC project will determine the Cluster crossings of the bow shock, magnetopause, electron radiation belt and neutral sheet
Three first events are needed to add quality flags to some observations, e.g. to flag the instrument mode – region mismatch (BS and MP) or to flag the increased MCP noise (RB)
A Matlab code for semi-automated boundary recognition will be created for the work
Manual verification and correction of the boundary crossing time, i.e. the first and last crossings are verified for multiple boundary crossings
A quality flag will be included to indicate the type crossing (including e.g. a flag for quasi-parallel bow shock crossing or an unclear magnetopause crossing through the exterior cusp) and caveat remarks when necessary
The code will be made available for scientific use through CAA.
Cluster Boundary Crossings
Example: Bow Shock Crossings
2011 ILWS Science Workshop 28 Aug – 1 Sept 2011 *  
Double Star Raw Data Reprocessing Workshops
The original raw data was found to have a large number of data gaps, noisy intervals, wrong time stamps etc.
it was decided that it is better to regenerate all raw data
New issues were found in regenerated data
When the problems have been resolved, the final science datasets will be generated and ingested on the CAA
The minutes of the meeting, with all presentations, can be found at
ftp://ftp.rssd.esa.int/pub/Cluster/MoM/CAA/DoubleStar/
ECLAT provides context for the Cluster observations (EU-FP7)
How do Cluster observations relate to other observations of the magnetosphere, made from the ground or other space-borne observatories?
The Cluster Ground-based Working Group has provided contextual information for the duration of the Cluster mission to date on an ad hoc basis
This will be formalized, improved, and combined with a data-serving functionality within ECLAT
European Cluster Assimilation Technology
ECLAT Data Products for the CAA
A knowledge of the magnetospheric regions and boundaries Cluster is encountering
Region and boundary identification (OEAW)
An ability to know where Cluster is relative to other observatories in space or on the ground
Magnetic field tracing (SPSU)
The ionospheric conditions at the mapped footprint of the Cluster spacecraft
Miracle equivalent current mapping (FMI)
Knowledge of the large-scale magnetospheric state and behaviour
SuperDARN ionospheric flow patterns and global auroral imagery (ULEIC)
2011 ILWS Science Workshop 28 Aug – 1 Sept 2011 *  
CFA vs CAA
The final Cluster data archive (CFA) will be located at ESAC (Madrid)
The CAA to CFA transition activities have started
CFA will use standard ESAC archive architecture (in order to simplify the maintenance in the future)
Services and user interfaces of CFA should be similar to the those of CAA
Role of CAA
the only interface to the data producers
CAA will validate and ingest the new files and new datasets on its database
Data will then be transferred to the CFA and be added in the CFA database
Up to 1 year after the end of operations, CAA will stay active
The rough transition schedule is
2011: the CFA specification and a major part of implementation done
2012: both CAA and CFA user interfaces will coexist, and CFA has further developments while the CAA GUIs are frozen
2013 onwards: CFA is the sole interface for the access of Cluster data
2011 ILWS Science Workshop 28 Aug – 1 Sept 2011 *  
Delivery Schedule of 2010-12 Data
All teams must deliver files more continuously than before (i.e. not once a year)
This helps the validation/ingestion tasks of the CAA
Files of some datasets are needed by other teams in their processing, and such files are needed even earlier
Delivery schedule for the last three years of spacecraft operations (assuming no further Cluster extension)
Measurements Delivery date
July-December 1 May 2011
July-December 1 May 2012
July-December 1 May 2013
2011 ILWS Science Workshop 28 Aug – 1 Sept 2011 *  
Near-Term Activities (Next 12 mo.)
Routine operations, data delivery and maintenance of the existing data access and value added services
System maintenance and upgrades
Production and ingestion of all dataset up to December 2011, and
Ingestion of WBD CDF data and translation to CEF for years 2001-TBD
Include the data mining tool to the CAA GUI
Calibration and cross-calibration of all measurements
14th Cross-Calibration Workshop, October 2011
15th Cross-Calibration Workshop, April 2012
Complete CAA cross-calibration analysis tools/plots
New data sources:
Completion of Double Star reprocessing and production of datasets
Start of the production of ECLAT datasets
CAA-CFA transition activities
Cluster Active Archive statistics
http://caa.estec.esa.int