8
EVT-SIAM: A tool based on Extreme-Value Theory for the assessment of accuracy and integrity assessment of SBAS Guillaume Buscarlet, Thales Alenia Space France Jean-Marc Azaïs, Sébastien Gadat, IMT, Institut de Mathématiques de Toulouse, Toulouse III University Norbert Suard, CNES, French Space Agency BIOGRAPHIES Guillaume Buscarlet, graduated from Supaero (ENSAE), Toulouse in 1997, worked at JPL, Pasadena, at Cap Gemini, Toulouse and joined Thales Alenia Space in 2007. He is a System Performance Engineer in TAS-F Navigation unit, working on EGNOS performance analysis and evolutions. He has been working on performance of EGNOS releases, EGNOS-like SBAS above Arctic areas and on GNSS messages performances. Jean-Marc Azaïs, graduated from ENS Paris in 1982, docteur es sciences in 1989. He has been a Professor at Toulouse III University since 1990. He authored about 50 papers in Probability and Statistics and in particular in the Theory of extremes. Sébastien Gadat , graduated from ENS de Cachan, France in 2004, obtained his position, at Toulouse III University in 2005. He is “Maître de conférences” and works in the team of Probability and Statistics. Norbert Suard, graduated from the ENSEEIHT (Toulouse, France) in 1982, joined the CNES in 1983. He is an Engineering Expert in the CNES Navigation System Division where he has over 20 years of experience in development, studies, performances analysis of navigation system augmentation like CE-GPS, EURIDIS, ESTB and now EGNOS, WAAS and GAGAN. He is currently more specifically in charge of the CNES Navigation and Time Monitoring Facility (NTMF) designed to monitor GPS and SBAS Signals In Space and Performances, he is member of the WGC of the UE-US agreement in the promotion, provision and use of civil GPS and GALILEO navigation and timing signals and services and is chairing the French group GEOPOS of the CNIG (National Council of Geographic Information) ABSTRACT Following the start of WAAS extensions in Canada, in Alaska and in Mexico, as well as the start of WAAS system evolutions between 2005-2008, and considering the noticed improvements of performances and procedures, the FAA suggested widening the WAAS services up to LPV200 (aka. category I approaches), with a vertical alert limit (VAL) of 35m. This objective has also been assigned to EGNOS in Europe, and ICAO Annex 10, Volume 1 has been amended in that direction (Amendment #85). Though most of the specifications of LPV200 are identical to those for APV1 service level (where VAL = 50m) in terms of integrity, horizontal alert limit, availability and continuity, different or additional specifications have been introduced in the field of positioning accuracy. Indeed, the specification of APV1 accuracy – that the 95th centile of the horizontal error (H-NSE) be below 16 m, and the 95th centile of the vertical error (V-NSE) be below 20 m – becomes for LPV200 in the vertical domain that - V-NSE_95 % < 4m - Proba(VPE_1sec > 10m) < 1E-7 in the absence of failure everywhere the operation is to be approved; - Proba(VPE_1sec > 15m) < 1E-5 in case of failure. The specification of APV1 and LPV200 integrity is Proba(VPE > VAL or HPE > HAL) < 2E-7 per 150 s where VAL (resp. HAL) the vertical (resp. horizontal) alert limits values are fixed by the International Civil Aviation Organisation for different flight phases. The failures to be taken into account are the ones that affect the used basic constellations and GNSS augmentation systems. This latter probability results from an allocation that takes into account the probability that a given failure occurs and of the probability of detection. Verifying such low probabilities with standard statistical methods require several months or years of observation data, even with a 1-second accuracy sampling. This is not realistic and compatible with industrial constraints that require system qualification time scales. For instance, such rare “events” are generally not observed within the data used to qualify a SBAS release because of their scarcity.

EVT-SIAM: A tool based on Extreme-Value Theory for the …gadat/Papers/Tool_EVT_SIAM_V4.pdf · 2012-07-15 · EVT-SIAM: A tool based on Extreme-Value Theory for the assessment of

  • Upload
    others

  • View
    7

  • Download
    0

Embed Size (px)

Citation preview

Page 1: EVT-SIAM: A tool based on Extreme-Value Theory for the …gadat/Papers/Tool_EVT_SIAM_V4.pdf · 2012-07-15 · EVT-SIAM: A tool based on Extreme-Value Theory for the assessment of

EVT-SIAM: A tool based on Extreme-Value Theory for the assessment of accuracy and

integrity assessment of SBAS

Guillaume Buscarlet, Thales Alenia Space France Jean-Marc Azaïs, Sébastien Gadat, IMT, Institut de Mathématiques de Toulouse, Toulouse III University

Norbert Suard, CNES, French Space Agency

BIOGRAPHIES Guillaume Buscarlet, graduated from Supaero (ENSAE), Toulouse in 1997, worked at JPL, Pasadena, at Cap Gemini, Toulouse and joined Thales Alenia Space in 2007. He is a System Performance Engineer in TAS-F Navigation unit, working on EGNOS performance analysis and evolutions. He has been working on performance of EGNOS releases, EGNOS-like SBAS above Arctic areas and on GNSS messages performances. Jean-Marc Azaïs, graduated from ENS Paris in 1982, docteur es sciences in 1989. He has been a Professor at Toulouse III University since 1990. He authored about 50 papers in Probability and Statistics and in particular in the Theory of extremes. Sébastien Gadat, graduated from ENS de Cachan, France in 2004, obtained his position, at Toulouse III University in 2005. He is “Maître de conférences” and works in the team of Probability and Statistics. Norbert Suard, graduated from the ENSEEIHT (Toulouse, France) in 1982, joined the CNES in 1983. He is an Engineering Expert in the CNES Navigation System Division where he has over 20 years of experience in development, studies, performances analysis of navigation system augmentation like CE-GPS, EURIDIS, ESTB and now EGNOS, WAAS and GAGAN. He is currently more specifically in charge of the CNES Navigation and Time Monitoring Facility (NTMF) designed to monitor GPS and SBAS Signals In Space and Performances, he is member of the WGC of the UE-US agreement in the promotion, provision and use of civil GPS and GALILEO navigation and timing signals and services and is chairing the French group GEOPOS of the CNIG (National Council of Geographic Information) ABSTRACT Following the start of WAAS extensions in Canada, in Alaska and in Mexico, as well as the start of WAAS system evolutions between 2005-2008, and considering the noticed improvements of performances and

procedures, the FAA suggested widening the WAAS services up to LPV200 (aka. category I approaches), with a vertical alert limit (VAL) of 35m. This objective has also been assigned to EGNOS in Europe, and ICAO Annex 10, Volume 1 has been amended in that direction (Amendment #85). Though most of the specifications of LPV200 are identical to those for APV1 service level (where VAL = 50m) in terms of integrity, horizontal alert limit, availability and continuity, different or additional specifications have been introduced in the field of positioning accuracy. Indeed, the specification of APV1 accuracy – that the 95th centile of the horizontal error (H-NSE) be below 16 m, and the 95th centile of the vertical error (V-NSE) be below 20 m – becomes for LPV200 in the vertical domain that

- V-NSE_95 % < 4m - Proba(VPE_1sec > 10m) < 1E-7 in the absence of failure everywhere the operation is to be approved; - Proba(VPE_1sec > 15m) < 1E-5 in case of failure.

The specification of APV1 and LPV200 integrity is Proba(VPE > VAL or HPE > HAL) < 2E-7 per 150 s

where VAL (resp. HAL) the vertical (resp. horizontal) alert limits values are fixed by the International Civil Aviation Organisation for different flight phases. The failures to be taken into account are the ones that affect the used basic constellations and GNSS augmentation systems. This latter probability results from an allocation that takes into account the probability that a given failure occurs and of the probability of detection. Verifying such low probabilities with standard statistical methods require several months or years of observation data, even with a 1-second accuracy sampling. This is not realistic and compatible with industrial constraints that require system qualification time scales. For instance, such rare “events” are generally not observed within the data used to qualify a SBAS release because of their scarcity.

Page 2: EVT-SIAM: A tool based on Extreme-Value Theory for the …gadat/Papers/Tool_EVT_SIAM_V4.pdf · 2012-07-15 · EVT-SIAM: A tool based on Extreme-Value Theory for the assessment of

This motivates the development of non-standard verification approaches. The statistical Extreme Value Theory (EVT) consists in an extrapolation of the error distributions tails under rough and conservative power law assumptions. The approach is not new (Fisher-Tippet,1928), but recent developments in quantile estimation have allowed its application in numerous domains. It happens to be suited to cases regardless of the underlying measurement error distributions. It avoids in particular the questionable assumption of Gaussian error distributions, which implicitly assumes exponential fast decaying tails. EVT derives instead some properties of the distribution tails from the measured data. This allows a meaningful extrapolation into the low-probability region, even when no (or very limited amounts of) samples are available. Following two previous CNES funded studies that revealed full relevance of this approach for the verification of system integrity [4] and accuracy requirements, a tool (EVT_SIAM: Extreme Value Theory Supporting Integrity and Accuracy Measurement) has been defined, prototyped and industrialized. This allows implementing this method in the verification of Navigation system performance. Thus, the EVT-SIAM Tool is a SBAS statistical analysis tool able to characterize or verify the behaviour of the tail of an error distribution deriving from measurement data. This tool, avoiding the assumption of Gaussian error distributions, allows extrapolating meaningfully the data into the region of misleading information, even when no (or very limited amounts of) sample values in this region are available. EVT-SIAM Tool is dedicated to integrity and accuracy of SBAS requirements. With a given confidence level, a quantile can be provided by EVT-SIAM Tool for the SBAS requirements analysis mentioned above. . The paper is organized in the following way. - A first part presents the main lines of the EVT approach and the Pareto-law assumption, and the conclusions of the previous studies are recalled. - The use cases (e.g. local user or global assessment, accuracy or integrity requirement assessment, orbit/clock or ionosphere correction etc areas) of the tool are described in a second part, and the main requirements that result are presented. - The tool architecture is explained, with its five modules. Two first modules monitor the input data and the configuration parameters. A third one adapts the navigation data to the constraints of assessing them against a Pareto distribution law. A fourth module validates the main assumptions (non correlation, cluster, stationary…) for the validity of the extreme value domain of attraction. The last module estimates the parameters of the error distribution law and deduces the values of the figures of merit to be compared to the requirements. - The ways to enforce representativeness and robustness of the computation results are then focused on. - The way the tool has been verified is then presented, and some results are given.

Finally, recommendations are formulated for using this technique in future assessment of such demanding GNSS LPV 200 accuracy performance requirements as for new EGNOS releases for which LPV200 is required. INTRODUCTION The European Satellite Based Augmentation System, called EGNOS (European Geostationary Navigation Overlay Service), provides to users in Europe an augmentation of three pseudo-GPS signals plus corrections/integrity information about the available GPS constellation [1, 2] enabling to compute a safe and precise position that can be dated in a legal time scale (UTC)[3].

Differentialcorrections

Integrity(Use / Don't Use)

+ ACCURACY+ AVAILABILITY+ CONTINUITY

+ SAFETY

GEO

Time Function

GPS-likesignals

Time information

Figure 1: SBAS Missions

These missions lead to some stringent integrity requirements (risk of loss of integrity is required to be less than 10-7/150 sec range). The experimental demonstration of such a requirement by the classical methods taking into account the time correlations require several tens years of observations and such Loss of Integrity (LOI) is generally not observed among limited data set because of their scarcity. Moreover, accuracy performances have been claimed through LPV200 aeronautical service to consider a risk of more than 10 m. accuracy, which have to be less than 10-7/150 sec range. Same confidence levels require identical answers. So the aim of this paper is to present a tool performed during an IMT/CNES/TAS Action for using the extreme value theory in these domains enabling verifications of high confidence level requirements using a time limited data set. . 1. EXTREME VALUE THEORY: RECALL Studies led by CNES and TAS [4] with the support of two French Universities demonstrated the gain that could be obtained by the application of the Extreme Value Theory in the navigation domain for the Integrity Risk evaluation and Accuracy evaluation. A protocol for the correct application of the method has been defined and relies on two main steps: estimates the parameters to quantify the tail of the distribution and extrapolates this tail to extreme values. These steps involve a mathematical application of some statistical inference for Pareto distributions (through the Pick Over Threshold -POT- method). The POT method is based on the following theorem [5, 6]:

Page 3: EVT-SIAM: A tool based on Extreme-Value Theory for the …gadat/Papers/Tool_EVT_SIAM_V4.pdf · 2012-07-15 · EVT-SIAM: A tool based on Extreme-Value Theory for the assessment of

If Fu(X) is the distribution of a random variable X conditionally to be over a given threshold u, more precisely, if

Fu(X) = (X ≤ x+u | X > u), then, under mild assumptions on stationary, independency, sizing conditions and attraction domain validity, when u goes to infinity, Fu(X) can be approximated as

1( ) 1 (1 )u

XF X γγσ

≈ − + if 0γ ≠

( ) 1X

uF X e σ−

≈ − if 0γ = where γ and σ are parameters must be estimated from the data. But this has been only developed experimentally, during the former protocol, including conditions through data visualisation, γ and σ parameters are estimated through a human supervision and it is thus subject to misinterpretation. Hence, the EVT-SIAM (Extreme Value Theory Supporting Integrity and Accuracy Measurement) tool has been set up to industrialized the human part of the protocol. Moreover, in case of EVT failure or with no compliant results, pre-investigation tasks can also be performed to understand reasons or limitations emerged through the protocol. Troubleshooting through automatic tasks will become easier with EVT-SIAM tool. 2. APPLICATION DOMAIN 2.1. SBAS Integrity and accuracy In former studies, two kinds of navigation domains errors had been analysed: user level and system level. At user level, precision error (xPE) and protection level (xPL) are known through an analysis tool [7], dedicated for performance analysis (integrity, accuracy, availability) at user level, computing EGNOS positions and protection level in compliance with the standard. These parameters lead to analyse accuracy through xPE and integrity through xPE / xPL. In fact, a loss of integrity (LOI) is defined as xPE > xAL (HMI), xPE > xPL (MI) with MI + HMI = LOI. At system level, the residuals range error for each satellite (SREW) under its protection (UDRE) and the residuals error for each iono grid point (GIVDe) under its protection (GIVE) are also known through a navigation system analysis tool, dedicated for performance analysis (integrity, accuracy, availability) at system level (SiS ad ionosphere errors). Thus, accuracy estimation can be analyzed with SREW and GIVDe and Integrity estimation through SREW/UDRE and GIVDe/GIVE. In fact, a loss of integrity is defined if SREW > UDRE5.33 or GIVDe > GIVE5.33 [8]. These parameters are, in fact, the input of EVT-SIAM Tool and defined its application domain. Thus, three kinds of daily files can be used as described in the EVT-SIAM Tool ICD (Interface Control Document):

• User data files, containing HPE, HPL and VPE, VPL per second. (Precision error and protection level for Horizontal and Vertical position). Each epoch proposes also position status information for Precision Approach or Non-Precision Approach.

• SiS data files, containing SREW and UDRE per second. Each epoch proposes also satellite status information for Don’t Use analysis purpose.

• Iono data files, containing GIVDe and GIVE per second. Each epoch proposes also IGP status information for Don’t Use analysis purpose.

“Don’t Use” information are exploited because of the EVT-SIAM tool capacity to remove 6 seconds of “potential DU impacts”. Regarding the 6s Time To Alert, the system could have carried a MI within the 6 seconds before the reception of the Don’t Use message. So, the tool allows removing a possible cause of non-compliance, and it analyses also automatically the number of Don’t Use for troubleshooting purpose. Moreover, the meaning of information is not specified inside EVT-SIAM Tool. Thus, as long as ICD is verified, more use cases can be analysed and application domain can be easily extended: for example, subsets of Iono files filtered on GIVE value, sets of SiS files containing several GNSS constellations, or use of SRE (Satellite residual error) calculated per satellite per several users. 2.2. Main requirements Based on previous studies, a dedicated tool has then been specified and built. Today, a first release of this tool is running at TAS-F and CNES premises. The main requirements have been summarized below: The Extreme Value Theory Supporting Integrity and Accuracy Measurement Tool (EVT-SIAM Tool) is an analysis tool able to manage and analyse large sets of data by applying EVT algorithms. Thus, the EVT-SIAM Tool shall provide an analysis environment:

• To enable data importation (data are made available from external analysis platforms and consist in SBAS integrity and accuracy figures at user and SiS level)

• To characterize statistical performances of data set by applying EVT algorithms, which include Pareto law use and check of data conditions (stationary, independency…)

• To provide reporting and result visualization capabilities

Data to be analysed mainly consists in SBAS integrity and accuracy data in the pseudo-range (SiS) and position domain (User). EVT-SIAM Tool aims at characterising or verifying the integrity/accuracy performances with a given probability and a given step. Thus, EVT-SIAM Tool shall have the capability to configure, through MMI/GUI, explicitly every input parameter to define a scenario, i.e. an EVT analysis. This includes:

Page 4: EVT-SIAM: A tool based on Extreme-Value Theory for the …gadat/Papers/Tool_EVT_SIAM_V4.pdf · 2012-07-15 · EVT-SIAM: A tool based on Extreme-Value Theory for the assessment of

• Name of the scenario • Definition of the period (typically T0-Tf) • Source (GEO) • Value type to be analysed (SREW,

SREW/UDRE,GIVDe, GIVDe/GIVE, xPE, xPE / xPL…)

• Probability target for the confidence interval (typically 2.10-7)

• Planification of the scenarios executions (Automatic for operation use case, Manual for expertise use case)

• Configuration (through dedicated panels) of all algorithms

The outputs produced by the tool are: • Report files, • Log files, • Results files containing statistical figures and

parameters estimation for each configured scenario.

Thus, for each scenario EVT-SIAM Tool shall produce algorithms results report. And within troubleshooting purpose, for each new data successfully introduced within an EVT database, for each scenario, EVT-SIAM Tool shall produce statistical indicators within a dedicated file report. These statistical indicators are the following:

• For several value type (SREW, SREW/UDRE… including differential analysis), Mean, Stdev, Median, 95percentile, 99 percentile, Maximum, Date of the maximum value, Histogram

• Loss of integrity: if exists, epochs and PRN / IGP of the loss of integrity

• Don’t Use: if exists, epoch and PRN / IGP of the Don’t use

• Number of available data • Number of epoch with missing data and missing

days • Number of holes with missing data (If missing

data exists: mean and stdev duration of missing data for a hole, maximum duration of missing data plus is epoch)

3. TOOL ARCHITECTURE EVT-SIAM Tool architecture has been proposed to ease the handling and the processing of huge amount of data (daily data over 4 years will require 2To data disk) and to allow the evolution of algorithms (addition / modification). Thus, EVT-SIAM Tool is modular and includes:

• A database module –aka F0 • A scenario management module to define

analysis to be run –F1 3 functions (sequentially activated) to analyse each configured scenario

• A pre-processing module for preparation of data specifically to each configured EVT scenario –F2

• A data control module to check applicability of extreme value theory to configured EVT scenario –F3

• A statistical analysis module implementing algorithms from extreme values theory –F4

Functional architecture highlights links between modules. Modules executions are basically performed in order from F0 to F4. (F0 is a background task)

Observables

Database Management

(F0)

Data Control(F3)

EVT algorithms(F4)

Results

Pre-processing(F2)

EVT scenario management

(F1)Exchange areafor external data

EVT Inputs Database(File System)

Scenario reports and

logs

Scenario reports and

logs

Figure 2: EEEVVVTTT___SSSIIIAAAMMM TTToooooolll Functional Architecture

General capabilities of EVT modules are detailed hereafter: 3.1. Data base management module (F0) The main purpose of the database management module is to:

• Manage the external interfaces of EVT-SIAM Tool, i.e. check data made available by external platform and import them within EVT own database structure

• Provide means to control and monitor the EVT database, i.e.: Produce indicators w.r.t to the database completion status

• Monitor easily the size of the database with regards to available bytes on the disk

3.2. Scenario management module (F1) The main purpose of the scenario management module is:

Page 5: EVT-SIAM: A tool based on Extreme-Value Theory for the …gadat/Papers/Tool_EVT_SIAM_V4.pdf · 2012-07-15 · EVT-SIAM: A tool based on Extreme-Value Theory for the assessment of

• To enable, through configuration files, the full configuration of analysis scenarios, i.e.:

• Definition of period and data source to be analysed

• Configuration of functions involved in analysis (including parameterization of algorithms)

• To provide means to visualize scenario executions reports, logs and results

MMI for F1 module has been designed as follows

FFFiiiggguuurrreee 3::: EEEVVVTTT___SSSIIIAAAMMM TTToooooolll MMMMMMIII

The 5 parts MMI design follows the modularity. And each module MMI proposes the possible parameters configurations. 3.3. Pre-processing module (F2) The main purpose of the pre-processing module is to build or complete a data set specific to a scenario configuration in order to feed the data check and EVT algorithms modules. This consists in performing the following tasks (in accordance with the configured scenario):

• Check of data in input files within EVT database,

• Data selection and conversion from EVT database into internal format, i.e.: application of sampling rules. Several sampling rules are proposed (such as non sampling rule to apply) but the main one derived from previous studies is the maximum value amongst a period of data (typically 150 s.)

• In case of integrity analysis, integrity ratio computation

• Data set construction The pre-processing module aims also at computing elaborated troubleshooting indicators for the corresponding data set. 3.4. Data check module (F3) The main purpose of the data check module is to check the applicability of the Extreme Values Theory toward the

internal data of a given scenario against the use of EVT algorithm module. These checks provide a data check report composed by outputs leading to appreciate the EVT applicability. The data check consists in performing the following tasks

• Samples number check, to verify if the number of samples is sufficient to perform the EVT for a given probability.

• Excursions study: This study consists in providing estimation of the excursion length, and of the length between excursions. An excursion is a sequence of consecutive values over a threshold. Typically this threshold is the threshold value K, estimated in the EVT algorithm module.

• Series discontinuity, to evaluate the presence of a break (or change-point) in the series. A break is an index in the series that implies two different behaviours in the input data.

• Multiple breaks, to evaluate the number of series discontinuities in the input data. This multiple breaks algorithm is reduced to a limited number of recursivity depths.

• Cluster event: a cluster is defined as a small number of consecutive values that have a distribution that is different from the rest of the series. Typically it is a set of high values. The clusters differ from breaks that concern long-term changes. A cluster implies at least two different behaviours in the input data.

• Multiple clusters, to evaluate the number of cluster in the input data. This multiple cluster algorithm is reduced to a limited number of recursivity depths.

• Consecution estimation, to estimate the consecution of the input data. The consecution is a sequence of identical values.

• Reliability estimator, to estimate the reliability of the EVT applicability. The estimator output is based on Kolgomorov-Smirnov test.

Each task provides one or several outputs to indicate a way of appreciate the applicability (including indicators to define alternate scenario in case of clusters or break detection). 3.5. EVT algorithms module (F4) The main purpose of the EVT algorithms module is to apply the Extreme Values Theory toward the internal data of a given scenario. This application provides mainly an estimation of the quantile with a confidence interval. The EVT algorithms consist in performing the following tasks:

• Estimation of a threshold value K, which yields Generalized Pareto Distribution (GPD) for the excess law.

Page 6: EVT-SIAM: A tool based on Extreme-Value Theory for the …gadat/Papers/Tool_EVT_SIAM_V4.pdf · 2012-07-15 · EVT-SIAM: A tool based on Extreme-Value Theory for the assessment of

• Estimation of the GPD parameters (scale and shape) and of the quantile of the tail of distribution with a confidence interval.

For each estimation, two methods are applied and a check of coherency between the two methods is performed. This threshold value K estimation is performed at least with the two following algorithms:

• GraphSum algorithm, which analyses the linearity behaviour of the sum of logarithmic ratio.

• Bootstrap method, which analyses the linearity behaviour of the sum of logarithmic difference applied on several samples of the population.

An estimation of the reliability of the chosen threshold is provided, through a comparison between both algorithm responses. EVT quantile estimation for a given probability plus a confident interval is performed at least with the two following algorithms:

• Maximum Likelihood Method, this is the application of Generalized Pareto distribution with Maximum Likelihood estimation.

• Bootstrap estimation, this is an application of Generalized Pareto distribution with Maximum Likelihood estimation, applied on several bootstrapped samples of the population.

An estimation of the reliability of the confident intervals algorithm obtained by the different methods is provided. 4. REPRESENTATIVENESS AND ROBUSTNESS Before the final implementation of EVT-SIAM Tool, a prototyping phase has been carried out to analyse representativeness of such a tool. Its execution allowed discovering some tricky events within F3 parameters. Fitting changes in F3 parameters can make a break or a cluster appearance or disappearance. These parameters have been deeply tuned to fit the set of scenarios that were available for the representativeness of the tool, knowing where a real break or cluster should appear. Nevertheless, a final and unique tuning has not been obtained. So, appearance and disappearance shall be explained through an external event (change of system / SW release for example). If the termination of analysis lead to satisfactory results, these observations are not troublesome. If the termination of analysis does not lead to satisfactory results, operators are able to configure alternative scenarios to obtain satisfactory results. In case of break attendance detected through F3 module, two alternative scenarios shall be chosen:

• A scenario before the break • A scenario after the break

If terminations of each analysis of the two new scenarios lead to satisfactory results, a certain amount of confidence

in the satisfactory results on the total data can be expected, bringing external explanation of the break appearance In case of clusters presence detected through the F3 module, two alternative scenarios shall be chosen:

• A scenario with the removal of each detected clusters

• A scenario with the reducing of each detected clusters

Reducing a cluster is an operation that changes the cluster with only one epoch equals to the maximum value appeared during the whole temporal period of cluster. The number of data to reduce or to remove is generally a period of time including the cluster with a certain amount of time from each side of the cluster. This period of time is chosen using a multiplicative factor on the length of the cluster period of time identified by the tool (usually 2). If the termination of the first analysis with removal of clusters leads to satisfactory results, then the second analysis with reducing clusters can be performed. If the termination of the second analysis with reducing clusters leads to satisfactory results, a certain amount of confidence in the satisfactory results of the total data can be highlighted. The idea is to stick to the EVT theory with dotted and without correlation excesses. Analysis on the F2 log files and then on the F0 log files shall be performed to understand the attendance of clusters. 5. VERIFICATION Initially, the tool has been verified using scenarios that have been defined through previous studies. Magnitude orders of the results have been checked (especially because of statistical Bootstrap method, results are not exactly identical), and the correlation between data check module (F3) and population visualisation have been verified. Then, an analysis has been performed through an actual large observation, from 2011, July 1st to 2012, January 3rd.

FFFiiiggguuurrreee 4::: PPPooopppuuulllaaattt iiiooonnn,,, from 2011, July 1st to 2012,

January 3rd

Page 7: EVT-SIAM: A tool based on Extreme-Value Theory for the …gadat/Papers/Tool_EVT_SIAM_V4.pdf · 2012-07-15 · EVT-SIAM: A tool based on Extreme-Value Theory for the assessment of

Different quantile have been estimated using different probabilities on this population. These statistical quantile are then compared with empirical data. The regular one (10^_7 for integrity prediction) has been used to check the consistency of the data and the extrapolation in line with previous studies (SREW/UDRE 2011-extra scenario). As a reminder, quantile less than 5.33 leads to satisfactory results (SREW > UDRE5.33). Then, interpolations have been verified. The large observation runs over 6 months. For this period of time, the percentage for the quantile represents 12*150 /(86400*365*6) • 0.9.10^-5. Thus, 6 months population with 10^-5 probability is an interpolation (SREW/UDRE 2011-interp scenario). 6 months population with 10^-2 probability shall identify the threshold for 1% of the data (SREW/UDRE 2011-2 scenario). A sample of this population has also been selected to predict the quantile, which should be obtained through the large observation, only 2 months population have been selected from 2011, July 1st to 2011, September 1st with 10^-5 probability as an extrapolation. (SREW/UDRE 2011-2months)

Scenario name

Sampling Rate /

Probability

Number of data /

Maximum

Expected Quantile

Min / Max SREW/UDRE 2011-extra

150 s. 10^-7

103044 4.04

3.97 / 4.18

SREW/UDRE 2011-interp

150 s. 10^-5

103044 4.04

3.85 / 3.99

SREW/UDRE 2011-2

150 s. 10^-2

103044 4.04

2.95 / 3.00

SREW/UDRE 2011-2months

150 s. 10^-5

33870 3.64

3.39 / 3.84

Results of the quantile at the required probability propose a value (within a confidence interval) for which, the tail of the distribution lies above. Sample of the tail of the large distribution is […3.87, 3.88, 4.04]

FFFiiiggguuurrreee 5::: HHHiiissstttooogggrrraaammm ooofff ttthhheee 222000111111 pppooopppuuulllaaattt iiiooonnn

Thus, results of SREW/UDRE 2011-interp and SREW/UDRE 2011-2months are in line with this

distribution, because the proposed quantile follows the empirical tail of the population behaviour. SREW/UDRE 2011-extra scenario quantile follows results of fault-free SREW/UDRE 2008 previous scenario [4] (between 3.70 and 4.06, function of the period) Moreover, for SREW/UDRE 2011-2 scenario, it has been checked that the number of values above 2.95 in the population is exactly 1029. And 1029/103044 • 10^-2 fits really well with the 10^-2 required probability. CONCLUSION First uses of EVT-SIAM Tool have demonstrated the interest of the procedure implementation to analyse Integrity Risk and Accuracy performances in the navigation domain by the application of the Extreme Value Theory. Data collection, began at CNES since 2011, September allows first data analysis to verify and qualify the EVT_SIAM Tool functional architecture, from the pre-processing module, which select data within a database, to the application of EVT algorithm, which proposes a threshold to keep the data above which interpolation with Generalized Pareto Distribution is applied, passing through the check of the data, to know their level of independency, their stationary state, excursions behaviour and to evaluate the confidence in the results. One of the main teachings is that when the EVT analysis leads to satisfactory results, a large confidence can be held in the results. When results are not in line with the aimed purpose, parallel scenario shall be proposed to understand if results are due to the non-stationary system or any other theoretical assumptions or because of the application of the extreme value theory. Of course, future studies shall confirm the confidence in the satisfactory results, Gaussian distribution is avoided but excursions behaviour as known as overbounding in line with EVT application shall be well understood in terms of independency and peaks and shall confirm the first results from EVT-SIAM Tool. At least and hopefully, stringent performances such as integrity, continuity or LPV200 service accuracy wouldn’t be anymore “a pain in the neck” to demonstrate. ACKNOWLEDGMENTS TAS-F and UPS-IMT thank CNES for promoting EVT-SIAM Tool. This work has been carried out under contract #104118. The author wishes to thank TAS-F Tool Team (C. Germa and S. Auriol), CNES Tool Team (I. Bailly and M. Gesson) and SII Team for their involvement in EVT-SIAM Tool making, TAS-F M. Van-Den-Bossche and TAS-F B. Rols for their advices and reviews.

Page 8: EVT-SIAM: A tool based on Extreme-Value Theory for the …gadat/Papers/Tool_EVT_SIAM_V4.pdf · 2012-07-15 · EVT-SIAM: A tool based on Extreme-Value Theory for the assessment of

REFERENCES [1] J. Benedicto et al., EGNOS: the European Satellite

Based Augmentation to GPS and GLONASS, Proc. of the GNSS 98.

[2] D. Flament et al., EGNOS, the European Based Augmentation to GPS and GLONASS – mission and system architecture, Proc. of the GNSS 98.

[3] N. Suard et al, Performance Assessment of the Time Difference between EGNOS Network Time and UTC, ION-GNSS 2009, 22-25 Sep 2009

[4] JM Azaïs, S. Gadat et al, GNSS Integrity Achievement by using Extreme Value Theory (2009)

[5] Coles S., an Introduction to Statistical Modelling of Extreme Values, Springer-Verlag (2001)

[6] Rassmussen, P.F., et al., The POT method for flood estimation: a review, Stoc. And Stat. Meth. in Hydro. and Environ. Eng, (1994)

[7] http://www.ecacnav.com/Tools/PEGASUS [8] DO229C, Minimum Operational Performance

Standard for Global Positioning System/Satellite Based Augmentation System Airbone Equipment, Nov 2001

GLOSSARY CPF Central Processing Facilities EVT-SIAM Extreme Value Theory Supporting Integrity

and Accuracy Measurement GEO Navigation payload on a geostationary

satellite GIVDe Grid Ionosphere Vertical Delay error GIVE Grid Ionosphere Vertical Error ICD Interface Control Document IGP Ionosphere Grid Point IMT Institut de Mathématiques de Toulouse LOI Loss of Integrity LPV200 Localizer Performance with Vertical

guidance approaches with 200-foot PEGASUS tool enabling the position computation using

EGNOS in accordance to DO229C POT Pick Over Threshold PRN Pseudo Range Number – GEO identifier SBAS Satellite Based Augmentation System SIS Signal In Space SREW Satellite Range Error at the Worst user

location TAS-F Thales Alenia Space France TGD Timing Group Delay UDRE User Differential Range Error UPS University Paul Sabatier