D v. u Intermediate results of the tests - HeERO · D v. u Intermediate results of the tests Version number: ... 7.8.2 METHODOLOGY OF DATA ANALYSIS ... 7.9.5 RESULTS DRIVE TEST

15/05/2014

Information and Communications Technologies Policy Support Programme (the “ICT PSP”) Information Society and Media Directorate-General Grant agreement no.: 270906 Pilot type A

D4.3 Intermediate results of the tests

Version number: Version 1.3

Main author: NavCert GmbH, Germany

Dissemination level: PU

Lead contractor: ERTICO – ITS Europe

Due date:

Delivery date:

Delivery date updated document

Page left intentionally blank

D4.3 Intermediate test results

15/05/2014 3 Version 1.3

Control sheet

Version history

Version Date Main author Summary of changes

0.1 05.08.2012 Stefan Götte First draft version

0.2 12.09.2012 Stefan Götte internal input

0.3 30.09.2012 Stefan Götte after review

0.4 15.10.2012 Monika Hentschinski Input from Germany

0.5 19.10.2012 Monika Hentschinski Input chapter 4

0.6 22.10.2012 Renato Filjar Input for Croatia

0.7-0.9 30.10.2012-

5.11.12

Monika Hentschinski Input of received reports,

formatting, chapter 5.8.9

0.10 update

0.11 12.11. Monika Hentschinski Actualization chapter 5.4

and formats

0.12 22.11.12 Martin Grzebellus Finalization for review

0.13 26.11.12 Renato Filjar Peer review

0.14 02.12.12 Martin Grzebellus Management summary

0.14_gr2 05.12.12 Gunilla Rydberg Modifications in

Management Summary,

and clarifications in

report of the Swedish

tests.

0.15 07.12.12 Martin Grzebellus

0.16 21.12.12 Martin Grzebellus Ertico review included

1.1 12.4.13 Monika Hentschinski Update after comments

from Technical Review

1.2 14.02.2014 Monika Hentschinski Greek input

1.3 13.5.14 Monika Hentschinski Update after comments

from Technical Review


15/05/2014 4 Version 1.3

2014

Name Date

Prepared Stefan Götte, NavCert GmbH 05.08.2012

Reviewed Renato Filjar 26.11.2012

Authorized Andy Rooke 15.05.2014

Circulation

Recipient Date of submission

Project partners 24.5.2013 Re-Submitted 14.02.2014 and 15.05.2014

European Commission 24.05.2013 Re-Submitted 14.02.2014 and 15.05.2014


15/05/2014 5 Version 1.3

Table of contents

1 TERMS AND ABBREVIATIONS ................................................................................................................. 15

2 INTRODUCTION ..................................................................................................................................... 17

2.1 PURPOSE OF DOCUMENT...................................................................................................................... 17

2.2 STRUCTURE OF DOCUMENT .................................................................................................................. 17

2.3 HEERO CONTRACTUAL REFERENCES ..................................................................................................... 17

3 MANAGEMENT SUMMARY .................................................................................................................... 19

4 CONCLUSIONS ........................................................................................................................................ 21

5 RECOMMENDATIONS ............................................................................................................................ 23

6 VALIDATION OF RESULTS ....................................................................................................................... 25

6.1 BACKGROUND ....................................................................................................................................... 25

6.2 VALIDATION PROCEDURE ...................................................................................................................... 26

6.3 APPROACH PROPOSAL .......................................................................................................................... 26

6.4 DATA COLLECTION ................................................................................................................................ 27

QUALITY MANAGEMENT ............................................................................................................... 27 6.4.1

DATA COLLECTION PLAN ............................................................................................................... 27 6.4.2

6.5 RECORDING OF RESULTS ....................................................................................................................... 27

6.6 DATA ANALYSIS ..................................................................................................................................... 27

6.7 REPORTING ............................................................................................................................................ 29

6.8 WP4 ACTION PLAN ................................................................................................................................ 29

6.9 OVERVIEW OF KPIS ................................................................................................................................ 30

6.10 CONSOLIDATED KPIS ............................................................................................................................. 37

AUTOMATICALLY AND MANUALLY INITIATED ECALLS .................................................................. 37 6.10.1

AUTOMATICALLY INITIATED CALLS................................................................................................ 39 6.10.2

MANUALLY INITIATED ECALLS ....................................................................................................... 40 6.10.3

7 MEMBER STATES RESULTS ..................................................................................................................... 41

7.1 CROATIA ................................................................................................................................................ 41

PURPOSE AND SCOPE .................................................................................................................... 41 7.1.1

CROATIAN 2012 WP4 ACTION PLAN ............................................................................................. 41 7.1.2

METHODOLOGY OF DATA ANALYSIS ............................................................................................. 41 7.1.3

DESCRIPTION OF EQUIPMENT ....................................................................................................... 41 7.1.4

TESTING AND VALIDATION SCENARIOS DESCRIPTION .................................................................. 42 7.1.5

DESCRIPTION OF CONSOLIDATED PHASE I T&V DATA .................................................................. 42 7.1.6

EVALUATION RESULTS ................................................................................................................... 43 7.1.7


15/05/2014 6 Version 1.3

DISCUSSION OF EVALUATION RESULTS ......................................................................................... 61 7.1.8

CONCLUSION ................................................................................................................................. 63 7.1.9

RECOMMENDATIONS .................................................................................................................... 63 7.1.10

7.2 CZECH REPUBLIC ................................................................................................................................... 65

PURPOSE OF DOCUMENT .............................................................................................................. 65 7.2.1

TESTING INFRASTRUCTURE ........................................................................................................... 65 7.2.2

TESTING DESCRIPTION .................................................................................................................. 68 7.2.3

SUMMARY OF TEST RESULT .......................................................................................................... 70 7.2.4

CONCLUSIONS ............................................................................................................................... 82 7.2.5


7.3 FINLAND ................................................................................................................................................ 85

INTRODUCTION ............................................................................................................................. 85 7.3.1

HEERO KPI MEASUREMENT ........................................................................................................... 85 7.3.2

FINNISH ECALL PILOT SYSTEM ....................................................................................................... 88 7.3.3

ECALL TESTING – KEY PERFORMANCE INDICATORS ...................................................................... 90 7.3.4

ECALL TESTING – TESTING AND ANALYSIS METHODS ................................................................... 92 7.3.5

TEST RESULTS ................................................................................................................................ 93 7.3.6

CONCLUSIONS ............................................................................................................................... 99 7.3.7


REFERENCES ................................................................................................................................ 100 7.3.9

7.4 GERMANY ............................................................................................................................................ 101

TEST SCENARIOS .......................................................................................................................... 101 7.4.1

USED TEST TRACKS FOR ECALL TESTS .......................................................................................... 102 7.4.2

DETAILS OF USED IVS ................................................................................................................... 105 7.4.3

DETERMINATION OF KPIS OUT OF ECALL LOG FILES IN THE DATABASE ..................................... 105 7.4.4

RESULTS OF KPIS .......................................................................................................................... 106 7.4.5

EVALUATION OF MSD PRESENTATION TIME (KPI 5).................................................................... 107 7.4.6

EVALUATION OF VOICE CHANNEL BLOCKING TIME (KPI 007A) ................................................... 109 7.4.7

AUTOMATIC TEST RESULTS ......................................................................................................... 111 7.4.8

INTEROPERABILITY TESTS WITH CZ ............................................................................................. 112 7.4.9

CONCLUSIONS ............................................................................................................................. 113 7.4.10

RECOMMENDATIONS .................................................................................................................. 113 7.4.11

7.5 GREECE ................................................................................................................................................ 114

METHODOLOGY .......................................................................................................................... 114 7.5.1

RESULTS ....................................................................................................................................... 116 7.5.2


15/05/2014 7 Version 1.3

CONCLUSIONS ............................................................................................................................. 124 7.5.3


7.6 ITALY .................................................................................................................................................... 126

PRELIMINARY TESTS .................................................................................................................... 126 7.6.1

VARESE PILOT TESTS .................................................................................................................... 126 7.6.2

CONCLUSIONS ............................................................................................................................. 132 7.6.3


7.7 ROMANIA ............................................................................................................................................ 133

PURPOSE AND SCOPE .................................................................................................................. 133 7.7.1

STRUCTURE OF DOCUMENT ........................................................................................................ 133 7.7.2

DESCRIPTION OF EQUIPMENT ..................................................................................................... 133 7.7.3

MEASURED KPIS .......................................................................................................................... 133 7.7.4

TESTING AND VALIDATION SCENARIOS DESCRIPTION ................................................................ 136 7.7.5

TEST SESSION ANALYSIS .............................................................................................................. 137 7.7.6

CONCLUSION ............................................................................................................................... 159 7.7.7


7.8 SWEDEN .............................................................................................................................................. 161

GENERAL...................................................................................................................................... 161 7.8.1

METHODOLOGY OF DATA ANALYSIS ........................................................................................... 166 7.8.2

TESTING AND VALIDATION SCENARIOS DESCRIPTION ................................................................ 166 7.8.3

DESCRIPTION OF CONSOLIDATED PHASE 1 WP3 DATA ............................................................... 167 7.8.4

EVALUATION RESULTS ................................................................................................................. 167 7.8.5

DESCRIPTION OF THE EQUIPMENT .............................................................................................. 175 7.8.6

DISCUSSION OF EVALUATION RESULTS ....................................................................................... 176 7.8.7

CONCLUSIONS ............................................................................................................................. 178 7.8.8

CONCLUSIONS ............................................................................................................................. 179 7.8.9


REFERENCE .................................................................................................................................. 180 7.8.11

7.9 THE NETHERLANDS ............................................................................................................................. 182

IN GENERAL ................................................................................................................................. 182 7.9.1

METHODOLOGY .......................................................................................................................... 183 7.9.2

DESCRIPTION OF KPI’S ................................................................................................................. 187 7.9.3

DATA SELECTION ......................................................................................................................... 188 7.9.4

RESULTS DRIVE TEST .................................................................................................................... 191 7.9.5

CONCLUSIONS ............................................................................................................................. 198 7.9.6


15/05/2014 8 Version 1.3


APPENDIX A ................................................................................................................................. 200 7.9.8

8 REFERENCE .......................................................................................................................................... 205

9 ANNEX ................................................................................................................................................. 207

9.1 ANNEX I: CHANGE REQUESTS TO CEN ................................................................................................. 207


15/05/2014 9 Version 1.3

Figures

FIGURE 1: NUMBER OF MEASUREMENTS ............................................................................................................. 21

FIGURE 2: HISTOGRAM OF KPI 5 ........................................................................................................................... 22

FIGURE 3: HISTOGRAM OF KPI 7A ......................................................................................................................... 23

FIGURE 4: TIME SERIES OF KPI007A (LEFT) AND KPI008 (RIGHT) FOR L1 SCENARIO (HR) .................................... 44

FIGURE 5: HISTOGRAMS OF KPI007A (LEFT) AND KPI008 (RIGHT) FOR L1 SCENARIO (HR) .................................. 44

FIGURE 6: CORRELOGRAMS OF KPI007A (LEFT) AND KPI008 (RIGHT) FOR L1 SCENARIO (HR) ............................. 45

FIGURE 7: CROSS-CORRELOGRAM FOR L1 SCENARIO (HR) ................................................................................... 46

FIGURE 8: TIME SERIES OF KPI007A (LEFT) AND KPI008 (RIGHT) FOR L2 SCENARIO (HR) .................................... 47

FIGURE 9: HISTOGRAMS OF KPI007A (LEFT) AND KPI008 (RIGHT) FOR L2 SCENARIO (HR) .................................. 47

FIGURE 10: CORRELOGRAMS OF KPI007A (LEFT) AND KPI008 (RIGHT) FOR L2 SCENARIO (HR) ........................... 48

FIGURE 11: CROSS-CORRELOGRAM FOR L2 SCENARIO (HR) ................................................................................. 48

FIGURE 12: TIME SERIES OF KPI007A (LEFT) AND KPI008 (RIGHT) FOR L3 SCENARIO (HR) .................................. 49

FIGURE 13:HISTOGRAMS OF KPI007A (LEFT) AND KPI008 (RIGHT) FOR L3 SCENARIO (HR) ................................. 50

FIGURE 14: CORRELORGRAMS OF KPI007A (LEFT) AND KPI008 (RIGHT) FOR L3 SCENARIO (HR) ......................... 50

FIGURE 15: CROSS-CORRELOGRAM FOR L3 SCENARIO (HR) ................................................................................. 51

FIGURE 16: TIME SERIES OF KPI007A (LEFT) AND KPI008 (RIGHT) FOR R1 SCENARIO (HR) .................................. 52

FIGURE 17: HISTOGRAMS OF KPI007A (LEFT) AND KPI008 (RIGHT) FOR R1 SCENARIO (HR) ................................ 52

FIGURE 18: CORRELOGRAMS OF KPI007A (LEFT) AND KPI008 (RIGHT) FOR R1 SCENARIO (HR) .......................... 53

FIGURE 19: CROSS-CORRELATION FOR R1 SCENARIO (HR) ................................................................................... 53




FIGURE 23: CROSS-CORRELOGRAM FOR R2 SCENARIO (HR) ................................................................................ 56









FIGURE 32: ECALL ARCHITECTURE (CZ) ................................................................................................................. 66

FIGURE 33: DIFFERENT TIMERS AND MAIN KPIS (CZ) ........................................................................................... 72

FIGURE 34 KPI_008: TIME FOR CALL ESTABLISHMENT – IVS SHERLOG TRACELL (CZ) .......................................... 74

FIGURE 35 KPI_023: GSM NETWORK LATENCY– IVS SHERLOG TRACELL (CZ) ....................................................... 74

FIGURE 36 KPI_024: 112 NATIONAL NETWORK LATENCY– IVS SHERLOG TRACELL (CZ) ....................................... 75

FIGURE 37 KPI_007: DURATION OF VOICE CHANNEL BLOCKING– IVS SHERLOG TRACELL (CZ) ............................ 75

FIGURE 38: KPI_005: DURATION UNTIL MSD IS PRESENTED IN PSAP– IVS SHERLOG TRACELL (CZ) ..................... 76

FIGURE 39: IVS SHERLOG TRACE ECALL HANDLING TIME - BASIC PHASES [SEC] (CZ) ........................................... 77

FIGURE 40: KPI_008: TIME FOR CALL ESTABLISHMENT- IVS TELEMATIX (CZ) ....................................................... 78

FIGURE 41 KPI_023: GSM NETWORK LATENCY- IVS TELEMATIX (CZ) ................................................................... 79

FIGURE 42 KPI_024: 112 NATIONAL NETWORK LATENCY- IVS TELEMATIX (CZ) ................................................... 79

FIGURE 43 KPI_007: DURATION OF VOICE CHANNEL BLOCKING- IVS TELEMATIX (CZ) ........................................ 80

FIGURE 44: KPI_005: DURATION UNTIL MSD IS PRESENTED IN PSAP - IVS TELEMATIX (CZ) ................................ 80

FIGURE 45: IVS TELEMATIX ECALL HANDLING TIME - BASIS PHASES [SEC] (CZ) ................................................... 81


15/05/2014 10 Version 1.3

FIGURE 46 HEERO FINNISH PILOT SYSTEM ARCHITECTURE OUTLINE (FI) ............................................................. 88

FIGURE 47: GECKO IVS PROTOTYPE (FI) ................................................................................................................ 89

FIGURE 48: INDAGON IVS PROTOTYPE (FI) ........................................................................................................... 89

FIGURE 49: PROCESSING OF TEST RESULTS (FI) .................................................................................................... 93

FIGURE 50: TEST TRACK 1 (DE) ............................................................................................................................ 103

FIGURE 51: TEST TRACK 2 (DE) ............................................................................................................................ 104

FIGURE 52 TEST TRACK 3 (DE) ............................................................................................................................. 105

FIGURE 53: DISTRIBUTION OF THE MSD PRESENTATION TIME WITH DIFFERENT IVS (DE) ................................ 107

FIGURE 54: HISTOGRAMS OF THE MSD PRESENTATION TIME WITH DIFFERENT IVS (DE) ................................. 108

FIGURE 55: DISTRIBUTION OF THE VOICE CHANNEL BLOCKING TIME WITH DIFFERENT PROVIDERS (DE) ........ 109

FIGURE 56: HISTOGRAMS OF VOICE CHANNEL BLOCKING TIME WITH DIFFERENT IVS (DE) .............................. 110

FIGURE 57: HISTOGRAM OF KPI 5 WITH CZECH IVS (DE) .................................................................................... 112

FIGURE 58: HISTOGRAM OF KPI 7 WITH CZECH IVS (DE) .................................................................................... 112

FIGURE 59: NUMBER OF ECALLS PER DAY AND REGION (GR) ............................................................................. 115

FIGURE 60: TABLES TO EXTRACT THE KPIS IN AN MS SQL SERVER (GR) ............................................................. 115

FIGURE 61: NUMBER OF ECALLS WITH SIMULATED AUTOMATIC INITIATION (KPI_001A) (GR) ......................... 116

FIGURE 62: NUMBER OF ECALLS WITH SIMULATED AUTOMATIC INITIATION PER MNO (KPI_001A) (GR) ........ 116

FIGURE 63: NUMBER OF ECALLS WITH MANUAL INITIATION (KPI_001B) (GR) .................................................. 117

FIGURE 64: NUMBER OF ECALLS WITH MANUAL INITIATION PER MNO (KPI_001B) (GR) .................................. 117

FIGURE 65: SUCCESS RATE OF COMPLETED ECALLS USING LONG NUMBER (KPI_002B) (GR) ............................ 118

FIGURE 66: SUCCESS RATE OF COMPLETED ECALLS USING LONG NUMBER BY MNO (KPI_002B) (GR) ............. 118

FIGURE 67: SUCCESS RATE OF RECEIVED MSDS (KPI_003) (GR) ......................................................................... 119

FIGURE 68: SUCCESS RATE OF RECEIVED MSDS PER MNO (KPI_003) (GR) ......................................................... 119

FIGURE 69: SUCCESS RATE OF CORRECT MSDS (KPI_004) (GR) .......................................................................... 120

FIGURE 70: SUCCESS RATE OF CORRECT MSDS PER MNO (KPI_004) (GR) .......................................................... 120

FIGURE 71: SUCCESS RATE OF ESTABLISHED VOICE TRANSMISSIONS (KPI_006) (GR) ....................................... 121

FIGURE 72: SUCCESS RATE OF ESTABLISHED VOICE TRANSMISSIONS PER PROVIDER (KPI_006) (GR) ............... 121

FIGURE 73: SUCCESS RATE OF VIN DECODING WITHOUT EUCARIS (KPI_014) (GR) ........................................... 122

FIGURE 74: SUCCESS RATE OF VIN DECODING WITHOUT EUCARIS PER MNO (KPI_014) (GR) ......................... 122

FIGURE 75: NUMBER OF SUCCESSFUL CALLBACKS (KPI_021) (GR) ..................................................................... 123

FIGURE 76: NUMBER OF SUCCESSFUL CALL-BACKS PER MNO (KPI_021) (GR) ................................................... 123

FIGURE 77: SUCCESS RATE OF CALLBACKS (KPI_022) (GR) ................................................................................. 124

FIGURE 78: SUCCESS RATE OF CALL-BACKS PER MNO (KPI_022) (GR) ................................................................ 124

FIGURE 79: ITALIAN PILOT IN VARESE AREA (IT) ................................................................................................. 127

FIGURE 80: PSAP LOG FILE(IT) ............................................................................................................................. 128

FIGURE 81: IVS1 DISTRIBUTION (IT) .................................................................................................................... 129





FIGURE 86: TIMESTAMPS MEASURED IN THE ROMANIAN PILOT SITE (RO) ....................................................... 135

FIGURE 87: RESULTS FOR KPI_007 DURING SESSION L1 (RO) ............................................................................. 138

FIGURE 88: RESULTS FOR KPI_023 DURING SESSION L2 (RO) ............................................................................. 139

FIGURE 89 RESULTS FOR KPI_024 DURING SESSION L2 (RO) .............................................................................. 139

FIGURE 90: RESULTS FOR KPI_007 DURING SESSION R1 (RO) ............................................................................ 142




15/05/2014 11 Version 1.3



FIGURE 95 RESULTS FOR KPI_007 DURING SESSION R3 (RO) ............................................................................. 145




FIGURE 99 RESULTS FOR KPI_005 DURING SESSION R7(RO) .............................................................................. 149

FIGURE 100: TIMELINE IVS R&S TOPEX (RO) ....................................................................................................... 149

FIGURE 101: TEST ROUTE SESSION R9 (RO) ........................................................................................................ 152

FIGURE 102: TEST ROUTE SESSION R10 (RO) ...................................................................................................... 152






FIGURE 108: TIMELINE IVS CIVITRONIC (RO) ...................................................................................................... 158

FIGURE 109: R1 TIME SERIES OF KPI_005 AND KPI_007A (SE) ............................................................................ 168

FIGURE 110: R1 HISTOGRAM OF KPI_005 AND KPI_007A (SE) ........................................................................... 168











FIGURE 121: R3 SCENARIO TIME SERIES ANALYSIS (SE) ...................................................................................... 177

FIGURE 122: THE SKI-BOX ATTACHED TO A TEST-VEHICLE (LEFT) AND D-FACTS INSIDE THE SKIBOX (LEFT)(NL) 183

FIGURE 123: THE ROUTE OF THE DRIVE TEST (NL) .............................................................................................. 185

FIGURE 124: THE PROCESS OF LINKING THE LOGGINGS OF THE SYSTEMS (NL) ................................................. 186

FIGURE 125: OVERVIEW OF THE DATA USED FOR ANALYSING THE KPIS (NL) .................................................... 189

FIGURE 126: OVERVIEW OF THE SUBSETS USED FOR ANALYSING THE KPI’S (NL) .............................................. 191


15/05/2014 12 Version 1.3

Tables

TABLE 1: PERFORMANCE PARAMETER EXAMPLES ............................................................................................... 26

TABLE 2: RECOMMENDATIONS OF SPECIFIC KPIS ................................................................................................. 32

TABLE 3: TESTING OF RECOMMENDED KPIS IN PHASE 1 ...................................................................................... 34

TABLE 4: KPIS TO BE EVALUATED WITHIN THE MEMBER STATES ......................................................................... 36

TABLE 5: STATUS OF RECEIVED INPUTS ................................................................................................................ 37

TABLE 6: EVALUATED RESULTS ............................................................................................................................. 38

TABLE 7: EVALUATED RESULTS OF ALL AUTOMATICALLY INITIATED ECALLS........................................................ 39

TABLE 8: EVALUATED RESULTS OF ALL MANUALLY INITIATED ECALLS ................................................................. 40

TABLE 9: DESCRIPTION OF EQUIPMENT (HR) ........................................................................................................ 42

TABLE 10: CROATIAN LABORATORY ECALL T&V SCENARIOS (HR) ........................................................................ 42

TABLE 11: CROATIAN ECALL COMMUNICATIONS T&V SCENARIOS (HR) .............................................................. 42

TABLE 12: CROATIAN ECALL T&V SUMMARY (HR) ................................................................................................ 43

TABLE 13: L1 STATISTICAL RESULTS SUMMARY (HR) ............................................................................................ 43

TABLE 14:L2 STATISTICAL RESULTS SUMMARY (HR) ............................................................................................. 46

TABLE 15: L3 STATISTICAL RESULTS SUMMARY (HR) ............................................................................................ 49

TABLE 16: R1 STATISTICAL RESULTS SUMMARY (HR) ........................................................................................... 51




TABLE 20: TECHNICAL SPECIFICATION .................................................................................................................. 67

TABLE 21: LIST OF TESTED KPIS (CZ) ...................................................................................................................... 71

TABLE 22: LIST OF KPI RESULTS WITH THEIR STATISTICAL EVALUATION – IVS SHERLOG TRACELL (CZ) ............... 73

TABLE 23: AVERAGE TIMING VALUES OF SIGNIFICANT ECALL PHASES – IVS SHERLOG TRACELL (CZ) .................. 76

TABLE 24: OF KPI RESULTS WITH THEIR STATISTICAL EVALUATION - IVS TELEMATIX (CZ) ................................... 77

TABLE 25: AVERAGE TIMING OF SIGNIFICANT ECALL PHASES- IVS TELEMATIX (CZ) ............................................. 81

TABLE 26: KEY PERFORMANCE INDICATORS PLANNED FOR EVALUATION OF FINNISH ECALL ............................. 91

TABLE 27: KPIS MEASURED IN PHASE 1 IN FINNISH ECALL PILOT (FI) ................................................................... 92

TABLE 28: DESCRIPTION OF TEST ROUTE (FI) ........................................................................................................ 93

TABLE 29: SUMMARY OF TEST RESULTS FOR GECKO IVS PROTOTYPE (FI) ........................................................... 94

TABLE 30: SUMMARY OF TEST RESULTS FOR GECKO IVS PROTOTYPE (FI) ........................................................... 94

TABLE 31: SUMMARY OF ACTIVATED ECALLS, STARTED MSDS AND SUCCESSFUL MSDS, GECKO PROTOTYPE (FI)

..................................................................................................................................................................... 95

TABLE 32:.SUMMARY OF TEST RESULTS FOR INDAGON IVS PROTOTYPE (FI) ....................................................... 96

TABLE 33: SUMMARY OF TEST RESULTS FOR INDAGON IVS PROTOTYPE(FI) ........................................................ 97

TABLE 34: SUMMARY OF ACTIVATED ECALLS, STARTED MSDS AND SUCCESSFUL MSDS, INDAGON PROTOTYPE

(FI) ................................................................................................................................................................ 97

TABLE 35: CONFIDENCE INTERVALS FOR SUCCESS RATE OF RECEIVED MSDS (KPI_003), GECKO PROTOTYPE (FI)

..................................................................................................................................................................... 98

TABLE 36: CONFIDENCE INTERVALS FOR SUCCESS RATE OF RECEIVED MSDS (KPI_003), INDAGON PROTOTYPE.

(FI) ................................................................................................................................................................ 98

TABLE 37: KPIS TO BE EVALUATED (DE) .............................................................................................................. 101

TABLE 38: MAPPING OF TEST SCENARIOS TO KPI (DE) ....................................................................................... 102

TABLE 39: DETAILS OF IVS CONTINENTAL (DE) ................................................................................................... 105

TABLE 40: DETAILS OF IVS S1NN (DE) .................................................................................................................. 105

TABLE 41: RESULTS OF THE GERMAN KPIS ......................................................................................................... 107


15/05/2014 13 Version 1.3

TABLE 42: STATISTICAL EVALUATION OF MSD PRESENTATION TIME (DE) ......................................................... 109

TABLE 43: STATISTICAL EVALUATION OF VOICE CHANNEL BLOCKING TIME (DE)............................................... 111

TABLE 44: AUTOMATIC TEST RESULTS S1NN AND CONTINENTAL (DE) .............................................................. 111

TABLE 45: KPIS MEASURED IN THE GREEK PILOT TESTS (GR) ............................................................................. 114

TABLE 46: EQUIPMENT USED IN THE ROMANIAN PILOT SITE (RO)..................................................................... 133

TABLE 47: KPIS MEASURED IN THE ROMANIAN PILOT SITE (RO) ........................................................................ 134

TABLE 48: OVERVIEW OF LABORATORY TESTS (RO) ........................................................................................... 136

TABLE 49: OVERVIEW OF FIELD TESTS (RO) ........................................................................................................ 137

TABLE 50: OVERVIEW OF TMC TESTS (RO) .......................................................................................................... 137

TABLE 51: ROMANIAN RESULTS FOR KPI_007 DURING SESSION L1 ................................................................... 137

TABLE 52: RESULTS FOR KPI_003 AND KPI_004 DURING SESSION L3 (RO) ......................................................... 140



TABLE 55: TEST RESULTS DURING SESSION R1 (RO) ........................................................................................... 141



TABLE 58: ROMANIAN TEST RESULTS DURING SESSION R4 ................................................................................ 145




TABLE 62: IVS R&S TOPEX ONE-WAY ANOVA (RO) ............................................................................................. 150

TABLE 63: DESCRIPTIVE STATISTICS IVS R&S TOPEX (RO) ................................................................................... 151



TABLE 66: TEST RESULTS DURING SESSION R10 (RO) ......................................................................................... 153






TABLE 72: IVS CIVITRONIC ONE WAY ANOVA (RO) ............................................................................................. 158

TABLE 73: DESCRIPTIVE STATISTICS IVS CIVITRONIC (RO)................................................................................... 159

TABLE 74: KPI DISCREPANCY VERSUS D4.1 (SE) .................................................................................................. 165

TABLE 75: TEST SCENARIOS DESCRIPTION (SE) ................................................................................................... 167

TABLE 76: CONSOLIDATED PHASE 1 WP3 DATA (SE) .......................................................................................... 167

TABLE 77: R1 SCENARIO KPI STATISTICS (SE) ...................................................................................................... 167






TABLE 83: INCLUDED KPIS IN THE DRIVE TESTS (NL) ........................................................................................... 183

TABLE 84: COMBINATIONS OF MNOS AND IVSS TESTED DURING THE DRIVE TEST (NL) .................................... 184

TABLE 85: RELEVANT LOGGING PER SYSTEM (NL) .............................................................................................. 185

TABLE 86: OVERALL RESULTS OF THE DRIVE TEST(NL) ........................................................................................ 192

TABLE 87: RESULTS OF THE DRIVE TEST PER IVS (NL) ......................................................................................... 194

TABLE 88: RESULTS OF THE DRIVE TEST PER MNO FOR ALL IVSS (NL) ................................................................ 195


15/05/2014 14 Version 1.3

TABLE 89: NUMBER OF TESTS DISTRIBUTED PER MNO AND PER IVS (NL) ......................................................... 195

TABLE 90: RESULTS OF THE DRIVE TEST PER MNO FOR IVS 1 AND 2(NL) ........................................................... 196

TABLE 91: RESULTS OF THE DRIVE TEST PER AREA-TYPE (NL) ............................................................................. 197

TABLE 92: RESULTS OF THE DRIVE TEST PER ENCOUNTERED OBJECT (NL) ......................................................... 198

TABLE 93: OUTLIERS KPI_009 (NL) ...................................................................................................................... 204


15/05/2014 15 Version 1.3

1 Terms and abbreviations Abbreviation

Definition

3GPP Third Generation Partnership Project

CAN Controller Area Network

CEN Comité Européen de Normalisation

CIP Competitiveness and Innovation Framework Programme

DoW Description of Work

EC European Commission

EGNOS European Geostationary Navigation Overlay System

ENT Ericsson Nikola Tesla

ETSI European Telecommunication Standards Institute

EUCARIS EUropean CAR and driving License Information System

ESO European Standards Organization

GDOP Geometric dilution of precision

GIS Geographic Information System

GLONASS

Globalnaja Nawigazionnaja Sputnikowaja Sistema

GNSS Global Navigation Satellite System

GPS Global Positioning System

GPRS General Packet Radio System

GSM Global System of Mobile telecommunications

ISO International Standardization Organization

IVS In-Vehicle System

KPI Key Performance Indicator

MNO Mobile Network Operator

MSD Minimum Set of Data

MSISDN Mobile Subscriber Integrated Services Digital Network Number


15/05/2014 16 Version 1.3

NIST National Institute of Standards and Technology

NMEA National Marine Electronics Association

PLMN Public Land Mobile Network

PSAP Public Safety Answering Point

SBAS Satellite Based Augmentation System

SIM Subscriber Identity Module

TMC Traffic Management Centre

UMTS Universal Mobile Telecommunication System

USB Universal Serial Bus

VAS Value Added Services

VIN Vehicle Identification Number

Term Definition

Process The method of operation in any particular stage of development of the material part,

component or assembly involved.


15/05/2014 17 Version 1.3

2 Introduction

2.1 Purpose of Document This document D4.3 presents the test results from the various pilot sites. Fine-tuning of the

remaining tests will be possible based on this information. In addition, this document presents the

preliminary results of the eCall service in Phase I, evaluating performance and operation in the

respective technology and business environments in which eCall is expected to be deployed. Data

was collected and processed as defined in D4.1 and D4.2 to allow evaluation of the results from all

pilot sites in a common way. Also this document provides a first overview of the

Key Performance Indicators (KPIs) as measured in the different eCall implementations at the various

pilot sites. In addition to the national experience with lessons learnt, it allows to compare measured

KPIs across the sites. This is possible due to the fact that identical test scenarios and test

methodologies were implemented at all pilot sites.

2.2 Structure of Document This document is structured into three main sections, one describing the agreed procedure on how

collected data shall be evaluated and processed, and the other two containing actual results with

recommendations on how to improve the outcome. The first section introduces the results on

pan-European level derived from and valid for all pilot sites. The other two sections describe the

individual findings at the various pilot sites.

2.3 HeERO Contractual References HeERO (Harmonised eCall European Pilot) is a Pilot type A of the ICT Policy Support Programme (ICT

PSP), Competitiveness and Innovation Framework Programme (CIP).

The Grant Agreement number is 270906 and project duration is 36 months, effective from 01 January

2011 until 31 December 2013. It is a contract with the European Commission, DG INFSO.

The principal EC Project Officer is:

Aude ZIMMERMANN

EUROPEAN COMMISSION

DG CONNECT

Office: BU 31 – 6/35

B - 1049 Brussels

Tel: +32 296 2188

E-mail: [email protected]

One other Project Officer will follow the HeERO project:

mailto:[email protected]


15/05/2014 18 Version 1.3

Dimitrios AXIOTIS

[email protected]

Address to which all deliverables and reports have to be sent:

Aude ZIMMERMANN

EUROPEAN COMMISSION

DG CONNECT BU 31 – 6/35 B - 1049 Brussels Tel: +32 296 2188

By mail: [email protected]

Any communication or request concerning the grant agreement shall identify the grant agreement

number, the nature and details of the request or communication and be submitted to the following

addresses:

European Commission

DG CONNECT B-1049 Brussels

Belgium

By electronic mail:

[email protected]

mailto:[email protected]


15/05/2014 19 Version 1.3

3 Management Summary The purpose of this document is to present the test results from all pilot sites in a concise manner to

allow fine-tuning of the remaining tests. The evaluation of performance and operation generated

preliminary results for each pilot site. Data was collected and processed as defined in D4.1 and D4.2

so that now the results of all pilot sites can be evaluated in a common way. Key Performance

Indicators (KPIs) were measured in the different eCall implementations at the various sites applying

identical test scenarios with common test methodologies. Measured KPIs can now be compared

across pilot sites.

The overall evaluation is based on results of 9 pilot sites (Croatia, Czech Republic, Finland, Greece,

Italy, Germany, Romania, Sweden and The Netherlands). Each pilot site was requested to provide

statistical evaluations of the measured KPIs, recommendations and conclusions.

Between the pilot sites the number of initiated calls varied between a few hundred and ten

thousands of calls. One major requirement for the execution of tests was to initiate calls not only

from moving vehicles but also from non-moving vehicles to provide realistic scenarios for initiation of

an eCall, e.g. for cars that are stopped after a crash. These two scenarios are identified as “automatic

test” for moving vehicles and “manual test” for non-moving vehicles completely independent of the

eCall flag in call set up.

During the preparation phase all pilot sites proposed individual KPIs, so that in total more than 30

KPIs were defined. One of the most important KPIs operationally is KPI 018 (time to activate rescue

forces), as the overall objective for the introduction of eCall is to reduce the alert time for rescue

forces and to allow them to arrive earlier at the location of an incident. Typically this value might not

be available as no real PSAP is used but only “test PSAP”. Two pilot sites have indicated to measure

this KPI. However as this KPI is dependent also on human interaction like behaviour of the call taker

in the PSAP, from an operating aspect the most important KPIs are KPI 005 (duration until MSD is

presented in PSAP) and KPI07a (voice channel blocking time). ECall provides additional information in

comparison to the 112 call and this information is transmitted in the MSD, therefore the duration of

the transmission is very important. KPIs 005 and 007a have been measured and evaluated by 6 pilot

sites.

The KPIs measured by the majority of pilot sites are the various success rates KPI 002a (success rate

of completed eCalls using 112), KPI 002b (success rate of completed eCalls using long number), KPI

003 (success rate of received MSDs), KPI 004 (success rate of correct MSDs) and KPI 006 (success rate

of established voice transmissions). Unfortunately not all pilot sites were able to evaluate a 112 call

set up when the eCall flag was not supported by the MNO or due to other restriction in the test set

up.

The success rate is evaluated taking into account the different set ups (112 vs. long number and

automated vs. manual): The highest success rate is for 112 with a median of 94% versus 71% for long

number. The success rate for successful MSD transmission KPI 003 and 004 is very good with a mean

of 90% or above.


15/05/2014 20 Version 1.3

KPI 005 (duration until MSD is presented in PSAP) varies between 8s and 30s with a weighted mean

of 13s. A value of around 10s seems to be reasonable and feasible with some optimizations. KPI 007a

(voice channel blocking time) is between 5s and 16s with a weighted mean of 7.5s. The proposed

changes in the High Level Application Protocol (HLAP) will further decrease the upper value by 1s to 2

s.

In Phase 2 emphasis should be put on the analysis of the value of KPI 007a which is currently

between 3s and 6s for two pilot sites and between 6s and 9s for three other pilot sites. The overall

goal should be to optimize all components to achieve values in the range of 3s to 5s for this KPI.

To further improve evaluation of measured KPIs in phase 2 all pilot sites should apply a reasonable

number of test sets, minimum a few hundred, so that statistically more relevant conclusions can be

derived. In addition the recommendations of chapter 5 should be implemented by all pilot sites.


15/05/2014 21 Version 1.3

4 Conclusions In this chapter the overall conclusions are provided as derived from all pilot sites. Individual

conclusions per pilot site are provided in chapter 7.

Figure 1 shows the number of measurements per pilot site. The large number of manually initiated

calls both in the Netherlands and the Czech Republic and the large number of automatically initiated

eCalls in Germany are remarkable.

Figure 1: number of measurements

The duration until the MSD is presented in a PSAP (KPI 5) ranges between 9 and 30 s. The mean is

about 17 s with a standard deviation of 7. Romania uses different procedures than the other

countries resulting into a longer MSD Presentation and Voice Channel Blocking time.


15/05/2014 22 Version 1.3

Figure 2: Histogram of KPI 5

The mean of the voice channel blocking time (KPI 7a) is 8 s with a standard deviation of 3.7 s. Figure 3

illustrates that the majority of countries have a voice channel blocking time of less than 9 s. The time

mainly depends on the quality of the connection and on the interaction between IVS and PSAP. The

quality of the connection cannot be affected and therefore the main focus for optimization is the

interaction between PSAP and IVS.


15/05/2014 23 Version 1.3

Figure 3: Histogram of KPI 7a

The interaction between PSAP and IVS is mainly described by the HLAP protocol. Detailed analysis

resulted into several recommendations for optimization. The change requests to CEN have been

attached hereto as “Annex I: change requests to CEN”.

In any case it is strongly recommended to implement proposed optimizations like “PULL Mode by

PSAP” to evaluate the influence on KPI 07a. In addition, IVS implementation and hardware have a

considerable effect on KPI 07 as was seen comparing the values for KPI 07a for different IVS with the

same PSAP. It is recommended for phase 2 to conduct an in-depth analysis of the different results to

guide IVS manufacturers.

The low success rates and the large number of blocked voice channels result from the initial

challenges of setting up both laboratory and real-network test environments with in many cases

prototypes for IVS and PSAP.

In the following the recommendations of the pilot sites are summarized:

5 Recommendations

In this chapter overall recommendations derived from all pilot sites are provided. Individual

recommendations for each pilot site are provided in chapter 7.

The following measures are recommended for Phase 2 by the pilot sites:


15/05/2014 24 Version 1.3

(Better) synchronization with a common time framework (supposedly controlled either by

GPS/GNSS or by internet-accessible atomic clock-based time synchronization services) of

timers in PSAP and IVS to increase reliability and comparability of observed results

More detailed analysis based on new dedicated test sets of identified anomalies (both IVS

and PSAP related)

Evaluation of inconsistencies or other problems in encoding of MSD

Validation that if a busy tone is received, a redial shall be initiated; the system must not

‘hang up’ i.e. stop the call

Discussion on proper content of resent MSD (same as already sent or modified one with

updated information) with eventually comment to standards

Validation that if an eCall session is terminated (with clear down command), that there is

always a possibility to call-back the IVS.

More interoperability tests both with long numbers and with roaming within pilot sites

Analysis of all critical time consuming phases like building MSD within IVS to optimize voice

channel blocking time

A distinction should be made between problems specifically related to eCall and more

general problems (like i.e. bad coverage) not specific to eCall.

(More) tests of the eCall performance for dormant IVSs.

Interoperability testing with Russia and other non-EU partners

Test results have led to changes in modems and application. A 2nd round of drive-tests and

scenario tests is needed, with a larger number of calls.

Interoperability testing shall be executed to show whether performance is merely a national

issue or if there will be implications for other pilot countries as well

Evaluate the impact of the speed of the IVS. So far tests were performed with vehicles driving

at high speed. This scenario is relevant for manually triggered eCalls, but is untypical for

automatically triggered eCalls. To understand the impact of speed mainly tests with standing

or slow moving IVS should be performed.

The definition of additional data in the standard MSD needs to be changed to make it usable.

This will also reduce the risk that future implementations of the “spare” bits in the standard

MSD could have negative impact on the deciphering of the standard MSD.

The analysis of test results implies that EN 16062 should be modified to improve

performance. To reduce disturbances on the line imposed by algorithms for improvement of

voice transmission, echo cancellation should optionally be used. The timers’ t3 and t5 should

be increased to allow higher success rate of data transmission. The duration of the data

transmission can significantly be reduced by allowing a PULL mode on the PSAP side. The

standard should be modified accordingly

In phase II both IVS and PSAP should implement the proposed changes in the standard

independent of the update status of the standard. Therefore the proposed changes are

attached in Annex I: change requests to CEN


15/05/2014 25 Version 1.3

6 Validation of results The validation of results is performed taking into account the various preconditions. The experiment

is divided into sets. All measurements within one set are expected to have the same base, e .g. same

HW and SW for the In-vehicle-system/on-board-unit and for the PSAP system. Typically a new release

of software or firmware will imply starting a new set of measurements. Therefore it is not necessary

to document all preconditions for each sample test; it is sufficient to set preconditions for a set of

tests. The benefit of this differentiation is a clear picture, i.e. one is able to see if a newer version of

hardware, software of firmware influences the overall achieved results and it hopefully provides

proof if intended improvements are achieved.

6.1 Background Method validation is a well-established procedure in analytical science. Methodology of statistical

data analysis may be seen as a compilation of historical achievements from various fields of science

and engineering, including, but not limited to: Chemical engineering, medicine, astrophysics, quality

control etc. Those contributions are now considered the foundation framework for a

scientific/research work. The result of the test determines to what extent a measurement method is

suitable for the intended application, that it is “fit for purpose”.

The statistical framework for the validation of measurement methods is laid out in ISO-5725,

“Accuracy (trueness and precision) of measurement methods and results”. The approach aims in

determining estimates for a series of parameters1, which enable a user to determine if a certain

system is suitable or not for the application.

Precision: relates to the spread of experimentally determined data. The smaller the data

variation, the greater the precision of the analytical method. In parcel area

measurement, precision relates to the width of the confidence interval;

Bias: Error systematically occurring during the measurement. Bias can result from a lack

of calibration, is constant within a method and thus should be predictable. For a method

to be useful, the avoidance of bias through the use correct measurement protocols

needs to be ensured.

Accuracy: defined as a measure of the difference between the "true value" and a set of

experimentally determined data, and is affected by both systematic error (bias) and

random error (precision).

Repeatability: the standard deviation of a series of quantitative measurements

performed with the same method and sample under similar conditions (e.g. instrument,

analyst, etc.) over a short period of time.

1 Definitions taken from http://www.vam.org.uk/biomeasurement/biomeasurement_quality.asp


15/05/2014 26 Version 1.3

Reproducibility: the standard deviation of a series of quantitative measurements

performed with the same method and sample under different conditions (e.g. different

instrument, analyst, laboratory etc.) over a long period of time.

Range: the useful range for which measurements can be made.

Robustness: sometimes called ruggedness, is the measure of a method’s capacity to

remain unaffected by small variations in the conditions of use.

Examples of these definitions of performance parameters, as applied to parcel measurement

techniques, are given in Table 1.

Term Example

Precision Can be described as the range of values that might occur with a certain level of

probability, for example a buffer calculated from the Standard Deviation or

RMSE of differences between a reference area and measured areas.

Bias The instrument should, when following the standardized measurement

protocol, produce a result that is on average very close to the expected result,

and not consistently larger or smaller.

Repeatability The variability of a value estimate if it was measured in the same set in quick

succession.

Reproducibility The variability of a value area estimate if it was measured by a different set, on

different occasions.

Range In terms of values, the minimum and maximum sizes that can be measured and

achieve a certain level of accuracy

Robustness The sensitivity of an instrument to various extraneous effects, such as battery

low power conditions, rain, tree cover, electric power cables, satellite

constellation changes etc.

Table 1: Performance parameter examples

6.2 Validation procedure A number of approaches are presented in ISO-5725, the simplest of which is a balanced design

involving independent sets of measurements of reference object(s). Each set of measurements must

include a number (at least two, preferably four) of repeated measurements of the objects.

6.3 Approach proposal The basic design of the test consists of:

Collection of preliminary, pre-validation data.

Analysis of these data and possible simulations, design of the main validation trial.

Finalization of the operating protocol.


15/05/2014 27 Version 1.3

Collection of independent (with respect to date) sets of measurements, for example at

different times of the day over several days.

In each set of measurements, repetitions will be made; the number of repetitions should

be determined at the beginning of the trial and be fixed.

All results, including those which are determined by the operator to contain gross

(blunder) errors will be recorded.

Statistical processing of the data collected will be performed in order to assess the

quality of performance of the receiver.

6.4 Data collection

Quality Management 6.4.1

The data (i.e. test result data) collection and documentation process should be in accordance with

the ISO/IEC 17025 Quality Management System Model, or equivalent.

Data collection plan 6.4.2

In case of GNSS measurements, different measurement sets should be collected with different

satellite constellations (e.g. at different times of the day).

Data within one measurement set should be collected in the shortest time possible. This way the

stability of the GNSS satellite constellation can be assumed to be within a set of measurements.

6.5 Recording of results Results should be recorded electronically and include a full set of data normally available at the end

of the measurement.

Typically, the following parameters should be recorded:

ID of vehicle

ID of IVS

ID of the measurement (from experimental design protocol)

Date, time of measurement

Operator identification

KPI to be measured

Other observations, anomalies

6.6 Data analysis The results should be evaluated using the analysis of variance procedures defined in ISO 5725 in

order to identify the significant factors influencing the results. Outlier measurements will be

determined using the defined procedures, namely Grubbs and Cochran outlier tests. The number of

excluded measurements sets will be limited to a maximum of 2/9.


15/05/2014 28 Version 1.3

The repeatability limit represents the maximum expected difference at a 95% confidence

level between two measurements made under repeatability conditions (usually same set,

same occasion);

The reproducibility limit represents the maximum expected difference at a 95%

confidence level between two measurements made under reproducibility conditions

(usually different sets, different occasions)

Procedures for creation of the KPI time series diagrams and for fundamental KPI statistical

description are described in [12], and conducted in accordance to [11] and [13]. Processing software

scripts have been developed by ENT in R statistical environment [14]. The exact description of

statistical parameters to be examined is given in a table below.

Statistical parameter Definition Comments

Time series diagram of KPI - Graphical representation of time series values vs. measurement time stamps

Mean x=

1

n∑i= 1

n

x i

A numerical measure of the central location of the data values

Median The value at the middle when the data is sorted in ascending order

Variance s

2=

1

n− 1∑i= 1

n

( x i− x )2

A numerical measure of data values dispersion around the mean

Standard deviation σ=

s

√n

An observation variable proportional to the square root of its variance

Skewness γ1=μ3

μ2

3 / 2

A measure of the symmetry of the data distribution

Kurtosis γ 2=μ4

μ2

2− 3

A measure of the peakedness of the data distribution

Histogram with normal probability diagram

- A graphical representation of the frequency distribution of a KPI values

Correlogram Sample auto covariance

sxx=1

n− 1∑i = 1

n

(x i− x)( x ' i− x ')

Autocorrelation coefficient

r xx=sxx

(sx s x)

A graphic representation of the values of the autocorrelation coefficient r(τ) between the original and τ-delayed time series of the same KPI

Correlation between the time series of different KPIs

Sample covariance:

A graphic representation of the values of the correlation coefficient


15/05/2014 29 Version 1.3

sxy=1

n− 1∑i = 1

n

(x i− x)( yi− y)

Correlation coefficient:

r xy=sxy

s x s y

r(τ) between two KPIs' time series with their individual variances sx and sy, their covariance s xy, and the time lag τ

Correlation between KPI time series and geospatial environment

- A qualitative evaluation of correlation between KPI value extremes and geospatial conditions

6.7 Reporting A detailed report for the validation should be prepared, including:

detailed description of the measurements system tested: model version of the hardware,

including external antennas if used, software version used, correction signals if used,

results of statistical analysis of the data: number of outliers detected and rejected,

problems identified by use of ANOVA, bias and repeatability and reproducibility limits,

time series diagrams

6.8 WP4 Action Plan The HeERO WP4 action plan has been proposed and agreed for the first-phase raw data analysis as

following:

Action Due

Agreement on proposal of the common data analysis procedures and

processes

20 July, 2012

Consolidation of the common data analysis procedures and processes 1 August, 2012

Agreement on consolidated common data analysis procedures and

processes

15 August, 2012

Collection of raw WP3 data (WP4 input data) by Member States 15 August 2012

Processing the WP4 input data according to agreed common data

analysis procedures and processes

31 August 2012

Processed WP4 input data to be arranged in WP4 reports, based on

WP4.3 report templates

15 September, 2012

Draft first-phase WP4 consolidated results report 30 September, 2012

Completed internal review of the Draft first-phase WP4 consolidated

results report

15 October, 2012


15/05/2014 30 Version 1.3

Consolidated and reviewed First-phase WP4 consolidated results

report presented to ERTICO

31 October, 2012

6.9 Overview of KPIs To summarize the definitions done in D4.1, the following table gives an overview about which KPIs

are recommended to be tested by all pilot sites.

ID of KPI Name of KPI

recommended Remark

KPI_001a Number of automatically initiated eCalls

This KPI measures mainly coverage of GSM

network but not eCall specific aspects, as typically

vehicle is driving, no voice communication

KPI_001b Number of manually initiated eCalls

Y Describes number of “real” eCall scenarios with

vehicle not moving and voice communication

KPI_002a Success rate of completed eCalls using 112

Y It is recommended to use eCall flag for call

establishment with 112

KPI_002b Success rate of completed eCalls using long number

Only if eCall via 112 is not possible as eCall flag not

supported in member state or due to other

technical restrictions

KPI_003 Success rate of received MSDs

Y Measures exactly what differs eCall from 112 call

KPI_004 Success rate of correct MSDs

Y Measures proper en-/de-coding of MSD

KPI_005 Duration until MSD is presented in PSAP

Y Measures time until information is available to

operator

KPI_006 Success rate of established voice transmissions

Y Measures basics of eCall, MSD and voice

transmission

KPI_007a Duration of voice channel blocking

Y

Most important to minimize, as during this time

passengers in the vehicle do not know if eCall does

work or not

KPI_007b

Duration of voice channel blocking: automatic retransmission of MSD

If for some reason the transmission of MSD was

not successful, operator may require

retransmission of MSD, but voice communication

has been established already

KPI_008 Time for call establishment

Allows calibration between timer in PSAP and IVS

if for some KPIs only one and not the other is


15/05/2014 31 Version 1.3

available; if all timers are available not required

KPI_009 Accuracy of position

Provides accuracy information of IVS system, This

will mainly be influenced by the later integration

into the vehicle and as such not high priority for

pilot phase

KPI_010 Number of usable satellites

Provides additional information on good or bad

accuracy

KPI_011 Geometric dilution of precision

Provides additional information on good or bad

accuracy

KPI_012 Time between successful positioning fixes

Typically within IVS not varying too much

KPI_013 Success rate of heading information

Y This value is calculated by IVS and is critical to

identify right side on highways

KPI_014 Success rate of VIN decoding without EUCARIS

It is critical that VIN is properly encoded and

transmitted via MSD. VIN decoding is not eCall

specific

KPI_015 Success rate of VIN decoding with EUCARIS


transmitted via MSD. EUCARIS is not eCall specific

KPI_016 Time for VIN decoding with EUCARIS


transmitted via MSD. EUCARIS is not eCall specific

KPI_017 Dispatch time of incident data to rescue forces

Typically this value might not be available as no

real PSAP is used but only “test PSAP”. In case of

unique alarms not enough values for statistical

analysis

KPI_018 Time to activate rescue forces transmissions




analysis.

KPI_019 Dispatch time of incident data to TMC

Potential value added Service for eCall but critical

part is more when to provide information

KPI_020 Success rate of presented incident data in TMC

Potential value added Service for eCall but critical

part is more when to provide information

KPI_021 Number of

In case of errors, operator may request call back to


15/05/2014 32 Version 1.3

successful call-backs vehicle; mainly question of proper

implementation of dormant eCalls (validated by

certification)

KPI_022 Success rate of call-backs

In case of errors, operator may request call back to

vehicle; mainly question of proper

implementation of dormant eCalls (validated by

certification)

KPI_023 GSM network Latency

Depends on network implementation and no

significance on eCall timers (units are measured in

seconds not fraction of seconds)

KPI_024 112 National network latency

Depends on network implementation and no

significance on eCall timers (units are measured in

seconds not fraction of seconds)

KPI_025 112 Operator reaction time

Depending on loads in PSAP, no influence to eCall

KPI_026

Time for acknowledgement of emergency services




analysis

KPI_027 Total response time




analysis

KPI_028a Number of cross-border tests

Y

Required tests and should be specified per

member site with which cross border was

performed

KPI_028b Number of interoperability tests

Y

Required tests and should be specified per

member site with which interoperability was

performed

Table 2: recommendations of specific KPIs

In Table 3 the testing and evaluation of recommended KPIs in phase 1 is illustrated. means the Red

KPI was recommended but not tested by this pilot site, green that a recommended KPI was tested

and yellow that this KPI was measured although it was not recommended.

In chapter 7 each Pilot sites gives reasons why recommended KPI hadn’t been evaluated in Phase I.

In the following, this information is consolidated on per KPI basis:


15/05/2014 33 Version 1.3

HR CZ FI DE GR IT RO SE NL

KPI_001a

KPI_001b

KPI_002a

KPI_002b

KPI_003

KPI_004

KPI_005

KPI_006

KPI_007a

KPI_007b

KPI_008

KPI_009

KPI_010

KPI_011

KPI_012

KPI_013

KPI_014

KPI_015

KPI_016

KPI_017

KPI_018

KPI_019

KPI_020

KPI_021


15/05/2014 34 Version 1.3

KPI_022

KPI_023

KPI_024

KPI_025

KPI_026

KPI_027

KPI_028

Table 3: Testing of recommended KPIs in phase 1

KPI1b: Not tested by SE. In Sweden nearly all eCalls were computer-generated (only a few were

manually-initiated). The number of computer-initiated eCalls is reported under KPI1a, although the

call was manually initiated. The number of manually-initiated eCalls (KPI_001b) is not reported. Only

a handful manually-initiated eCalls were performed.

KPI2a: Not tested in CZ, FI, DE, GR and NL: In Czech Republic the eCall tests are carried out with the

short number 162 instead of 112. In Finland, Germany, Greece and the Netherlands, the eCall flag

has not been implemented by any of the national mobile network operators and won’t be available

during the HeERO project. Therefore all tests are performed using the long number of the PSAPs.

KPI5: Not tested in HR and FI. The presentation time couldn’t be measured in Finland because no real

PSAP with user interface was available for the tests. Croatia has not planned to test KPI005, since,

given the technology and business environment of the Croatian eCall Pilot test-bed, the very same

results are to be achieved by testing KPI007a.

KPI6: Not tested in FI: At the time of test preparation there was no requirement for implementation

of the voice connection in the two IVS prototypes. Therefore voice connection related KPIs were not

tested in Phase 1. This KPI will be measured in the Phase 2 of HeERO in Finland

KPI7: Not tested in FI, GR and IT: In Finland this KPI will be tested in phase 2 due to the fact that the

implementation of voice connection was not available for the tests in Phase 1. In Italy, KPI 7 is not

measured because on IVSs side it is impossible to acquire low level mobile modem related data. In

phase 2 this might be revised as the KPI may be measured based on PSAP information only as well.

KPI13: Not tested in HR, FI, GR, DE, IT and SE: In Italy, KPI 13 has not been measured mainly because

it was not of interest to check GPS related information with IVSs still in prototype version. In

Germany, heading information was not evaluated during the first test period, but is part of the

second test phase. In Finland the eCall IVS devices were early prototypes and the focus was on eCall

specific new features during the first phase. KPI 13 is planned to be measured in the Phase 2 of

HeERO tests in Finland. In Sweden this KPI is not measured, as there is a vendor specific (Actia)

solution successfully used in Volvo On Call for many years with the evidence of providing correct

information on location. Croatia has not tested KPI013, since, the technology environment of the


15/05/2014 35 Version 1.3

eCall Pilot test-bed and equipment, the Croatian eCall Pilot has not got the technical and operational

means for reliable testing of the KPI013.

KPI28: not systematically tested in all pilot sites. There were some tests which showed that

interoperability is possible, e.g. at the General Assembly in Zagreb or before the eCall Days in Berlin.

However not enough test cases were executed to measure KPIs during interoperability tests. This will

be done in the second phase by all pilot sites.

The following table gives an overview which part of the eCall-system was evaluated via a KPI in which

country as stated in D4.1 for the pilot sites. As such the overall findings have been derived only from

those pilot sites measuring the specific KPI as indicated below.

The most important KPIs have been measured by nearly all member states. This refers to the number

of established calls, the resulting success rate and the amount of time passed until a direct voice

communication between passenger and operator in the PSAP can take place. All other KPIs have

been measured by less than 50% of the participating pilot sites, so it might be difficult to derive

overall conclusions from the consolidated data valid for all pilot sites. If less than 3 pilot sites have

measured a KPI, this KPI was not consolidated on pan-European level.

ID o

f K

PI

Nam

e o

f K

PI

Member States, where KPI is evaluated

Cro

atia

Cze

ch R

ep

ub

lic

Fin

lan

d

Ger

man

y

Gre

ece

Ital

y

Ro

man

ia

Swe

de

n

The

Ne

the

rlan

ds

KPI_001a Number of automatically initiated eCalls

X X -- X -- 0 X X X

KPI_001b Number of manually initiated eCalls

X -- X X X X X X X

KPI_002a Success rate of completed eCalls using 112

X 0 -- 0 0 X X X 0


X -- X X -- -- 0 (X) X


X X X X -- X X X X

KPI_004 Success rate of correct MSDs X X 0 X X X X X X


-- X -- X X X X (X) X


X X -- X X X X X X

KPI_007a Duration of voice channel blocking

X -- -- X -- -- X (X) (X)

KPI_007b Duration of voice channel blocking: automatic retransmission of MSD

-- (X) -- -- -- -- 0 0 --


15/05/2014 36 Version 1.3

X Tested as planned (X) Not Tested in phase 1, committed as “will be tested if possibly”

0 Not tested in Phase 1, eventually scheduled for Phase 2 -- additionally tested

-- No commitment

In chapter 7 the inputs of the pilot sites are listed. For Greece the input had been delivered in February 2014 and had then been integrated in the document. The reason for missing statistical analyses is that only success rates had been tested (FI) or the evaluation was focussed on decreasing some of the observed failures (NL).

Collected

data

Statistical

evaluation

Recommendations

conclusions

KPI_008 Time for call establishment X X -- -- 0 X -- -- X

KPI_009 Accuracy of position 0 -- 0 -- -- -- -- X

KPI_010 Number of usable satellites 0 X -- -- -- -- -- -- --


0 X -- -- -- -- -- -- --


0 (X) -- -- -- -- -- -- --

KPI_013 Success rate of heading information

-- -- -- 0 -- -- -- -- X


X X 0 -- -- -- X -- --

KPI_015 Success rate of VIN decoding with EUCARIS

-- -- -- 0 -- 0 0 -- 0

KPI_016 Time for VIN decoding with EUCARIS

-- -- -- -- -- -- 0 -- 0


0 X -- -- -- 0 0 -- --

KPI_018 Time to activate rescue forces

-- 0 -- -- -- -- 0 -- --


-- (X) 0 -- -- -- 0 -- X


-- (X) -- -- -- -- 0 -- X

KPI_021 Number of successful call-backs

-- 0 -- -- -- 0 X -- --

KPI_022 Success rate of call-backs -- 0 -- -- -- 0 X -- 0

KPI_023 GSM network latency -- X -- -- -- -- X -- --

KPI_024 112 National network latency -- (X) -- -- -- -- X -- --

KPI_025 112 Operator reaction time -- -- -- -- -- -- X -- --

KPI_026 Time for acknowledgement of emergency services

-- 0 -- -- -- -- 0 -- --

KPI_027 Total response time -- 0 -- -- -- -- 0 -- --

KPI_028 Number of cross-border tests -- (X) -- -- -- (X) 0 0 --

Table 4: KPIs to be evaluated within the Member States


15/05/2014 37 Version 1.3

Croatia x x x

Czech x x x

Finland x (x) x

Germany x x x

Greece x (x) x

Italy x x x

Netherlands x (x) x

Romania x (x) x

Sweden x x x

Table 5: Status of received inputs

X Received as planned

(x) Not received in favoured quality or only in part

6.10 Consolidated KPIs

Automatically and manually initiated eCalls 6.10.1

Looking to the KPIs, the most relevant ones are: 02a and 02b, 03, 04, 05, 06, 07a, 08. They provide

information on reliability (success rate – 02a, 02b, 03, 04 and 06), additional waiting time compared

to a normal 112 call (07a), time passed prior to information available to call handler (05) and

comparison to call set up times (08). As the number of tests initiated per pilot site varies quite a lot,

this should be reflected in the statistical evaluation. For better possibilities of interpreting a weighted

mean has been introduced. The mean is weighted with the total number of measurements a pilot

site did however only if more than 3 pilot sites provided results. As the standard deviation is huge,

the values have no real significance. As one example we have a closer look to KPI 07a “Duration of

voice channel blocking” in seconds. The voice channel blocking time is the most important KPI for the

later success of eCall. This KPI indicates the additional time which the passenger in a vehicle is

waiting to be connected to the call handler compared with a standard 112 call. Therefore all

measures should be taken to assure that this time is as short as possible. This KPI has been measured

by 6 pilot sites. The mean is 8.0s; the weighted mean 7.5s, median 6.5s, minimum value is 5s and

maximum value is 15.8s after exclusion of outliers. The standard deviation is 3.7. Therefore the

analysis and conclusions have to be made based on the individual results per pilot sites and the

degree of similarity between different pilot sites.


15/05/2014 38 Version 1.3

KP

I

Explanation Mean

Wei

ghte

d

mea

n2

Med

ian

Min Max

test

ing

cou

ntr

ies

Std dev.

1a Number of automatically initiated eCalls 1918.3 644 79 1069

7 8

3359.1

1b Number of manually initiated eCalls 771.5 286 60 3791 8 1180.

7

2a Success rate of completed eCalls using 112 [%]

91 93 94 79 98 4 7.0

2b Success rate of completed eCalls using long number[%]

76 71 70 57 100 6 16.4

3 Success rate of received MSDs [%] 89 81 97 71 100 9 11.5

4 Success rate of correct MSDs [%] 92 93 93 81 100 7.5

5 Duration until MSD is presented in PSAP [s]

16.6 13.4 17.0 8.6 29.8 7 7.2

6 Success rate of established voice transmissions [%]

92 92 92 83 100 8 5.2

7a Duration of voice channel blocking [s] 8.0 7.5 6.5 5.0 15.8 6 3.7

08 Time for call establishment [s] 9.1 10.3 11.4 4.7 12.0 5 3.6

09 Accuracy of position [m] 131.0 131 131.

0 131.

0 131.

0 1 -

10 Number of usable satellites 8.6 8.6 8.6 8.6 8.6 1 -

11 Geometric dilution of precision 2.0 2.0 2.0 2.0 2.0 1 -

12 Time between successful positioning fixes [s]

2.5 2.5 2.5 2.5 2.5 1 -

13 Success rate of heading information [%] 91 79 99 73 100 3 12.5

14 Success rate of VIN decoding 86 83 91 60 100 4 15.5

16 Time for VIN decoding with EUCARIS [%] 1.3 1.3 1.3 1.3 1.3 1 -

17 Dispatch time of incident data to rescue forces [s]

61.4 61.4 61.4 61.4 61.4 1 -

19 Dispatch time of incident data to TMC [s] 21.0 17.1 21.0 15.0 26.9 2 6.0

20 Success rate of presented incident data in TMC [%]

89 84 89 81 97 2 8.0

21 Number of successful call-backs 117.7 162.

7 56.2 38.0

259.0

3 100.2

22 Success rate of call-backs [%] 83.3 84.4 78.0 75.6 96.4 3 9.3

23 GSM network latency [s] 2.9 3.0 2.9 2.6 3.1 2 0.3

24 112 National network latency [s] 1.4 0.1 1.4 1.0 1.7 2 0.3

25 112 Operator reaction time [s] 1.3 2.6 1.3 0.0 2.6 2 1.3 Table 6: evaluated results

2 Weighted with the total number of measurements per country


15/05/2014 39 Version 1.3

Automatically initiated calls 6.10.2

KPI Explanation Mean

Wei

ghte

d

me

an3

Med

ian

Min Max

test

ing

cou

ntr

ies

Std dev.

01a Number of automatically initiated eCalls

2011 307 79 1069

7 7

3581.4


90 92 94 79 98 3 8.0


66 59 80 0 100 5 36.7


04 Success rate of correct MSDs [%] 95 93 94 90 100 5 4.0


18.8 15.7 17.3 10.6 29.8 4 6.9


94 91 95 87 100 5 4.8

07a Duration of voice channel blocking [s] 8.4 8.3 6.7 5.2 15.8 5 3.9


09 Accuracy of position [m] 5.8 5.8 5.8 5.8 5.8 1 -




2.0 2.0 2.0 2.0 2.0 1 -

13 Success rate of heading information [%]

91 77 99 73 100 3 12.5

14 Success rate of VIN decoding without EUCARIS [%]

94 94 94 89 100 2 5.6


48.4 48.4 48.4 48.4 48.4 1 -

19 Dispatch time of incident data to TMC [s]

15.1 15.1 15.1 15.1 15.1 1 -


90 81 90 80 100 2 9.9

21 Number of successful call-backs 47.1 52.9 47.1 38.0 56.2 2 9.1

22 Success rate of call-backs [%] 86.0 92.6 86.0 75.6 96.4 2 10.4

23 GSM network latency [s] 2.9 2.7 2.9 2.6 3.1 2 0.2

24 112 National network latency [s] 1.4 1.2 1.4 1.0 1.7 2 0.3

25 112 Operator reaction time [s] 2.6 2.6 2.6 2.6 2.6 1 - Table 7: evaluated results of all automatically initiated eCalls

3 Weighted with the number of automatically initiated measurements per country


15/05/2014 40 Version 1.3

Manually initiated eCalls 6.10.3

KPI Explanation Mean

Wei

ghte

d

me

an4

Med

ian

Min Max

test

ing

cou

ntr

ies

Std dev.

01b Number manually initiated eCalls 987.7 347 110 3791 6 1292


96 97 96 94 98 2 2.1


73 73 68 57 100 4 16.1


04 Success rate of correct MSDs [%] 94 84 100 76 100 5 9.2


12.9 16.2 13.1 8.6 16.8 4 3.7


93 92 92 83 100 5 6.2

07a Duration voice channel blocking [s] 7.1 5.8 7.1 5.3 8.9 3 1.5


09 Accuracy of position [m] 193.7 193.7 193.7 193.7 193.7 1 -




3.1 3.1 3.1 3.1 3.1 1 -

13 Success rate of heading information [%]

74 74 74 74 74 1 -

14 Success rate of VIN decoding without EUCARIS [%]

80 70 80 60 100 2 20.2

16 Time for VIN decoding with EUCARIS

1.3 1.3 1.3 1.3 1.3 1 -


74.3 74.3 74.3 74.3 74.3 1 -

19 Dispatch time of incident data to TMC [s]

20.9 17.6 20.9 14.9 26.9 2 6.0


87 84 87 81 94 2 6.7

23 GSM network latency [s] 3.1 3.1 3.1 3.1 3.1 1 -

24 112 National network latency [s] 1.6 0.3 1.6 1.6 1.6 1 - Table 8: evaluated results of all manually initiated eCalls

4 Weighted with the number of manually initiated measurements per country


15/05/2014 41 Version 1.3

7 Member States results

7.1 Croatia

Purpose and scope 7.1.1

This report outlines: the Croatian 2012 WP4 Phase I results. Croatian eCall Pilot intends to examine

and the procedure of the KPI values calculation, the WP4 input data format, and the statistical

methods and procedures to be used by Croatian eCall Pilot for WP4 input data analyses. Finally, data

analyses results are presented in the consolidated WP4 output data report format.

Croatian 2012 WP4 Action Plan 7.1.2

Croatian 2012 WP4 Action Plan has been developed in compliance to the HeERO WP4 action plan for

the first-phase raw data analysis.

Methodology of data analysis 7.1.3

Every operation of a WP3 scenario (Groups 1, 2 and 4, as described in DoW) results in a batch (a set)

of raw data (WP4 input data set) that is to be statistically processed separately, generating a

separate examination report.

The examination report consists of the following:

Time series diagrams of the values of relevant KPIs

Fundamental KPI statistical description for every time series (mean, median, variance,

standard deviation, skewness, kurtosis, histogram with normal probability and

correlogram)

Results of examination of correlation between the time series of different KPIs, as to

identify potential causality

Results of examination of correlation between KPI values and geospatial environment

Discussion of the outliers

Description of equipment 7.1.4

The equipment was used in the eCall Phase I testing and validation in Croatia is described in the table

below.

Component Description

IVS

Manufacturer Skymeter Corp.

Commercial name of the product (if available), or provisional prototype name NA

Hardware ID NA

Software ID / revision NA

PSAP

Manufacturer Ericsson LM

Commercial name of the product (if available), or provisional prototype name CoordCom


15/05/2014 42 Version 1.3

Hardware ID NA

Software ID / revision 4.6

Table 9: Description of equipment (HR)

Testing and validation scenarios description 7.1.5

The eCall Testing and validation Phase I in Croatia comprises the scenarios of Group 1, as described in

HeERO deliverable WP 3.1. A summary of Group 1 Scenarios requirements is given in the table

below.

Laboratory eCall T&V

Code No of IVS units involved

No of IVS

units in

roaming

eCall

initiation

No of repeated

initiations

No of

tests

L1 1 0 A 0 > 1000

L2 1 1 A 0 > 1000

L3 1 0 M 0 > 1000

L4 1 1 M 0 > 1000

L5 1 0 M 3 > 1000

L6 1 1 M 3 > 1000

L7 1 0 A 3 > 1000

Table 10: Croatian Laboratory eCall T&V scenarios (HR)

ECall Communications T&V


No of IVS

units in

roaming

eCall

initiation

No of repeated

initiations

No of

tests

R1 1 1 M 0 > 100

R2 1 1 M 0 > 100

R3 1 0 A 0 > 1000

R4 1 0 A 0 > 1000

Table 11: Croatian eCall Communications T&V scenarios (HR)

Description of consolidated Phase I T&V data 7.1.6

The description of received consolidated Phase 1 WP3 results is comprehensively given below.


15/05/2014 43 Version 1.3

Scenario code Consolidated data received No of tests

L1 YES 211

L2 YES 164

L3 YES 212

L4 NO NA

L5 NO NA

L6 NO NA

L7 NO NA

R1 YES 156

R2 YES 214

R3 YES 105

R4 YES 103

Table 12: Croatian eCall T&V summary (HR)

Evaluation results 7.1.7

The evaluation results are presented in this chapter by scenarios.

7.1.7.1 L1 scenario

The results of statistical analyses by KPIs are systematically presented in this chapter. Scalar

statistical indices by KPIs are presented below.

The time series of KPI 007a and KPI 008 values are presented below.

Table 13: L1 statistical results summary (HR)


15/05/2014 44 Version 1.3

Figure 4: Time series of KPI007a (left) and KPI008 (right) for L1 scenario (HR)

The histograms of KPI 007a and KPI 008 values are presented below.

Figure 5: Histograms of KPI007a (left) and KPI008 (right) for L1 scenario (HR)

The correlograms of KPI 007a and KPI 008 values are presented below.


15/05/2014 45 Version 1.3

Figure 6: Correlograms of KPI007a (left) and KPI008 (right) for L1 scenario (HR)

Finally, the KPI007a and KPI008 cross-correlogram is presented below.


15/05/2014 46 Version 1.3

7.1.7.2 L2 scenario



Table 14:L2 statistical results summary (HR)


figure 7: Cross-correlogram for L1 scenario (HR)


15/05/2014 47 Version 1.3



Figure 9: histograms of KPI007a (left) and KPI008 (right) for L2 scenario (HR)



15/05/2014 48 Version 1.3

Figure 10: correlograms of KPI007a (left) and KPI008 (right) for L2 scenario (HR)


Figure 11: Cross-correlogram for L2 scenario (HR)


15/05/2014 49 Version 1.3

7.1.7.3 L3 scenario



Table 15: L3 statistical results summary (HR)





15/05/2014 50 Version 1.3

Figure 13: Histograms of KPI007a (left) and KPI008 (right) for L3 scenario (HR)


Figure 14: correlograms of KPI007a (left) and KPI008 (right) for L3 scenario (HR)



15/05/2014 51 Version 1.3

Figure 15: cross-correlogram for L3 scenario (HR)

7.1.7.4 R1 scenario



Table 16: R1 statistical results summary (HR)



15/05/2014 52 Version 1.3

Figure 16: Time series of KPI007a (left) and KPI008 (right) for R1 scenario (HR)


Figure 17: histograms of KPI007a (left) and KPI008 (right) for R1 scenario (HR)



15/05/2014 53 Version 1.3

Figure 18: correlograms of KPI007a (left) and KPI008 (right) for R1 scenario (HR)


Figure 19: cross-correlation for R1 scenario (HR)


15/05/2014 54 Version 1.3

7.1.7.5 R2 scenario








15/05/2014 55 Version 1.3

Figure 21: Histograms of KPI007a (left) and KPI008 (right) for R2 scenario (HR)



Finally, the KPI007a and KPI008 cross-correlogram is presented in figure below.


15/05/2014 56 Version 1.3

Figure 23: cross-correlogram for R2 scenario (HR)

7.1.7.6 R3 scenario






15/05/2014 57 Version 1.3






15/05/2014 58 Version 1.3





15/05/2014 59 Version 1.3

7.1.7.7 R4 scenario








15/05/2014 60 Version 1.3






15/05/2014 61 Version 1.3


Discussion of evaluation results 7.1.8

The results of the analysis are given per testing scenarios. Croatia has not planned to test the eCall

system and service for KPI005, since, given the technology and business environment of the Croatian

eCall Pilot test-bed, the very same results are to be achieved by testing KPI007a. It is not planned to

test the eCall system and service for KPI013, since, given the available funding, the technology

environment of the eCall Pilot test-bed and the available eCall equipment, the Croatian eCall Pilot

has not got the technical and operational means for reliable testing of the KPI013.

Croatia plans to test the eCall system and service for KPIs: 009, 010, 011, 012 in both the eCall

Testing and Validation Phases I and II, and issue the final report after the Phase II. Extended testing is

planned in order to allow for the time to equip the testing vehicle with the suitable equipment, and

to challenge the GPS performance in various positioning environments.

Croatia plans to test the eCall system and service for KPI028 in the eCall Testing and Validation Phase

II, when the Joint field exercise with participation of the relevant emergency services is scheduled.

7.1.8.1 Scenario L1

The 211 L1 test cases were conducted. Of 211 cases, 99.04% of long-number eCalls were successfully

completed, and 72.73% MSD transfers were successfully completed. 100% of successfully

transmitted MSD were successfully interpreted at PSAP, of which 100% were successfully decoded


15/05/2014 62 Version 1.3

without EUCARIS. Of all established voice transmission test cases, 99.04% were successful. Mean

time of duration of voice channel blocking was 8.792 s; with standard deviation of 0.940 s. Mean

time of time for the eCall establishment was 4.671 s, with standard deviation of 0.245 s.

7.1.8.2 Scenario L2







7.1.8.3 Scenario L3







7.1.8.4 Scenario R1

The 156 L1 test cases were conducted. Of 156 cases, 98.72% of eCalls were successfully completed,

and 96.79% MSD transfers were successfully completed. 100% of successfully transmitted MSD were

successfully interpreted at PSAP, of which 100% were successfully decoded without EUCARIS. Of all

established voice transmission test cases, 98.72% were successful. Mean time of duration of voice

channel blocking was 5.131 s; with standard deviation of 0.601 s. Mean time of time for the eCall

establishment was 3.939 s, with standard deviation of 0.129 s.

7.1.8.5 Scenario R2







7.1.8.6 Scenario R3








15/05/2014 63 Version 1.3

7.1.8.7 Scenario R4







7.1.8.8 Observed issues

The following issues have been observed:

insufficient number of test cases in several scenarios,

low success-rate of MSD transmissions in several laboratory test, due to initial internal

connection between MSC and PSAP (the appropriate corrections already deployed and

tested),

significantly delayed eCall initiation in specific cases of incompatible networks (Tele2 to

T-Mobile, former eCall-enabled, latter not, in several R2 cases) - first 50 or so R2 cases

conducted with moving vehicle, the rest of all R scenarios with stopped vehicle,

drawbacks in deployment of agreed T&V methodology and procedure, especially in

relation to synchronization of all components involved to the same (GPS-based) time

standard,

shortcomings in T&V raw data consolidation,

Shortcomings in T&V scenario design that allows for several potentially interesting cases

to remain non-validated (increased KPI08 values for vehicles on the move in the areas of

the weak MN signal reception).

Conclusion 7.1.9

This report presents the results of the evaluation of the reduced set of the Croatian eCall Pilot

experimental data. In general, the performance of the eCall in the Croatian eCall Pilot environment

meets the requirements set up by the EC.

The results of the WP4 Evaluation lead to the set of recommendations to the eCall operators,

component manufacturers and system integrators with the aim to mitigate the sources of

disturbances and improve the quality and performance of the eCall service.

Those are to be evaluated during the second Phase of the Croatian eCall Pilot WP3 testing and

validation, schedule to start in January 2013.

Recommendations 7.1.10

After a thorough consideration of the observed issues and drawbacks, the following

recommendations are suggested for implementation in Phase II Testing and Validation, as a set of

improvement-related corrective actions:


15/05/2014 64 Version 1.3

the proper (planned) number of test and validation cases should be conducted for every

scenario in a manner described by agreed T&V methodology and procedure,

the addition of a timing device in laboratory MSC to be performed for advanced

monitoring of the eCall propagation,

deployment of PSAP synchronising device,

applications of advanced consolidated data quality management and control,

All existing R (real-network) scenarios to be conducted with stopped vehicles, additional

R scenario to be designed to observe the possible issues of the eCall initiated from the

moving vehicle.


15/05/2014 65 Version 1.3

7.2 Czech Republic

Purpose of Document 7.2.1

The purpose of this document is to describe the architecture and method of testing eCall Phase I and

statistically evaluate measured KPIs. All recommended KPIs were measured in phase 1 except of KPI

02a due to the fact that the tests are carried out with the short number 162.

The document consists of the following parts:

description of the test environment

logging options

KPIs evaluation

method of testing

test results, observations

conclusion, open issue

Testing infrastructure 7.2.2

7.2.2.1 ECall architecture

IVS dials short number 162 after automatic/manual activation of eCall. Mobile network of Telefónica

Czech Republic recalculates this number to the NRN+112. The number is consequently routed to the

eCall PSAP test platform in Prague.


15/05/2014 66 Version 1.3

Figure 32: eCall architecture (CZ)

Test PSAP Prague

MSD +

VIN data

TCZ mobile

networkTCZ fixed

network

GPS

MSD source

IVS data

modem

NAD

I V S

162

162 - E2191 112 2xxxx

Event Ringing :

ANI : +420 602602602

DNIS : 21911122xxxx

attached data : KVP with MSD

Tserver

Genesys

CCIVR

voice

PBX

eCall

operator

PSAP modem

(3GPP TS 26268 rel. 10.0)

PSAP application

Call Taker Application

SW phone

MSD visualisation

VIN data

Topography Helper

GIS VIN decoder

interface

TCP/IPVIN

database

data

record

ES system

responder

PSAP

modem

2191112...

SS7 monitoringIVS logging

PSAP logging

TMS

interface

VIN

request


15/05/2014 67 Version 1.3

7.2.2.2 Technical specification

Table 20: Technical specification

7.2.2.3 Logs in eCall technology

a. IVS

Number of automatic/manual activations

Number of unsuccessful attempts

Number of successful eCalls

MSD content

Number of transaction with no LL ACK)

Number of initiated MSD transmissions

Time of „emergency setup“ (T0-IVS)

Time of MSD transfer start (T1-IVS)

T2-IVS

time between activation and N-1 position

number of satellites during eCall activation

GPS fix

GDOP

b. MNO/Fix network

T0-PSAP (Initiated eCall is indicated at PSAP) – Answer message


15/05/2014 68 Version 1.3

point in time the call reaches the 112 network (T0-FIX)

c. PBX


d. PSAP modem


T1-PSAP (Start of MSD reception at PSAP) - time of START message (SEND MSD request)

T1-IVS IVS starts the MSD transmission (start of phase “data transmission”) – time of Initiation message

T2-IVS End of phase “data transmission” - time of HL-ACK

e. PSAP - CTI


T2-PSAP (Start of phase “voice transmission”)

f. PSAP - ASW

T2-PSAP (Start of phase “voice transmission”)

T3-PSAP (Start of dispatching information about incident to emergency services)

T4-PSAP (Start of dispatching information about incident to TMC)

T3-ES (Start of confirmation about incident handling to PSAP)

Testing description 7.2.3

The object of this chapter is to describe the basic assumptions and parameters of testing. Based on

experience from previous test cycles and in order to meet the requirements WP4 - KPI evaluation

was determined time period for intensive testing called “Testing period I”. This period was primarily

focused on fulfil the KPI data for evaluation IVS and PSAP.

7.2.3.1 Testing period

The testing period of phase I: 3.9 – 7.9.2012

7.2.3.2 Testing equipment

Car & IVS

5x eCall car equipped by IVS Sherlog Trace (IVS_SHT_1-5)

5x eCall car equipped by IVS Telematix (IVS_TMX_2,3,4,7,8)

Technical details for IVS are specified in 5.2.2.2Technical specification

PSAP

For testing purposes, is used eCall testing platform that is directly integrated in the test environment

of PSAP 112th. This environment includes a separate installation of application eCall with emergency

call dispatching system flagged with eCall:


15/05/2014 69 Version 1.3

The number of active workplaces for handling incoming calls: 5

Maximum number of simultaneously possible dispatched eCalls: 3

Operating time availability: 24x7

Operator environment: PC, 2 monitor workplace, operator application eCall, application 112

GIS for eCall, phone, headset, etc.

PSAP location : Prague, TCZ NMC

For more detailed description please see chapter Testing infrastructure.

7.2.3.3 Method of generating eCall (IVS)

All vehicles equipped with IVS units may generate two possible ways of eCall call.

1. Automated system of generating eCalls

a. Unattended generate a periodic eCall

b. Interval generation of eCall: 4 minutes

c. Occupants of the car do not communicate with PSAP; the whole procedure from call

initiation to the termination is fully automated. The driver is informed about

activated and on-going calls either via a unit display or via the colour change of the

activation button. Acoustic signalling is inactive.

d. Re-send MSD and Call-back is unused.

2. Manual system of generating eCalls

a. Operating of IVS is monitored by a vehicle crew

b. Generating of eCall (interval, frequency) is on crew decision or based on the testing

scenarios instructions.

c. The voice communication between crew and PSAP is to be conducted

d. The re-send of MSD and Call-back by PSAP operator is to be conducted

Majority of tests were automatically generated. Manual mode of generation eCalls was assigned to a

predetermined day in relation to manual handling system on the PSAP

7.2.3.4 Procedure of receiving and handling eCall (PSAP)

The two modes of call establishment in the IVS (automatic/ manual) are handled in two different

ways in the PSAP.

1. Automatic mode of handling eCalls

a. PSAP will automatically answer calls

b. The PSAP during processing of incoming calls automatically creates an "log entry"

and assigns it to an incoming call

c. The PSAP ensure the control of received MSD, VIN decoding and display eCall

position in GIS

d. The PSAP automatically fills defined data to the "log entry" based on the content of

MSD (localization of event, event classification, regionalization and selection of

operation units for cooperation).

e. The PSAP automatically creates the data for transmission to testing TRAFFIC

MANAGEMENT CENTRE and testing OPERATION CENTRE.


15/05/2014 70 Version 1.3

f. The PSAP ends the eCall after 30 s

g. During that automatic process (and before PSAP ends eCall) can be “event” edited by

the PSAP operator, this makes that the whole process is interrupted and PSAP goes

to manual handling mode.

Events data in the PSAP are stored and incoming calls are recorded.

2. Manual mode of handling eCalls

a. The PSAP will automatically assigns the eCall to an operator

b. All activities related to handling incoming calls are supervised by a trained eCall PSAP

operator. This mode is used primarily for testing functionalities that does not cover

auto Mode:

i. Request for MSD (resend)

ii. Call-back

iii. Call-back (backup phone number)

iv. Request MSD during call-back

v. Voice communication between occupants and PSAP operator

vi. The visual control (by operator) of submitted position and direction against

the values reported by crew.

Events data in the PSAP are stored and incoming calls are recorded.

During the first test period, primarily automatic mode was used in the PSAP. Manual mode was

applied primarily for application testing. The results of manual tests are not reviewed in KPI

evaluation.

7.2.3.5 Data collect and evaluation

Data collection is performed automatically for the following systems:

IVS log

SS7 monitoring system

PSAP modem application log

Application server log

PSAP database

Operator workstation

Manual data collection is performed PSAP operators.

The logs are manually processed and analysed (results are given in the following chapter - Summary

of test result)

Summary of test result 7.2.4

During eCall operational Phase I we have concentrated to the following KPIs.

KPI_001a Number of automatically initiated eCalls -

KPI_001b Number of manually initiated eCalls -


15/05/2014 71 Version 1.3

KPI_002b Success rate of completed eCalls using long number %

KPI_003 Success rate of received MSDs %

KPI_004 Success rate of correct MSDs %

KPI_005 Duration until MSD is presented in PSAP s

KPI_006 Success rate of established voice transmissions %

KPI_007 Duration of voice channel blocking s

KPI_008 Time for call establishment s

KPI_010 Number of usable satellites -

KPI_011 Geometric dilution of precision -

KPI_012 Time between successful positioning fixes s

KPI_014 Success rate of VIN decoding without EUCARIS %

KPI_016 Time for VIN decoding without EUCARIS s

KPI_017 Dispatch time of incident data to rescue forces s

KPI_019 Dispatch time of incident data to TMC s

KPI_020 Success rate of presented incident data in TMC %

KPI_023 GSM network latency s

KPI_024 112 national network latency s Table 21: list of tested KPIs (CZ)

7.2.4.1 General test conditions

The eCall was routed through mobile and fixed network of Telefónica (see figure below) before it

reached testing PSAP. Thus pilot was implemented in such a way that all later conditions of real

deployment were met. The 112 PSAP is connected to the fixed network in Czech Republic without

direct connection to the mobile networks.

There is only one exception - instead of 112 we used in the testing a short number 162. This is

because of false eCalls that we detected in the beginning of this year (eCall flag 112 calls from several

162

PLMN

TelefónicaGPS

MSD source

IVS data modem

NAD

I V S

M S C

Testing

PSAPPSTN

Telefónica

IVS environment Fixed networkMobile Network

Local

Exchange

E2191112E2191112

PSAP environment


15/05/2014 72 Version 1.3

types of mobile phones). Number 162 was then translated in the mobile network to the E2191112

number that was routed to the appropriate testing PSAP.

If 112 + eCall flag was dialled, voice connection set up time would be shorter. Results of special

measurement of 112 eCalls shows approximately 0.5 s shorter time for call set up.

The eCall test handling proceeded according to the scenarios described in chapter 7.2.3.

The different timers and main KPIs are shown in the following diagram.

Figure 33: different timers and main KPIs (CZ)

In the following chapters statistical results and time series graphs of most important KPIs are

enclosed for both IVS systems (Sherlog Trace and Telematix)

T2

-PS

AP

Call establishment MSD transmission Agent routing Voice communicationVIN decoding

T3

-PS

AP

T0

-PS

AP

T1

-PS

AP

T0

-IV

S

T1

-IV

S

INITSETUP

ANSWER SEND

AL-ACK

T2

-IV

S

MSD presented

operator desktop

T4

-PS

AP

TMC dispatch

EC dispatch

MSD presentation time

Voice channel blocking

Time for call establishment

Dispatch time to rescue forces

Dispatch time of incident data to TMC

T0

-FIX

GSM networklatency

national networklatency


15/05/2014 73 Version 1.3

7.2.4.2 Statistical evaluation – IVS Sherlog Tracell

Table 22: List of KPI results with their statistical evaluation – IVS Sherlog Tracell (CZ)

Comments to the selected KPIs:

The eCall was generated under short number 162 and in all test cases we had set up in MSD manual

initiated eCall.

KPI_017 – because of automated eCall handling in the PSAP we set up a fixed period for event

handling and closure. Therefore this KPI is around 45 sec. In the real situation it would be influenced

by PSAP operator.

KPI_019 – again, because of automated eCall handling in the PSAP a predefined delay was used prior

to sending eCall information data to the TMC system – it was done just after the MSD was presented

in Call Taker application. In the real situation PSAP operator can decide when these data are to be

send to the TMC system (in the period between MSD is presented and event is closed).

KPI_020 – in fact, this KPI is influenced by the fact that in case of PositionCanBeTrusted parameter

value is set by IVS to False, call taker application in automated mode doesn´t create event so no data

are sent to TMC.

a. Time series diagrams

Most important KPIs related to the voice connection establishment, MSD transmission and MSD

presentation in the call taker application are visualised on the following time series diagrams.


15/05/2014 74 Version 1.3

Figure 34 KPI_008: Time for call establishment – IVS Sherlog Tracell (CZ)

Figure 35 KPI_023: GSM network latency– IVS Sherlog Tracell (CZ)


15/05/2014 75 Version 1.3

Figure 36 KPI_024: 112 national network latency– IVS Sherlog Tracell (CZ)

Figure 37 KPI_007: Duration of voice channel blocking– IVS Sherlog Tracell (CZ)


15/05/2014 76 Version 1.3

Figure 38: KPI_005: Duration until MSD is presented in PSAP– IVS Sherlog Tracell (CZ)

b. ECall handling phases diagram

Following diagram shows average timing values of significant eCall phases:

connection establishment (GSM+fixed network latency)

In-band modem synchronisation (IVS-PSAP)

actual MSD transmission

call routing to the PSAP call taker + MSD presentation in his/her screen

Average values are calculated across all IVS units during one week testing period.

Table 23: average timing values of significant eCall phases – IVS Sherlog Tracell (CZ)

eCall session phaseAverage value

(s)

GSM network latency 3,33

Fixed network latency 1,61

Modem synchronisation 3,31

MSD transmission 3,63

Call routing + MSD presentation 4,32

Total time 16,20


15/05/2014 77 Version 1.3

Figure 39: IVS Sherlog Trace eCall handling time - basic phases [sec] (CZ)

7.2.4.3 Statistical evaluation – IVS Telematix

List of KPI results with their statistical evaluation:

Table 24: of KPI results with their statistical evaluation - IVS Telematix (CZ)


15/05/2014 78 Version 1.3

Comments to the selected KPI :

ECall was generated under short number 162 and in all test cases we had set up in MSD manual

initiated eCall.

KPI_017 – because of automated eCall handling in the PSAP we set up a fix period for event handling

and closure. Therefore this KPI is around 45 sec. In the real situation it would be influenced by PSAP

operator.

KPI_019 – again, because of automated eCall handling in the PSAP we set up a fix moment when we

sent eCall information data to the TMC system – it was done just after the MSD was presented in Call

Taker application. In the real situation PSAP operator can decide when these data are to be send to

the TMC system (in the period between MSD is presented and event is closed).

KPI_020 – in fact this KPI is influenced by the fact that in case of PositionCanBeTrusted parameter

value is set by IVS to False, call taker application in automated mode doesn´t create event so no data

are sent to TMC.

a. Time series diagrams

Most important KPIs related to the voice connection establishment, MSD transmission and MSD

presentation in the call taker application are visualised on the following time series diagrams.

Figure 40: KPI_008: Time for call establishment- IVS Telematix (CZ)


15/05/2014 79 Version 1.3

Figure 41 KPI_023: GSM network latency- IVS Telematix (CZ)

Figure 42 KPI_024: 112 national network latency- IVS Telematix (CZ)


15/05/2014 80 Version 1.3

Figure 43 KPI_007: Duration of voice channel blocking- IVS Telematix (CZ)

Figure 44: KPI_005: Duration until MSD is presented in PSAP - IVS Telematix (CZ)


15/05/2014 81 Version 1.3

b. ECall phases diagram

Following diagram shows average timing values of significant eCall phases:

connection establishment (GSM+ fixed network latency)

In-band modem synchronisation (IVS-PSAP)

actual MSD transmission

call routing to the PSAP call taker + MSD presentation in his/her screen

Average values are calculated across all IVS units during one week testing period.

Table 25: average timing of significant eCall phases- IVS Telematix (CZ)

Figure 45: IVS Telematix eCall handling time - basis phases [sec] (CZ)

eCall session phaseAverage value

(s)

GSM network latency 2,93

Fixed network latency 1,65

Modem synchronisation 3,65

MSD transmission 4,02

Call routing + MSD presentation 4,00

Total time 16,25


15/05/2014 82 Version 1.3

Conclusions 7.2.5


KPI in both IVS types is 100%. We can say that in all the test cases GSM signal was on the level

allowing call set up and eCall was properly routed to the testing PSAP.


KPI is nearly 100%, in several cases MSD timeout elapsed. In the next operational test phase we will

concentrate on it and try to find the reason.


One IVS type has sometimes a problem with correctness of MSD and we will try to fix it.


Median value is 15.8 / 15.8 s. This complex KPI is influenced by set up time, MSD transmission and

call routing to the PSAP operator.

Call set up and routing + MSD presentation screen phase is most stable as far as time duration is

concerned, nevertheless there is still certain space to speed up it, especially routing phase.

In the test phase II we will rather concentrate on the duration of MSD transmission – see KPI 007

comments.


No problem with speech channel was recognised during eCall testing, KPI is 100% for all IVSs .

KPI_007 Duration of voice channel blocking

Median value is 6.5 / 6.6 s there are certain “extreme values” that need deeper analysis. Moreover

we plan to perform a benchmark tests of our PSAP modem solution with Qualcomm people to prove

that the specification SW code is properly designed and implemented in the Telco part of PSAP.


15/05/2014 83 Version 1.3

KPI_008 Time for call establishment

Median value is 4.8 / 4.6s. It is relatively stable KPI, average values of both IVS types are under 5 s.

This duration, consisting of GSM network and fixed network latency, can be in the future slightly

truncated by using 112+eCall flag instead of current testing number 162.

KPI_010 Number of usable satellites

Parameters of particular IVS are more or less without any chance to influence it.


No special observation.


Both IVS types behave differently, we have to investigate it. In one case there is a constant value of 4

s, second IVS type sets this time call by call. It will be a subject of discussion with IVS vendor.


In the automated eCall handling mode, there is set up a fix period for event termination time,

particularly 45 s. Therefore this KPI is around this value, since current configuration sends data to

rescue forces system at the time of event closure. In the real situation it would be influenced by PSAP

operator.


In automated eCall handling mode there is a defined delay after which PSAP sends eCall information

data to the TMC system – it is done just after the MSD is presented in Call Taker application. In the

real situation PSAP operator will decide when these data are to be send to the TMC system - in the

period between MSD is presented and event closure.



15/05/2014 84 Version 1.3

In our case this KPI calculation is influenced by the fact that in case of PositionCanBeTrusted MSD

parameter set by IVS to False value, application SW in automated mode doesn´t create event so no

data are sent to TMC.

KPI_023 GSM network latency

Median value is 3.3 / 3. 0 s. Due to the usage of short number 162, this part of connection set up is a

bit longer than it is expected for a real eCall using 112. Comparative test with 112+eCall flag shows

that realistic mean value of GSM network latency would be in this case approximately 2,5 s.

KPI_024 112 national network latency

Median value is 1.5 / 1.7 s. The most stable KPI without any chance to influence it.


Based on KPI evaluation and feedback from operational Phase 1 and testing we will improve in Phase

2:

analysis of KPI parameters which are outside of the expected range (both IVS and PSAP

related)

additional tests focusing on eCall behaviour in impaired conditions of GSM/GPS signal

false eCalls analysis - report for eCall standardisation group; elimination of its impact for real

service phase

identification of further improvements in EN16062 standard together with an suggestion of

possible modification

IVS bugs fixes and functional improvements (areas: logging procedure, GPS „freezing, in-band

modem transmission reliability)

benchmark test of PSAP modem with Qualcomm

PSAP modem change request implementation (better identification of incorrect MSD

parameters)


15/05/2014 85 Version 1.3

7.3 Finland

Introduction 7.3.1

7.3.1.1 Background

This report is prepared as a part of HeERO project and it provides input for deliverable D4.3.

7.3.1.2 Objectives

The objective is to evaluate the results of the Finnish eCall pilot implemented in Phase 1 of HeERO

project. The main focus in the evaluation of the Finnish eCall pilot will be to evaluate the

successfulness of MSD transmission and the whole eCall session. The evaluation process will also

identify points for improvement and provide recommendations when necessary.

7.3.1.3 Methods

The Finnish eCall pilot realized during Phase 1 of HeERO is documented in chapter 2 (Finnish eCall

pilot system). The key performance indicators used for evaluation are presented in chapter 3 (eCall

testing – key performance indicators) and testing and analysis methods used in chapter 4 (eCall

testing – testing and analysis methods).

HeERO KPI measurement 7.3.2

7.3.2.1 Basic definitions

The point of time, where the IVS starts the process to get in contact with the PSAP is called “call

connection initiation”, the corresponding phase starting here is called “call establishment”.

The phase, where the transmission of the MSD happens is called “data transmission”.

The phase, where the voice communication happens is called “voice transmission”.


15/05/2014 86 Version 1.3

Significant instants defined with respect to the module where the measurement takes place:

T0-IVS: IVS initiated the eCall (start of phase “call establishment”) – time of Call setup

T1-IVS: IVS starts the MSD transmission (start of phase “data transmission”) – time of Initiation message

T2-IVS: End of phase “data transmission” - time of HL-ACK

T0-PSAP: Initiated eCall is indicated at PSAP – Answer message

T1-PSAP: Start of MSD reception at PSAP - time of START message (SEND MSD request)

T2-PSAP: Start of phase “voice transmission”

T3-PSAP: Start of dispatching information about incident to emergency services

T4-PSAP: Start of dispatching information about incident to TMC

T3-ES: Start of confirmation about incident handling to PSAP

T4-ES: Start of dispatching rescue forces

T0-FIX: point in time when the call enters the 112 national network

Time instants measurement points:

IVS SS7

monitorin

g

PSAP Telco part PSAP ASW part

IVS PSAP TMCES

T0-IVS

T2-IVS

T1-IVS

T0-PSAP

T3-PSAP

T2-PSAP

T4-PSAP

T1-PSAP

T0-FIX

INIT msg

AL-ACK

SEND MSD

msg

Start MSD transmission

End MSD transmission

ES

T3-ESPSAP

TMC

Call establishment phase

Data transmission phase

Voice communication phase

PSAP eCall processing phase

Answer

msg

SETUP

eCall initiation

Confirm

Dispatch

Dispatch


15/05/2014 87 Version 1.3

T0-

IVS

Log IVS :

SETUP msg time

CDR MOB

:

SETUP

msg

- -

T1-

IVS Log IVS :

INIT msg time (push req) -

PSAP modem log:

Ecall::onPsapCatchEve

nt

PSAP_CONTROL_SYNC

-

T2-

IVS Log IVS :

Time of HL ACK

-

PSAP modem log:


nt

PSAP_SENDING_HLAC

K

-

T0-

PSA

P

CDR FIX :

ANM msg

SIP message log :

SIP/2.0 200 OK after

INVITE

PSAP modem log:

DateOfBirth

-

T1-

PSA

P Log IVS :

Time of SEND MSD reception -

PSAP modem log:


nt

PSAP_SENDING_STAR

T

-

T2-

PSA

P - -

T server log :

message

EventEstablished

DM3TCTV.log :

MSG:IDM_PHONE_RINGI

NG


15/05/2014 88 Version 1.3

T3-

PSA

P

- - - DM3TCTV.log :

IDM_SET_ID_UDALOSTI

T4-

PSA

P - - -

DM3TCTV.log :

MSG_TOBE_SENT_TO_T

MC

Finnish eCall pilot system 7.3.3

The following figure (Figure 46) outlines the Finnish eCall pilot system and its basic components (see

D2.3 Finnish pilot implementation plan).

Figure 46 HeERO Finnish pilot system architecture outline (FI)

The main parts of the system include:

eCall client simulator (eCall IVS)

PSAP simulator (eCall test bed)

ECall pilot system control and administrator’s UI.

eCall IVS

Software

modem

GPS

User interfaceMessage activation and configuration

eCall client simulatorPSAP

PSAP1

PSAP2

eCall testbed

Software

modemMSD

encodingMSD

extraction, decoding,

validation

Voice + MSD

ELS

ELS SOAP API

ELS

ELS SOAP API

Risk assessment by a

human user

PSAP2 API

MSD handling,

data complementing

and verification

PSAP1 API

call to test number

https

eCall testbed control UI

Control and configuration

Show result logs

www.ecall.fi/xyzwww.ecall.fi/xyz

Logs, configurations

PSAP2

PSAP simulator

Control and administrator’s UI


15/05/2014 89 Version 1.3

7.3.3.1 ECall client simulator (IVS)

Two IVS prototypes were used in Phase 1 of HeERO in Finland. The first of them (Gecko) implements

only transmission of MSD but provides no voice connection (Figure 47). However, it has features for

automatic activation of MSD transmission. The second prototype used in Finland (Indagon) provides

both voice connection and transmission of MSD but no features for automatic activation of eCall

(Figure 48).

Figure 47: Gecko IVS prototype (FI)

Figure 48: Indagon IVS prototype (FI)


15/05/2014 90 Version 1.3

Both prototypes have been photographed without cables used for power supply or data logging.

While the Gecko prototype has external GSM and GNSS antennas, the Indagon prototype has internal

antennas mounted within its plastic enclosure.

Both prototypes provide interfaces suitable for data logging at IVS side. However, the data logging

interfaces as well as the formats of log files are different for the two prototypes. Therefore, both

prototypes required their own tools for post processing of log files.

The test results are presented separately for each IVS prototype. Because the Gecko prototype

provides no voice connection to PSAP, KPI_002B (Table 8) is measured only for the Indagon

prototype.

7.3.3.2 PSAP simulator

The PSAP simulator part of the system consists of eCall test bed and eCall test bed control user

interface (Figure 46). These components were used in Phase 1 tests of the Finnish eCall pilot as a

PSAP simulator.

The eCall test bed is the eCall message receiver part of the system. It includes functionality for

handling incoming eCall phone calls. It receives and decodes eCall message data, includes interfaces

for PSAP1 and PSAP2 subsystems, provides logs for analysing results and includes facility for

configuring the operation of the system.

A landline phone number (other than 112) was configured for test bed to receive eCall phone calls.

The test bed used the standardized in-band modem (Version 10.0.0) implemented with the ANSI-C

reference code to extract MSD data from the call. The incoming MSD messages were assumed to be

encoded according to the standard CEN EN 15722 (eCall minimum set of data). The test bed decoded

and validated MSD messages. For analysing results, the eCall test bed provided data logging features.

ECall testing – key performance indicators 7.3.4

The following key performance indicators were planned to be used to evaluate the Finnish eCall pilot after the first round of tests carried out in HeERO (Table 26).

KPI Name of KPI Definition

KPI_001B Number of manually initiated eCalls

Every manual initiation of an eCall is counted up to get an overview of the total number of manually initiated eCalls.

KPI_002B Success rate of completed eCalls using long number

eCall success rate = successful eCalls / all initiated eCalls * 100 % Successful eCalls = initiated eCalls - failed eCalls General definition of successful eCall: Voice call path was established, MSD data transfer was done and MSD content was shown at operator’s desk. Initiated eCall: eCall triggered by IVS Failed eCall: Either no establishment of a voice path connection at all, or no stable connection at all, or no voice call possible or


15/05/2014 91 Version 1.3

no MSD transmission or faulty MSD transmitted


MSD success rate = successful MSDs / all initiated MSDs * 100 % Successful MSDs = initiated MSDs - failed MSDs General definition of successful MSD: Content is presented at operator’s desk in PSAP Initiated MSD: Start of MSD-transmission in push mode (comes from IVS) Failed MSD: No MSD data transmission or faulty transmission: voice call started without content of MSD is presented at operator’s desk in PSAP or MSD transmission is not successfully completed.


MSD correctness rate = correct MSDs / all received MSDs * 100 % correct MSDs = received MSDs - incorrect MSDs


VIN success rate = correct reported information about vehicle by database / all requests at database * 100 %; “Correct” if provided VIN is identical and presented data fits to type of vehicle (interface to database is correctly implemented), otherwise “Incorrect”.


Required time until incident data is presented = point of time of presentation of incident data at operator’s desk in TMC - point of time for IVS initiated the eCall

Table 26: Key performance indicators planned for evaluation of Finnish eCall

Limitation of measured key performance indicators

Because the Gecko IVS prototype had an automatic activation feature, KPI_001A (Number of

automatically initiated eCalls) had to be used for that prototype instead of KPI_001B. KPIs requiring a

voice connection could not be measured for the Gecko IVS prototype. This was caused by the fact

that the Gecko prototype implemented only the transmission of MSD but not voice connection.

All other KPIs in Table 8 except KPI_014 and KPI_019 were evaluated for the Phase 1 implementation

of the Finnish eCall pilot.. KPI_014 was not measured in Phase 1 tests because the Finnish PSAP

organization has indicated that they are not going to use the VIN number at least in current situation.

Some of the recommended KPIs (listed in ) could not be measured in the Phase 1 tests in Finland:

KPI_002a (Success rate of completed eCalls using 112): The eCall-flag has not been

implemented yet in mobile networks in Finland. It will be available in Finland only after the

HeERO project.

KPI_005 (Duration until MSD is presented in PSAP): There was no real PSAP (with user

interface) available for the tests in Finland. Therefore, presentation time could not be

measured.

KPI_006 (Success rate of established voice transmissions): At the time of test preparation

and selection of KPIs there was no information about implementation of the voice

connection in the two IVS prototypes. Therefore voice connection related KPIs were not

tested in Finland Phase 1. This KPI will be measured in the Phase 2 if HeERO tests in Finland.

KPI_007a (Duration of voice channel blocking): [The same as above]]


15/05/2014 92 Version 1.3

KPI_013 (Success rate of heading information): In Finland the eCall IVS devices were early

prototypes and we wanted to concentrate on eCall specific new features. Positioning rated

optimization of the devices (e.g. GPS antennas etc.) was not in the production level.

Therefore, positioning related KPIs were not tested in Finland Phase 1. This KPI is planned to

be measured in the Phase 2 if HeERO tests in Finland.

KPIs measured for the two prototypes during tests carried out in 2012 are summarized in Table 27.

KPI IVS prototype

Gecko Indagon

KPI_001A – Number of automatically initiated eCalls X

KPI_001B – Number of manually initiated eCalls X

KPI_002B – Success rate of completed eCalls using long

number

X

KPI_003 – Success rate of received MSDs X X

KPI_004 – Success rate of correct MSDs X X Table 27: KPIs measured in Phase 1 in Finnish eCall pilot (FI)

ECall testing – testing and analysis methods 7.3.5

7.3.5.1 Overview

Tests were carried out by installing the IVS prototypes in a test vehicle and driving a test route (Table

28). The Gecko IVS prototype was activated automatically using the periodic automatic activation

feature provided by the prototype. The number of activations (KPI_001A or KPI_001B) was

documented separately for each route section.

The number of activated eCalls was documented manually for the Indagon prototype (KPI_001B) by

drawing a line each time the prototype was activated and making notes of the times each route

section was started or finished. KPI_003 and KPI_004 were then determined on the basis of notes

written during the test and log files provided by the test bed.

For the Gecko prototype, the number of activations was determined on the basis of a log file

collected on the IVS side, notes on the times each route section was started or finished and log files

provided by the eCall test bed.

For both prototypes, the number of successful MSD transmissions was determined on the basis of

test bed log files and the times each route section had been started or finished.

The successfulness of eCall session (KPI_002B) was determined by checking that a two-way voice

connection had been established and that the MSD was transmitted successfully. The voice

connection was checked manually by persons talking and listening on IVS and test bed side. The

times of voice calls were recorded during the tests, and corresponding modem sessions were then

identified among the test bed log files to see whether the MSDs related to the tested voice


15/05/2014 93 Version 1.3

connections had been successfully received. The contents of those MSDs were not checked at this

stage.

The processing of test results is illustrated in Figure 49.

Figure 49: Processing of test results (FI)

7.3.5.2 Test route

A description of the test route used is provided in Table 28.

Route section Length / km Estimated driving time

Environment

1 Otaniemi-Helsinginkatu-Karhusaarentie

25,3 38 min Urban /suburban

2 Karhusaarentie-Hanko railway station

119,6 1 h 23 min Interurban road

3 Hanko railway station - Nummi-Pusula

103,9 1 h 39 min Regional and rural roads

Hanko railway station - Tenhola 48,2 50 min

Tenhola-Nummi-Pusula 55,7 49 min

4 Nummi-Pusula-Otaniemi 60,9 37 min Motorway VT1 and Ring Road I

Total 314,6 4 h 17 min Table 28: Description of test route (FI)

Test results 7.3.6

7.3.6.1 Gecko IVS prototype

The results for Gecko IVS prototype are presented in and summarized in Table 29 .

Gecko IVS

prototype

(USB output)

Indagon IVS

prototype

(Ethernet output)

Data logging to

a text file

(Laptop with

PuTTY)

Data logging to

a text file

(Laptop with

PuTTY)

Device-specific

postprocessing

of log f iles

Device-specific

postprocessing

of log f iles

Analysis of test

results

- Analysis with

IVS-specif ic

options

depending on

the capabilities

and limitations of

IVS prototypes

eCall testbed

(output as a text

f ile)

Presentation of

test results

- Presentation of

calculated KPIs

and other test

results

A B C D

A: IVS specif ic information

received via vendor-specif ic

interfaces

B: IVS specif ic information as a

text f ile

C: Events and other data

generated by IVS prototypes

expressed in a common f ile

format (plain text)

D: Values of selected KPIs

Notes

(notes made

during the test)


15/05/2014 94 Version 1.3

Table 29: Summary of test results for Gecko IVS prototype (FI)

Table 30: Summary of test results for Gecko IVS prototype (FI)

Success rate of correct MSDs

The MSDs sent by the Gecko IVS prototype were received and decoded by the eCall test bed. All

MSDs received during the test had problems with latitude, longitude and timestamp fields. In other

words, the values decoded from MSDs were implausible. For this reason, the value of KPI_004

(Success rate of correct MSDs) was considered to be zero.

ID of test set:

Unit Result 1 Result 2 Result 3 Result 4

KPI_001a Number of automatically initiated eCalls - 22 94 144 47

KPI_001b Number of manually initiated eCalls - 0 0 0 0

KPI_002a Success rate of completed eCalls using 112 %


KPI_006 Success rate of established voice transmissions %

KPI_007 Duration of voice channel blocking s

KPI_007a

Duration of voice channel blocking:

automatic retransmission of MSD s

KPI_009 Accuracy of position m

KPI_010 Number of usable satellites -

KPI_011 Geometric dilution of precision -

KPI_012 Time between successful positioning fixes s

KPI_013 Success rate of heading information %

KPI_021 Number of successful call-backs -

KPI_022 Success rate of call-backs %

KPI_027 Total response time s

KPI_028 Number of cross-border tests -

KPI_003 Success rate of received MSDs % 100 % 97,50 % 93,94 % 86,67 %

KPI_004 Success rate of correct MSDs % 0 0 0 0

ID of test set: 1

Date: 13.-15.6.2012

Time:

Type of

initiation: a a a a

Roaming (y) (y) (y) (y)

Environment urban interurban rural motorway

Moving vehicle y y y y

No. of involved

vehicles in

incident 1 1 1 1

Name of KPI1

ID of

test set:

Result 1 Unit

KPI_001a Number of automatically initiated eCalls 307 -

KPI_001b Number of manually initiated eCalls 0 -



KPI_003 Success rate of received MSDs 94,69 %

KPI_004 Success rate of correct MSDs 0,00 %

1 Name of KPI

IVS 1

PSAP 1

MNO 1


15/05/2014 95 Version 1.3

Other observations during the test

The IVS activated 307 times in total during the test. However, only 226 calls were successfully

established between IVS and PSAP. This means that in 81 cases of 307 some kind of failure occurred

in call setup or network registration phase before the in-band modem could start transmitting the

MSD. These cases were observed on all route sections (Table 31).

Route section

1 2 3 4

Successful MSDs 17 78 93 26

Started MSDs (cases with call setup from IVS to test bed)

17 80 99 30

Activations with failed network registration or call setup

5 14 45 17

Automatic activations (KPI_001A) 22 94 144 47 Table 31: Summary of activated eCalls, started MSDs and successful MSDs, Gecko prototype (FI)

7.3.6.2 Indagon IVS prototype

Number of manually initiated eCalls (KPI_001B)

The number of manually initiated eCalls was determined on the basis of notes made during the test.

Success rate of completed eCalls using long number (KPI_002B)

Success rate of eCalls using long number was determined by checking the successfulness of both

voice connection and the MSD for a sample of all initiated eCalls made with the Indagon IVS

prototype. In this context, a successful MSD was defined as a MSD which had been successfully

received by the test bed. The contents of the MSDs were not checked at this stage.

Success rate of received MSDs (KPI_003)

The success rate of received MSDs was determined by dividing the number of MSDs successfully

received by the test bed by the number of times the test bed had started the reception of MSD using

the in-band modem.

Success rate of correct MSDs (KPI_004)

The correctness of the MSDs transmitted by the Indagon IVS prototype was checked by decoding

them in the eCall test bed and comparing the results against the MSD standard (EN15722). The

version number and message identifier fields decoded from the MSDs were zeros in all of the MSDs

transmitted by the Indagon prototype. At least some MSDs also had implausible latitude and

longitude values. This result suggests that the prototype does not set correct values for variables


15/05/2014 96 Version 1.3

“MSD version” and “messageidentifier” nor has some other problems with encoding of MSD.

Therefore, the success rate of correct MSDs (KPI_003) was considered to be zero.

The test results for the Indagon IVS prototype are presented in Table 32 and summarized in Table 34.

Table 32:.Summary of test results for Indagon IVS prototype (FI)

ID o

f te

st s

et:

Un

itR

esu

lt 1

Re

sult

2R

esu

lt 3

Re

sult

4R

esu

lt 5

Re

sult

6R

esu

lt 7

Re

sult

8R

esu

lt 9

Re

sult

10

KP

I_00

1aN

um

be

r o

f au

tom

atic

ally

init

iate

d e

Cal

ls-

00

00

00

00

00

KP

I_00

1bN

um

be

r o

f m

anu

ally

init

iate

d e

Cal

ls-

3457

3321

3352

1012

5813

KP

I_00

2aSu

cce

ss r

ate

of

com

ple

ted

eC

alls

usi

ng

112

%

KP

I_00

2bSu

cce

ss r

ate

of

com

ple

ted

eC

alls

usi

ng

lon

g n

um

be

r%

56 %

--

--

-50

%67

%-

-

KP

I_00

6Su

cce

ss r

ate

of

est

abli

she

d v

oic

e t

ran

smis

sio

ns

%

KP

I_00

7D

ura

tio

n o

f vo

ice

ch

ann

el b

lock

ing

s

KP

I_00

7a

Du

rati

on

of

voic

e c

han

ne

l blo

ckin

g:

auto

mat

ic r

etr

ansm

issi

on

of

MSD

s

KP

I_00

9A

ccu

racy

of

po

siti

on

m

KP

I_01

0N

um

be

r o

f u

sab

le s

ate

llit

es

-

KP

I_01

1G

eo

me

tric

dil

uti

on

of

pre

cisi

on

-

KP

I_01

2Ti

me

be

twe

en

su

cce

ssfu

l po

siti

on

ing

fixe

ss

KP

I_01

3Su

cce

ss r

ate

of

he

adin

g in

form

atio

n%

KP

I_02

1N

um

be

r o

f su

cce

ssfu

l cal

l-b

acks

-

KP

I_02

2Su

cce

ss r

ate

of

call

-bac

ks%

KP

I_02

7To

tal r

esp

on

se t

ime

s

KP

I_02

8N

um

be

r o

f cr

oss

-bo

rde

r te

sts

-

KP

I_00

3Su

cce

ss r

ate

of

rece

ive

d M

SDs

%59

%47

,92

%30

,30

%42

,86

%48

,00

%50

,00

%55

,56

%66

,67

%43

,40

%46

,15

%

KP

I_00

4Su

cce

ss r

ate

of

corr

ect

MSD

s%

0 %

0 %

0 %

0 %

0 %

0 %

0 %

0 %

0 %

0 %

ID o

f te

st s

et:

2

Dat

e:

25.9

.201

225

.9.2

012

25.9

.201

225

.9.2

012

26.9

.201

226

.9.2

012

26.9

.201

226

.9.2

012

26.9

.201

226

.9.2

012

Tim

e:

Typ

e o

f

init

iati

on

:m

mm

mm

mm

mm

m

Ro

amin

gy

yy

yy

yy

yy

y

Envi

ron

me

nt

urb

anin

teru

rban

rura

lm

oto

rway

urb

anin

teru

rban

rura

lm

oto

rway

rura

lm

oto

rway

Mo

vin

g ve

hic

ley

yy

yy

yy

yy

y

No

. of

invo

lve

d

ve

hic

les

in

inci

de

nt

11

11

11

11

11

1N

ame

of

KP

I


15/05/2014 97 Version 1.3

Table 33: Summary of test results for Indagon IVS prototype (FI)

Other observations during the test

The IVS prototype was activated 323 times in total during the test. However, only 290 calls were

successfully established between the IVS and the eCall test bed. This means that in 33 of 323 cases

some kind of failure occurred in network registration or call setup phase before the in-band modem

could start transmitting the MSD (Table 34).

Route section

1 2 3 4

Successful MSDs 31 45 38 23

Started MSDs (cases with call setup from IVS to test bed)

57 92 95 46

Activations with failed network registration or call setup

10 17 6 0

Manual activations (KPI_001B) 67 109 101 46 Table 34: Summary of activated eCalls, started MSDs and successful MSDs, Indagon prototype (FI)

The prototype had problems in obtaining position and time from GNSS. This was documented in the

log files generated by the prototype.

7.3.6.3 Calculation of confidence intervals for success rate of received MSDs

The results of the Finnish eCall pilot for year 2012 include the shares of successful MSDs of all

received MSDs (KPI_003). Because only a finite number of trials were carried out for each

combination of prototype and route section, the impact of random variation on the results has to be

estimated by calculating confidence intervals for MSD success rate.

When estimating confidence intervals for the MSD success rate, assumptions on the properties of

the system under analysis are needed. After that, suitable mathematical tools have to be employed

to solve the problem.

ID of

test set:

Result 2 Unit




KPI_002b Success rate of completed eCalls using long number 57 % %



1 Name of KPI

IVS 2

PSAP 1

MNO 1


15/05/2014 98 Version 1.3

When calculating confidence intervals for the success rate of received MSDs (KPI_003), it was

assumed that transmission of a MSD is in fact a Bernoulli trial which can result in either success or

failure. The successes or failures of individual MSDs were assumed to be independent of each other

and each MSD transmitted by the same prototype within the same route section (in the same

operating environment) was assumed to have equal probability of success or failure. When the

previous assumptions are both true and the experiment consists of a fixed number of trials which can

result in either success or failure, the number of successful MSDs transmitted by the same prototype

within the same route section follows binomial distribution.

Means for calculating confidence intervals for a binomially distributed variable can be found in

textbooks (Milton and Arnold 1995). In case of a binomially distributed random variable, confidence

intervals (with probability of 0.95) can be calculated with formula

√ ( )

(1), when and ( )

(Milton and Arnold 1995), where is the probability of success estimated on the basis of the sample

under analysis.

The confidence intervals calculated for KPI_003 using (1) and figures from Chapters 5.1 and 5.2 are

presented in Table 35 and Table 36.

Route section

1 2 3 4

Started MSDs, n 17 80 99 30

MSDs received successfully 17 78 93 26

MSD success rate ( , KPI_003) 100% 97.50% 93.94% 86.67%

Upper confidence interval for p (KPI_003)

- - 98.64% -

Lower confidence interval for p (KPI_003)

- - 89.24% -

Estimation of confidence intervals successful: n x p > 5 and n x (1-p) > 5

no no yes no

Table 35: Confidence intervals for success rate of received MSDs (KPI_003), Gecko prototype (FI)

Route section

1 2 3 4

Started MSDs, n 57 92 95 46

MSDs received successfully 31 45 38 23

MSD success rate ( , KPI_003) 54.39% 48.91% 40% 50%

Upper confidence interval for p (KPI_003)

67.32% 59.13% 49.85% 64.45%

Lower confidence interval for p (KPI_003)

41.46% 38.70% 30.15% 35.55%

Estimation of confidence intervals successful: n x p > 5 and n x (1-p) > 5

yes yes yes yes

Table 36: Confidence intervals for success rate of received MSDs (KPI_003), Indagon prototype. (FI)


15/05/2014 99 Version 1.3

Conclusions 7.3.7

Success rate of received MSDs

The tests have indicated that there is a large difference between prototypes in success rate of MSD

transmission. While the Gecko prototype was able to successfully transmit 94.69% of initiated MSDs,

the corresponding success rate was only 47.24% for the Indagon prototype. This difference has most

likely been caused by differences in hardware and software between the two prototypes. While the

94.69% success rate is reasonably close to the required success rate, 47.24% leaves much room for

improvement. In any case, the evaluated devices were prototypes under development, and the MSD

success rate can likely be improved by solving possible hardware-related problems and by updating

the software of the prototypes.

However, the figures mentioned above do not take into account the number of cases in which some

kind of failure occurs in the call setup phase. In these cases, no call is connected from IVS to test bed

and the in-band modem never starts the transmission of MSD. In case of the Gecko prototype, 81 of

307 calls attempted by the IVS (26.38%) were never connected to the test bed. In case of the Indagon

prototype, this was the result for 33 of the 323 calls attempted (10.22%). The cause behind the large

number of failures in call setup or network registration should be investigated in detail. The

difference between the two prototypes may be related to the differences in the status of mobile

network at the time of testing, the manner the test was carried out or some unknown factor which is

likely related to the mobile network or the SIM card used in the tests.

Success rate of completed eCalls

The success rate of completed eCalls was measured only for the Indagon IVS prototype. The success

rate of completed eCalls (57%) was limited by the relatively low success rate of received MSDs.

However, all MSDs for which the test bed sent acknowledgement to IVS were classified as successful

without looking at the contents. The most important measures to improve the success rate is to

ensure that the MSD is encoded and decoded correctly and to improve the MSD success rate for

example by solving possible hardware-related problems and updating the software of the prototype.

MSD encoding and decoding

Both of the tested prototypes had at least some inconsistencies or other problems in encoding of

MSD. This underlines the need to verify the conformance of the test bed MSD decoder to the

specification in EN15722 and to make the required changes to IVS prototypes to ensure correct

encoding of MSD.


The Indagon prototype used in the tests should be checked to detect possible hardware related

problems which may have affected the success rate of MSD transmission. The encoding of MSD

carried out by the prototypes should be investigated in detail to make sure that the MSD received by

the test bed complies with EN15722. It is also recommended that the firmware of the prototypes

should be updated to make sure that the best possible implementation of the in-band modem is

used in HeERO tests to be carried out in 2013. The causes of calls not connected by the mobile


15/05/2014 100 Version 1.3

network should also be investigated in cooperation with the mobile network operator. Tests to be

carried out in 2013 should also cover a larger geographical area in Finland and involve several MNOs

instead of just one.

References 7.3.9

Götte, S. and Filjar, R 2012. KPIs, test specification and methodology, Final version. Deliverable D4.2 of HeERO. http://www.heero-pilot.eu/ressource/static/files/heero_wp4_d4-2-v1-1-_projectplace_112147_.pdf [accessed 1st October 2012] Milton, J. S. and Arnold, J. C. 1995. Introduction to probability and statistics. McGraw-Hill Book Co, Singapore.

http://www.heero-pilot.eu/ressource/static/files/heero_wp4_d4-2-v1-1-_projectplace_112147_.pdf

http://www.heero-pilot.eu/ressource/static/files/heero_wp4_d4-2-v1-1-_projectplace_112147_.pdf


15/05/2014 101 Version 1.3

7.4 Germany

Test scenarios 7.4.1

The following table gives an overview about the KPIs which were planned to be evaluated in

Germany. In addition to the tests in Germany cross border tests with the Czech Republic were done

where Czech IVS called the German PSAP. The evaluation of these calls is done in chapter 7.4.9

KPI_001a Number of automatically initiated eCalls X

KPI_001b Number of manually initiated eCalls X

KPI_002a Success rate of completed eCalls using 112 (X)

KPI_002b Success rate of completed eCalls using long number X

KPI_003 Success rate of received MSDs X

KPI_004 Success rate of correct MSDs X

KPI_005 Duration until MSD is presented in PSAP X

KPI_006 Success rate of established voice transmissions X

KPI_007 Duration of voice channel blocking X

KPI_008 Time for call establishment (X)

KPI_009 Accuracy of position X

KPI_013 Success rate of heading information X

KPI_015 Success rate of VIN decoding with EUCARIS (X) Table 37: KPIs to be evaluated (DE)

The reasons why some KPIs were not measured are the following:

KPI_002a: The eCall-flag has neither been implemented yet in mobile networks nor in any pilot

environment. It will be available in Germany only after October 2014 and therefore this KPI won’t be

measured during the HeERO project.

KPI_008: It is possible to measure this KPI, but as long as no E112 calls are possible, the measured

times will not reflect the planned reality. In addition, due to different equipment installed within the

networks, using long numbers could lead to very different call setup times. Furthermore the current

used test IVS are always connected to the mobile network.

KPI_013: Heading information was not evaluated during the first test period. In the second test

phase the driver will record the direction indicated by the board instrument (RNS) for later

evaluation.

KPI_015: There is so far no real connection to EUCARIS. For test purposes a data base with only a few

data sets was implemented. Success rate with this test database is always 100% and access times do

not reflect reality.

7.4.1.1 Automatic test scenarios (ATS)

Automatic tests were performed to collect a large amount of data during the test phase. Due to the

fact that the IVS can be reconfigured by SMS commands, normally no attendance is necessary from

IVS manufacturers after installation of the IVS in test vehicles. Some changes of the SIM cards were

necessary, but could easily be performed by the driver. It was possible to monitor the start and end

time of the test and the duration between two eCall initiations.


15/05/2014 102 Version 1.3

At first correct system function was verified with several manual tests. Thereafter the system was

configured for automatic eCalls and one automatic eCall was sent each hour. All data necessary to

evaluate the above mentioned KPIs was logged in parallel by IVS and PSAP. Logged parameters can

also be found in this document.

In Phase 2 the frequency of initiated eCalls will be increased in order to have more data for statistical

analysis and for load tests, to get an impression about the capacity of the PSAP system.

7.4.1.2 Manual test scenarios (MTS)

In manual testing two different test scenarios were performed:

1) MTS-1 – Dedicated test sessions

To verify certain functionalities or to evaluate erroneous behaviour, dedicated test sessions were

defined and then executed. In particular voice communication between drivers of the test vehicle

and PSAP was tested at difficult locations, where environmental conditions for GPS or GSM

connections were not an optimum. When locations and/or other problems were identified during

automatic test sessions, further manual tests were done to identify reasons for the problems. A close

team effort was necessary between IVS manufacturer, PSAP operator and test fleet manager to

coordinate dedicated test sessions.

2) MTS-2 – Additional eCalls during test drives

In addition to the automatic tests, the drivers of the test vehicle were asked to initiate random eCalls

whenever they wanted. These eCalls were only initiated, when the vehicle was not moving, to get

reasonable values concerning the heading and positioning information and to get a realistic scenario

of GSM conditions ‘at the roadside’ to simulate future real-life eCall situations.

7.4.1.3 Mapping of KPIs to test scenarios

The following table provides an overview, how different test scenarios were mapped to defined KPIs.

Used test tracks for eCall tests 7.4.2

Figure 50 shows the main test area, where most of the automatic and manual eCalls were initiated.

KPI_001a Number of automatically initiated eCalls ATS

KPI_001b Number of manually initiated eCalls MTS-1, MTS-2

KPI_002b Success rate of completed eCalls using long number MTS-1, MTS-2

KPI_003 Success rate of received MSDs ATS, MTS-1, MTS-2

KPI_004 Success rate of correct MSDs MTS-1, MTS-2

KPI_005 Duration until MSD is presented in PSAP MTS-1, MTS-2

KPI_006 Success rate of established voice transmissions MTS-1, MTS-2

KPI_007 Duration of voice channel blocking ATS, MTS-1, MTS-2

KPI_009 Accuracy of position MTS-1, MTS-2

KPI_013 Success rate of heading information MTS-1, MTS-2 Table 38: Mapping of test scenarios to KPI (DE)


15/05/2014 103 Version 1.3

Figure 50: Test track 1 (DE)

Test track 2 (Figure 51) was used to evaluate the behaviour of the IVS under non-ideal conditions

concerning GNSS and GSM coverage, for example multi-story car-parks, avenues, urban city area,

mountainous areas, etc.


15/05/2014 104 Version 1.3

Figure 51: Test track 2 (DE)

Test track 3 (see Figure 52) was used for automatic testing (ATR)


15/05/2014 105 Version 1.3

Figure 52 Test track 3 (DE)

Details of used IVS 7.4.3

Table 39 and Table 40 list the information on the providers, vehicles and phone numbers used.

Vehicle MSISDN MNO

H-VW 5562 +4915152258969 Telekom, Congstar

H-VW 5566 +4915205404588 Vodafone


H-VW 5563 +4917692190758 O2,Tchibo

H-VW 5564 +4915737472190 E-Plus, Blau Table 39: details of IVS Continental (DE)

Vehicle MSISDN MNO


H-VW 2771 +491742685107 Vodafone, Fyve


H-VW 2770 +4917692162809 O2,Tchibo

H-VW 2772 +4915734030788 E-Plus, Blau Table 40: details of IVS S1nn (DE)

Determination of KPIs out of eCall log files in the database 7.4.4

All logs of the performed eCalls are recorded within a database for later analysis. Out of this

database, all values for the defined KPIs can be determined as described in the following:


15/05/2014 106 Version 1.3

KPI_001a – Number of automatically initiated eCalls: To determine this KPI, the logged parameter

concerning the activation is evaluated based on information from the PSAP log file.

KPI_001b – Number of manually initiated eCalls: To determine this KPI, the logged parameter

concerning the activation is evaluated based on information from the PSAP log file.

KPI_002b – Success rate of completed eCalls using long number: To determine this KPI, the PSAP

operator and test driver talked to each other at various occasions during the manual test session.

During the “emergency call” both parties filled in a test protocol describing the quality of voice

communication. Successfully completed eCalls are highlighted in the test protocol.

KPI_003 – Success rate of received MSDs: As every MSD is logged, each of them was checked for

invalid data. If – worst case - no MSD was received during the eCall, this was also noted. To

determine pass-fail criteria for this KPI, invalid MSDs were flagged and counted.

KPI_004 – Success rate of correct MSDs: As every MSD is logged on IVS side and on PSAP side, the

data pairs were compared. To evaluate this KPI, incorrect MSDs were flagged and counted.

KPI_005 – Duration until MSD is presented in PSAP: To assess this KPI, two timestamps were

evaluated, one in the IVS log file indicating the eCall activation and one in the PSAP log file indicating

that the MSD was received by the PSAP.

KPI_006 – Success rate of established voice transmissions: To assess this KPI, the PSAP operator and

test driver talked to each other a various occasions during the manual test session. During the

“emergency call” both parties filled out a test protocol concerning the quality of the voice

communication. Successful documented voice transmissions within eCall sessions are counted.

KPI_007 – Duration of voice channel blocking: To assess this KPI, two timestamps were evaluated,

one in the PSAP log file indicating the beginning of the log file and another one in the PSAP log file

indicating that the voice connection was established.

KPI_009 – Accuracy of position: To assess this KPI, the PSAP operator and test driver recorded the

vehicle’s position during the “emergency call”. In the post processing, the test protocols were

compared concerning the position information.

KPI_013 – Success rate of heading information: To assess this KPI, the heading parameter of the IVS log file was evaluated. In the post processing, the value of this parameter was compared with the direction of the road at the position logged by the IVS at the time of the eCall activation, assuming that the vehicle was parked in traffic direction.

Results of KPIs 7.4.5

KPI_001a Number of automatically initiated eCalls 10,697

KPI_001b Number of manually initiated eCalls 248


15/05/2014 107 Version 1.3

KPI_002b Success rate of completed eCalls using long number [%] 68

KPI_003 Success rate of received MSDs [%] 72

KPI_004 Success rate of correct MSDs [%] 100

KPI_005 Duration until MSD is presented in PSAP [s] 9.9

KPI_006 Success rate of established voice transmissions [%] 92

KPI_007a Duration of voice channel blocking [s] 8.7

Table 41: results of the German KPIs

In the following the MSD Presentation Time (KPI 5) and Voice Channel Blocking Time (KPI 7a) are

evaluated in detail, because these are the most relevant parameters for acceptance of eCall.

Evaluation of MSD Presentation Time (KPI 5) 7.4.6

At first we show the distribution of all values (including outliers) of the two IVS.

Figure 53: Distribution of the MSD Presentation Time with different IVS (DE)

The variation of the values in the tests of Continental is wider and the focus seems to be higher than

in the tests with S1nn. The same shows Figure 53 where the relative histograms are plotted.

In the data basis of the next plots (relative histograms) the outliers are already removed.

M S D

IV S

[s]

S1n

n

Co

nti

nen

tal

0

1 0

2 0

3 0

4 0

5 0


15/05/2014 108 Version 1.3

Figure 54: histograms of the MSD Presentation Time with different IVS (DE)

Relative histograms show the relative distribution of the data which are separated in classes. The left

histogram shows that the S1nn values are smaller than the MSD Presentation Time with an IVS by

Continental.

The figure shows like Table 42 that there are more measurements with Continental than with S1nn.

Therefore the dispersion is much higher (standard deviation of 2.3 s versus 0.8 s).

IVS S1nn Continental

Analysed values 65 112

Number of outliers 0 1

Number of values 65 111

Minimum 7 9

25% Percentile 8 10

Median 9 12

75% Percentile 9 14

Maximum 10 20

Mean 8.6 12.3

Std. Deviation 0.75 2.35

Std. Error of Mean 0.093 0.22

S 1 n n

M S D P re s e n ta t io n T im e [s ]

Re

lati

ve

fre

qu

en

cy

[%

]

7 8 910

0

1 0

2 0

3 0

4 0

5 0

C o n t in e n ta l

M S D P re s e n ta t io n T im e [s ]

Re

lati

ve

fre

qu

en

cy

[%

]

910

11

12

13

14

15

16

17

18

19

20

0

5

1 0

1 5

2 0

2 5


15/05/2014 109 Version 1.3

Lower 95% CI of mean 8.4 11.9

Upper 95% CI of mean 8.7 12.8

Geometric mean 8.6 12.1

Skewness 0.16 0.81

Kurtosis -0.34 0.24

Table 42: statistical evaluation of MSD Presentation Time (DE)

Evaluation of Voice Channel Blocking Time (KPI 007a) 7.4.7

Figure 55: Distribution of the Voice Channel Blocking Time with different providers (DE)

The Voice Channel Blocking time with Continental takes longer than with S1nn. Figure 55 also shows

a wider distribution of the Continental values.

V C B

IV S

[s]

S1n

n

Co

nti

nen

tal

0

5

1 0

1 5

2 0

2 5


15/05/2014 110 Version 1.3

Figure 56: histograms of Voice Channel Blocking Time with different IVS (DE)

IVS S1nn Continental

Analysed values 65 186

Number of outliers 0 5

Number of values 65 181

Minimum 5 7

25% Percentile 5 8

Median 6 9

75% Percentile 6 10

Maximum 8 15

Mean 5.9 9.2

Std. Deviation 0.82 1.62

Std. Error of Mean 0.10 0.12

Lower 95% CI of mean 5.8 9

Upper 95% CI of mean 6.2 9.4

Geometric mean 5.9 9.1

S 1 n n

V o ic e C h a n n e l B lo c k in g T im e [s ]

Re

lati

ve

fre

qu

en

cy

[%

]

5 6 7 8

0

1 0

2 0

3 0

4 0

5 0

C o n t in e n ta l

V o ic e C h a n n e l B lo c k in g T im e [s ]

Re

lati

ve

fre

qu

en

cy

[%

]

7 8 910

11

12

13

14

15

0

1 0

2 0

3 0

4 0


15/05/2014 111 Version 1.3

Skewness 0.57 1.13

Kurtosis -0.15 1.28

Table 43: statistical evaluation of Voice Channel Blocking Time (DE)

The statistic values in Table 43 confirm the figures before. The standard deviation of Continental is

even larger than the whole distribution of the values.

Automatic test results 7.4.8

MSISDN

Total

no. of

calls

No. of

received

MSD

Success

probability

%

S1nn +4915734030788 1917 118 6.15

S1nn +491701737904 13 13 100

S1nn +491701748772 905 118 13.04

S1nn +491742685107 125 22 17.6

S1nn +4917692162809 317 93 29.34

Conti +4915152258969 2558 1913 74.78

Conti +4915205404588 2003 1245 62.16

Conti +4915737472190 1211 983 81.17

Conti +4917692190758 1896 1475 77.79

Table 44: automatic test results S1nn and Continental (DE)

For the automatic tests only the success rate has been determined as the success rate of both

systems is quite low.

The success probability of the S1nn tests is much too low and was caused by a synchronisation timer

issue.

The success probability of Continental is better but also not sufficient and needs to be improved for

Phase 2.


15/05/2014 112 Version 1.3

Interoperability tests with CZ 7.4.9

In September and October 2012 interoperability tests with the Czech Republic were done, in which

75 Czech eCalls reached the German PSAP. In the following figure the most important KPIs are

evaluated for these calls.

Figure 57: histogram of KPI 5 with Czech IVS (DE)

In Figure 57 one can see that the majority of eCalls resulted in a MSD Presentation time between 5 to

10 s. An analysis of the calls in which this parameter is smaller showed, that in these cases no MSD

transmission took place. Calls in which KPI 5 is higher than 10 but less than 20 s were caused by

retransmission of MSD due to transmission errors (CRC). The calls with a value of larger than 20 s are

calls in which a retransmission of the MSD was requested after voice communication.

Figure 58: histogram of KPI 7 with Czech IVS (DE)

0

10

20

30

40

50

60

<5

5 -

10

10

- 1

5

15

- 2

0

20

- 2

5

25

- 6

5

65

- 7

0

70

- 7

5

75

- 1

50

15

0 -

15

5

15

5 -

16

0

>1

60

KPI 5 MSD Presentation Time[s]

0

5

10

15

20

25

30

35

40

45

50

< 3 3 -6 6 - 9 9 - 12 12 - 15 15 - 18 18 - 21 > 21

KPI 7 Voice Channel Blocking Time [s]


15/05/2014 113 Version 1.3

In Figure 58 one can see that the majority of eCalls resulted in a voice channel blocking time between

6 to 9 s. An analysis of the calls in which this parameter is smaller showed, that in these cases no

MSD transmission took place. Calls in which KPI 7 is higher than 9 but less than 18 s were caused by

retransmission of MSD due to transmission errors (CRC). The calls with a value of larger than 18 s are

calls in which a retransmission of the MSD was requested after voice communication. The overall

performance of the Czech IVS is better than the two IVS from German manufacturers resulting into a

slightly lower mean value for the 6 to 9 second class of performance.

Conclusions 7.4.10

To generate large amounts of data, automatic tests were performed during the test phase, but for

several reasons the success rate was quite low. Although the number of manually initiated eCalls was

much smaller, they provided much better results for evaluation. In addition to the scheduled tests,

cross border tests with the Czech Republic were performed.

The results of the various KPIs depend on which IVS is used. One IVS achieved for both KPI 5 and

KPI 7 better performance than the other. Same good performance was measured in the cross border

tests with the Czech Republic.

One reason for the bad performance is the missing of time stamps in the IVS logs which prevented

further evaluation of the calls.

Furthermore problems with the timers were identified which have been solved in the meantime.

Another finding is that it was possible to disturb the IVS in such a way, that normal operation was not

possible any longer. This happened for example when a busy number was dialled in the PSAP.

Although dialling 112 is not expected to get a busy tone, due to network errors a busy signal might

still be sent back to the IVS. Even if that is the case, the IVS should continue with standard behaviour

and for example try redialling the number.


In the next phase the log information provided by the IVS manufacturers should to be improved, so

that necessary information is provided with more reliability, but based on a comparable data

structure. The success rate should be increased to a level close to 100%.

It should be evaluated how the overall performance mainly of KPI 5 and KPI 7 can be increased.

As in Germany the MNOs do not support the eCall flag and will not implement it in 2013, the IVS

should be tested together with pilot sites which support the eCall flag like in the Czech Republic.

The analysis of the test results identified the necessity to increase timer t3 and t5 to allow higher

success rates of data transmission. To reduce disturbances on the line imposed by algorithms for

improvement of voice transmission, echo cancellation should optionally be used. The standard

should be modified accordingly.


15/05/2014 114 Version 1.3

7.5 Greece The Greek Pilot Site has experienced significant difficulties since October 2011. The difficulties revolve

around the ability of the administration to complete the procurement process for the purchase of all

technical and logistical provision for the installation of Pan European eCall in Greece. The HeERO

management team have continued to support the pilot site as they continued to secure the necessary

permissions to secure a contractor to fulfil these tasks. The practical impact was that the Greek Pilot

Site has been unable to fulfil any of the operational and performance based activities for WP3 and

WP4 for HeERO 1. The results had been delivered in February 2014 and have been included in all

relevant chapters.

Methodology 7.5.1

On-road eCall tests in Greece were conducted in November-December 2013. We analysed log files

from a total of 1446 eCalls, manual and simulated automatic ones, equally distributed in Thessaloniki

and Athens and in several traffic environments. SIM cards by all three Greek MNOs were used; none

of the networks had the eCall discriminator software available. A long number was used. Two IVS

from Civitronic were used. The eCall PSAP was installed at the MINGR premises.

The following KPIs were evaluated in phase 1 of the Greek pilot tests.

KPI Description

1 KPI_001a Number of automatically initiated eCalls

2 KPI_001b Number of manually initiated eCalls

3 KPI_002b Success rate of completed eCalls using long number

4 KPI_003 Success rate of received MSDs

5 KPI_004 Success rate of correct MSDs

6 KPI_005 Duration until MSD is presented in PSAP

7 KPI_006 Success rate of established voice transmissions

8 KPI_014 Success rate of VIN decoding without EUCARIS

9 KPI_021 Number of successful call-backs

10 KPI_022 Success rate of call-backs

Table 45: KPIs measured in the Greek pilot tests (GR)

The following figure graphically depicts the number of tests performed in Athens and Thessaloniki

per day.


15/05/2014 115 Version 1.3

Figure 59: Number of eCalls per day and region (GR)

To calculate the KPIs log files were automatically stored by the IVS and the PSAP. The data were

stored in an electronic file for each call, which was available at the end of the process for further

analysis. The data were introduced in relational database in an MS SQL Server in order to perform

the analysis and to extract the required KPIs.

Figure 60: Tables to extract the KPIs in an MS SQL server (GR)

Devices

id

IMEI

CallNumber

Base

IVSLog

ID

IVSID

TypeOfECall

NetworkRegistrationMode

Moment

PositionLat

PositionLon

PSAP

IMEI

IMSI

MSD

Network

Log1

Log2

Log3

Log4

Log5

Log6

Log7

Log8

Log9

Log10

Log11

Log12

Log13

Log14

Log15

Log16

Log17

Log18

Log19

Log20

Log21

Log22

Log23

Log24

Log25

Log26

Log27

Log28

Log29

KPI_14

KPIID

Text

CreatedDateTime

FullId

KPI_1A

KPIID

CallID

DateTime

ID

Message_Identifier

Control_automaticActivation

Control_testCall

Control_posCanBeTrusted

Control_vehicleType

VIN

vehPropStorageType_gasoline

vehPropStorageType_diesel

vehPropStorageType_compNat...

vehPropStorageType_liqPropGas

vehPropStorageType_electric

vehPropStorageType_hydrogen

Timestamp

VehLoc_latitude

VehLoc_longitude

Vehicle_direction

RecVehLoc_n_1_latitude

RecVehLoc_n_1_longitude



No_of_Passengers

Optional_additional_data

MSDReceived

CallerID

InbandID

Ticketnum

KPI_1B

KPIID

CallID

DateTime

ID

Message_Identifier


Control_testCall


Control_vehicleType

VIN







Timestamp

VehLoc_latitude

VehLoc_longitude

Vehicle_direction





No_of_Passengers


MSDReceived

CallerID

InbandID

Ticketnum

KPI_21_22

KPIID

FullId

recorder_local_time

Comment

KPI_2B

KPIID

CallID

DateTime

ID

Message_Identifier


Control_testCall


Control_vehicleType

VIN



vehPropStorageType_compNatGas




Timestamp

VehLoc_latitude

VehLoc_longitude

Vehicle_direction





No_of_Passengers


MSDReceived

CallerID

InbandID

Ticketnum

KPI_2BB

KPIID

FullId

recorder_local_time

Comment

KPI_3

KPIID

db_id

msd_record_local_time

msd_status

CallerID

Ticketnum

ticket_creation_time

KPI_4

KPIID

db_id

msd_record_local_time

ID

Message_Identifier


Control_testCall


Control_vehicleType

VIN







ivs_local_time

VehLoc_latitude

VehLoc_longitude

Vehicle_direction

No_of_Passengers


msd_status

CallerID

Ticketnum

ticket_creation_time

KPI_5

KPIID

FullId

Customer

ESN

EventDate

CreatedDate

AssignedDate

CallAuth

Closed

TicketState

TicketOwner

KPI_6

KPIID

FullId

recorder_local_time

Comment

KPI_7A

KPIID

FullId

Customer

ESN

EventDate

CreatedDate

AssignedDate

CallAuth

Closed

TicketState

TicketOwner

KPI_7B

KPIID

FullId

recorder_local_time

Comment


15/05/2014 116 Version 1.3

Results 7.5.2

In total there were 1268 eCalls with simulated automatic initiation. The data for KPI_001a are shown

below.

Figure 61: Number of eCalls with simulated automatic initiation (KPI_001a) (GR)

Figure 62: Number of eCalls with simulated automatic initiation per MNO (KPI_001a) (GR)

In total there were 178 eCalls with manual initiation. The data for KPI_001b are shown below.


15/05/2014 117 Version 1.3

Figure 63: Number of eCalls with manual initiation (KPI_001b) (GR)

Figure 64: Number of eCalls with manual initiation per MNO (KPI_001b) (GR)

The success rate of completed eCalls (KPI_002b) was 58.8% for Attica region and 65.8% for

Thessaloniki.


15/05/2014 118 Version 1.3

Figure 65: Success rate of completed eCalls using long number (KPI_002b) (GR)

Figure 66: Success rate of completed eCalls using long number by MNO (KPI_002b) (GR)

The success rate of received MSDs (KPI_003) is 73.3% for Attica and 80.2% for Thessaloniki.


15/05/2014 119 Version 1.3

Figure 67: Success rate of received MSDs (KPI_003) (GR)

Figure 68: Success rate of received MSDs per MNO (KPI_003) (GR)

The success rate of correct MSDs (KPI_004) was 79.5% for Attica and 81.6% for Thessaloniki.


15/05/2014 120 Version 1.3

Figure 69: Success rate of correct MSDs (KPI_004) (GR)

Figure 70: Success rate of correct MSDs per MNO (KPI_004) (GR)

As regards the KPI_005, duration until MSD is presented in PSAP, there were inconsistencies in the

timestamps of the log files generated by the IVS and the PSAP, therefore this KPI could not be reliably

evaluated based on the log files. Because of the observed difficulties in calculating it there were

another 30 verification eCalls conducted at a later stage in Attica area. In these verification eCalls the

mean KPI_005 was 23 s with a standard deviation of 3 s.

The success rate of established voice transmissions (KPI_006) was 88% for Attica and 87.3% for

Thessaloniki.


15/05/2014 121 Version 1.3

Figure 71: Success rate of established voice transmissions (KPI_006) (GR)

Figure 72: Success rate of established voice transmissions per provider (KPI_006) (GR)

VIN decoding without EUCARIS was successful in 736 of the 784 attempts (KPI_014). The success rate

is therefore very high. It is noted that decoding could not be implemented on some of the trial days;

this is why the success rate is around 94%.


15/05/2014 122 Version 1.3

Figure 73: Success rate of VIN decoding without EUCARIS (KPI_014) (GR)

Figure 74: Success rate of VIN decoding without EUCARIS per MNO (KPI_014) (GR)

In total there were 259 successful call-backs from the PSAP to the IVS (KPI_021). The data about

KPI_021 are shown below:


15/05/2014 123 Version 1.3

Figure 75: Number of successful call-backs (KPI_021) (GR)

Figure 76: Number of successful call-backs per MNO (KPI_021) (GR)

In total there were 330 attempts for call-backs, so the success rates of call-backs (KPI_022) was high.


15/05/2014 124 Version 1.3

Figure 77: Success rate of call-backs (KPI_022) (GR)

Figure 78: Success rate of call-backs per MNO (KPI_022) (GR)

Conclusions 7.5.3

The KPI-002b was rather low, for both regions and for all three MNOs. This should be rather

due to low mobile network coverage at some locations, in which cases the eCall could not

be sent to the PSAP.

The success rate of received MSDs (KPI_003) was satisfactory for both regions and all three

MNOs.

The success rate of established voice transmissions (KPI_006) was satisfactory for both

regions and all three MNOs.

The success rate of correct MSDs (KPI_004) is a bit low. The main cause should be the low

GPS signal coverage in specific locations, which was clearly noted during the trials. Still it

was noted that the IVS failed to store log files in a significant number of eCalls. It has to be

checked further, if the IVS log files, which were used for calculating this KPI, were correct in

general.


15/05/2014 125 Version 1.3

It was noted that the system was sensitive in bad weather conditions, as in cloudy days the

number of problematic eCalls was higher in the same area than in days with good weather

conditions. On such occasions, there were problems in voice connection; no voice was

transmitted to the PSAP or to the vehicle, and problems in the correct MSD transmission, in

which cases the PSAP operator had to request a new MSD.


Based on the results of first phase, the following must be further analysed in Phase 2:

Further analyse specific times in the eCall chain, so as to identify the reasons for delays.

Possibly perform eCalls with the eCall discriminator software and check how this affects

KPIs, especially those relevant to time.

Additional tests focusing on bad weather conditions and on locations with low GSM/GPS

signal and how they affect the KPIs.

Solve the synchronisation problem between IVS and PSAP, so that all time-related KPIs can

be reliably calculated from the log files.

Check the IVS logging and ensure that they are correct.


15/05/2014 126 Version 1.3

7.6 Italy

Preliminary tests 7.6.1

Preliminary tests have been performed in order to evaluate the Inband modem communication and

the MSD transmission between IVS and Varese PSAP. For these tests a fixed long number for the

PSAP has been used.

From the list with recommended KPIs, KPI 7 has not been measured because on IVSs side it is

impossible to acquire low level mobile modem related data. KPI 13 has not been measured mainly

because it was not of our interest to check or verify GPS related information with IVSs still in

prototype version; the major Italian Pilot goal is to check and test the overall eCall chain.

89 manual calls received and with voice connection:

75 calls with MSD correctly received and decoded

14 calls without MSD

6 different IVS calling (2 Italian SIMs in roaming, 2 SIMs from UK, 2 SIMs from Luxemburg)

For these tests there wasn’t the possibility to register the log files from IVSs; for this reason the test

results are based on PSAP log data:

Average MSD reception time: 11.9 sec.

Min MSD reception time: 6 sec

Max reception MSD: 35 sec

Varese pilot tests 7.6.2

The eCall complete chain has been validated in Varese Area with 112 calls and eCall discriminator

management by the Italian Mobile Operator.


15/05/2014 127 Version 1.3

Figure 79: Italian Pilot in Varese Area (IT)

Two different IVSs have been tested and their log files have been stored.

Manual activated calls: 38

MSD received correctly by PSAP: 33

Min PSAP MSD reception time: 5sec

Max PSAP MSD reception time: 18sec

The voice channel has been established between driver and PSAP operator as soon as MSD has been

received. In some cases the Operator couldn’t take in charge the call, probably due to other real

incoming call, and the MSD was transmitted again by the IVS. This problem will be analysed during

the forthcoming tests.


15/05/2014 128 Version 1.3

Figure 80: PSAP log file (IT)

The following KPIs have been evaluated for IVS1 and IVS2:


15/05/2014 129 Version 1.3

Figure 81: IVS1 distribution (IT)

Figure 82: IVS2 Distribution (IT)

Other tests have been performed in November and December; the previous IVSs were used together

with a third IVS. Again some of the analysed data are difficult to be interpreted, specially the success

ID of test set:

Result Unit Result Unit

KPI_001a Number of automatically initiated eCalls 0 - 0 -

KPI_001b Number of manually initiated eCalls 21 - 11 -

KPI_002a Success rate of completed eCalls using 112 % %

KPI_003 Success rate of received MSDs 100 % 100 %

KPI_004 Success rate of correct MSDs 80,95238 % 92,30769 %

KPI_005 Duration until MSD is presented in PSAP 8,411765 s 11,58333 s

KPI_006 Success rate of established voice transmissions 57,14286 % 100 %

KPI_008 Time for call establishment 3,238095 s 23,72727 s

KPI_015 Success rate of VIN decoding with EUCARIS - % - %

KPI_017 Dispatch time of incident data to rescue forces - % - %

KPI_021 Number of successful call-backs - - - -

KPI_022 Success rate of call-backs - % - %

KPI_028 Number of cross-border tests - - - -

Name of KPI

Combination of

IVS/MNO/PSAP:

Combination of

IVS/MNO/PSAP:

11/1/1 2/1/1

0

2

4

6

8

10

12

14

16

18

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17

IVS1 - MSD Presentation Time

MSD Presentation Time

0

2

4

6

8

10

12

14

1 3 5 7 9 11 13 15 17 19 21

IVS1 - Establishment call time

Establishment call time

0

2

4

6

8

10

12

14

16

18

20

1 2 3 4 5 6 7 8 9 10 11 12


Serie1

0

5

10

15

20

25

30

35

40

45

1 2 3 4 5 6 7 8 9 10 11


Serie1


15/05/2014 130 Version 1.3

rate of the completed eCall; this because the information logged in the IVSs state that the voice

channel hasn’t been correctly opened after MSD transmission, but is not possible to understand the

reason from the PSAP log where this information is not recorded yet. During the 2013 test campaign

more data will be recorded in log files.

Further analysis on test results will be performed before 2013 test campaign start in order to resolve

all pending issues on data interpretation.

The following KPIs have been evaluated for IVS1 and IVS3:


ID of test set:

Result Unit Result Unit

KPI_001a Number of automatically initiated eCalls 0 - 0 -

KPI_001b Number of manually initiated eCalls 33 - 13 -

KPI_002a Success rate of completed eCalls using 112 % %

KPI_003 Success rate of received MSDs 100 % 100 %

KPI_004 Success rate of correct MSDs 100 % 100 %

KPI_005 Duration until MSD is presented in PSAP 7,30303 s 8,307692 s

KPI_006 Success rate of established voice transmissions 75,75758 % 84,61538 %

KPI_008 Time for call establishment 3,714286 s 8,153846 s

KPI_015 Success rate of VIN decoding with EUCARIS - % - %

KPI_017 Dispatch time of incident data to rescue forces - % - %

KPI_021 Number of successful call-backs - - - -

KPI_022 Success rate of call-backs - % - %

KPI_028 Number of cross-border tests - - - -

Name of KPI

Combination of

IVS/MNO/PSAP:

Combination of

IVS/MNO/PSAP:

21/1/1 3/1/1

0

2

4

6

8

10

12

14

16

18

20

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33


Serie1

0

0,5

1

1,5

2

2,5

3

3,5

4

4,5

1 3 5 7 9 11 13 15 17 19 21 23 25 27


Serie1


15/05/2014 131 Version 1.3


The following KPIs have been evaluated for IVS2:


0

2

4

6

8

10

12

14

1 2 3 4 5 6 7 8 9 10 11 12 13


Serie1

7,4

7,6

7,8

8

8,2

8,4

8,6

8,8

9

9,2

1 2 3 4 5 6 7 8 9 10 11 12 13


Serie1

ID of test set:

Result Unit



KPI_002a Success rate of completed eCalls using 112 93,75 %



KPI_005 Duration until MSD is presented in PSAP 7,606061 s

KPI_006 Success rate of established voice transmissions 96,875 %

KPI_008 Time for call establishment 24,53125 s

KPI_015 Success rate of VIN decoding with EUCARIS - %

KPI_017 Dispatch time of incident data to rescue forces - %

KPI_021 Number of successful call-backs - -

KPI_022 Success rate of call-backs - %

KPI_028 Number of cross-border tests - -

Name of KPI

Combination of

IVS/MNO/PSAP:

22/1/1

0

2

4

6

8

10

12

14

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33


Serie1

21,5

22

22,5

23

23,5

24

24,5

25

25,5

26

26,5

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31


Serie1


15/05/2014 132 Version 1.3

Conclusions 7.6.3

All performed tests indicated that the end to end emergency call has been correctly developed and

works properly; in order to calculate HeERO KPIs it is needed an improvement in log data

management. Before the 2013 test campaign it is necessary to improve this capability.


In some cases preliminary test results were difficult to analyse. It seems that not all the time

counters were aligned between IVSs and PSAP: KPI related to MSD displayed at PSAP operator desk is

dependent on the fact that operator is available to manage the call and display the data. This

because the information logged in the PSAP log file was not complete. A deeper analysis on PSAP log

files requirements shall be performed before the 2013 test campaign.


15/05/2014 133 Version 1.3

7.7 Romania

Purpose and Scope 7.7.1

This document contains the operating eCall tests conducted in the Romanian pan European eCall

pilot and presents a report of results from Phase 1 testing.

This report presents the results of the evaluation of the reduced set of the Romanian eCall Pilot

experimental and real data.

Structure of Document 7.7.2

The report consists of the following:

- Table with the measured KPI’s during the tests

- Short description of the conditions in which the tests were made

- Values measured on each KPI on a test session

- Time series diagrams of the values of relevant KPIs

- Conclusions regarding the technical aspects identified in the tests

- Conclusions regarding the operational aspects identified in the tests

Description of Equipment 7.7.3

The equipment that were used in the eCall Phase 1 testing and validation in Romania eCall Pilot is

described in the table.


T-IVS Manufacturer Rohde&Schwarz Topex Commercial name of the product NA Hardware revision 0.320 Software revision 1.0 C-IVS Manufacturer Civitronic Commercial name of the product Ubiq eCall IVS Hardware revision NA Software revision NA MODEM PSAP Manufacturer Rohde&Schwarz Topex Commercial name of the product Qutex eCall Hardware revision pgtexF Software revision 4.3.88

Table 46: equipment used in the Romanian pilot site (RO)

Measured KPIs 7.7.4

ID of KPI Name of KPI


15/05/2014 134 Version 1.3

The only recommended KPIs that weren't measured in the Romanian pilot site are KPI_028a (Number of cross-border tests) and KPI_028b (Number of interoperability tests). KPI_028a wasn't measured due to the fact that none of the neighbouring countries have a working eCall system or are participating in HeERO, therefore making it impossible for doing cross-borders tests. KPI_028b wasn't measured because interoperability tests weren't planned for the first operational phase, but it will be measured in the second operational phase.

KPI_001a Number of automatically initiated eCalls KPI_001b Number of manually initiated eCalls KPI_002a Success rate of completed eCalls using 112 KPI_002b Success rate of completed eCalls using long number KPI_003 Success rate of received MSDs KPI_004 Success rate of correct MSDs KPI_005 Duration until MSD is presented in PSAP KPI_006 Success rate of established voice transmissions KPI_007a Duration of voice channel blocking KPI_007b Duration of voice channel blocking: automatic retransmission of MSD KPI_008 Time for call establishment KPI_009 Accuracy of position KPI_013 Success rate of heading information KPI_014 Success rate of VIN decoding without EUCARIS KPI_015 Success rate of VIN decoding with EUCARIS KPI_016 Time for VIN decoding with EUCARIS KPI_019 Dispatch time of incident data to TMC KPI_020 Success rate of presented incident data in TMC KPI_021 Number of successful call-backs KPI_022 Success rate of call-backs KPI_023 GSM network latency KPI_024 112 national network latency KPI_025 112 operator reaction time

Table 47: KPIs measured in the Romanian pilot site (RO)


15/05/2014 135 Version 1.3

Figure 86: Timestamps measured in the Romanian pilot site (RO)


15/05/2014 136 Version 1.3

Testing and Validation Scenarios Description 7.7.5

7.7.5.1 Test sessions – Laboratory tests

We tested all hardware and software for deployment using laboratory environment to ensure that

eCall integration doesn’t affect the existing 112 applications. All laboratory tests were made in a

training environment which is identical with the live system. We used only T-IVS because it permits

to modify MSD field’s value and the B-number used.

L1 to L5 means a laboratory test session.


No of IVS units in roaming

eCall initiation

No of tests made

No of tests measured

L1 5 0 A >100 4

L2 5 0 A >100 8

L3 5 0 A >100 11

L4 5 0 A >100 28

L5 5 0 A >100 64 Table 48: overview of laboratory tests (RO)

7.7.5.2 Test sessions – Real life tests

Tests were made using both Civitronic IVS (C-IVS) equipped with RDS - SIM card because the RDS

network is supporting the eCall flag and Topex IVS (T-IVS) equipped with VODAFONE, ORANGE,

COSMOTE, RDS – SIM cards. There were generated both manual and automatic eCalls by setting the

eCall flag to Automatic eCall or Manual eCall. We have tried to cover different period times (morning,

afternoon, evening, night) in 11 different counties (from 42 of Romania), covering mountain, hills,

plane, highways, national roads and rural roads.

R1 to R12 means a real environment test session.

Code IVS used No of IVS units involved

No of IVS units in roaming

eCall initiation

No of tests made

No of tests measured

R1 T-IVS 5 0 A/M >100 32

R2 T-IVS 3 0 A/M >100 32

R3 T-IVS 5 0 A/M >100 32

R4 T-IVS 5 0 A/M >100 32

R5 T-IVS 3 0 A/M >100 32

R6 T-IVS 4 0 A/M >100 32

R7 T-IVS 2 0 A/M >100 32

R8 C-IVS 2 0 A/M >100 26

R9 C-IVS 2 0 A/M >100 55

R10 C-IVS 2 0 A/M >100 15

R11 C-IVS 2 0 A/M >100 20

R12 C-IVS 2 0 A/M >100 50


15/05/2014 137 Version 1.3

R13 C-IVS 2 0 A/M >100 27

R14 C-IVS 2 0 A/M >100 35

R15 C-IVS 2 0 A/M >100 40

R16 T-IVS/ C-IVS

2 0 A/M >25 0

R17 T-IVS/ C-IVS

2 0 A/M >25 0

R18 T-IVS/ C-IVS

2 0 A/M >25 0

Table 49: overview of field tests (RO)

The test R16, R17 and R18 were made on a test cell in MNO Orange for testing eCall flag during

September - October 2012. We didn’t measure KPI’s.

7.7.5.3 Test sessions – Traffic Management Centre interface

Tests were done for the interface between the 112 PSAP and the Traffic Management Centre. Tests

were done for different scenarios:

Code Scenario

T1 Synchronizing the open cases between the TMC and 112 PSAP T2 Updating the cases presented in the TMC: creating new cases, modifying existing cases,

closing old cases T3 Positioning of the case on the map in the TMC application T4 Sending the correct data through the web interface

Table 50: overview of TMC tests (RO)

The time needed for the data to reach the TMC from the moment it was sent from the 112 PSAP was

measured. The average time for receiving data in the TMC was 23 s out of 96 tests, with a minimum

value of 10 s and a maximum value of 55 s.

Test Session Analysis 7.7.6

7.7.6.1 Laboratory sessions

a. Session L1 with T-IVS

In this session we focus on testing the PSAP modem interface with MSD decoder application. Value

measured was independent from the GSM network used because we generate MSD data directly

from PSAP modem.

KPI

measured

Test 1 Test 2 Test 3 Test 4

Value units Value units Value units Value units

KPI_007 9.4 s 10.7 s 8.5 s 9 s

Table 51: Romanian results for KPI_007 during session L1


15/05/2014 138 Version 1.3

Figure 87: results for KPI_007 during session L1 (RO)

b. Session L2 with T-IVS

In this session we focus on testing the GSM and national network latency. Average value is made on

2 calls/ operator.

KPI

measured

RDS ORANGE COSMOTE VODAFONE

Average units Average units Average units Average units

KPI_023 2.2 s 3 s 1.9 s 2.5 s

KPI_024 1.1 s 0.8 s 1.2 s 1 s

9,4

10,7

8,5 9

Test1 Test2 Test3 Test4

kpi v

alu

e m

easu

red

(se

c)


15/05/2014 139 Version 1.3

Figure 88: results for KPI_023 during session L2 (RO)

Figure 89 results for KPI_024 during session L2 (RO)

c. Session L3 with T-IVS

In this session we analysed the per cent of MSD received versus correct MSD received.

KPI

measured



KPI_003 98 % 97 % 100 % 98 %

KPI_004 96 % 93 % 95 % 90 %

2,2

3

1,9

2,5


kpi v

alu

e m

easu

red

(se

c)

1,1

0,8

1,2

1


kpi v

alu

e m

easu

red

(se

c)


15/05/2014 140 Version 1.3

Table 52: results for KPI_003 and KPI_004 during session L3 (RO)

d. Session L4 with T-IVS

In this session we made a number of 32 eCalls (KPI_001a = 14, KPI_001b = 14).

KPI

measured



KPI_002b 100 % 100 % 100 % 100 %

KPI_003 97 % 95 % 99 % 98 %

KPI_004 82 % 80 % 73 % 81 %

KPI_005 24.2 s 26 s 23.9 s 25 s

KPI_006 100 % 100 % 100 % 100 %

KPI_007a 14.3 s 15 s 13.8 s 15.7 s

KPI_008 4 s 4.5 s 5 s 5.8 s


e. Session L5 with T-IVS

In this session we made a number of 64 eCalls (KPI_001a = 44, KPI_001b = 20) and we also tested the

call-back function (KPI_021=12).

KPI

measured



KPI_002b 100 % 100 % 100 % 100 %

KPI_003 99 % 97 % 100 % 98 %

KPI_004 94 % 93 % 84 % 86 %

KPI_005 25.2 s 23.4 s 25.9 s 21.2 s

KPI_006 100 % 100 % 100 % 100 %

KPI_007a 14.4 s 15 s 13.7 s 16.2 s

KPI_008 3.3 s 4.1 s 4 s 3.8 s

KPI_023 2.1 s 2.8 s 3.1 s 1.9 s


15/05/2014 141 Version 1.3

KPI_024 1.2 s 1.2 s 1.8 s 1.9 s


7.7.6.2 Real life sessions

a. Session R1 with T-IVS

In this session we made 8 eCalls with flag using RDS SIM card and 24 eCalls without flag using a long

number and SIM cards from different GSM operators. We made 12 call-backs with success rate of

100%. KPI_007 and KPI_008 are presented also in diagram.

KPI measured



KPI_002a 100 % 0 % 0 % 0 %

KPI_002b 0 % 100 % 100 % 100 %

KPI_003 100 % 100 % 100 % 100 %

KPI_004 100 % 100 % 100 % 100 %

KPI_005 23.35 s 24.14 s 25.28 s 27 s

KPI_006 100 % 100 % 100 % 100 %

KPI_007 14.3 s 15.9 s 15.3 s 14 s

KPI_008 2.78 s 3.42 s 3.57 s 4.35 s

KPI_013 100 % 100 % 100 % 100 %

KPI_014 100 % 100 % 100 % 100 %

KPI_022 100 % 100 % 100 % 100 %

KPI_023 1.85 s 3.14 s 2.42 s 3.42 s

KPI_024 1.28 s 1.28 s 1.28 s 1.5 s

KPI_025 3.85 s 3.42 s 3.57 s 2.14 s

Table 55: test results during session R1 (RO)

14,3

15,9

15,3

14


kpi v

alu

e m

easu

red

(se

c)


15/05/2014 142 Version 1.3

Figure 90: results for KPI_007 during session R1 (RO)


b. Session R2 with T-IVS




KPI measured RDS ORANGE COSMOTE VODAFONE


KPI_002a 100 % 0 % 0 % 0 %

KPI_002b 0 % 100 % 100 % 100 %

KPI_003 100 % 100 % 100 % 100 %

KPI_004 100 % 100 % 100 % 100 %

KPI_005 23.42 s 24.28 s 25.42 s 24.85 s

KPI_006 100 % 100 % 100 % 100 %

KPI_007 14.9 s 16.6 s 16.3 s 16.1 s

KPI_008 2.85 s 3.42 s 3.28 s 3.42 s

KPI_013 100 % 100 % 100 % 100 %

KPI_014 100 % 100 % 100 % 100 %

KPI_022 100 % 100 % 100 % 100 %

KPI_023 1.78 s 3.57 s 2.42 s 3.57 s

KPI_024 1.28 s 1.28 s 1.28 s 1.14 s

KPI_025 3.57 s 3.42 s 3 s 3.42 s


2,78

3,42 3,57

4,35


kpi v

alu

e m

easu

red

(se

c)


15/05/2014 143 Version 1.3



c. Session R3 with T-IVS




KPI measured


Average units Average units Average units Average Units

KPI_002a 100 % 0 % 0 % 0 %

14,9

16,6

16,3 16,1


kpi v

alu

e m

easu

red

(se

c)

2,85

3,42 3,28

3,42


kpi v

alu

e m

easu

red

(se

c)


15/05/2014 144 Version 1.3

KPI_002b 0 % 100 % 100 % 100 %

KPI_003 100 % 100 % 100 % 100 %

KPI_004 100 % 100 % 100 % 100 %

KPI_005 23.42 s 25 s 26 s 26.14 S

KPI_006 100 % 100 % 100 % 100 %

KPI_007 16.8 s 17 s 15 s 17 S

KPI_008 2.85 s 3.57 s 3.28 s 4.57 S

KPI_013 100 % 100 % 100 % 100 %

KPI_014 100 % 100 % 100 % 100 %

KPI_022 100 % 100 % 100 % 100 %

KPI_023 1.85 s 3.57 s 2.42 s 3.28 S

KPI_024 1.28 s 1.14 s 1.28 s 1.14 S

KPI_025 3.57 s 3.14 s 3 s 2.14 S



23,42

25

26 26,14


kpi v

alu

e m

easu

red

(se

c)


15/05/2014 145 Version 1.3

Figure 95 results for KPI_007 during session R3 (RO)

d. Session R4 with T-IVS



100%. KPI_005 is presented also in diagram.



KPI_002b 0 % 100 % 100 % 100 %

KPI_003 100 % 100 % 100 % 100 %

KPI_004 100 % 100 % 100 % 100 %

KPI_005 22.42 s 25.71 s 25.42 s 27 s

KPI_006 100 % 100 % 100 % 100 %

KPI_007 14.9 s 16.6 s 16.4 s 16.3 s

KPI_008 2.85 s 4 s 3.35 s 4.35 s

KPI_013 100 % 100 % 100 % 100 %

KPI_014 100 % 100 % 100 % 100 %

KPI_021 2 - 2 - 2 - 2 -

KPI_022 100 % 100 % 100 % 100 %

KPI_023 2.21 s 3.85 s 3.35 s 3.42 s

KPI_024 1.5 s 1.42 s 1.28 s 1.5 s

KPI_025 2.07 s 1.85 s 2.21 s 2.14 s

Table 58: Romanian test results during session R4

16,8 17

15

17


kpi v

alu

e m

easu

red

(se

c)


15/05/2014 146 Version 1.3


e. Session R5 with T-IVS



100%. KPI_005 are presented also in diagram.



KPI_002b 0 % 100 % 100 % 100 %

KPI_003 100 % 100 % 100 % 100 %

KPI_004 100 % 100 % 100 % 100 %

KPI_005 23.71 s 25.42 s 25.42 s 26.71 s

KPI_006 100 % 100 % 100 % 100 %

KPI_007 14.8 s 16.5 s 16 s 16 s

KPI_008 3.42 s 4.14 s 3.14 s 4.21 s

KPI_013 100 % 100 % 100 % 100 %

KPI_014 100 % 100 % 100 % 100 %

KPI_022 100 % 100 % 100 % 100 %

KPI_023 2.57 s 3.42 s 3.71 s 3.57 s

KPI_024 1.57 s 1.21 s 1.28 s 1.21 s

KPI_025 1.92 s 1.28 s 1.71 s 1.57 s


22,42

25,71 25,42 27


kpi v

alu

e m

easu

red

(se

c)


15/05/2014 147 Version 1.3


f. Session R6 with T-IVS




KPI measured



KPI_002b 0 % 100 % 100 % 100 %

KPI_003 100 % 100 % 100 % 100 %

KPI_004 100 % 100 % 100 % 100 %

KPI_005 23.57 s 24.57 s 25.57 s 26.42 s

KPI_006 100 % 100 % 100 % 100 %

KPI_007 15.1 s 16.4 s 16.5 s 16.4 s

KPI_008 3.64 s 4.14 s 3.57 s 4.21 s

KPI_013 100 % 100 % 100 % 100 %

KPI_014 100 % 100 % 100 % 100 %

KPI_022 100 % 100 % 100 % 100 %

KPI_023 2.71 s 3.57 s 3.92 s 3.57 s

KPI_024 1.28 s 1.21 s 1.21 s 1.21 s

KPI_025 1.42 s 1.57 s 2 s 1.42 s


23,71

25,42 25,42

26,71


kpi v

alu

e m

easu

red

(se

c)


15/05/2014 148 Version 1.3


g. Session R7 with T-IVS






KPI_002b 0 % 100 % 100 % 100 %

KPI_003 100 % 100 % 100 % 100 %

KPI_004 100 % 100 % 100 % 100 %

KPI_005 23.57 s 24.57 s 19.28 s 27.57 s

KPI_006 100 % 100 % 100 % 100 %

KPI_007 15.1 s 16.4 s 16.5 s 16.4 s

KPI_008 3.64 s 4.14 s 4.71 s 4.42 s

KPI_013 100 % 100 % 100 % 100 %

KPI_014 100 % 100 % 100 % 100 %

KPI_022 100 % 100 % 100 % 100 %

KPI_023 2.71 s 3.57 s 3.42 s 3.14 s

KPI_024 1.28 s 1.21 s 1.28 s 1.14 s

KPI_025 1.42 s 1.57 s 5.14 s 2.14 s


23,57

24,57

25,57

26,42


kpi v

alu

e m

easu

red

(se

c)


15/05/2014 149 Version 1.3

Figure 99 results for KPI_005 during session R7 (RO)

Figure 100: Timeline IVS R&S Topex (RO)

Date KPI_005 KPI_007

1 10.05.2012 23.35 14.3

2 10.05.2012 24.14 15.9

3 10.05.2012 25.28 15.3

4 10.05.2012 27 14

23,57 24,57

19,28

27,57


kpi v

alu

e m

easu

red

(se

c)

0

5

10

15

20

25

30

KPI_005

KPI_007


15/05/2014 150 Version 1.3

5 16.05.2012 23.42 14.9

6 16.05.2012 24.28 16.6

7 16.05.2012 25.42 16.3

8 16.05.2012 24.85 16.1

9 24.05.2012 23.42 16.8

10 24.05.2012 25 17

11 24.05.2012 26 15

12 24.05.2012 26.14 17

13 05.06.2012 22.42 14.9

14 05.06.2012 25.71 16.6

15 05.06.2012 25.42 16.4

16 05.06.2012 27 16.3

17 13.06.2012 23.71 14.8

18 13.06.2012 25.42 16.5

19 13.06.2012 25.42 16

20 13.06.2012 26.71 16

21 21.06.2012 23.57 15.1

22 21.06.2012 24.57 16.4

23 21.06.2012 25.57 16.5

24 21.06.2012 26.42 16.4

25 27.06.2012 23.37 16.1

26 27.06.2012 24.5 14.8

27 27.06.2012 19.28 16.2

28 27.06.2012 22.57 16.4

Sum 689.96 444.6

Mean 24.64142857 15.87857143

Table 62: IVS R&S TOPEX One-Way ANOVA (RO)

Anova: Single Factor (Alpha=0,05) SUMMARY

Groups Count Sum Average Variance

KPI_005 28 689.96 24.64143 2.752939

KPI_007 28 444.6 15.87857 0.697302

KPI_005 KPI_007

Mean 24.64142857 15.87857

Standard Error 0.313558984 0.157809

Median 24.925 16.15


15/05/2014 151 Version 1.3

Mode 25.42 16.4

Standard Deviation 1.659198187 0.835046

Sample Variance 2.752938624 0.697302

Kurtosis 2.638977797 -0.5691

Skewness -1.163282136 -0.73168

Range 7.72 3

Minimum 19.28 14

Maximum 27 17

Sum 689.96 444.6

Count 28 28

Largest(1) 27 17

Smallest(1) 19.28 14

Confidence Level (95, 0%) 0.643369885 0.323797

Table 63: Descriptive Statistics IVS R&S TOPEX (RO)

h. Session R8 with C-IVS

In this session we made 20 automatically generated eCalls and 6 manually eCalls. We made 5 call-

backs with success rate of 50%.

KPI measured

RDS

Average units

KPI_002a 100 %

KPI_003 90 %

KPI_004 100 %

KPI_005 27.7 s

KPI_006 100 %

KPI_007 15.7 s

KPI_008 10.3 s

KPI_013 100 %

KPI_014 100 %

KPI_022 50 %

KPI_023 2.44 s

KPI_024 1.03 s

KPI_025 1.7 s


i. Session R9 with C-IVS


backs with success rate of 80%. This session was made by driving on A2 highway from Bucharest to

Cernavoda (from A point to B point).


15/05/2014 152 Version 1.3

Figure 101: test route Session R9 (RO)

KPI measured

RDS

Average Average

KPI_002a 95 %

KPI_003 95 %

KPI_004 100 %

KPI_005 30.2 s

KPI_006 100 %

KPI_007 16.5 s

KPI_008 12 s

KPI_013 100 %

KPI_014 100 %

KPI_022 80 %

KPI_023 2.64 s

KPI_024 1.03 s

KPI_025 1.7 s


j. Session R10 with C-IVS


backs with success rate of 0%. This session was made by driving on A2 highway from Cernavoda to

Constanta (from B point to C point). We encountered low radio signal.



15/05/2014 153 Version 1.3

KPI measured

RDS

Average Average

KPI_002a 20 %

KPI_003 100 %

KPI_004 80 %

KPI_005 28.9 s

KPI_006 20 %

KPI_007 15.4 s

KPI_008 12 s

KPI_013 100 %

KPI_014 100 %

KPI_022 0 %

KPI_023 2.44 s

KPI_024 1.03 s

KPI_025 1.5 s


k. Session R11 with C-IVS


backs with success rate of 80%. This session was made by driving in Constanta city.


KPI measured

RDS

Average Average

KPI_002a 100 %

KPI_003 100 %

KPI_004 90 %

KPI_005 26.5 s

KPI_006 100 %

KPI_007 14.3 s

KPI_008 10.2 s


15/05/2014 154 Version 1.3

KPI_013 100 %

KPI_014 100 %

KPI_022 80 %

KPI_023 2.64 s

KPI_024 1.03 s

KPI_025 2 s


l. Session R12 with C-IVS


backs with success rate of 50%. This session was made by driving from Constanta to Bucharest on

national roads.


KPI measured

RDS

Average Average

KPI_002a 50 %

KPI_003 100 %

KPI_004 52 %

KPI_005 32.7 s

KPI_006 100 %

KPI_007 18.2 s

KPI_008 10.5 s

KPI_013 100 %

KPI_014 100 %

KPI_022 50 %

KPI_023 2.64 s

KPI_024 1.03 s

KPI_025 4 s



15/05/2014 155 Version 1.3

m. Session R13 with C-IVS


backs with success rate of 100%. This session was made by driving from Bucharest to Sinaia on

national roads (mountain area).


KPI measured

RDS

Average Average

KPI_002a 100 %

KPI_003 100 %

KPI_004 100 %

KPI_005 30.4 s

KPI_006 100 %

KPI_007 15.7 s

KPI_008 11.5 s

KPI_013 100 %

KPI_022 100 %

KPI_023 2.24 s

KPI_024 1.03 s

KPI_025 3.2 s


n. Session R14 with C-IVS


backs with success rate of 100%. This session was made by driving from Sinaia to Pitesti via

Campulung on national roads (mountain area).


15/05/2014 156 Version 1.3


KPI measured

RDS

Average Average

KPI_002a 60 %

KPI_003 100 %

KPI_004 90 %

KPI_005 33.4 s

KPI_006 100 %

KPI_007 16.4 s

KPI_008 12 s

KPI_013 90 %

KPI_014 100 %

KPI_022 100 %

KPI_023 2.44 s

KPI_024 1.03 s

KPI_025 5 s


o. Session R15 with C-IVS


backs with success rate of 80%. This session was made by driving from Pitesti to Bucharest on A1

highway.


15/05/2014 157 Version 1.3


KPI measured

RDS

Average Average

KPI_002a 90 %

KPI_003 100 %

KPI_004 100 %

KPI_005 29 s

KPI_006 100 %

KPI_007 15.2 s

KPI_008 11.1 s

KPI_013 100 %

KPI_014 100 %

KPI_022 80 %

KPI_023 2.64 s

KPI_024 1.03 s

KPI_025 2.7 s


p. Session R16, R17, R18 with C-IVS and T-IVS

The test R13, R14 and R15 were made on a test cell in MNO Orange for testing eCall flag during

September - October 2012. We didn’t measure KPI’s. We only tested that eCalls generated from IVS

with ORANGE SIM cards are received correctly in 112 applications and that MSD is decoded and

presented to eCall operator. The eCalls were successful only in 3G radio network.


15/05/2014 158 Version 1.3

Figure 108: timeline IVS Civitronic (RO)

Date KPI_005 KPI_007

1 18.07.2012 27.7 15.7

2 20.07.2012 30.2 16.5

3 20.07.2012 28.9 15.4

4 20.07.2012 26.5 14.3

5 20.07.2012 32.7 18.2

6 25.07.2012 30.4 15.7

7 25.07.2012 33.4 16.4

8 25.07.2012 29 15.2

9 27.07.2012 29.3 14.6

Sum 268.1 142

Mean 29.78888889 15.77777778

Table 72: IVS Civitronic One Way ANOVA (RO)

Anova: Single Factor (Alpha=0,05) SUMMARY

Groups Count Sum Average Variance

KPI_005 9 268.1 29.7888889 4.861111111

KPI_007 9 142 15.7777778 1.354444444

KPI_005 KPI_007

Mean 29.78888889 15.7778

Standard Error 0.73493092 0.38794

Median 29.3 15.7

Mode #N/A 15.7

0

10

20

30

40

50

60K

PI

VA

LUES

KPI VALUES TIMELINE IVS C

KPI_007

KPI_005


15/05/2014 159 Version 1.3

Standard Deviation 2.204792759 1.16381

Sample Variance 4.861111111 1.35444

Kurtosis -0.273742237 1.51289

Skewness 0.390733563 0.99106

Range 6.9 3.9

Minimum 26.5 14.3

Maximum 33.4 18.2

Sum 268.1 142

Count 9 9

Largest(1) 33.4 18.2

Smallest(1) 26.5 14.3

Confidence Level (95, 0%) 1.694753739 0.89458

Table 73: Descriptive Statistics IVS Civitronic (RO)

Conclusion 7.7.7

All the tests have had a strong operational focus. The goal of this phase was to verify that eCall

standards used to implement eCall pilot in HeERO member states, defined and approved by the

European Standardization Bodies, and the operational procedures created in the project, are useful

and complete and not affecting the normal 112 workflow.

These operations were simulated real-life situations and tested the implemented components.

Also, these overall pilot outcomes will be included in the final recommendations for future eCall

deployment in Europe. These results will be transferred to the European Standards Organizations

(ESO) already involved with eCall standards (CEN and ETSI) to allow finalization and fine-tuning of the

eCall standardization process.

o The in-band modem seems to have a slightly lower robustness than the voice call itself.

o If an eCall session is terminated (with clear down command), there is no possibility to call-

back the IVS.

o Analyses upon the time in which an IVS is generating eCalls (registered / unregistered in a

MNO) must be done.


o The eCall in-band modem work well when radio signal is good. Where radio signal is low the

IVS is making a lot of retries before having a succeeded eCall. We encountered situation

when the voice call was possible from a regular phone, but the IVS didn’t succeed to

generate the call. We consider that an update of the standards is needed to force IVS to

switch to another mobile network after a number of repeated unsuccessful tries.

o In all eCall tests, during an eCall session the MSD has the same data (If a resend MSD

command is sent by the operator, a new MSD is presented to the operator but it contains the


15/05/2014 160 Version 1.3

same information as the initial one).We consider that when a resend MSD requests made by

the eCall operator, the IVS equipment should send actualized data.

o Further activities such as test of foreign IVSs roaming to and within Romania should be done

(also foreign SIM cards).

o The time for the IVS to build the MSD should be reduced. C-IVS used had around 10 sec

before the call was launched


15/05/2014 161 Version 1.3

7.8 Sweden

General 7.8.1

This chapter outlines the Swedish 2012 WP4 Phase 1 results, and presenting data analysis results.

This chapter also describes the statistical methods and procedures used.

The raw data input to the statistical analysis in this report is collected during operational tests

performed during Q1, Q2 and Q3 2012.

For details of the Swedish WP3.2 operation phase, see document “HeERO_WP3_DEL_D3.2-Pilot

Operational Report_v2.0_SE.pdf” or later, as uploaded on the Member State/Sweden area on HeERO

Project Place.

The Swedish pilot has tested all KPIs that are planned to be tested. These plans have a discrepancy

versus the D4.1 summary table for the Swedish pilot.

In the technical review there were many comments on what KPI´s the different pilots have

measured. When the KPI´s were designed at the workshop in Düsseldorf 2011-05-17 it was done

under the condition that all pilots could introduce KPI´s of their liking and all pilots were free to

choose KPI´s. This is because the pilots had different focus and different capability to make

measurements (as stated in the DoW).

The Swedish pilot tested: KPI_001a, KPI_002a, KPI_003, KPI_004, KPI_006, KPI_007a, KPI_021 and

KPI_022.

In addition, the Swedish pilot has provided results on additional KPIs, to better understand the

behaviour of eCall during different radio network conditions. These are reported in D 3.2.

KPI_003a_S: Number of MSD transmissions attempts (Call-back included)

Note: KPI_003a_S is the number of conditional MSD-transmissions attempts in all Test Sequences

(Test 1 and Test 2, see section “Test methodology”). Condition for PULL#1 and PULL#2 attempts in

Test 1 is that voice call was established, condition for PULL#3 in Test 2 is that voice call-back was

established in Test 2. Test 2 (call-back) is only performed, if Test 1 was successfully establishing the

first voice connection.

KPI_003b_S: Number of received and correct MSDs (Call-back included)

Note: This based on the same conditions as KPI_003a_S.

KPI_003c_S: This KPI describes the relation between the number of MSD transmissions attempts at a

given period of time versus the number of received and correct MSD (KPI_003b_S / KPI_003a_S x

100).

KPI_021_S: The number of computer-initiated Call-back attempts from CC-PSAP to VA-IVS.


15/05/2014 162 Version 1.3

In the Swedish Pilot Site this is identical to the number of successfully terminated Test 1 calls; see

section “Test methodology”.

KPI_030_S: This KPI refers to the number of “ABORT” messages the VA-IVS eCall Modem has sent to

the VA-IVS eCall application.

The MSDs were, however, correctly received by the CC-PSAP in these cases. Therefore the MSD

transmissions with “ABORT” message have been counted as “OK” in the Swedish Pilot Site KPI

calculations. For more details of the “ABORT” message see section “Issues found during functional

and operational tests” in D 3.2.

KPI_031_S: This KPI is calculated as follows: KPI_030_S / KPI_003b_S x 100.

KPI_network_001: The purpose of this KPI is to evaluate both the eCall standards and the

implementation from a human communication perspective.

Note: This was measured during test drives in real networks by manually triggered eCalls with real

voice communication and MSD-transmissions. It is a subjective evaluation of voice path blockade and

other artefacts.

KPI_network_002: The purpose of this KPI is to evaluate the behaviour of the eCall systems under

poor conditions concerning the mobile network.

Note: For details on how this KPI was measured see section “Laboratory Tests” TEST case IDs L001a

and L002 in D 3.2.

Will be done:

KPI_001b (success rate of manually initiated eCalls)

In some HeERO documents manually initiated eCall are defined as “the car standing still”. In

our design verification and lab test we see no significant impact of vehicle speed in the eCall

success rate (and there is no reason to think so).

KPI_001a: gives the number of IVS-initiated eCalls with eCall Flag AIeC.

KPI_001b: gives the number of IVS-initiated eCalls with eCall Flag MIeC.

Note: Nearly all eCalls in Swedish Pilot Site were computer-generated (only a few were manually-

initiated). All eCalls used only the eCall Flag “MIeC” for routing in the MSC to CC-PSAP. In contrast to

a later real deployment of eCall the differentiation between AIeC and MIeC makes absolutely no

difference for the KPIs under investigation. Swedish Pilot Site reports the number of computer-

initiated eCalls under KPI1a, although MIeC was used. The number of manually-initiated eCalls

(KPI_001b) is not reported. Only a handful manually-initiated eCalls are performed.

KPI_028: Number of cross-border tests is part of phase 2 testing

Not part of the Swedish trial (and will not be done):


15/05/2014 163 Version 1.3

KPI_002b-success rate of completed eCalls using long number- this KPI will not be

reported, as it is an alternative to KPI_002a, for those pilot sites that do not manage

to get the eCall flag implemented in the mobile networks. During design verification

we have used long numbers on many occasions so we know that it works as well.

KPI_005: Duration until MSD is presented in PSAP: We have measured the reception

of the MSD in the PSAP. There is no possibility to measure when the MSD is

presented on the operator monitor, as the operator needs to take the call in order to

see the MSD. This also depends on which organisation is responsible for this specific

area.

KPI_007b: Duration of voice channel blocking: automatic retransmission of MSD: This

KPI does not apply for Swedish Pilot Site . The VA-IVS follows the eCall standard and

does either send more MSD-redundancy versions (if the PSAP sends LL-NACK), then

this time is already included in other KPIs above, or VA-IVS stops the MSD-

transmission with timeout. No automatic retransmission of the MSD is initiated by

the IVS once the first MSD transmission failed.

KPI_008: Time for call establishment: Call establishment is a standard feature in the

mobile networks and is not at all affected by eCall.

KPI_009: Accuracy of position: Not a focus for the Swedish pilot. Furthermore there

are no requirements in the MSD standards on the accuracy.

KPI_010: Number of usable satellite: this has been a special focus for the Croatian

pilot.

KPI_011: Geometric dilution of precision: this has been a special focus for the

Croatian pilot.

KPI_012 Time between successful positioning fixes: this has been a special focus for

the Croatian pilot.

KPI_013: Success rate of heading information: This is not measured in the pilot, as

there is a vendor specific (Actia) solution to the possible success rate problem and

that has been in successful use in Volvo On Call for many years.

KPI_014: Success rate of VIN decoding without EUCARIS: No EUCARIS connection

from the Swedish PSAP.

KPI_015: Success rate of VIN decoding with EUCARIS: No EUCARIS connection from

the Swedish PSAP.

KPI_016: Time for VIN decoding with EUCARIS: No EUCARIS connection from the

Swedish PSAP.

KPI_017: Dispatch time of incident data to rescue forces: The rescue forces are not

involved in the pilot. In Sweden most 112 calls comes from mobile phones and they

are automatically positioned by mobile network positioning. This info is presented

instantly to the PSAP call taker in the GIS system and the rescue forces are sent to

the position (address). What eCall improves in Sweden is the automatic detection of

a crash (shorter “detection time) and an improved accuracy of the position (GPS-data

versus Mobile positioning), the rest of the procedure is the same as current 112

service.


15/05/2014 164 Version 1.3

KPI_018: Time to activate rescue forces: The rescue forces are not involved in the

pilot. In Sweden most 112 calls comes from mobile phones and they are

automatically positioned by mobile network positioning. This info is presented

instantly to the PSAP call taker in the GIS system and the rescue forces are sent to

the position (address). What eCall improves in Sweden is the automatic detection of

a crash (shorter “detection time) and an improved accuracy of the position (GPS-data

versus Mobile positioning), the rest of the procedure is the same as current 112

service.

KPI_019: Dispatch time of incident data to TMC. The TMCs are not involved in the

pilot. All accidents and emergencies are already today shared between authorities

and TMC´s through “Samverkanswebben” – the cooperative web place, eCall makes

no change to this procedure

KPI_020: Success rate of presented incident data in TMC. The TMCs are not involved

in the pilot. All accidents and emergencies are already today shared between

authorities and TMC´s through “Samverkanswebben” – the cooperative web place,

eCall makes no change to this procedure

KPI_023: GSM network latency. This is a standard feature in the mobile networks and

is not at all affected by eCall. This is a special test for Romania due to their special

eCall network architecture.

KPI_024: 112 network latency. This is a special test for Romania due to their special

eCall network architecture.

KPI_025: 112 Operator reaction time. This is a special test for Romania.

KPI_026: Time for acknowledgement of emergency services. This is a special test for

Romania.

KPI_027: Total response time. This is a special test for Romania.

ID o

f K

PI

Nam

e o

f K

PI

Swed

en–p

lan

ned

test

s as

per

D4

.1

Swed

en-a

ctu

al

KP

Is c

aptu

red

KPI_001a Number of automatically initiated eCalls X X

KPI_001b Number of manually initiated eCalls X Might be done

KPI_002a Success rate of completed eCalls using 112 X X

KPI_002b Success rate of completed eCalls using long

number (X)


15/05/2014 165 Version 1.3

Table 74: KPI Discrepancy versus D4.1 (SE)

Explanations: KPI_002b: Success-rate of completed eCalls using long number will not be of any value

to capture, as the pilot instead has been successful in enabling the eCall flag in two operational

mobile networks.

KPI_007b: Duration of voice channel blocking: automatic retransmission of MSD: This KPI is not

currently planned to be captured.

KPI_003 Success rate of received MSDs X X

KPI_004 Success rate of correct MSDs X X

KPI_005 Duration until MSD is presented in PSAP (X)

KPI_006 Success rate of established voice

transmissions X

X

KPI_007a Duration of voice channel blocking (X) X

KPI_007b Duration of voice channel blocking:

automatic retransmission of MSD X

KPI_008-

KPI_020 --

--

KPI_021 Number of successful call-backs -- X

KPI_022 Success rate of call-backs -- X

KPI_023 GSM network latency -- -

KPI_024 112 National network latency -- -

KPI_025 112 Operator reaction time (X)

KPI_026 Time for acknowledgement of emergency

services X

KPI_027 Total response time X

KPI_028 Number of cross-border tests X

Tests are done, KPI not

reported yet

KPI_NW_001 Voice Channel Disturbance X

KPI_NW_002 Weak Radio Signal Behaviour X


15/05/2014 166 Version 1.3

KPI_025 (112 Operator reaction time), KPI_026 (Time for acknowledgement of emergency services),

and KPI_027 (Total response time) have never been planned to be evaluated, and the X´s in the D4.1

summary table are incorrect.

X Tested as committed

X Not reported in Phase1 test results

Incorrectly marked

additionally tested

No tests are planned

Methodology of data analysis 7.8.2

Raw data from Sweden WP3.2 Operation test phase are used as input to this examination

The examination consists of the following:

- Time series diagrams of the values of relevant KPIs

- Fundamental KPI statistical description for every time series (mean, median, variance,

standard deviation, skewness, kurtosis and histogram with normal probability)

- Discussion

Testing and validation scenarios description 7.8.3

The eCall testing and validation Phase 1 in Sweden comprises of 18 tests session (test sets) as

described in the document “HeERO_WP3_DEL_D3.2-Pilot Operational Report_v2.0_SE.pdf”. The test

was performed in 3 main geographical conditions: Urban, Highway and Rural condition in south and

mid of Sweden.

In this report the consolidated data for each condition are presented, the scenarios (R1 to R6) are

given in the table below:

Code IVS No of

IVS

eCall

trigger

PSAP Operator eCall

flag

112 Moving

vehicle

Condition

R1 VA-IVS 1 A CC-PSAP A Yes Yes Yes Urban

R2 VA-IVS 1 A CC-PSAP B Yes Yes Yes Urban

R3 VA-IVS 1 A CC-PSAP A Yes Yes Yes Highway


15/05/2014 167 Version 1.3

R4 VA-IVS 1 A CC-PSAP B Yes Yes Yes Highway

R5 VA-IVS 1 A CC-PSAP A Yes Yes Yes Rural

R6 VA-IVS 1 A CC-PSAP B Yes Yes Yes Rural

Table 75: Test scenarios description (SE)

Description of consolidated Phase 1 WP3 data 7.8.4

Scenario code No of tests

R1 31

R2 40

R3 499

R4 26

R5 375

R6 80

Table 76: Consolidated Phase 1 WP3 data (SE)

Evaluation results 7.8.5

The results of the statistical analyses by KPIs are systematically presented in this chapter by

scenarios.

R1 Scenario

Scalar statistical indices by KPI are presented in the table below.

R1 Unit Single value

Mean Median Variance Standard deviation

Skewness Kurtosis

KPI_001a Unit less 31 NA NA NA NA NA NA

KPI_002a % 100 NA NA NA NA NA NA

KPI_003 % 100 NA NA NA NA NA NA


KPI_005 S NA 11.000 10.190 5.672 1.019 2.044 2.653


KPI_007a S NA 6.393 5.610 4.205 0.755 2.565 4.821

KPI_021 Unit less 29 NA NA NA NA NA NA

KPI_022 % 97,5 NA NA NA NA NA NA Table 77: R1 Scenario KPI statistics (SE)

The time series of KPI_005 and KPI_007a values are presented below


15/05/2014 168 Version 1.3

Figure 109: R1 time series of KPI_005 and KPI_007a (SE)

The histogram of KPI_005 and KPI_007a values are presented below

Figure 110: R1 histogram of KPI_005 and KPI_007a (SE)

R2 Scenario




Skewness Kurtosis


KPI_002a % 97,5 NA NA NA NA NA NA

KPI_003 % 97,5 NA NA NA NA NA NA



15/05/2014 169 Version 1.3

KPI_005 S NA 10.460 9.282 7.594 1.216 2.474 7.561


KPI_007 S NA 7.068 6.000 7.206 1.154 2.682 8.576


KPI_022 % 97,5 NA NA NA NA NA NA Table 78: R2 Scenario KPI statistics (SE)





R3 Scenario



15/05/2014 170 Version 1.3



Skewness Kurtosis


KPI_002a % 97.0 NA NA NA NA NA NA

KPI_003 % 98.3 NA NA NA NA NA NA


KPI_005 S NA 11.530 9.762 12.244 0.553 1.871 3.351


KPI_007 S NA 7.384 5.609 11.811 0.534 1.792 2.504


KPI_022 % 98.2 NA NA NA NA NA NA Table 79: R3 Scenario KPI statistics (SE)





15/05/2014 171 Version 1.3


R4 Scenario




Skewness Kurtosis





KPI_005 S NA 10.370 9.244 9.353 1.909 2.928 9.219


KPI_007 S NA 7.055 6.000 10.056 2.053 2.836 8.671





15/05/2014 172 Version 1.3




R5 Scenario




Skewness Kurtosis





KPI_005 S NA 10.090 9.524 6.804 0.361 4.686 21.731


KPI_007 S NA 6.073 5.594 6.671 0.355 4.847 22.707



15/05/2014 173 Version 1.3






R6 Scenario.




Skewness Kurtosis




15/05/2014 174 Version 1.3



KPI_005 S NA 11.290 9.315 12.892 1.664 1.839 3.158


KPI_007 S NA 7.861 6.000 11.092 1.432 1.751 2.674








15/05/2014 175 Version 1.3

Description of the equipment 7.8.6

The equipment used in the eCall Phase 1 testing and validation in Sweden is described in the table

below.


IVS

Manufacturer ACTIA

Commercial name of the product (if available), or provisional prototype name

TEM3 (VA-IVS)

Hardware ID TEM3 611159000D

Software ID / revision SWR7.6, build Feb 28 2012; 15:15:30 SWR7.6, build Apr 11 2012; 11:22:10 (improved log feature

Version of standard for eCall Modem 10.0.0

Version of standard for MSD CEN EN 15722 Date: 2010-11

PSAP

Manufacturer Ericsson

Commercial name of the product (if available), or provisional prototype name

CoordCom (CC-PSAP)

Hardware ID HP ProLiant ML350 including Dialogic CG6000

Software ID / revision CXP 901 7546/2 R2E, including eCall In Band adaptor CXC 173 2279/1 R1C

Version of standard for eCall Modem 10.0.0

Version of standard for MSD CEN EN 15722 Date: 2010-11

PLMNs:

The only modification necessary and performed in both participating Swedish PLMNs for Swedish

Pilot Site was the implementation of the eCall Flag handler in the MSCs of the networks. No other

modification was necessary.

Telenor MSC for eCall Drive Tests in Swedish Pilot Site: eCall Flag Handler

Release-Base: M14

Date: 2011-May.

TeliaSonera MSC for eCall Drive Tests in Swedish Pilot Site: eCall Flag Handler

Release-Base: MSS12A CM02

Reference: MSS12A/Part1/ECALLP1A01


15/05/2014 176 Version 1.3

Date: 2012-01-23

Discussion of evaluation results 7.8.7

Note that there is a discrepancy between KPI_001a “Number of initiated eCalls” and the count

number (index) in the time series diagram, the reason for this is that only eCall where a connection

between IVS and PSAP was possible to establish at all is included in the time series.

The number of initiated eCall is not sufficient for some test scenarios to give reliable statistics.

In rural areas the eCall setup failure rate was in general higher than under highway and urban

conditions. That is to be expected, as coverage in thinly populated areas is typically worse and

scattered (more holes in the coverage).

Also the MSD success rate after successful voice path establishment is somewhat lower in rural

areas, but this may be caused by the fact that the car was driving during these tests and in scattered

areas the radio conditions change quicker than on highway and urban conditions.

The biggest number of failures was observed in eCall setup attempts. 52 of 1051 eCall setup attempts

failed overall. This fact reflects the average radio conditions in general during these drive tests in

these regions at these times and dates. ECall setup attempts failed much more often in rural areas.

All other failures are “conditional” failures, i.e. they occur only after the connection between IVS and

PSAP was established, i.e. in comparably good, “selected” radio conditions.

The R3 scenario time series shows a clear pattern with long times for KPI_005 and KPI_007a around

the count index 25, 200, 300 and 420. The figure below shows a map plot of these areas (red dots =

long times), there is a clear correlation when driving the same rout at different occasions.

1: Test date: 2012-03-05, Test time: 19:59h – 21:36h, Göteborg – Trollhättan, Going north

2: Test date: 2012-04-18, Test time: 18:18h – 19:48h, Göteborg – Trollhättan, Going north

3: Test date: 2012-03-06, Test time: 18:49h – 22:54h, Mora – Sundsvall, Going north

4: Test date: 2012-03-07, Test time: 09:07 – 11:55, Sundsvall – Mora, Going south


15/05/2014 177 Version 1.3

Figure 121: R3 scenario time series analysis (SE)

Scenario R1: Of 31 initiated eCall, 100% was successfully completed, and 100% MSD transfers were

successfully completed. 100% of successfully transmitted MSD were successfully decoded at PSAP.

100% of all voice transmission was successful.

Mean time of duration of voice channel blocking was 6.393 s, with standard deviation of 0.755 s.

Mean time of duration until MSD presented in PSAP was 11.0 s, with standard deviation of 1.019 s.

Scenario R2: Of 40 initiated eCall, 97.5% was successfully completed, and 97.5% MSD transfers were


100% of all voice transmission was successful.



Scenario R3: Of 499 initiated eCall, 97.0% was successfully completed, and 98.3% MSD transfers

were successfully completed. 100% of successfully transmitted MSD were successfully decoded at

PSAP. 98.2% of all voice transmission was successful.





92.3% of all voice transmission was successful.


15/05/2014 178 Version 1.3



Scenario R5: Of 375 initiated eCall, 92.0% was successfully completed, and 97.5% MSD transfers

were successfully completed. 100% of successfully transmitted MSD were successfully decoded at

PSAP. 97.3% of all voice transmission was successful.





75.0% of all voice transmission was successful.



Conclusions 7.8.8

The intent of the Swedish HeERO pilot has been to evaluate if the requested performance of the

eCall service can be met with a deployment of the approved eCall standards in the existing public

mobile networks and within the existing 112 system. This means that the testing has had a strong

focus on the eCall standards and capturing the key performance indicators, the KPIs. Other issues,

such as the response time of the rescue services and ambulances, use of EUCARIS and use of VIN in

the operational rescue chain, as well as non-operational issues, like legal liability, periodic time

inspections, change of a car ownership, etc. have not been considered in this report.

The outcome of the tests performed by Swedish Pilot Site and reported in this document confirm

that the pan-European eCall is working according to expectations when used in communication

environments with good quality (high strength) of public mobile network radio signals. The results

show that the performance and reliability of eCall is lower in rural areas than in urban areas, as

expected as the coverage generally is better in urban areas. The In-Band Modem seems to perform

less robust than the 112 voice call itself.

The testing also gave us insight to the effects on the PSAP operator, the call taker, when answering

an eCall where the MSD transmission delays or fails.

Our test results are collected in the two major PLMNs in Sweden, where the networks are known to

generally provide a good and almost seamless coverage over populated areas.

Our results have verified this in selected parts of Sweden.

A concern raised by the Swedish Pilot Site partners is if eCall can be expected to work reliably enough

throughout the European Union, as more than 200 mobile network operators and an extensive

number of PSAPs are required to support the EU-wide eCall service.

The pilot also recognizes that the time expected for the eCall set-up time while the dormant mode is

used as another important issue. How will the resulting longer set-up time affect the number of


15/05/2014 179 Version 1.3

“silent” eCalls to SOS Alarm, the Swedish PSAP operator? A long call set-up time may give many

“silent” calls, as passengers that are able to, often try to leave a crashed car quickly after an airbag

deployment.

Conclusions 7.8.9

The intent of the Swedish HeERO pilot has been to evaluate if the requested performance of the

eCall service can be met with a deployment of the approved eCall standards in the existing public

mobile networks and within the existing 112 system. This means that the testing has had a strong

focus on the eCall standards and capturing the key performance indicators, the KPIs. Other issues,

such as the response time of the rescue services and ambulances, use of EUCARIS and use of VIN in

the operational rescue chain, as well as non-operational issues, like legal liability, periodic time

inspections, change of a car ownership, etc. have not been considered in this report.

The outcome of the tests performed by Swedish Pilot Site confirms that the pan-European eCall is

working according to expectations when used in communication environments with good quality

(high strength) of public mobile network radio signals. The results show that the performance and

reliability of eCall is lower in rural areas than in urban areas, as expected as the coverage generally is

better in urban areas. The In-Band Modem seems to perform less robust than the 112 voice call

itself.

The testing also gave us insight to the effects on the PSAP operator, the call taker, when answering

an eCall where the MSD transmission delays or fails.

Our test results are collected in the two major PLMNs in Sweden, where the networks are known to

generally provide a good and almost seamless coverage over populated areas.

Our results have verified this in selected parts of Sweden.

A concern raised by the Swedish Pilot Site partners is if eCall can be expected to work reliably enough

throughout the European Union, as more than 200 mobile network operators and an extensive

number of PSAPs are required to support the EU-wide eCall service.

The pilot also recognizes that the time expected for the eCall set-up time while the dormant mode is

used as another important issue. How will the resulting longer set-up time affect the number of

“silent” eCalls to SOS Alarm, the Swedish PSAP operator? A long call set-up time may give many

“silent” calls, as passengers that are able to, often try to leave a crashed car quickly after an airbag

deployment.


Some of the listed activities are planned and some would require additional funding.

- Test of foreign IVSs (other implementations) roaming to and within Sweden. These IVSs must

ensure that they use only PLMNs, where eCall Flag handling is provided.

- Test of eCall behaviour for dormant IVSs.


15/05/2014 180 Version 1.3

- Interoperability testing in Finland and over the border into Russia. This to confirm

interoperability with ERA GLONASS (support from the Russian project is required).

- Evaluate the impact of the speed of the IVS. So far Swedish Pilot Site performed tests with

high speed. This is relevant for manually triggered eCalls, but is untypical for automatically

triggered eCalls. To understand the impact of speed tests with standing or slow moving IVS

should be performed.

- Evaluate the impact of combined coverage of all PLMNs in a certain location, when using

112-eCall with eCall Flag handling. Theoretically the combined coverage should be higher. To

perform this evaluation the IVS must be standing still to have sufficient time for selecting the

best available PLMN. Of course all PLMNs must have eCall Flag handling active.

- Evaluate effect on MSD success rate with back-up solutions, for example back-up SMS.

Reference 7.8.11

[1] W.G. Cochran, The distribution of the largest of a set of estimated variances as a fraction of their

total, Annals of Human Genetics (London), 11(1), 47–52 (January 1941).

[2] D.L. Massart, B.G.M. Vandeginste, L.M.C. Buydens, S. de Jong, P.J. Lewi, J. Smeyers-Verbeke,

Handbook of Chemometrics and Qualimetrics: Part A, Elsevier, Amsterdam, The Netherlands, 1997,

ISBN 0-444-89724-0.

[3] P. Konieczka, J. Namieśnik, Quality Assurance and Quality Control in the Analytical Chemical

Laboratory – A Practical Approach, CRC Press, Boca Raton, Florida, 2009, ISBN 978-1-4200-8270-8.

[4] J.K. Taylor, Quality Assurance of Chemical Measurements, 4th printing, Lewis Publishers, Chelsea,

Michigan, 1988, ISBN 0-87371-097-5.

[5] W. Horwitz, Harmonized protocol for the design and interpretation of collaborative studies,

Trends in Analytical Chemistry, 7(4), 118–120 (April 1988).

[6] ISO Standard 5725–2:1994, “Accuracy (trueness and precision) of measurement methods and

results – Part 2: Basic method for the determination of repeatability and reproducibility of a standard

measurement method”, International Organization for Standardization, Geneva, Switzerland, 1994,

available at: http://bit.ly/P2AwhJ.

[7] R. Moore, Mathematics Department, Macquarie University, Sydney, Australia, 1999, available at:

http://bit.ly/QGUS1X.

[8] R.U.E.’t Lam, Scrutiny of variance results for outliers: Cochran's test optimized, Analytica Chimica

Acta 659, 68–84 (2010); doi:10.1016/j.aca.2009.11.032

[9] R.U.E.’t Lam, Variance Outlier Test, blog: http://rtlam.blogspot.com/

[10] Table of critical values of the F-distribution: NIS


15/05/2014 181 Version 1.3

[11] Hughes, I G and T P A Hase, Measurements and their Uncertainties: A Practical Guide to Modern

Error Analysis, Oxford University Press, Inc., New York, NY, 2010, ISBN 978-0199566334.

[12] Maindonald, J and W J Brown, Data Analysis and Graphics Using R - an Example-Based Approach

(3rd edition), Cambridge University Press, Cambridge, UK, 2010, ISBN 978-0521762939.

[13] Ott, R L and M Longnecker, An Introduction to Statistical Methods and Data Analysis (5th ed),

Duxbury, Thomson Learning, Inc., Pacific Grove, CA, 2000, ISBN 978-0534251222.

[14] R Development Core Team (2010). R: A language and environment for statistical computing. R

Foundation for Statistical Computing, Vienna, Austria. ISBN 3-900051-07-0, available at:

http://www.R-project.org, accessed on 1 June 2011.

[15] Wikipedia, Cochran's C-test, available at: http://bit.ly/OWuldq.

[16] Wikipedia, Grubbs' test for outliers, available at: http://bit.ly/NluMe6.

[17] National Institute of Standards and Technology (NIST), Detection of Outliers, NIST/SEMATECH e-

Handbook of Statistical Methods, 2012, available at: http://1.usa.gov/RXP1QU


15/05/2014 182 Version 1.3

7.9 The Netherlands

In General 7.9.1

The Netherlands is participating in the European HeERO-project. Commission by the authorities,

Grontmij has executed several drive tests to evaluate 4 IVS-systems and 3 MNO’s. In order to

appreciate the companies participated in these tests, the results are made anonymous. This prevents

that the companies will be judged on their prototypes and gives them the opportunity to improve

their products.

The anonymous IVS 1 and 2 is the same product from the same manufacturer. This IVS is doubled

tested because two prototypes were available and the test system accommodated an extra IVS.

Work package 4.2 describes which KPI should be tested by each participating country. Next table

shows which KPIs were included in the drive tests.

KPI_001a: Number of automatically initiated eCalls Tested

KPI_001n: Number of manually initiated eCalls Tested

KPI_002a: Success rate of completed eCalls using 112 Not tested, during the test the MNOs

were not equipped to use to eCall-flag

KPI_002b: Success rate of completed eCalls long number Tested

KPI_003: Success rate of received MSDs Tested

KPI_004: Success rate of correct MSDs Tested

KPI_005: Duration until MSD is presented in PSAP Tested

KPI_006: Success rate of established voice transmission Tested

KPI_007a: Duration of voice channel blocking Tested

KPI_008: Time for call establishment Tested

KPI_009: Accuracy of position Tested

KPI_013: Success rate of heading information Tested

KPI_015: Success rate of VIN decoding with EUCARIS Not tested, EUCARIS not part of the

Dutch test setup

KPI_016: Time for decoding VIN with EUCARIS Not tested, EUCARIS not part of the

Dutch test setup


15/05/2014 183 Version 1.3

KPI_019: Dispatch time of incident data to TMC Tested

KPI_020: Success rate of presented incident data in TMC Tested

KPI_022: Success rate of call-backs Not tested, already tested by J. Van

Hattem (Dutch HeERO (static)testing,

February 23/24 2012)

Table 83: Included KPIs in the drive tests (NL)

Not all of the recommended KPI´s The Netherlands have committed to test could be tested. This is caused by the fact that no MNO has already implemented the eCall flag. As a result the Dutch pilot could only perform the tests by using a long number instead of using the number 112. Only if a MNO implements the eCall Flag in 2013, The Netherlands can test KPI 002a (Success rate of completed eCalls) and KPI_028a (number of cross border tests). The Netherlands have started with interoperability tests (KPI_028b) and will perform more of these in Phase 2.

Methodology 7.9.2

7.9.2.1 Drive tests

This report describes the ‘drive test’ that was conducted as part of the HeERO-NL project. A drive test

is a test in which a specific route is driven, during which several eCalls are triggered. It is almost

impossible to execute such a test reliably without using specific test equipment. Therefor

Rijkswaterstaat has developed a specific test device that is able to do automatic triggering of

(currently) at maximum four (4) in-vehicle systems (IVSs). This device is assembled inside a ski-box

that can be mounted onto a car. In the remainder of this document this test device will be referred

to as ‘Dutch First Automated eCall Test Setup’ (D-FACTS)

Figure 122: the ski-box attached to a test-vehicle (left) and D-FACTS inside the ski box (left) (NL)

D-FACTS are capable of automatically triggering both ‘manual’ eCalls and ‘automatic’ eCalls (provided

the IVSs offer a triggering for both variants). The triggering can be location based, i.e. triggers for


15/05/2014 184 Version 1.3

specific locations, or time based: a trigger each X minutes. In this specific drive test eCalls were

triggered each minute, resulting in four eCalls per minute.

The eCalls were, when received at the PSAP, routed to a test setup which received the MSD and

automatically switched to a voice connection. After this the call was disconnected by the PSAP side.

In order to measure the correctness of the GPS-position determined by each IVS, the vehicle was

equipped with a separate commercial available GPS-logger. This system logged the GPS-position

every 10 meters.

Three mobile network operators (MNOs) participate in the test. As the drive test was carried out

over several days, each day different combinations of IVSs and MNOs were tested.

Day 1 Day 2 Day 3 Day 4

IVS 1 MNO 2 MNO 2 MNO 3 MNO 3




Table 84: Combinations of MNOs and IVSs tested during the drive test (NL)

The drive test took place on four days: August 31, September 4, 5 and 7. The driver of the vehicle

drove a predetermined route through the Rotterdam-Rijnmond region in the Netherlands (Figure

123). This route was designed so that specific situations would be encountered that might have an

effect on the GPS accuracy (urban canyon) or on the mobile reception:

Low mobile coverage (lage dekking)

Rotterdam Port (haven)

Rotterdam-The Hague Airport (vliegveld)

Rotterdam city centre (stedelijk gebied)

High voltage cables and pylons (hoogspanning)

Tunnel (tunnel)


15/05/2014 185 Version 1.3

Figure 123: the route of the drive test (NL)

7.9.2.2 Data collection and processing

The systems active during the drive test logged the important timestamps and the sent and received

data. In Table 85, per system the relevant data from the loggings are described.

System Logging

D-FACTS Telephone number, time eCall triggering per IVS,

PSAP Telephone number, CISID, time eCall reception, time back to speech, time voice

connection, time end eCall, MSD

TMC CISID, time MSD received, MSD

GPS-

logger

Time log, GPS-location

Table 85: Relevant logging per system (NL)

To evaluate individual eCalls, and to calculate the KPIs, the loggings from the systems had to be

matched. As soon as an eCall is received in the PSAP system a unique ID is tagged to it which can be

found in the other systems (emergency services and traffic management centre) as well. That ID can


15/05/2014 186 Version 1.3

however not be used to match the eCall triggering (D-FACTS) with receiving the call in the PSAP

system.

It is possible to identify the IVS by its phone number, so that in combination with the timestamp of

triggering the eCall should suffice to match the loggings from D-FACTS, PSAP and the GPS-logger.

Unfortunately it turned out that the PSAP system had not been synchronised correctly. This meant

that the timestamp of triggering the eCall (D-FACTS) could not be related to the timestamp the eCall

was received by the PSAP. To overcome this deficit a time window of one minute was used to match

the individual eCalls between D-FACTS and PSAP.

After matching all loggings were combined to one dataset.

Figure 124: The process of linking the loggings of the systems (NL)

There are two possibilities if an eCall triggered by D-FACTS cannot be matched with logging in the

PSAP system. Either the eCall is not received by the PSAP (e.g. no mobile network coverage), or the

previous eCall was not ended. D-FACTS have no knowledge about calls in progress on the IVSs. So it

can trigger a new eCall while the IVS is still in communication with the PSAP. Conformant to the

specification the IVS will ignore the triggering, but D-FACTS will log it anyhow. Such situations have

been filtered from the loggings.

7.9.2.3 Outliers

To detect mismatches of the loggings in the dataset, or to detect error in loggings themselves,

statistical analysis for outliers is required. An outlier is a value in a dataset which differs greatly from

the other values in the same dataset, that the assumption is that it is an error and not a part of the

distribution of the dataset.

For every IVS, outliers are detected using the Turkey’s Outlier Filter. The quartiles (Q1 and Q3) of the

dataset are used to define a lower and upper limit. Values outside those limits are assumed to be an

outlier.


15/05/2014 187 Version 1.3

lower limit = Q1 – 1, 5 * (Q3 – Q1)

upper limit = Q1 + 1, 5 * (Q3 – Q1)

Description of KPI’s 7.9.3

Several KPIs have been defined within work package 4.2 of the HeERO project. Not all are relevant

for the national situations, so each participating country will perform a subset of these. In the Dutch

HeERO project the KPIs described in this chapter have been calculated.

7.9.3.1 KPI_001a: Number of automatically initiated eCalls

This KPI measures the total number of ‘automatic’ eCalls. This simply is the number of ‘automatic’

eCalls that has been triggered by D-FACTS and is present in the dataset (not filtered).

7.9.3.2 KPI_001b: Number of manually initiated eCalls

This KPI measures the total number of ‘manual’ initiated eCalls. This simply is the number of ‘manual’

eCalls that has been triggered by D-FACTS and is present in the dataset (not filtered)

7.9.3.3 KPI_002b: Success rate of completed eCalls using long number

This KPI describes the relationship between the numbers of initiated eCalls versus the number of

successful eCalls while the long number of a PSAP is used as a telephone number for the emergency

call.

An eCall is considered to be successful when the voice connection between the IVS and PSAP is

established, and the received MSD in the PSAP contains a valid GPS-location, direction and VIN-

number.

7.9.3.4 KPI_003: Success rate of received MSDs


received MSD’s in the PSAP. A MSD is considered as received when a valid GPS-location, the vehicle

direction or the VIN-number is presented in the PSAP.

7.9.3.5 KPI_004: Success rate of correct MSDs


correctly received MSD’s in the PSAP. A MSD is considered as correctly received when a valid GPS-

location, the vehicle direction and the VIN-number are presented in the PSAP.

7.9.3.6 KPI_005: Duration until MSD is presented in PSAP

This KPI describes the duration from the initiation (automatically or manually) of an eCall to the

presentation of the MSD content in the PSAP.

Only eCalls with a correct MSD are included in this calculation.

7.9.3.7 KPI_006: Success rate of established voice transmissions

This KPI describes the relation between the number of initiated voice transmissions versus the

number of successfully established voice transmissions between the vehicle and the PSAP.


15/05/2014 188 Version 1.3

A voice transmission is initiated when the PSAP finishes the data transmission and starts the voice

transmission. A voice transmission is considered to be successful when the eCall is received in and

accepted by the voice taker application

7.9.3.8 KPI_007a: Duration of voice channel blocking

This KPI represents the time the transmission of the MSD blocks the voice channel. The time the

voice channel is blocked is defined as the moment from the eCall reception in the PSAP until the

moment the eCall is send to the voice taker application.

Only eCalls with a correct MSD are included in this calculation.

7.9.3.9 KPI_008: Time for call establishment

This KPI represents the time necessary for the eCall setup. The time needed for the eCall setup is

defined as the moment from the eCall imitation at the IVS until the moment the eCall is received at

the PSAP.

7.9.3.10 KPI_009: Accuracy of position

This KPI describes the difference between the reported position by the IVS in the MSD and the actual

position of the vehicle (GPS-logger). As described in section 2.2, the time of the eCall imitation is

matched with the GPS-logger to determine the actual position of the vehicle.

7.9.3.11 KPI_013: Success rate of heading information

This KPI describes the accuracy of the heading information of the vehicle reported in the MSD. The

actual heading information of the vehicle is calculated using the two most relevant points (based on

the timestamp) of the GPS-logger. These points never differ more than 10 meters and offer a good

basis to calculate the heading. The heading information in the MSD is correct if the absolute

deviation from the calculation heading is no more than 37.5 degrees.

7.9.3.12 KPI_019: Dispatch time of incident data to TMC

This KPI refers to the time it takes to inform the TMC operators after the initiation of the eCall. This

KPI is calculated using the moment the eCall is initiated by the IVS and the moment the MSD is

received in the TMC-application.

7.9.3.13 KPI_020: Success rate of presented incident data in TMC

This KPI describes the number of times the incident data received in the TMC is accurate and

received in an acceptable time (1 minute). This KPI is calculated by comparing the MSD in the PSAP

with the incident data received by the TMC, and the time it takes from initialising the eCall and the

reception of the data in the TMC.

Data selection 7.9.4

To calculate the in chapter 3 presented KPI’s, the data from the loggings needs to be analysed. This

chapter describes what data is used, which subsets are created and how outliers are detected.


15/05/2014 189 Version 1.3

7.9.4.1 Data used

Figure 125 shows what data from the different loggings is used for calculating the KPIs.

Figure 125: Overview of the data used for analysing the KPIs (NL)

7.9.4.2 Subsets

In Figure 126 the different subsets for calculating the KPI’s are presented.

Telefone

number

KPI_001a

Number of automatic eCalls

KPI_001b

Number of manual eCalls

Telefone

number

KPI_002b

Success rate completed eCalls

KPI_001a /

KPI_001b

Time voice

connection

GPS-location

Direction

VIN

KPI_003

Success rate received MSDs

KPI_004

Success rate correct MSDs

KPI_005

Duration MSD presented in PSAP

KPI_006

Success rate established voice

connections

KPI_007a

Duration voice channel blocking

KPI_008

Time for call establishment

KPI_009

Accuracy of position

KPI_013

Success rate of heading information

KPI_019

Dispatch time of incident data to TMC

KPI_020

Success rate presented incident data

TMC

KPI_001a /

KPI_001b

GPS-location

Direction

VIN

KPI_001a /

KPI_001b

GPS-location

Direction

VIN

KPI_001a /

KPI_001b

Time eCall

triggering

Time back to

speechMSD

KPI_001a /

KPI_001b

Time back to

speech

Time voice

connection

KPI_004Time eCall

reception

Time back to

speech

KPI_001a /

KPI_001b

Time eCall

triggering

Time eCall

reception

KPI_004 GPS position GPS position

Direction GPS positionKPI_004

KPI_004Time eCall

triggering

Time incident

data reception

KPI_019

GPS-location

Direction

VIN

GPS-location

Direction

VIN

Log

D-FACTSKPI

Log

PSAP

Log

GPS-logger

Log

TMC


15/05/2014 190 Version 1.3

During the drive tests, D-FACTS registered the initialisation of 6901 eCalls. 5378 eCalls (78%) of these

can be matched with an eCall in the PSAP-logging. The remaining eCalls are not received by PSAP.

This can be explained by multiple causes:

A new eCall is initialised by D-FACTS, while the previous eCall is not finished yet. According the

specifications of the IVSs, the new eCall is not initialised by the IVS and therefore not received by

PSAP.

During the drive tests, multiple times an IVS crashed. If D-FACTS initialised a new eCall while the

IVS was crashed, the new eCall is not initialised by the IVS and therefore not received by PSAP.

There were some technical issues regarding the PSAP. Due to these issues, not all eCalls are

received by the PSAP.

In the analysis of the Dutch test setup, only the eCalls received by the PSAP and matched to D-FACTS

are taken into account. The remaining initialised eCalls by D-FACTS are subtracted from the data.

All initialised eCalls received by the PSAP started the transmission of the MSD. Unfortunately, not all

MSD’s were received by the PSAP. 4836 eCalls were received (90%), and 4351 eCalls were correct

(81%). From the correct eCalls the accuracy of GPS-location and the heading direction are calculated.

The completed MSD’s (including the incorrect ones) are used to determine the success rate of

establishing the voice connection from the vehicle to the call taker application. The same subset is

used to calculate the success rate of transferring the data to the TMC.


15/05/2014 191 Version 1.3

Figure 126: Overview of the subsets used for analysing the KPI’s (NL)

7.9.4.3 Outliers

Some KPI’s are not arbitrary (yes or no), but refer for example to a time interval. For these KPI’s it is

important to detect outliers (section 2.3), in order to get a reliable analysis of the data. As described,

outliers are detected using the Turkey’s Outlier Filter.

Using the Turkey’s Outlier Filer, outliers are visualised with a boxplot (Appendix A). The median

together with the first and third quarter of the data are displayed with a box. 50% of all data is

located within this box. The outliers are displayed with dots above and under the limits of the filter.

Outliers are caused by different reasons. For example a mismatch of the loggings or a time

synchronisation error in the log files results in very high or low values. Using the outlier filter these

values are subtracted from the data.

Results drive test 7.9.5

In this chapter the results of the KPI’s are presented.

Initiated eCalls D-FACTS

6901

Matches D-FACTS <> PSAP

5378

MSD transmissions started

5378

MSD transmissions

completed

4836

KPI_001a automatic eCalls:

1587 (30%)

KPI_001b manual eCalls:

3791 (70%)

Containing:

- VIN: 4836

- Valid GPS: 4770

- Direction: 4351

- VIN & GPS & Direction: 4351

KPI_003 received MSD’s:

4836 (90%)

KPI_004 correct MSD’s:

4351 (81%)

KPI_009 accuracy of position:

131 m

KPI_013 Success rate of

heading information:

73%

Accepted eCalls call taker

application

4299

Containing:

- VIN: 4299

- Valid GPS: 4245

- Direction: 3837

- VIN & GPS & Direction: 3836

KPI_002b Completed eCalls:

3836 (71%)

KPI_006 Established voice

connections:

4299 (89%)

Received eCalls TMC

4352

Within 1 minute: 3505

MSD content correct: 3757

KPI_020 Successful incidentdata in

TMC:

3505 (81%)


15/05/2014 192 Version 1.3

7.9.5.1 Overall results

Table 86 summarizes the overall results of the four days of driving tests. A total of 5.378 eCalls were

generated with four IVSs, three MNOs and one PSAP. 71% of those calls were “completed”, which

means that the MSD was received, its contents was presented to the PSAP operator and a voice path

was established. 19% of the received MSDs were incorrect, mostly due to incorrect location and/or

heading.

It takes, on average, 17 s starting from the initiation of the eCall until the moment the phone of the

PSAP-operator is ringing, 12 s to establish the call and 5 s to send the MSD.

ID of

test set:

Name of KPI

All IVSs

All MNOs

PSAP

Result Unit



KPI_002b Success rate of completed eCalls using long number 71 %

KPI_003 Success rate of received MSDs 90 %

KPI_004 Success rate of correct MSDs 81 %

KPI_005 Duration until MSD is presented in PSAP 17 s

KPI_006 Success rate of established voice transmissions 89 %

KPI_007a Duration of voice channel blocking 5 s

KPI_008 Time for call establishment 12 s

KPI_009 Accuracy of position 131 m

KPI_013 Success rate of heading information 73 %

KPI_019 Dispatch time of incident data to TMC 15 s

KPI_020 Success rate of presented incident data in TMC 81 %

Table 86: Overall results of the drive test(NL)


15/05/2014 193 Version 1.3

7.9.5.2 Results per IVS

Table 87 shows the results of the drive test per IVS. The IVSs score differently on the KPIs. IVS 1 and 2

have the highest success rate, while IVS 3 is the fastest. In general IVS 4 scores worst: it has a

relatively low success rate and it takes long before a call is established.

The GPS-accuracy of IVS 3 is not good. According to the specs, an IVS is supposed to save the GPS-

location at the moment the eCall is initialised. When the MSD is sent, which is a few seconds later

(after call setup (KPI_008) and modem synchronisation), that saved GPS-location should be included

in the MSD. IVS 3 sends the GPS-location of the vehicle at the moment the MSD is send to the PSAP.

If an accident is mimicked this behaviour is less noticeable, as the vehicle isn’t moving at that time.

The drive test however is conducted from a moving vehicle, which makes this error shown more

prominently.

The duration to establish an eCall differs per IVS (6-21 s). When the call is established the voice

channel is blocked (for sending the MSD) for the same amount of time (5-6 s) which indicates that

the speed of sending the MSD is the same for the IVSs.

IVS 4 has a low success rate for completing the eCalls (41%). This is mostly caused by the low success

rate of transmitting the MSD: 43% of the MSDs are not received by the PSAP. Compared to other IVSs

this percentage is significant lower, which indicates that this problem is caused by the IVS and not by

the PSAP or MNOs.

ID of

test set: IVS 1 IVS 2 IVS 3 IVS 4

All MNOs All MNOs All MNOs All MNOs

PSAP PSAP PSAP PSAP

Result Unit Result Unit Result Unit Result Unit

KPI_001a 843 - 744 - 0 - 0 -

KPI_001b 728 - 895 - 1278 - 890 -

KPI_002b 83 % 82 % 65 % 41 %

KPI_003 95 % 96 % 99 % 57 %

KPI_004 93 % 94 % 65 % 56 %

KPI_005 17 s 17 s 11 s 26 s

KPI_006 89 % 86 % 98 % 74 %

KPI_007a 5 s 5 s 5 s 6 s


15/05/2014 194 Version 1.3

KPI_008 12 s 11 s 6 s 21 s

KPI_009 6 m 6 m 598 m 12 m

KPI_013 73 % 72 % 70 % 84 %

KPI_019 15 s 15 s 9 s 25 s

KPI_020 81 % 81 % 82 % 77 %

Table 87: Results of the drive test per IVS (NL)

7.9.5.3 Results per MNO

Table 88 shows the results of the drive tests per MNO. The most relevant KPI’s regarding the MNOs

are KPI_002b (success rate completed eCalls) and KPI_008 (time for call establishment).

Compared to the other MNOs, MNO 2 has a lower success rate of completed eCalls. This is partly

caused by success rate of transmitting MSD from the IVS to the PSAP (KPI_004) and partly by the

success rate of established voice transmissions (KPI_006).

By average, the time for the call establishment (KPI_008) is more or less the same for all MNOs.

ID of

test set: All IVSs All IVSs All IVSs

MNO 1 MNO 2 MNO 3

PSAP PSAP PSAP

Result Unit Result Unit Result Unit

KPI_001a 328 - 416 - 843 -

KPI_001b 1432 - 2051 - 308 -

KPI_002b 77 % 63 % 81 %

KPI_003 96 % 83 % 95 %

KPI_004 89 % 74 % 83 %

KPI_005 16 s 18 s 16 s

KPI_006 87 % 86 % 98 %


15/05/2014 195 Version 1.3

KPI_007a 5 s 5 s 5 s

KPI_008 11 s 13 s 10 s

KPI_009 123 m 146 m 118 m

KPI_013 71 % 75 % 75 %

KPI_019 15 s 16 s 14 s

KPI_020 80 % 82 % 78 %

Table 88: Results of the drive test per MNO for all IVSs (NL)

The results in Table 88 might be influenced by distribution of the MNOs over the IVSs and days.

Section 7.9.5.2 learns that the IVSs perform differently on the KPI’s. If one IVS is overrepresented in

the testing of a MNO, the results may be influenced by this IVS and therefore the comparison with

other MNOs becomes invalid.

Table 89 shows the distribution of the MNOs over the IVSs. As IVS 1 and 2 are of the same type and

perform the same on the KPIs (section 7.9.5.2), table 12 shows the results per MNO only for IVS 1

and 2.

Table 90 shows there are some minor differences between the investigated MNOs. The time to

establish an eCall is equal for all MNOs. However, MNO 1 and 2 have a lower success rate on

establishing a voice connection between the vehicle and the PSAP compared to MNO 3. This results

in a lower success rate of completed eCalls.

IVS 1 IVS 2 IVS 3 IVS 4

MNO 1 0 1233 381 156

MNO 2 427 416 589 734

MNO 3 843 0 308 0

Table 89: Number of tests distributed per MNO and per IVS (NL)

ID of

test set: IVS 1+2 IVS 1+2 IVS 1+2

MNO 1 MNO 2 MNO 3

PSAP PSAP PSAP


15/05/2014 196 Version 1.3

Result Unit Result Unit Result Unit

KPI_001a 328 - 416 - 843 -

KPI_001b 895 - 728 - 0 -

KPI_002b 78 % 81 % 90 %

KPI_003 96 % 97 % 93 %

KPI_004 94 % 94 % 93 %

KPI_005 17 s 17 s 17 s

KPI_006 82 % 86 % 97 %

KPI_007a 5 s 5 s 5 s

KPI_008 11 s 12 s 11 s

KPI_009 6 m 6 m 6 m

KPI_013 70 % 75 % 74 %

KPI_019 15 s 16 s 15 s

KPI_020 87 % 74 % 82 %

Table 90: Results of the drive test per MNO for IVS 1 and 2(NL)

7.9.5.4 Results per area-type

The route of the drive test was designed for encountering different areas with different

characteristics. Table 91 shows the results of the drive tests per area type.

The eCall chain performs differently in different circumstances. The success rate of establishing voice

transmission between the vehicle and the PSAP (KPI_006) is influenced by the area the vehicle is

driving in. Remarkable: the success rate is the highest in an area with low mobile coverage and the

lowest in city centre and near the airport.

Also the GPS-accuracy (KPI_009) depends on the area type. In the city centre and near the airport the

accuracy is much higher than in the other area. This is caused by the speed of the vehicle: IVS 3

reports the position of the vehicle when this MSD is send, and not the position when the eCall is

initialised. The lower the speed of the vehicle, the closer these positions is. This results in a better

accuracy of the GPS-location.

Regular Port Low coverage City centre Airport

ID of All IVSs All IVSs All IVSs All IVSs All IVSs


15/05/2014 197 Version 1.3

test set:

All MNOs All MNOs All MNOs All MNOs All MNOs

PSAP PSAP PSAP PSAP PSAP

Result Unit Result Result Result Unit Result Unit Result Unit

KPI_001a 957 - 225 - 72 - 242 - 37 -

KPI_001b 2438 - 480 - 160 - 515 - 70 -

KPI_002b 72 % 76 % 83 % 63 % 66 %

KPI_003 90 % 90 % 93 % 90 % 93 %

KPI_004 81 % 82 % 84 % 78 % 81 %

KPI_005 17 s 16 s 16 s 18 s 17 s

KPI_006 89 % 93 % 100 % 83 % 84 %

KPI_007a 5 s 5 s 5 s 5 s 5 s

KPI_008 12 s 11 s 11 s 12 s 12 s

KPI_009 150 m 142 m 159 m 23 m 44 m

KPI_013 78 % 72 % 67 % 58 % 59 %

KPI_019 15 s 15 s 15 s 16 s 15 s

KPI_020 82 % 98 % 51 % 83 % 62 %

Table 91: Results of the drive test per area-type (NL)

During the drive tests also several objects (tunnels, high voltage cables) were encountered, and on

fixed locations the test vehicle stopped for several minutes. The results from these locations are

given in Table 92.

In a tunnel, the accuracy GPS location and the heading direction is lower than in regular situations.

Also the number of correct send MSDs is lower, resulting in a lower success rate of completed eCalls.

High voltage cables have no significant influence on the eCalls. Only the accuracy of the heading

information is lower, which can be explained that one stop-location was close to a high voltage cable.

Most likely, this lower accuracy is caused by this particular stop.


15/05/2014 198 Version 1.3

Stops have both a positive and a negative influence on the eCalls. The GPS accuracy is higher, while

the accuracy of the heading information is lower. Also less correct MSDs are sent to the PSAP.

Regular Tunnel

High voltage

cables Stop

ID of

test set: IVS 1 IVS 2 IVS 3 IVS 4

All MNOs All MNOs All MNOs All MNOs

PSAP PSAP PSAP PSAP

Result Unit Result Unit Result Unit Result Unit

KPI_001a 1347 - 17 - 99 - 120 -

KPI_001b 3257 - 43 - 220 - 251 -

KPI_002b 72 % 52 % 73 % 63 %

KPI_003 90 % 87 % 94 % 94 %

KPI_004 82 % 57 % 80 % 73 %

KPI_005 17 s 17 s 16 s 17 s

KPI_006 89 % 92 % 92 % 88 %

KPI_007a 5 s 6 s 5 s 5 s

KPI_008 12 s 12 s 11 s 11 s

KPI_009 135 m 188 m 137 m 43 m

KPI_013 76 % 62 % 61 % 36 %

KPI_019 15 s 16 s 14 s 15 s

KPI_020 83 % 100 % 91 % 70 %

Table 92: Results of the drive test per encountered object (NL)

Conclusions 7.9.6

The main conclusion from the first round of drive tests is that eCall seems technically feasible but the

results aren’t satisfactory yet. For an emergency system the results are below par:


15/05/2014 199 Version 1.3

The variance in performance between the different IVSs is substantially although the used

IVSs are supposed to be according to standards. This implies that the standards might not be

specific enough

IVSs that behave differently will be a menace for PSAP operators when they have to assess

the incoming emergency-calls

The variance in performance between the different MNOs is less significant.


Testing is only useful when all manufacturers have implemented the standards correctly; our

research shows that this may not be the case.

Insight in every part of the eCall chain is necessary in order to be able to realise and interpret

test results.

Every test method has its own presumptions on the functioning of the systems that are being

tested. One should make all these assumptions explicit.

The analysis of the test results implies that some standards need to be adjusted (change of

additional data definitions, timestamp definitions, a newly asked MSD should contain new

information)

Good understanding of the problems occurred while testing needs more analytical tools then

presently available.

Test results have led to changes in modems and application. A 2nd round of drive-tests and

scenario tests is needed, with a larger number of calls.

Interoperability testing will show whether the disappointing performance is merely a Dutch

issue or that there will be implications for other pilot countries as well.

The implication of the results in the HeERO pilot sites should be aggregated and discussed on

European level.

A distinction should be made between problems specifically related to eCall and more

general problems (like i.e. bad coverage) not related to eCall. These are not the scope of the

eCall project.

The definition of additional data in the standard MSD needs to be changed to make it usable.

Those will also take away the risk that future implementations of the “spare” room in the

standard MSD could have negative impact on the deciphering of the standard MSD.

The technical implementation of the first additional data within the present standard now

used by HGV eCall asks for more is the active involvement of other HeERO members.


15/05/2014 200 Version 1.3

Appendix A 7.9.8


Measurements 1467 1551 890 499

Outliers 137 149 129 26


15/05/2014 201 Version 1.3



Outliers 140 149 82 17


15/05/2014 202 Version 1.3


Measurements 1587 1658 1407 999

Outliers 122 278 102 83


15/05/2014 203 Version 1.3


15/05/2014 204 Version 1.3

Table 93: Outliers KPI_009 (NL)



Outliers 88 225 71 5



Outliers 163 157 33 60


15/05/2014 205 Version 1.3

8 Reference [1] W.G. Cochran, The distribution of the largest of a set of estimated variances as a fraction of their

total, Annals of Human Genetics (London), 11(1), 47–52 (January 1941).

[2] D.L. Massart, B.G.M. Vandeginste, L.M.C. Buydens, S. de Jong, P.J. Lewi, J. Smeyers-Verbeke,

Handbook of Chemometrics and Qualimetrics: Part A, Elsevier, Amsterdam, The Netherlands, 1997,

ISBN 0-444-89724-0.

[3] P. Konieczka, J. Namieśnik, Quality Assurance and Quality Control in the Analytical Chemical

Laboratory – A Practical Approach, CRC Press, Boca Raton, Florida, 2009, ISBN 978-1-4200-8270-8.

[4] J.K. Taylor, Quality Assurance of Chemical Measurements, 4th printing, Lewis Publishers, Chelsea,

Michigan, 1988, ISBN 0-87371-097-5.

[5] W. Horwitz, Harmonized protocol for the design and interpretation of collaborative studies,

Trends in Analytical Chemistry, 7(4), 118–120 (April 1988).

[6] ISO Standard 5725–2:1994, “Accuracy (trueness and precision) of measurement methods and

results – Part 2: Basic method for the determination of repeatability and reproducibility of a standard

measurement method”, International Organization for Standardization, Geneva, Switzerland, 1994,

available at: http://bit.ly/P2AwhJ.

[7] R. Moore, Mathematics Department, Macquarie University, Sydney, Australia, 1999, available at:

http://bit.ly/QGUS1X.

[8] R.U.E. 't Lam, Scrutiny of variance results for outliers: Cochran's test optimized, Analytica Chimica

Acta 659, 68–84 (2010); doi:10.1016/j.aca.2009.11.032

[9] R.U.E. 't Lam, Variance Outlier Test, blog: http://rtlam.blogspot.com/

[10] Table of critical values of the F-distribution: NIS

[11] Hughes, I G and T P A Hase, Measurements and their Uncertainties: A Practical Guide to Modern

Error Analysis, Oxford University Press, Inc., New York, NY, 2010, ISBN 978-0199566334 .

[12] Maindonald, J and W J Brown, Data Analysis and Graphics Using R - an Example-Based Approach

(3rd edition), Cambridge University Press, Cambridge, UK, 2010, ISBN 978-0521762939.

[13] Ott, R L and M Longnecker, An Introduction to Statistical Methods and Data Analysis (5th ed),

Duxbury, Thomson Learning, Inc., Pacific Grove, CA, 2000, ISBN 978-0534251222.

[14] R Development Core Team (2010). R: A language and environment for statistical computing. R

Foundation for Statistical Computing, Vienna, Austria. ISBN 3-900051-07-0, available at:

http://www.R-project.org, accessed on 1 June 2011.

[15] Wikipedia, Cochran's C-test, available at: http://bit.ly/OWuldq.

[16] Wikipedia, Grubbs' test for outliers, available at: http://bit.ly/NluMe6.


15/05/2014 206 Version 1.3

[17]National Institute of Standards and Technology (NIST), Detection of Outliers, NIST/SEMATECH e-

Handbook of Statistical Methods, 2012, available at: http://1.usa.gov/RXP1QU


15/05/2014 207 Version 1.3

9 Annex

9.1 Annex I: change requests to CEN


15/05/2014 208 Version 1.3


15/05/2014 209 Version 1.3


15/05/2014 210 Version 1.3


15/05/2014 211 Version 1.3


15/05/2014 212 Version 1.3


15/05/2014 213 Version 1.3

Documents

D v. u Intermediate results of the tests - HeERO · D v. u Intermediate results of the tests Version number: ... 7.8.2 METHODOLOGY OF DATA ANALYSIS ... 7.9.5 RESULTS DRIVE TEST