Embed Size (px)
DESCRIPTION
Methodological challenges and solutions in euroFOT
Citation preview
www.eurofot-ip.eu
Methodological challenges and solutions in euroFOT
Samantha Jamson
September 2009University of Leeds
www.eurofot-ip.eu
euroFOT Objectives• Apply a common European
approach for Field Operational Tests
• Perform multiple coordinated tests of Intelligent Vehicle Systems with ordinary drivers in real traffic
• Investigate performance, driver behaviour and user acceptance
• Assess the impacts on safety, efficiency and the environment,
Test Management Centres
www.eurofot-ip.eu
Functions• Longitudinal control functions
FCW Forward Collision WarningACC Adaptive Cruise ControlSL Speed Limiter
• Lateral control functionsBLIS Blind Spot Information SystemLDW Lane Departure WarningIW Impairment Warning
• OthersCSW Curve Speed WarningFEA Fuel Efficiency AdvisorSafeHMI Safe Human Machine Interaction
www.eurofot-ip.eu
FESTAFunction Identification and
Description
Performance Indicators
Use Cases
Research Questions and Hypotheses
Measures and Sensors
Study Design
Research Question and Hypotheses Analysis
Socio-Economic Impact Assessment
System and Function Analysis
Ethical and Legal Issues
Imp
lem
en
t at io
n P
lan
Database
Performance Indicators
Measures
Data Decoding
Data Acquisition
Data Analysis
The FOT Chain
www.eurofot-ip.eu
From Research Questions to Hypotheses #I• Research Questions (Example)
ACC: Does ACC improve the drivers’ distance keeping behaviour?
• Research questions are the reason for conducting the study.
• They are often rather general.• More detail is required to plan the study well
formulate hypotheses …
www.eurofot-ip.eu
From Research Questions to Hypotheses #2• Hypotheses (Examples)
ACC: The percentage of TTC(min) decreases by at least 20 per cent.
ACC: The variance in headway in a following situation decreases.
• Hypotheses must be testable.• They are specific, usually directional and identify
the Performance Indicator(s) (PI) in question.
www.eurofot-ip.eu
From Research Questions to Hypotheses #3• From PIs to Sensors
Percentage of TTC(min): Necessary to collect speed, distance to vehicle ahead, speed of vehicle ahead.Variance in headway: Necessary to collect distance to vehicle ahead.
• PI determine which measures are necessary.• The measures have to be logged via sensors.
www.eurofot-ip.eu
Procedure for euroFOT
• The research question has been asked.• The research question has to be broken down into
hypotheses (specific, testable, unidirectional). This is a vital step!!!
• A hypothesis contains at least one PI.• The PI is looked up in the PI matrix, which informs about
the measures necessary for the PI.• The necessary measures are looked up in the measures
matrix, which informs about the sensors that can be used to log the measures.
• (PI & measures matrix started within FESTA)
www.eurofot-ip.eu
Defining the experimental framework• Top-down procedure, setting out various
recommendations & assumptions. • Face-to-face clinics helped us to brainstorm and use
the expertise of the consortium• Produce a design that would enable the collection
and analysis of data that could be used to produce statistically reliable results.
www.eurofot-ip.eu
Identifying the constraintsi. Budget – the euroFOT workplan and budget are, as in all research projects, the
major deciding factor when developing the experimental procedures. ii. Vehicles – the types of vehicles will depend largely on the availability of the
functions on a particular model. Higher class vehicles which inevitably have more functions available appeal to older individuals with more disposable income than say a younger driver who may favour the lower insurance premiums and running costs of a smaller vehicle.
iii. Economic climate – reduced vehicle sales. European new passenger car sales fell 7.8 percent in 2008, the high-end markets suffering most as drivers may opt for smaller, cheaper alternatives. This squeezes the available participants.
iv. Drivers – drivers will mostly be self-selected (with the exception of the truck drivers who are selected by the fleet manager). Bias will be noted and compared to the country specific population statistics
v. Multiple functions – the vehicles are models which are manufactured for the current market. They are, in most cases, not specific “test” vehicles but are owned or leased by the driver. In this respect the eight functions under consideration in euroFOT are often sold in bundles – groups of functions.
www.eurofot-ip.eu
Power analysisPower analyses are undertaken to determine the required sample size in
order to make accurate and reliable statistical conclusions, and to determine how likely the statistical test will be able to detect effects of a given size in a particular situation.
When planning a FOT three interrelated questions arise regarding the experimental design:
• how large is the effect of the system under investigation, • how many cars have to be equipped to find this effect and • how long do they have to be driven to obtain a sufficient amount of
data.
www.eurofot-ip.eu
Power analysis conclusionsi. Monte Carlo simulations have shown that when at least 120
participants are included, with a mileage of 15k per year, even with the small effect size that can be expected in a FOT, a sufficient power will be reached.
ii. Including more cars or more unique participants should take precedence over measuring for longer periods: e.g. measuring for a year with 60 participants could fail where measuring for half a year with 120 participants could provide a significant result.
iii. Reduce the variance measured between participants - choose a homogenous group of drivers for example male drivers between 30-40 years of age with similar mileage. This would improve the power, but at the cost of the generalisability (external validity) of the results.
iv. Baseline and treatment phase should be as near to equal as possible
www.eurofot-ip.eu
Harmonised experimental designs
www.eurofot-ip.eu
Harmonised questionnaires
All VMCs are using the same core questionnaires (same scales, items etc) with flexibility for additional items
www.eurofot-ip.eu
How has this procedure helped euroFOT?
The approach taken in this workpackage, has, as far as possible given the identified constraints, enabled the VMCs within euroFOT to develop experimental procedures that are both robust and unified
With regards to the experimental designs, we have been able to achieve a coordinated effort in realising a common experimental with only a couple of deviations
We have also been successful with regards to developing the questionnaires that provide the subjective data in a unified format for use in the impact analysis at the end of the project.
www.eurofot-ip.euOctober A. Etemad – XYZ Workshop www.eurofot-ip.eu
