Use of Dempster-Shafer Conflict Metric to Detect InterpretationInconsistency

Jennifer Carlson and Robin R. Murphy∗

Safety Security Rescue Research CenterUniversity of South Florida

Tampa, FL 33620

Abstract

A model of the world built from sensor datamay be incorrect even if the sensors are func-tioning correctly. Possible causes include theuse of inappropriate sensors (e.g. a laser look-ing through glass walls), sensor inaccuraciesaccumulate (e.g. localization errors), the apriori models are wrong, or the internal rep-resentation does not match the world (e.g.a static occupancy grid used with dynami-cally moving objects). We are interested inthe case where the constructed model of theworld is flawed, but there is no access to theground truth that would allow the system tosee the discrepancy, such as a robot enter-ing an unknown environment. This paperconsiders the problem of determining whensomething is wrong using only the sensordata used to construct the world model. Itproposes 11 interpretation inconsistency in-dicators based on the Dempster-Shafer con-flict metric, Con, and evaluates these in-dicators according to three criteria: abilityto distinguish true inconsistency from sensornoise (classification), estimate the magnitudeof discrepancies (estimation), and determinethe source(s) (if any) of sensing problems inthe environment (isolation). The evaluationis conducted using data from a mobile robotwith sonar and laser range sensors navigatingindoor environments under controlled condi-tions. The evaluation shows that the Gam-bino indicator performed best in terms ofestimation (at best 0.77 correlation), isola-tion, and classification of the sensing situa-tion as degraded (7% false negative rate) ornormal (0% false positive rate). While theevaluation is limited to 2D world models, theuse of the Dempster-Shafer conflict metric to∗Email: {jcarlso1,murphy}@csee.usf.edu

detect significant inconsistencies in data ap-pears useful and could lead to systems whichautonomously switch information sources toensure the best mission performance, learnthe relative contribution and reliability ofsources for different environments, and rea-son about which sources to use under whatcircumstances.

1 INTRODUCTION

In an unknown environment, an intelligent agent, beit a human, software agent, or a robot, has only itssenses (or sensors) to guide its actions. On the otherhand, the suitability of each sense depends on the en-vironment’s characteristics (e.g. dark or noisy). If weassume that the agent does not know these characteris-tics, how can it determine which sensors to use and/orhow to adapt when the sensing situation changes?This study addresses the problem of determining whenan artificial agent’s ability to sense the environment isimpaired due to the use of inappropriate sensors, em-ploying only the sensor data used to construct its worldmodel.

This study considers the case of a mobile robot navi-gating in a controlled, but unknown, indoor environ-ment in the presence of sensing anomalies. In thispaper a sensing anomaly refers to cases in which thephysical sensor(s) are working within the manufac-turer’s specifications but the readings would lead to anincorrect interpretation of the environment. For ex-ample, a sonar sensor has difficulty detecting smoothwalls due to specular reflection, and laser range scan-ners cannot detect (clean) glass surfaces. The presenceof sensing anomalies is an indication that the selectedsensor, or set of sensors, are inappropriate for a givenenvironment.

Features of the Dempster-Shafer weight of conflictmetric, Con, are examined in this study as potential

solutions to the problem of detecting the use of inap-propriate sensors. The rationale for exploring Shafer’smodel of conflict (Shafer 1976) arises from previouswork in the field of robots (see Section 2) and the as-sumption that all environments are consistent, there-fore a metric that measures inconsistency could po-tentially detect sensing problems without relying ona ground truth. This potential was initially exploredin (Carlson et al. 2005) where the Con metric wasshown to provide an estimate (0.7–0.9 correlation) ofthe quality of the world model (a 2D map) for sonarreadings in indoor and confined space environments.In this paper the performance of 11 distinct featureswere evaluated according to three criteria: the abilityto estimate the state of a robot’s sensing capabilities(estimation), distinguish true sensing anomalies fromsensor noise (classification), and determine the sourceof the sensing problem within the environment (isola-tion).

The results of 30 experiments, using the real sonarand laser readings, showed that the Gambino indica-tor adapted from (Gambino, Ulivi, & Vendittelli 1997)could serve as a general solution to the problem of de-tecting, estimating, and isolating sensing problems inunknown environments. It was found to estimate theoverall error in the occupancy grid (0.77 correlation atbest) and classify the sensing situation with a 0% falsepositive and a 7% false negative rate. In addition itperformed at least as well as the other indicators onthe task of isolating problems within a 2D occupancygrid. The performance of the remaining 10 interpre-tation inconsistency indicators varied across the threetests. The maximum increase, increase frequency andarea indicators in particular showed potential as spe-cialized tools.

2 RELATED WORK

Inconsistency has been used as a tool in two over-lapping fields relevant to this work: mobile roboticsand fusion. In both fields the information used for ac-tion and decision making is often drawn from multiple,possibly unreliable and/or incompatible sources. Theprevalence of formal models in these fields allows thedegree of inconsistency to be measured.

In mobile robotics measures of inconsistency, suchas conflict and entropy, have been used to solve theproblem of localization in partially known or un-known environments. Examples of these can be foundin (Ayrulu & Barshan 2002), (Baltzakis, Argyros,& Trahanias 2003), (Davidson & Murray 2002), and(Moreno & Dapena 2003). These approaches often relyon environment-specific assumptions, ranging from aground truth map to loose constraints like “the ground

plane is flat” to reduce complexity. They typicallychoose actions or make decisions which minimize in-consistency.

Three studies found in the robotics literature developapproaches for solving sensing problems using incon-sistency. In (Gambino, Ulivi, & Vendittelli 1997) ametric was developed based on conflict (Smets’ formu-lation (1990)) to determine when information becomesirrelevant in unknown dynamic environments. The re-sults showed that this metric allowed the simulatedrobot to respond to changes in the environment fasteras compared to the traditional Dempster-Shafer map-ping approach. Yi et al. (2000) developed a metric toadapt their sonar sensor model using the conflict gen-erated by the latest measurement. The resulting map,built using a Nomadics Super Scout II, was more in-ternally consistent (fewer outliers) but not accuratecompared to the ground truth. In (Shayer et al. 2002)a novel inconsistency metric is presented based on anadaptive fuzzy-logic sensor fusion technique developedin previous work. The metric compares each pair ofsensors by creating a fused map from their readings.The agreement rate between the two sensors’ originalmap and the fused map was used to provide a rank-ing of each individual sensor’s relative accuracy. Thesensors were correctly ranked 83% of the time.

An overview of the broad area of fusion that highlightsthe problem of inconsistency and various approachesfor handling it, can be found in (Appriou et al. 2001).Related work along these lines can be found in (2001)where Ayoun and Smets present an approach to theassociation problem which selects the assignment ofsensor readings to targets which minimizes the con-flict (Smets formulation (1990)) of the fused informa-tion. The approach is validated by a series of simpli-fied (static world, binary sensor output) examples andmethods for generalizing the approach.

Dempster-Shafer theory was selected as a potential so-lution partly due to earlier work showing equal or bet-ter map building performance compared to Bayesianand fuzzy approaches when data from multiple sensorsare fused. The analysis of evidential structures pre-sented in (Zhu & Basir 2003) showed that Dempster-Shafer theory is equivalent to, and has the same pos-itive characteristics as both Bayesian and fuzzy ap-proaches. Pagac, Nebot, & Durrant-Whyte (1998)showed that Dempster-Shafer outperformed Bayesianupdating methods for building occupancy grids fromsensor readings.

3 APPROACH

The approach taken in this study examines featuresof Con as potential solutions to the problem of de-

tecting when sensors are inappropriate in unknownenvironments. Given the assumption that an envi-ronment is consistent, the Dempster-Shafer weight ofconflict metric, Con, provides a means of measuringthe degree of inconsistency and therefore the severityof sensing problems in unknown environments. Eleveninterpretation inconsistency indicators were developedto explore the potential of this approach. A inter-pretation inconsistency indicator is defined as an in-dicator that uses features of conflict to determine ifobservations and/or interpretations used to update agiven model are suspect, i.e. should not be trusted,or normal Three performance metrics, Fisher’s LinearDiscriminant (FLD), Pearson’s correlation coefficient,and Baddeley’s ∆2 metric were used to test the abilityof each interpretation inconsistency indicator to de-termine when the current sensor(s) are inappropriate,measure the extent to which this is the case, and toisolate problem areas.

3.1 CONSISTENCY ASSUMPTION

It is presupposed that any environment will conformto the consistency assumption, for example a point inspace cannot be both occupied and empty in the sameinstant. Drawing from logic (Ebbinghaus, Flum, &Thomas 1994), this assumption is modeled as the ab-sence of conflict (φ), i.e. ψ ∩ ¬ψ = φ, where ψ is anyhypothesis and φ must be the empty set.

Given this consistency assumption, if conflict appearsin a sensor-based world model it may be caused by oneor more of the following: sensor noise (e.g. introducedby discretization), the use of inappropriate sensors, in-accurate a priori models, or the use of a flawed inter-nal representation (e.g. a static occupancy grid usedfor dynamically moving objects). Therefore, depend-ing on the circumstances in which sensor readings aregathered and interpreted, conflict can theoretically beused to detect a variety of sensing problems.

3.2 DEMPSTER-SHAFER CONFLICT

The Dempster-Shafer weight of conflict metric, Conmeasures the support for conflicting evidence in a setof observations, thus given the consistency assumption,this metric can be applied to detect sensing problemsin a world model. The metric itself is given in Equa-tion (1)

Con(Bel1, Bel2) = log(1

1− k) (1)

where

k =∑

Ai∩Bj=∅

m1(Ai)m2(Bj)

where Bel1 and Bel2 represent degrees of belief for thesets of hypotheses θ1 and θ2 respectively, Ai ∈ θ1, Bj ∈θ2, and m1(x) and m2(x) give the belief assigned to ahypothesis x by Bel1 and Bel2 respectively. Con takesa value between 0 and ∞; as k → 0.0, Con→ 0.0, andas k → 1.0, Con → ∞. It is additive, which meansthat the conflict from more than two belief functionscan be attained easily.

This study also examines features of Smets’ conflictbelief mass as potential solutions to the problem ofdetecting when sensors are inappropriate in unknownenvironments. In Smets’ model (1990) φ is treatedas a valid hypothesis that accumulates belief, ratherthan an error to be factored out. It remains to be seenwhich of these two models of conflict is more effectivefor solving the problem of detecting sensing problems.

3.3 EVALUATION

The interpretation inconsistency indicators were eval-uated using three tests. Each test used a quanti-tative map quality metric as a measure of the ac-tual sensing situation. In this study a traditionaloccupancy grid (Elfes 1989) was used. This dividesa two dimensional space into a set of equally sizedcells, each labeled occupied or empty with some levelof certainty (probabilistic (Thrun 2003), possibilistic(Lopez-Sanchez, de Mantaras, & Sierra 1997), and ev-idential models (Murphy 2000) have been used withsuch grids in mobile robot mapping). The quality ofan occupancy grid given a ground truth occupancygrid can be quantified using Equations (2–4),

Error =w∑

x=0

h∑y=0

MAX(erroro, errore) (2)

erroro = |grido(x, y)− trutho(x, y)| (3)errore = |gride(x, y)− truthe(x, y)| (4)

where width and height together give the size ofthe occupancy and ground truth grids, grido(x, y),gride(x, y) give the occupancy and empty values fromthe occupancy grid, and trutho(x, y) and truthe(x, y)the same values from the ground truth grid, respec-tively. Lower Error scores indicate a better matchwith the ground truth.1

The ability to estimate the overall map quality, i.e. theextent to which a robot’s current sensing is inappro-priate, was measured using Pearson’s correlation co-efficient (Dowdy, Weardon, & Chilko 2004). In this

1Any cells which were not considered in the groundtruth (for example cells on the other side of walls), or notyet scanned by the sensors were excluded from the overallerror calculation.

case the test was used to determine if the averagevalue assigned by an interpretation inconsistency indi-cator over an occupancy grid (see Section 4.1) variedlinearly with the grid’s Error score.

Fisher’s linear discriminant (FLD) (Fisher 1936) wasused to test each interpretation inconsistency indica-tor’s ability to correctly detect a situation in whichthe current sensor(s) are inappropriate. The two-classformulation of FLD, given in Equation (5) where µ isthe mean value assigned to a class of grids (accurateversus inaccurate in this case) and σ is the variancewithin each group, provides a straightforward meansof testing the separability of classes provided by anindicator.

FLD =|µclass 1 − µclass 2|2

σclass 1 + σclass 2(5)

Before the classification test (FLD) could be applied,the ground truth classification of the occupancy gridsfrom each run had to be determined. The k-meansclustering technique (Hartigan & Wong 1979) was usedto find sets of Error scores that were statistically closein value. This way three distinct sets were identified,the largest of which contained error values subjectivelydetermined to reflect accurate occupancy grids, withthe other two reflecting inaccurate grids. A thresholdvalue of 300 was selected that lay between these twogroups.

The third test applied the ∆2 metric (Baddeley 1992)to report how well an interpretation inconsistency in-dicator can isolate sensing problems. The ∆2 metric isa position-sensitive (but shape insensitive) metric forevaluating binary images produced by computer visionalgorithms. It applies Equation (6) to a pair of images(typically the estimate image B and the ground truthimage A), i.e. ∆2(A,B) =

√1N

∑x∈X

(min(d(x,A), c)−min(d(x,B), c))2 (6)

whereX is the set of highlighted pixels in either image,d(x, I) corresponds to the distance between a pixel xand the nearest highlighted pixel in image I, and c isan arbitrary constant (in this study c = 100). ∆2 pro-duces a single scalar measure which describes how wellthe two images match in terms of both false negativesand false positives.

4 IMPLEMENTATION

A series of 30 experiments using sonar or laser datafrom a Nomad 200 robot were performed to determine

how well each of the interpretation inconsistency in-dicators could estimate the overall map quality, clas-sify the sensing situation, and isolate difficult to senseregions within the environment. The indicators ap-plied threshold values to determine which cells weresuspicious and which were considered normal. Thosethreshold values were varied for each indicator result-ing in the 355 potential indicators examined in thisstudy. Fifteen runs in three indoor hallways (five each)were evaluated using an indicator and the map qual-ity metric. This process was repeated individually forthe sonar and the laser readings, resulting in 30 ex-periments used to measure each of the 355 combina-tions’ capabilities. The experimental procedure usedcontrolled environments to ensure that the use of in-appropriate sensors would generate significantly moreconflict than sensor modeling or representation errors(see Section 3.1).

4.1 INTERPRETATION INCONSISTENCYINDICATORS

Eleven different interpretation inconsistency indica-tors were developed using either Shafer’s Con met-ric or Smets’ conflict belief mass (both of which willbe referred to as conflict, except where the distinc-tion is important). The indicators labeled each cell inthe grid as suspect, if the conflict indicated a problemwith sensing, or normal otherwise. The label was cho-sen on a cell by cell basis (with the exception of thearea indicator, see Table 1) by thresholding a functionof the conflict value. The average of this function’svalue (after thresholding) over the entire grid is calledthe conflict score for that indicator. In addition eachindicator produces a conflict map which is a binaryimage in which suspect cells are labeled true and therest false.

Table 1 lists each indicator along with the set of thresh-old values tested for that indicator. In the analysis thethresholds were increased in constant intervals, givenin the Step column, from the minimum to the max-imum value shown in the Threshold Range column.Note that in this first exploratory study the appropri-ate threshold values to test were determined empiri-cally.

The 11 indicators can be characterized by describingthe function that each applied to the conflict prior tothresholding. The total indicator takes the sum of theCon metric generated by each update, i.e. Shafer’sformulation. Three additional indicators normalize to-tal ’s function prior to thresholding according to thecurrent sensor’s angular resolution, range resolution,and average update rate respectively. Maximum in-crease takes the maximum value of Con achieved in asingle update. The average and average sequence in-

Table 1: The interpretation inconsistency indica-tors evaluated in this study. Note that the area andincrease frequency use a pair of threshold values todetermine if a given cell is suspect.

Interpretation incon-sistency indicator

Threshold Range Step

area 0.25–5 0.25size 50–250 50

average 0.025–0.5 0.025average sequence 0.05–1.0 0.05frequency 0.05–0.95 0.05Gambino 0.5–10.0 0.5increase frequency 0.05–0.95 0.05

magnitude 0.5–2.0 0.5maximum increase 0.1–2.0 0.1normalized by:angular resolution, 0.025–0.5 0.025range resolution, 0.25–5.0 0.25and update rate 0.025–0.5 0.025total 0.25–5.0 0.25

dicators both use the average Con generated per up-date, with the latter restricted to the last contiguousset of conflicting updates. Frequency and increase fre-quency both consider the percentage of updates thatgenerated conflict, where frequency counts all updateswith Con > 0 but increase frequency counts onlythose where Con ≥ magnitude. The Gambino indi-cator uses a heuristic developed in (Gambino, Ulivi,& Vendittelli 1997): if either occupied or empty wereat least 50% and Smets’ conflict belief mass increasedby at least 10% in a single reading then something iswrong. In this study the number of times in whichthese conditions held was counted and then thresh-olded. Finally the area indicator uses connected com-ponents to find contiguous regions in a conflict mapgiven by a total indicator (using the first thresholdlisted in Table 1), and then applies the size thresholdto remove smaller (assumed to be noise) regions.

4.2 EXPERIMENTAL PROCEDURE

A Nomadic Technologies Nomad 200 robot (seeFig. 1(a)) was used to simultaneously collect sonar andlaser readings in indoor hallways. Three different un-cluttered hallways with the characteristics presentedin Table 2 were selected. The robot was teleoperateddown the center of the hallway for a distance of sixmeters while readings from the robot’s single ring of16 sonar sensors, and a Sick laser sensor mounted justabove them, were collected for offline analysis. Thefirst five good runs in each hallway, in terms of con-sistent sampling rates and the absence of errors in thedata collection process, were used in this paper.

(a) Nomad 200 robot (b) The window hallway

Figure 1: The robot used for data collection and oneof the hallways used as a testbed environment.

Table 2: Characteristics of the three indoor hallwaysused to test the performance of the interpretation in-consistency indicators. The width (W) and length (L)are reported in meters. Smooth refers to painted sheet-rock walls.

Hallway Sonar Laser W L Wallsnarrow poor good 1.8 11.2 Smoothwide poor good 2.5 14.2 Smoothwindow poor poor 2.0 27 Smooth

& largewindows

The offline analysis system used sensor models to reg-ister either the sonar or the laser readings to a 28 me-ter square occupancy grid, with a cell resolution of10.16 cm (four inches) in both the x and y directions.This paper follows the approach described in (Murphy2000) for building occupancy grid maps from sensorreadings using Dempster-Shafer theory. In this ap-proach sensor models are used to convert range read-ings, d, into belief masses, m, distributed over a cone.The cone has a radius of R and a half-angle β whereβ is commonly tuned to the particular environment.

The Dempster-Shafer (or Smets) occupancy grid wasupdated using the sensor readings until the robothad traveled 1.0 meters, at which point the grid wasevaluated using an interpretation inconsistency indica-tor and the quantitative map quality metric. (In orderto use this metric, a ground truth occupancy map wasmanually created for each hallway.) This process wasrepeated every half meter, resulting in 10 sampled lo-cations per experiment.

For each of the 15 runs this procedure was repeated forthe sonar, then the laser data, resulting in 300 sampled

locations (in 30 time series) recorded for use in a post-hoc analysis. The following three values were recordedfor each point: the Error score, the conflict score, andthe value given by the ∆2 metric which compared theconflict map generated by a given interpretation in-consistency indicator to an error image. The error im-age was produced by applying standard binarizationtechniques to the grayscale image produced from eachcell’s error, i.e. MAX(erroro, errore) from Equations(2–4). The post-hoc analysis generated Pearson’s cor-relation coefficient, Fisher’s Linear Discriminant, andthe ∆2 score for each indicatorand sampled location.The results of this analysis are presented in Section 5.

5 RESULTS

The performance of the 11 distinct interpretation in-consistency indicators were evaluated using the threeperformance metrics presented in Section 3.3. Boththe average and the best performance of each indica-tor (across the threshold values tested) were used toevaluate that indicator’s ability to perform each of thethree tasks: classification, estimation, and isolation ofsensing problems. In general, the interpretation incon-sistency indicators showed varying performance acrossthe three tests. Area and increase frequency for ex-ample ranked among the top three indicators on theestimation and isolation test, but achieved only av-erage performance on the classification test. Maxi-mum increase showed the inverse trend by achievingthe best overall performance on the classification testwhile showing average performance on the other twotests. The Gambino indicator was the only indica-tor that performed well on all three. At the thresholdvalue of 2.0 it was found to classify the sensing sit-uation with a false negative rate of 7%, estimate theError metric (0.77 correlation), and isolate sensingproblems within the grid at least as well as the otherindicators.

For each of the three tests the mean (average), vari-ance (σ2), and best performance (minimum or maxi-mum depending on the metric) of each interpretationinconsistency indicator are presented in both tabularand graphical form. Also included are the number ofthreshold combinations that contributed to each indi-cator’s results (see Table 1). (For example area usedtwo thresholds to determine which cells were suspect.These were varied over 20 and five values respectively,resulting in 100 combinations tested.) In the graphs(Figures 2(b), 3(b), and 4(b)) the means are repre-sented as a bar graph with variances shown as errorbars. The best performance is plotted as a line graphalong the same y axis. For all three tests the resultsare presented alphabetically to facilitate comparison.

Table 2(a) and Figure 2(b) give the results of the esti-mation test. This test used Pearson’s correlation coef-ficient which varies between −1.0 for a perfect neg-ative linear relationship and 1.0 for a perfect posi-tive linear relationship. Note that the conflict scoreis only bounded for the frequency and increase fre-quency indicators (i.e. a strong negative relationshipis just as useful for estimation with these indicators asan equally strong positive relationship). All peak cor-relations were tested for statistical significance withthe appropriate corrections for autocorrelation (Meko2005) with and without Bonferroni correction (assum-ing 355 trails).

Table 2(a) and Figure 2(b) show that the majorityof the interpretation inconsistency indicators providepoor estimates of Error. Only area and the Gambinoindicator, at their peak performance, showed the abil-ity to estimate the total error score for both sonar andlaser readings, with the Gambino indicator performingbest in terms of both average (mean) and best per-formance. This result is interesting when comparedto previous results in indoor hallways (see Carlson etal. (2005)) which showed a very strong (0.92) correla-tion between the total indicator and the overall errorscore for sonar readings alone. It is clear from thisresult that a more sophisticated indicator was neededto handle readings from both sonar and laser sensors.

The estimation results also demonstrate the sensitiv-ity of each interpretation inconsistency indicator tovariance in its parameter(s). Total and its normal-ized variants appear to be insensitive (σ2 < 0.0004) tochanges in their respective threshold values. Area wasthe most sensitive (σ2 = 0.092), with the Gambinoindicator coming in second (σ2 = 0.090). A detailedanalysis of the raw correlation results shows that threeof the indicators, average sequence, average, and theGambino indicator, produced a monotonic increase ordecrease (relative to the threshold value) from theirbest performance to near zero correlation. This sug-gests that these three indicators are good candidatesfor systems in which tuning (or learning) of suitablethresholds can be performed.

The isolation test used Baddeley’s ∆2 metric to com-pare the error and conflict maps, the results of whichare shown in Table 3(a) and Figure 3(b). For the ∆2

metric smaller values indicate a better ability to iso-late sensor problems within the grid. This metric doesnot have a set bound like the correlation coefficient,thus it is only useful for comparing the performance ofthe 11 indicators.

The isolation results given in Table 3(a) showed littlevariance both within (at most 0.4) and between (0.70)the interpretation inconsistency indicators. This re-

Indicator Mean σ2 Best NAngular -0.492 0.000 -0.484 20resolutionArea 0.090 0.092 0.677** 100Average -0.314 0.025 0.033 20Average -0.218 0.064 0.064 20sequenceFrequency -0.041 0.026 -0.265 19Gambino 0.331 0.090 0.765* 20Increase 0.033 0.048 0.412* 35frequencyMaximum -0.452 0.011 -0.275 20increaseRange -0.474 0.000 -0.464 20resolutionTotal -0.474 0.000 -0.464 20Update -0.396 0.000 -0.351 20rate

(a) (b)

Figure 2: Results of the estimation test showing the correlation between the average conflict score and Error foreach of the 11 indicators. N is the number of thresholds values tested for each metric. Statistically significantcorrelations are mark with * (α= 0.05) or ** (Bonferroni corrected α= 1.41E-4).

Indicator Mean σ2 Best NAngular 9.68 0.00 9.65 20resolutionArea 7.95 0.40 7.39 100Average 8.48 0.18 7.86 20Average 8.33 0.09 7.91 20sequenceFrequency 8.84 0.18 8.04 19Gambino 7.61 0.36 7.37 20Increase 7.59 0.19 7.37 76frequencyMaximum 9.20 0.11 8.60 20increaseRange 9.30 0.04 9.06 20resolutionTotal 9.30 0.04 9.06 20Update 9.12 0.04 8.82 20rate

(a) (b)

Figure 3: Results of the isolation test showing the ∆2 metric for each of the 11 indicators. N is the number ofthresholds values tested for each metric.

sult is in part due to the small size of the grid andsubsequently that of the error image and conflict mapcompared using the ∆2 metric. As with the estima-tion test, total and its normalized variants performedpoorly compared to the other indicators and showedthe least sensitivity to their threshold values. Area andthe Gambino indicator again showed the most senti-tivity. Although the results of the estimation and iso-lation results are similar they are not identical. Onthe isolation test the increase frequency indicator per-formed marginally better than the Gambino indica-tor with half the variance.

The classification test results are given in Table 4(a)and Figure 4(b). This test was performed usingFisher’s Linear Discriminant (FLD) which, like ∆2,does not vary within a set scale. In this case largervalues indicate a better ability to classify the sensingsituation.

The classification results in Table 4(a) showed thatthe interpretation inconsistency indicators were morevaried in their ability to distinguish between normaland degraded sensing situations as compared to theirisolation ability. Variance in performance within theinterpretation inconsistency indicators was marginal,with the Gambino indicator and maximum increaseshowing the largest variances. Note that although theoverall variance is low, for the majority of the indica-tors the difference between mean and best performanceis noticeably different. In this test total and its nor-malized variants performed better (in a relative sense)then in the other two tests. They ranked highest interms of mean or average behavior, with the Gambinoindicator and maximum increase only performing bet-ter for a single threshold value.

Due to the Gambino indicator’s performance on theestimation and isolation tests its ability to classify thesensing situation was examined further. For all mapswith error scores below the 300 threshold, the Gam-bino indicator (at a threshold of 2.0) had an averageconflict score of 0.0, indicating that not a single cellwithin the grid had been flagged as degraded. Formaps with degraded error scores (≥ 300) only 7% alsohad scores of 0.0, while the rest had average conflictscores of 3.0 or more.

Overall the results show that good performance onone of the tests does not imply good performanceon the other two. For example maximum increaseranked low on the estimation and isolation tests, butoutperformed the the other indicators on classifica-tion. Increase frequency performed best on isolation,worst on classification, and estimated the Error met-ric moderately well (0.42 correlation). The area in-dicator ranked high on two of the tests: estimation

(0.68 correlation) and isolation. The only interpreta-tion inconsistency indicator that performed well on allthree was the Gambino indicator which achieved thebest overall performance on estimation, and secondbest on classification and isolation. This indicator’s0% false positive and 7% false negative rates also in-dicate that it would be a good candidate solution forthe problem of detecting sensing problems in unknownenvironments.

6 CONCLUSIONS

The analysis of 11 interpretation inconsistency indica-tors, developed and tested using real sensor data froma mobile robot operating in controlled indoor environ-ments, show that the Gambino indicator could serveas a general solution to the problem of detecting, es-timating, and isolating sensing problems in unknownenvironments. A series of 30 experiments showed thatthe Gambino indicator, using a threshold of 2.0, couldestimate the overall error in the occupancy grid (0.77correlation), classify the sensing situation with a falsenegative rate of 7%, and perform at least as well asthe other indicators on isolating problems within thegrid. The analysis also shows that the area, maximumincrease, and increase frequency indicators (which alsoperformed well on at least one of these tasks) could bedeveloped into specialized tools.

This exploratory study has developed a general ap-proach to the problem of detecting sensing problemsin unknown environments, relying only on the assump-tion of consistency and Dempster-Shafer theory. Theexperiments reported in this study represent one pos-sible application of this approach, i.e. to detecting in-consistency in sensor readings with 2D models. Morework is needed to explore its full potential to captureinconsistency in a variety of information sources andbelief spaces.

One problem for any inconsistency-based approach likethis one is the homogeneity of the sources, especially interms of the conditions in which they will produce in-appropriate (or simply incorrect) information. For ap-plications like mapping in mobile robots, where only afew sources are used and the conditions (environment)change rapidly, this issue rarely appears. Alternate ap-plications of this approach in which many sources (e.g.websites on the internet) are consulted and the infor-mation that is gathered is at a higher (e.g. symbolic ordecision) level, homogeneity may be more of a prob-lem. As potential applications of this approach areexplored, the failure conditions of the sources and thesensitivity of a specific interpretation inconsistency in-dicator to the homogeneity of those conditions shouldbe examined.

Indicator Mean σ2 Best NAngular 9.94E-3 7.94E-8 1.03E-2 20resolutionArea 1.71E-3 5.33E-6 9.29E-3 100Average 1.72E-3 1.04E-6 2.97E-3 20Average 1.80E-3 3.26E-6 6.29E-3 20sequenceFrequency 1.05E-3 1.59E-6 3.36E-3 19Gambino 3.02E-3 2.95E-5 1.83E-2 20Increase 3.39E-4 1.45E-7 1.59E-3 35frequencyMaximum 7.18E-3 2.31E-5 1.95E-2 20increaseRange 8.84E-3 1.84E-7 1.00E-2 20resolutionTotal 8.84E-3 1.84E-7 1.00E-2 20Update 6.30E-3 5.66E-7 7.29E-3 20rate

(a) (b)

Figure 4: Results of the classification test showing Fisher’s Linear Discriminant (FLD) for each of the 11indicators. N is the number of thresholds values tested for each metric.

Despite this vulnerability the field of fusion, in addi-tion to robotics, could benefit from the approach pre-sented in this paper which treats inconsistency as atool to evaluate the fused result. Such an approachcould provide feedback for a fusion system, to help re-solve the problems of association (determining whichbits of information are describing the same thing)and reliability estimation described as important chal-lenges in (Appriou et al. 2001). This could lead to sys-tems which adaptively switch information sources tooptimize mission performance, learn the relative con-tribution and reliability of sources for different envi-ronments, and reason about which information sourcesto use under what circumstances.

Acknowledgments

This effort was supported by grants from the Office ofNaval Research (N00773-99PI543), the Department ofEnergy (DE-FG02-01ER45904), and SAIC. The au-thors would like to thank Jennifer Burke and theanonymous reviewers for their helpful comments.

References

Appriou, A.; Ayoun, A.; Benferhat, S.; Besnard, P.;Cholvy, L.; Cooke, R.; Cuppens, F.; Dubious, D.;Fargier, H.; Grabisch, M.; Kruse, R.; Lang, J.;Moral, S.; Prade, H.; Saffiotti, A.; Smets, P.; andSossai, C. 2001. Fusion: General concepts and char-acteristics. International Journal of Intelligent Sys-tems 16(10):1107–1134.

Ayoun, A., and Smets, P. 2001. Data association inmulti-target detection using the transferable beliefmodel. International Journal of Intelligent Systems16(10):1167–1182.

Ayrulu, B., and Barshan, B. 2002. Reliability measureassignment to sonar for robust target differentiation.Pattern Recognition 35(6):1403–1419.

Baddeley, A. 1992. An error metric for binary im-ages. In Proceedings of the International Workshopon Robust Computer Vision, 59–78.

Baltzakis, H.; Argyros, A.; and Trahanias, P. 2003.Fusion of laser and visual data for robot motionplanning and collision avoidance. Machine Visionand Applications 15(2):92–100.

Carlson, J.; Murphy, R.; Christopher, S.; and Casper,J. 2005. Conflict metric as a measure of sensingquality. In Proceedings of the IEEE InternationalConference on Robotics and Automation (ICRA), toappear.

Davidson, A. J., and Murray, D. W. 2002. Simul-taneous localization and map-building using activevision. IEEE Transactions on Pattern Analysis andMachine Intelligence 24(7):865–880.

Dowdy, S.; Weardon, S.; and Chilko, D. 2004. Statis-tics for Research. Wiley Series in Probability andStatistics. Wiley-Interscience, third edition edition.

Ebbinghaus, H.-D.; Flum, J.; and Thomas, W. 1994.Mathematical Logic. Springer, 2nd edition.

Elfes, A. 1989. Using occupancy grids for mobile robotperception and navigation. IEEE Computer 46–57.

Fisher, R. 1936. The use of multiple measurements intaxonomic problems. Annals of Eugenics 7(2):179–188.

Gambino, F.; Ulivi, G.; and Vendittelli, M. 1997. Thetransferable belief model in ultrasonic map build-ing. In Proceedings of 6th IEEE International Con-ference on Fuzzy Systems, volume 1, 601–608.

Hartigan, J., and Wong, M. 1979. Algorithm as 136:A k-means clustering algorithm. Applied Statistics28(1):100–108.

Lopez-Sanchez, M.; de Mantaras, R. L.; and Sierra,C. 1997. Incremental map generation by low costrobots based on possibility / necessity grids. In Pro-ceedings of the Conference on Uncertainty in Artifi-cial Intelligence.

Meko, D. M. 2005. Applied time series analysis, cor-relation. Lecture notes, University of Arizona.

Moreno, L., and Dapena, E. 2003. Path quality mea-sures for sensor-based motion planning. Roboticsand Autonomous Systems 44(2):131–150.

Murphy, R. 2000. Introduction to AI Robotics. MITPress.

Pagac, D.; Nebot, E.; and Durrant-Whyte, H. 1998.

An evidential approach to map-building for au-tonomous vehicles. IEEE Transactions on Roboticsand Automation 14(4):623–629.

Shafer, G. 1976. A Mathematical Theory of Evidence.Princeton University Press.

Shayer, G.; Cohen, O.; Edan, Y.; and Korach, E. 2002.An adaptive fuzzy-logic procedure for ranking logi-cal sensory performance. In Proceedings IEEE Sen-sors, volume 2, 1317–1322.

Smets, P. 1990. The combination of evidencein the transferable belief model. IEEE Transac-tions on Pattern Analysis and Machine Intelligence12(5):447–458.

Thrun, S. 2003. Learning occupancy grids with for-ward sensor models. Autonomous Robots 15:111–127.

Yi, Z.; Khing, H. Y.; Seng, C. C.; and Wei, Z. X. 2000.Multi-ultrasonic sensor fusion for mobile robots. InProceedings of the IEEE Intelligent Vehicles Sympo-sium (IV), 387–391.

Zhu, H., and Basir, O. 2003. A scheme for construct-ing evidence structures in dempster-shafer evidencetheory for data fusion. In Proceedings IEEE Inter-national Symposium on Computational Intelligencein Robotics and Automation, volume 2, 960–965.