Pontius y Malason (2005).pdf

7/30/2019 Pontius y Malason (2005).pdf

1/24

This article was downloaded by:[Universidad De Chile 2205]

On: 14 July 2008

Access Details: [subscription number 791781823]

Publisher: Taylor & Francis

Informa Ltd Registered in England and Wales Registered Number: 1072954

Registered office: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK

International Journal of GeographicalInformation SciencePublication details, including instructions for authors and subscription information:

http://www.informaworld.com/smpp/title~content=t713599799

Comparison of the structure and accuracy of two land

change modelsGil R. Pontius a; Jeffrey Malanson a

a Clark University, Department of International Development, Community and

Environment, Graduate School of Geography, Worcester MA 01610-1477, USA

Online Publication Date: 01 February 2005

To cite this Article: Pontius, Gil R. and Malanson, Jeffrey (2005) 'Comparison of the

structure and accuracy of two land change models', International Journal ofGeographical Information Science, 19:2, 243 265

To link to this article: DOI: 10.1080/13658810410001713434

URL: http://dx.doi.org/10.1080/13658810410001713434

PLEASE SCROLL DOWN FOR ARTICLE

Full terms and conditions of use: http://www.informaworld.com/terms-and-conditions-of-access.pdf

This article maybe used for research, teaching and private study purposes. Any substantial or systematic reproduction,

re-distribution, re-selling, loan or sub-licensing, systematic supply or distribution in any form to anyone is expressly

forbidden.

The publisher does not give any warranty express or implied or make any representation that the contents will be

complete or accurate or up to date. The accuracy of any instructions, formulae and drug doses should be

independently verified with primary sources. The publisher shall not be liable for any loss, actions, claims, proceedings,

demand or costs or damages whatsoever or howsoever caused arising directly or indirectly in connection with or

arising out of the use of this material.
http://www.informaworld.com/smpp/title~content=t713599799http://dx.doi.org/10.1080/13658810410001713434http://www.informaworld.com/terms-and-conditions-of-access.pdfhttp://www.informaworld.com/terms-and-conditions-of-access.pdfhttp://dx.doi.org/10.1080/13658810410001713434http://www.informaworld.com/smpp/title~content=t713599799


2/24


3/24

therefore the scientist would want to select a model that has predictive power.

Hence, it is important to have a method to compare the accuracy of two models and

of several runs for any particular model to see how the models structure relates to

its accuracy.

It can be a challenge to compare various runs of a single model, because a model

can have a large number of options and implied assumptions that interact.Consequently, when one run is different than another run, it is not necessarily

immediately clear which of the possible interactions among which assumptions is

responsible for the difference. Comparison of runs from different models can be

even more challenging, because it can be difficult to tell whether the difference in the

runs is attributable to the models detailed assumptions or to the models more basic

assumptions. LUCC scientists are likely to learn most efficiently when we examine

the influence of the most basic assumptions before the highly-complex assumptions.

To this end, this paper compares several runs of two models that are different in

fundamental ways.

Overview of two models

The first model, Cellular Automata Markov (CA_Markov), allows any number of

categories and can simulate the transition from any category to any other category.

The second model, Geomod, uses exactly two categories and can simulate only the

transition from the first category to the second category. That is, Geomod can not

simulate an additional simultaneous transition of the second category to the first. In

this respect, CA_Markov is more complex than Geomod. For some applications,

the added complexity of CA_Markov may be desirable, for other applications the

opposite may be true. Therefore, it is important to compare models head-to-head. In

the process of comparing these two models, we illustrate a new generalized methodto measure the predictive power of land change models.

Markov-type models constitute some of the historically most common methods of

predicting change among various categorical states. At their core is the mathematics

of Markov chains, derived by the Russian Scientist Andrei A. Markov (1907). The

first Markov models were not spatially explicit, but since the middle of last century,

the Markov model has been used to simulate changes in maps (Baltzer et al. 1998).

Spatio-temporal Markov chain (STMC) models can be used to model changes over

time among categories in a landscape, whereby each pixel on the landscape is

classified as exactly one category, and each pixel has some probability of

transitioning to some other category at every time step. The Cellular Automata(CA) component of the CA_Markov model allows the transition probabilities of

one pixel to be a function of neighbouring pixels. Baltzer (2000) gives many

examples of how Markov-type models are used.

Geomod is a newer land use change model that was originally designed to

simulate the loss of tropical forests and to estimate the resulting carbon dioxide

emissions (Pontius 1994). Pontius et al. (2001) offers the single most complete

description of Geomod, which used to be known as Geomod2. Geomod predicts a

one-way conversion from one category to one other category, similar to other

popular land change models such as SLEUTH (Clarke et al. 1996, Clarke 1998). The

Geomod approach has been used to simulate deforestation in Massachusetts(Schneider and Pontius 2001), Costa Rica (Hall 2000, Pontius 2002), India (Pontius

and Batchu 2003, Pontius and Pacheco in press) and the tropics globally (Hall et al.

1995a, Hall et al. 1995b). It is rapidly gaining popularity in designing baseline

244 R. G. Pontius Jr and J. Malanson


4/24

scenarios of tropical deforestation in the context of carbon offset projects called for

by the Clean Development Mechanism of the Kyoto Protocol (Brown et al. 2002b).

It is now being used in private businesses, non-governmental organizations, and

research institutes (Brown et al. 2002c).

The methods section of this paper describes the qualitative and mathematical

details of each model, and then applies each model to a landscape in central

Massachusetts, USA. Several runs of each model are compared, so naturally, there

must be a criterion for comparison. The next two subsections of this introduction

motivate the selection of a criterion, which the methods section then explains in

technical detail.

Modelling strategy of calibration and validation

In land-use change modelling, there is no agreed upon criterion to assess the

performance of one model versus another model, or to compare one run versus

another run of the same model. This lack of a uniform method of assessment isrelated to the fact that there is not agreement among scientists concerning the

purpose of land change modelling. Some scientists prefer models that express the

theory of the mechanisms of the processes of land change, while others place more

weight on a models ability to extrapolate the observed pattern of change based on

past empirical patterns. It either case, most scientists would like for models to be

able to simulate true patterns. Therefore, there is a need to quantify a models

predictive power.

In order to quantify the predictive power, scientists should examine the goodness-

of-fit of the validation, which should not be confused with goodness-of-fit of

calibration. Calibration is the process whereby the scientist uses information aboutthe landscape to help select the parameters of the model. The information used for

calibration should be at or before some specific point in time (t1), which is the point

in time at which the predictive extrapolation begins. For example, CA_Markov

models are typically calibrated with reference maps from two points in time, t0 and

t1; whereas Geomod requires maps from only one point in time, t1. In both cases,

the model then extrapolates land change beyond time t1 to some subsequent time t2.

Validation is the process of comparing the models prediction for time t2 to a

reference map of time t2, where the reference map is considered a much more

accurate portrayal of the landscape at time t2. In order for the extrapolation to be a

legitimate prediction, the information used in the calibration must have existed at orbefore t1.

Pontius and Pacheco (in press) stress that a good fit of calibration does not

necessarily imply a good fit of validation, and that the later is the appropriate

indicator of a models predictive power. Mertens and Lambin (2000) and Brown et

al. (2002a) are two of the few examples that we have been able to find that clearly

distinguish calibration from validation. Rykiel (1996) and Oreskes et al. (1994) give

a more detailed and philosophical discussion of validation.

Criteria for accuracy assessment

Separation of the calibration process from the validation process is necessary, but

not sufficient, in order to compute the goodness-of-fit of validation. The scientist

must also select a criterion to compare the predicted map of time t2 to the reference

Comparison of two land change models 245


5/24

map of time t2. Perhaps the most popular criterion is the percent of pixels classified

correctly.

The percent correct is popular because of its ease of computation and apparent

ease of interpretation; however a high level of sophistication should be used to

interpret the percent correct. Most importantly, the interpreter must realize that a

large percent correct does not necessarily imply that the model has good predictive

power, due to a variety of reasons. Most importantly, it is common to attain a large

percent correct from a null model that predicts pure persistence (i.e. no change)

between time t1 and time t2, due to temporal autocorrelation between the reference

maps of t1 and t2. In most of the land change modelling literature that we examined,

the typical amount of change on the landscape was about 10%, so a null model of

pure persistence would be 90% correct, which is a measure of accuracy that most

nave interpreters would consider high. Hagen (2002, 2003) is the only other author

that we have found that articulates the need to compare a predictive model to a null

model of pure persistence.

The criterion of percent correct has additional complications. The percent correctis usually based on a pixel-by-pixel analysis, so it is possible that percent correct will

not correspond to a visual assessment of the maps, because pixel-by-pixel analysis

fails to consider spatial patterns of the pixels. For example, if a pixel is classified as

the wrong category, then the entire pixel is in error, regardless whether the correct

category is found in the neighbouring pixel or nowhere near the pixel (Pontius 2002).

Lastly, a large portion of the percent correct can be attributable to random chance

(Pontius 2000).

Consequently, this paper uses the null resolution as the criterion for the

goodness-of-fit of validation. The null resolution is the resolution at which the

accuracy of the predictive model is the same as the accuracy of a null model, whereaccuracy is measured in terms of percent correct of the entire landscape. The finer

the null resolution the more accurate and precise is the models prediction. The

methods section describes the multiple-resolution procedures necessary to compute

the models predictive power in terms of the null resolution.

Methods

Data

The data for this paper derives from the State of Massachusetts Executive Office of

Environmental Affairs, which makes GIS files available through the agency calledMassGIS. Vector files of historic land use are available for three times: t051971,

t151985, and t251999. Each map shows 20 categories. For each year, these

categories are reclassified into two categories, built or non-built, based on the

Anderson level 1 classification system (Anderson et al. 1976). For this study area,

built consists mostly of Residential, and includes Commercial, Industrial,

Transportation, Recreation and Waste Disposal. All of the maps are converted to

raster format on a 30-by-30 meter grid of 1116 rows and 1008 columns, which

contains 651,951 pixels in the study area.

The study site consists of the town of Worcester and the nine adjacent towns in

central Massachusetts. Worcester is a common example of a post-industrialAmerican city that has undergone suburbanization over the last three decades.

Recently, Worcester has become one of the nations fastest growing housing

markets, as many people prefer to live in central Massachusetts as opposed to the



6/24

nearest major population centre, Boston, where the housing prices are approxi-

mately double what they are in the Worcester area. As a result, real estate developers

are converting much of the land in the Worcester region to residential uses, which is

causing tremendous concern for the existing residents of central Massachusetts.

Figure 1 shows that from 1971 to 1985, approximately 3.7% of the landscape

converted from non-built to built and a negligible amount converted from built tonon-built. Figure 1 is the signal used to calibrate the model. Figure 2 is the true

pattern used eventually to assess the accuracy of the models prediction through the

validation process. Figure 2 shows that between 1985 and 1999, approximately 5.4%

of the landscape converted from non-built to built and 0.8% converted from built to

non-built, resulting in a net increase in quantity of built on the landscape of 4.6%.

The legend of figure 3 shows the detailed categories of land use of 1971. Historic

land use is important as a predictive factor of future change because historic land

use is related to the ease with which land can be converted to built uses. For

example, old agricultural land has a higher propensity to transition to built than

does forest. This is probably related to the fact that agricultural land is alreadycleared and has a closer proximity to infrastructure such as roads and water services.

Figure 4 shows the categories of legal protection that restrict the conversion of

land to new built uses. In Massachusetts, legal restrictions are clearly important in

determining which parcels of land convert to the built category. Unfortunately the

map of legal restrictions does not specify the years in which the laws were enacted;

therefore it is unclear whether the map of legal restrictions is legitimate for

calibration, since some of the laws portrayed in the map were probably created

Figure 1. Change in built category from 1971 to 1985, which is the signal used forcalibration.



7/24

Figure 2. Change in built category from 1985 to 1999, which is the pattern used forvalidation.

Figure 3. Land use of 1971, which is a driver used to calibrate the suitability maps.



8/24

subsequent to 1985. This paper uses the map of legal constraints in half of the model

runs in order to demonstrate both the impacts that such constraints can have on

land change predictions, and also to show the maximum level of accuracy attainable

if we were to use this potentially illegitimate data for model calibration.

Structural comparison

CA_Markov and Geomod have some major qualitative differences in basic

structure, and other minor differences in more subtle aspects. Table 1 states thedifferences between the models in general order of most fundamental to most

detailed. The first two characteristics in table 1 show that CA_Markov can simulate

two-way transitions among any number of categories, whereas Geomod simulates

either a one-way gain or a one-way loss from exactly one category to one alternative

category. In this sense, CA_Markov can theoretically model a wider variety of

simultaneous processes. There are a suite of challenges and complications associated

with simulating transitions among several categories. For example, many different

categories might simultaneously compete to claim any particular pixel. To deal with

this situation, CA_Markov implements a multiple objective land allocation

algorithm. Geomod avoids these complications by assuming only one type oftransition from one category to one other category. For many applications, the

process of interest can be expressed in terms of a transition from a non-disturbed

category to a disturbed category, in which case Geomods approach could suffice.

Figure 4. Contemporary legal constraints to development, which is an optional additionaldriver used to calibrate the suitability maps.



9/24

The remaining characteristics of table 1 are grouped in terms of the two basic

tasks that a model must perform. Specifically, the model must predict the quantity

of each category and the model must predict the location of each category. BothCA_Markov and Geomod specify the quantity of the predicted categories

independently from the location of the predicted categories. In table 1,

characteristics 34 refer to the rules by which each model predicts the quantity of

each category. Characteristics 57 refer to the rules by which each model predicts

the location of each category. The sections below give the details of how each

characteristic manifests in the example.

The characteristics in table 1 are related closely to figure 5, which shows the

specific options that this paper uses to model the land change for the example.

The first option of figure 5 concerns which model to use: CA_Markov or Geomod.

The next option concerns the method to extrapolate the quantity of each land

category. The Markov approach extrapolates both the gain and loss of each

category based on the proportion estimated from the calibration information.

Geomod recommends linear extrapolation to predict the net quantity of the built

category. The remaining options concern the decision rules that each model can use

to determine the location of the predicted land change. One important decision

concerns the factors to include in the creation of a suitability map. Another option

concerns whether or not to implement a contiguity rule that encourages pixels of the

same category to be adjacent. Yet another option concerns the time step, which is

the shortest interval of time over which the model predicts change. This final option

is relevant only if the contiguity rule is selected.

The options shown in figure 5 give a range of the most important options

available, but they are not exhaustive. For Geomod, an additional option allows for

stratification of the analysis. For CA_Markov, an additional option allows for

consideration of the accuracy of the reference data. The sections below discuss the

details of each option. In total, this paper examines the 24 different model runs

shown in figure 5 to extrapolate the change between 1985 and 1999, based on

calibration from 1971 to 1985.

CA_Markov

Predicting Quantity of Change. CA_Markov predicts the quantity of each categoryat time t2 by extrapolating both gain and loss of each category from time t1. This

extrapolation is calibrated by computing the proportional gain and loss between the

years of calibration, which are t051971 and t151985 in our example. The

Table 1. Qualitative characteristics of CA_Markov and Geomod.

Characteristic CA_Markov Geomod

1) Number of Categories Two or more Two2) Transitions Gains and Losses Gain or Loss

3) Information for Quantities Computed from 2 maps Must be supplied4) Quantity Propagation Multiplicative Additive5) Suitability Map Must be supplied for

each transitionCan be created from drivermaps and a land-use mapof 1 point in time

6) Proximity Method Filter Constraint7) Stratification Not part of the design Part of the design



10/24

extrapolated quantity for each category is a function of the transition matrix asdictated by the matrix algebra of Markov chains. Specifically, the Markov

calculation assumes that for every 14-year time interval, the proportion of land

that transitions from one category to another category is the same as the proportion

of land that made that particular transition during the 14-year calibration period.

Figure 6 shows this extrapolation where both the calibration interval and the

extrapolation interval are 14 years. During the calibration interval, about 3.7% of

the landscape shows gain in the built category and negligible loss of built. Therefore,

over the subsequent 14 years, CA_Markov predicts gain of built and negligible loss

of built. The open triangles of figure 6 show the true gross gain in built, whereas the

closed triangles show the true net gain in built after the loss of built is considered.The open diamond shows the extrapolated gain of built, and the closed diamond

below the open diamond indicates the net gain of built after the loss is taken into

consideration. The extrapolation of quantity is computed independently of the

decision of where to locate the change among categories on the landscape.

Predicting Location of Change. There are two principles that determine the location

where CA_Markov predicts land change. The first is based on the concept of a

suitability map and the second is based on the concept of a contiguity rule.

CA_Markov uses a suitability map for each transition that it extrapolates. At

every time step, CA_Markov determines the number of pixels that must undergo

each transition, then selects the pixels according the largest suitability for theparticular transition. The user can generate the suitability maps in a variety of ways,

for example by using a deductive approach such as Multi-Criteria Evaluation, or an

inductive approach such as logistic regression. In our example, CA_Markov predicts

Figure 5. Combinations of options for model runs.



11/24


12/24

the transition from non-built to built. The subsection on Geomod describes the

detail of that suitability map (figure 7), including the option to use the map of legal

constraints in the creation of the suitability map. The suitability maps remain static

for the duration of the simulation.

A contiguity rule is the second principle that CA_Markov uses to determine the

location of predicted change. The contiguity rule usually has the effect of predictingthe growth of a category near locations where that category already exists. At every

time step, the suitability value in each candidate pixel is temporarily recalculated to

show spatial dependence, such that the suitability for the transition to a particular

category is influenced by whether that category exists in nearby pixels. If few of the

nearby pixels belong to the category, then the suitability value is down-weighted. If

many of the nearby pixels belong to the category, then the suitability value is

maintained. The definition of nearby is determined by a spatial filter that the user

specifies. Our example uses a filter which causes the suitability at a particular pixel

to be weighted as a function of its nearest 24 neighbouring pixels, such that closer

neighbours exert stronger influence on the calculation. This temporary recalculationis performed at each time step; therefore the duration of the time step can have some

influence on the location of predicted land change when the contiguity rule is used.

When the time step is small, the model must apply the time step for many iterations

in order to achieve the desired duration of the extrapolation. Therefore, smaller time

steps lead to more frequent applications of the filter, hence more frequent updates of

the spatial dependence. If the user does not want to simulate spatial dependence

Figure 7. Suitability for conversion from non-built to built, based on slope and land use of1971.



13/24

explicitly, then the most appropriate filter has a 1 in the centre surrounded by 0s.

CA_Markov always assumes some spatial dependency, but this filter exerts the least

possible spatial dependency, so when it is used, the selection of time step is negligible.

Geomod

Predicting Quantity of Change. Geomod has no explicit method to extrapolate the

quantity of change from its one category to the alternative category, because the

purpose of Geomod is to predict the location of a one-way transition. Nevertheless,

Geomod must specify some quantity of change in order to make any prediction;

therefore Geomod allows the user to predict any quantity, based on any

extrapolation method that the user desires. We recommend a simple approach,

unless it is clear that a more complicated approach is justified. Therefore, the

example in this paper uses linear extrapolation of the net quantity of built, based on

the line that is interpolated through the quantities at 1971 and 1985. Figure 6 shows

that Geomods linear extrapolation of the quantity is nearly identical toCA_Markovs extrapolation of the gain and loss. The reason for the similarity of

the extrapolation is that the percent of change that took place on the landscape

during the calibration interval is small.

Half of the Geomod runs use the quantity obtained from the linear extrapolation

and the other half of the Geomod runs use the net quantity obtained from the

Markov extrapolation (figure 5). This allows eventual comparison between the

Geomod runs and the CA_Markov runs. Among all runs, there are three possible

quantities: Markov, net Markov, and linear.

Predicting Location of Change. There are four principles that determine the location

where Geomod predicts land change. First, and most importantly, Geomod predictspersistence for the category that grows during the extrapolation. For the example,

the built category grows in net quantity, therefore if a pixel is built in 1985, then

Geomod predicts that it will remain built in 1999. This is different than CA_Markov

and the true process, both of which show some loss of the built category. As a

consequence, Geomod is doomed to fail to predict any real conversion from built to

non-built, which is about 0.8% of the landscape as shown in figures 2 and 6.

Next, Geomod has the ability to stratify the entire analysis, meaning that

Geomod can specify the quantity in each stratum and calibrate the suitability map

separately in each stratum. This is helpful because it is common that data are

available by political unit, and that different political units experience differentpatterns of land change. Our example is not stratified in order to allow a more direct

comparison with CA_Markov, which does not allow for stratification.

Geomods last two principles that determine the predicted location of land change

are similar to the two principles of CA_Markov. Specifically, Geomod can use a

suitability map and/or a contiguity rule.

Similar to CA_Markov, Geomod examines a suitability map to find the pixels

with the largest suitability value, and then predicts conversion of new built at the

non-built pixels that have the largest suitability for built. The suitability map is

completely independent of the duration and the time step of the extrapolation, and

the suitability map remains static for the duration of the simulation.Unlike CA_Markov, Geomod has a method to create empirically and

automatically a suitability map based on several driver maps and a land cover

map from a single point in time. In our example, the single land cover map is the



14/24

built versus non-built map of 1985. The driver maps are slope and land use of 1971.

The information to compute the suitability map should be restricted to information

that would have been available during the calibration interval, 19711985.

The legend of figure 3 shows the various categories of land use of 1971, where the

darker shade indicates a larger proportion of each category that is built in 1985.

Categories that are pure black are categories that were built in 1971, of which more

than 90% remained built in 1985. The non-black categories were non-built in 1971.

For example, in the 1971 landscape, mining is a non-built category that transitioned

to 25% built by 1985. The non-built categories of Cropland, Water Based

Recreation, Pasture, and Woody Perennial are between 5% and 20% built in

1985. All other non-built categories experienced less than 5% conversion to built

during the interval 19711985.

Information of figure 3 is combined with information of a slope map to generate a

map of suitability for the built category. In general, flatter slopes have a larger

suitability for the built category. Figure 7 is the suitability map for conversion of

non-built to built for both Geomod and CA_Markov. Pontius et al. (2001) discussthe details of the method to calibrate the suitability map from any number of driver

maps. In the example, equal weights are applied to the slope driver and the 1971

land use driver.

A map of the present-day legal restrictions on development is available (figure 4),

so it could be used as another driver to create the suitability map. The date of the

information in the map is not well documented, as is typical of freely-available data.

Presumably many of the laws that regulate land conversion today, were in place in

1985. But several laws were probably created subsequent to 1985, which would

render our map of legal restrictions illegitimate to predict change between 1985 and

1999. Nevertheless, the legal restrictions are probably one of the most importantfactors in determining the land conversion; therefore we perform the model runs

both with and without the map of legal restrictions as a factor in order to measure

the maximum additional predictive power that the information of legal restrictions

could possibly offer. When the map of legal restrictions is used, it forces the

suitability for conversion to the built category to be zero in any pixel that has any

minimal level of legal restriction, meaning that any pixel that is classified as

permanent, limited, or unknown is given a suitability value of 0.

If the contiguity option is used, then Geomods search for the largest suitability

values is constrained to the non-built pixels that are near existing built pixels. This

option is useful for landscapes where growth of a land category occurs adjacent tothat category. The definition of near can be set by the user to include pixels that are

within any number of pixels of the edge of the two categories. Our example uses a

5-by-5 window in order to make the spatial dependency consistent with

CA_Markovs spatial filter. The 5-by-5 window forces the predicted land change

to be within two pixels of the edge including the diagonal. The definition of edge is

updated at every iteration of the time step; therefore it is possible for the time step to

have some influence on the location of the predicted landscape when the contiguity

rule is used. If the extrapolation has small time steps, then Geomod can predict

incremental growth at the locations with large suitability values. If the extrapolation

has a large time step, then Geomod is forced to locate the change near the edge, withless consideration of the suitability values. Similar to CA_Markov, smaller time

steps lead to more iterations and hence more frequent updates of the spatial

dependency. If the contiguity rule is not used, then the time step has no influence on



15/24

the predicted results. Figure 8 gives an example of the output of a Geomod run,

which is overlaid with maps of the real landscape for 1985 and 1999. The red and green

in figure 8 show the locations where Geomod simulated additional built land between

1985 and 1999.

We also consider three model runs that distribute the predicted quantity of change

evenly across the landscape. According to the mathematics of statistical expectation,the accuracy of these runs is the accuracy that one would expect from a model that

distributes the change at randomly selected locations. The three runs correspond to

the three different methods to predict the quantity of change: Markov, net Markov,

and linear. The net Markov and linear quantities give an increase in the built category,

whereas the Markov quantities give simultaneous gain and loss of the built category.

Goodness-of-fit

Percent correct. Two criteria measure the goodness-of-fit of each model run. The

first criterion is the percent of pixels classified correctly on the entire landscape at

the resolution of the raw data, which are 30-by-30 meter pixels. The percent correct

criterion is extremely popular and frequently misinterpreted. This paper shows how

to interpret the percent correct in an appropriate and sophisticated manner.

There are two crucial issues that a scientist should consider when using the

percent correct criterion. First, the scientist should compare the percent correct of

the predictive model to the percent correct of a null model that predicts pure

persistence. In all cases that we have seen, a null model of pure persistence gives a

larger percent correct than a predictive model at the resolution of the raw data. This

indicates that the resolution of the raw data is finer than the resolution at which the

simulation model can accurately predict the process of change. Second, the scientist

should examine the percent correct at multiple-resolutions in a way similar toCostanza (1989) and Pontius (2002), whereby a pixel aggregation procedure simply

averages neighbouring pixels into coarser pixels in order to quantify the agreement

between the maps at coarser resolutions. As the resolution of the accuracy

assessment becomes coarser, the percent correct tends to rise. The percent correct

Figure 8. Overlay of the real 1985, the real 1999, and the simulated 1999 for the model runthat is most accurate. The two shades of gray and the red are classified correctly by thesimulation model, whereas the blue, green and yellow are errors by the simulation model. Anull model would correctly predict the green and the two shades of gray.



16/24

increases rapidly at resolutions that correspond to distances at which the model is

accurate. At the coarsest resolution for which one extremely coarse pixel contains

the entire study area, the only error in the accuracy assessment is the error due to

misspecification of quantity.

Figures 9 and 10 show this phenomenon for a null model of pure persistence and

the best model prediction respectively. In each figure, the reference map for 1999 iscompared to the modelled map of 1999. The horizontal axis shows resolution

changing from fine to coarse as one moves from left to right. The vertical axis is the

percent of the entire landscape. The figures show three components of agreement

(agreement due to chance, agreement due to quantity, and agreement due to

location) and two components of disagreement (disagreement due to location and

disagreement due to quantity). The border that separates the agreement due to

location from the disagreement due to location is the percent correct. At the coarsest

resolution, location no longer has meaning, so there are no components associated

with location and the only error is due to quantity. Readers who are especially

interested in the details of the calculations should read Pontius (2000), Pontius(2002) and Pontius and Suedmeyer (in press).

The change in resolution is accomplished by aggregating neighbouring pixels at

the fine resolution into coarser pixels. The aggregation rule computes the aggregate

membership of each category for each coarse pixel as the proportion of fine

resolution pixels of each category that constitute the coarse pixel. Each additional

progression in the aggregation process averages the membership of four

neighbouring pixels, therefore the size of the resolutions progress as a geometric

sequence, where the side of a pixel in each additional resolution is twice the length of

the side of the pixels at the previous resolution.

Null resolution. The second criterion to assess a model run is the null resolution.

Figure 11 shows how to determine the null resolution, which is the resolution at

Figure 9. Components of agreement and disagreement between real 1985 map and real 1999map at multiple resolutions.



17/24

which the percent correct of the predictive model crosses the percent correct of the

null model. Figure 11 has the same axes as figures 9 and 10, since figure 11 shows an

overlay of the crucial information in figures 9 and 10. In figure 11, the solid line with

the circles (or triangles) is the percent correct for the predictive model (or null

model) respectively. The dashed line with the circles (or triangles) shows the error

due to quantity for the predictive model (or null model) respectively. The solid lines

Figure 10. Components of agreement and disagreement between best predicted 1999 mapand real 1999 map at multiple resolutions.

Figure 11. Illustration of the method to determine that the null resolution is 32 times thelength of the raw pixel side.



18/24

approach the dashed lines as resolution becomes coarser, and reach the dashed lines

by at least the coarsest possible resolution, which is the resolution at which one

coarse pixel contains the entire study area.

At fine resolutions, the percent correct for the predictive model is less than the

percent correct for the null model. As resolution becomes coarser, the percent

correct for both the predictive model and the null model rises, however the percentcorrect for the null model is constrained by its relatively large error due to quantity.

Since the predictive model has a smaller error due to quantity, the percent correct

for the predictive model becomes larger than the percent correct for the null model

at some resolution, which is called the null resolution. At resolutions finer than the

null resolution, the predictive model is less accurate than the null model of pure

persistence. At resolutions coarser than the null resolution, the predictive model is

more accurate than the null model of persistence only. More accurate models have

finer null resolutions because the null resolution is the resolution at which the

predictive model starts to be more accurate than a null model. For example, figure

11 shows a null resolution of 32 times the size of the side of a pixel of the raw data.

Results

Figure 8 displays the most important information of the best simulation run, which

is 92% correct at the 30-meter resolution. Figure 8 overlays the 1985 truth map, the

1999 truth map, and the 1999 simulated map of a Geomod run that uses net Markov

quantities, considered laws, and is unconstrained. The two shades of gray and the

red are classified correctly by the simulation model, but the gray would be classified

correctly also by a null model that predicts persistence only. Blue, green and yellow

are errors by the simulation model, but the green locations would be predicted

correctly by a null model. Both the null model and the simulation model fail topredict the blue and yellow. Figure 8 shows that the null model has a higher percent

correct than the best simulation run because there is more green than red.

Figure 12 plots the null resolution versus the percent correct at the 30-by-30 meter

resolution for all 24 model runs. Among all runs, the percent correct at the finest

resolution ranges from 91.06 to 92.15. These accuracies are less than the percent

correct of the null model, which is 93.75, but are greater than the accuracies of the

runs that distribute the change evenly across the landscape, which range from 90.75

to 90.91 percent correct. The null resolution for nearly all runs is 32 times the

resolution of the raw 30-meter data, which corresponds roughly to a 1-kilometer

resolution. Four of the runs have a null resolution of approximately 2 kilometres.Figure 12 shows that the model runs are grouped into distinct clusters that are

based more on the options for each model and less on the selection of CA_Markov

versus Geomod. The cluster that has the largest percent correct (ranging from 91.87

to 92.15 percent correct) contains the runs that do not use a contiguity rule. These

runs allow predicted change to occur at the largest suitability values, regardless of

whether those locations are near existing built areas. Within this most accurate

cluster, the runs with the finer null resolution of 32 tend to be from Geomod and to

use the laws in the suitability map. Among the runs that use the contiguity rule,

there are three clusters. Among these three clusters, the one with the largest percent

correct (ranging from 91.52 to 91.71 percent correct) contains all the constrainedGeomod runs. Among these constrained Geomod runs, the ones that predict a

smaller quantity of change have a larger percent correct. The filtered CA_Markov

runs are organized in two sub-clusters. The first of these sub-clusters (ranging from



19/24

91.30 to 91.39 percent correct) contains the CA_Markov filtered runs that have a

one 14-year time step. The other sub-cluster (ranging from 91.06 to 91.10 percent

correct) contains the CA_Markov filtered runs that have fourteen 1-year time steps.Within these sub-clusters, the runs that predict the smaller quantity of change, give

greater accuracy. In all cases, the use of the laws improves the predictive power of

the extrapolation. The laws account for variation of percent correct within the

clusters, but not variation among the clusters.

The cluster in the far left of figure 12 shows the statistically expected results from

runs that distribute the change randomly. The null resolution of the random runs is

32. In terms of percent correct, all the CA_Markov and Geomod runs perform

better than the random runs, but they also perform worse than the null model.

Furthermore, none of the predictions were more accurate in spatial location than

would be expected by random chance.The range in percent correct among the CA_Markov and Geomod runs is about 1

percentage point. This should be viewed in the context that the range in percent

correct from the random models to the null model is 3 percentage points. A visual

assessment of the various prediction maps shows that the percent correct

corresponds with the pattern agreement, meaning that the larger the percent correct

the more similar the spatial pattern. Most of the variation in the visual assessment is

attributable to the contiguity rule.

Discussion

Interpretation of Results

Percent correct. For any particular modelling application, it is a challenge to

separate the results that are true in general from those that true for only a specific

Figure 12. Null resolution versus percent correct at finest resolution for all model runs.Diamonds denote Geomod, squares denote CA_Markov, triangles denote smooth distribu-tion of location. Gray denotes Markov quantities, black denotes linear quantities. Smallsymbols denote that laws are considered, large symbols denote laws are ignored. Unfilledsymbols denote no contiguity rule, filled symbols denote contiguity rule.



20/24

landscape. We attempt to interpret our results in terms of general concepts that are

likely to apply to other landscapes and other models.

First, it is extremely important that the modeller be able to control the contiguity

rule. The behaviour of the contiguity rule explains the differences among the clusters

and sub-clusters in Figure 12. The cluster with the largest percent correct uses no

principles of contiguity, meaning that it does not force the prediction of new built tobe near existing built. The next most accurate cluster is the unconstrained Geomod

runs. The effect of the contiguity rule in Geomod is weaker than in CA_Markov.

Geomods contiguity rule merely restricts the candidates for new Built to be near the

existing Built, then Geomod selects the pixels based on the suitability map.

CA_Markovs contiguity rule applies a spatial filter to the entire suitability map,

with strong weighting towards predicting new built to be near existing built. This

filter rule is applied at every time step. As a result, there are two distinct sub-clusters

in figure 12 for the constrained CA_Markov runs. The least accurate cluster is the

one where there are fourteen 1-year time steps, as the filter is applied at each of the

fourteen steps.Some models assume that land change can be predicted accurately using spatial

dependency as expressed in a contiguity rule. However, a quick glance at figures 2

and 3, show clearly that new built areas do not grow from existing built areas.

Therefore, any model that forces spatial dependency is doomed on such a landscape.

This is an important observation because the contiguity assumption is hard-coded

into some LUCC models. For other landscapes at other scales, the best predictor of

the location of new development may indeed be the proximity to existing

development (Mertens and Lambin 2000, Pontius and Batchu 2003, Pontius and

Pacheco in press), but the Massachusetts landscape is an example for which the land

change occurs in entirely new patches, not connected to existing patches (Schneiderand Pontius 2001). Therefore, it is important that models have an option to control

the level of spatial dependency. Both CA_Markov and Geomod have such an

option.

Second, an important basic principle is that when the model predicts change, it

usually predicts it in the wrong location with respect to the fine resolution, therefore

models that predict a small quantity of change tend to have a higher percent correct

than models that predict a larger quantity of change. This phenomenon explains the

variation in accuracy within the clusters. There are three quantities in this analysis.

They are in order of smallest quantity of change to largest: net Markov, Linear, and

Markov. In all cases, the accuracy within clusters follows this ordering, where net

Markov is most accurate and Markov is least accurate.

In no cases did the small 1-year time step improve the accuracy beyond the level

attained by the corresponding run that has one 14-year time step. This supports

Ockhams razor in terms of both accuracy and CPU time.

Null resolution. Nearly all of the runs have a null resolution of 32 times the 30-meter

pixel side, which is about 1 kilometre. The only exceptions are four of the

unconstrained runs, which have a null resolution of about 2 kilometres. It makes

sense that the unconstrained runs can have coarser null resolutions because those

runs allow predicted change to occur far from existing built locations, since the

predictions are unconstrained in space. All the models predictions are less accuratethan a null model at distances less than 1 kilometre. But at distances greater than

2 kilometres, all the models predictions are more accurate than a null models

prediction.



21/24

The model runs that distribute the change smoothly or randomly across the

landscape also have a null resolution of 32. Therefore, there are no scales at which

any of the model predictions are simultaneously more accurate than both a null

prediction and random prediction. We suspect that this characteristic is typical of

the accuracy of contemporary LUCC models. We know of no spatially explicit

models that have been shown to predict for any landscape better than both a nullmodel and a random model at any spatial resolution.

Signal versus Noise

The results show that it is important to focus on the most important signals of

change on the landscape, as opposed to modelling the detail of the noise. In our

example, the major signal is persistence, with a modest amount of gain of the built

category arranged in patches distributed widely across the landscape. The some of

the noise is the small patches of transition from built to non-built. The predictive

power of the runs is higher for the cases where the model focuses on the signal and

ignores the noise. For example, Geomod ignores the small conversion from built tonon-built, whereas CA_Markov attempts to model the weak empirical pattern of

conversion from built to non-built. When CA_Markov tries to predict this rare

phenomenon, it usually predicts incorrectly. Consequently, Geomod gives more

accurate results by ignoring the rare phenomenon.

The importance of focusing on the signal and ignoring the noise is especially

crucial in light of the quality of the data. Noise derives from both unsystematic

trivial change on the landscape and error in the maps. We do not know exactly how

much error there is in the data because proper metadata for the maps does not exist

and a complete error assessment of the maps was not done by the map producer. It

could be that the small conversion from built to non-built is primarily error. In thiscase, it is wise to ignore this transition. Models that attempt to reproduce all the fine

detail in the calibration data run the risk of modelling the error, which would be

counter-productive.

Scientists need to improve our understanding of how map error influences the

level of certainty we should have in models (Foody and Atkinson 2002). For now,

one necessary step is to apply rigorous procedures that separate calibration

information from validation information. When scientists fail to distinguish between

the calibration process and the validation process, there is a temptation to refine the

model parameters until the model reproduces the noise in the data. This flawed

approach can cause a models output to be a close match with the reference maps,but the apparent accuracy is a result of modelling the noise, i.e. of over-fitting the

model. Such a practice can lure scientists into thinking that a model has a greater

predictive power than it really does.

Importance of goodness-of-fit analysis

Techniques to measure the goodness-of-fit of validation are the least sophisticated

tools in the standard toolbox of the contemporary land change modeller. In the past,

scientists assessed the accuracy of models by a qualitative visual assessment; more

recently modellers have been looking for more sophisticated measures (Foody 2002).

Intuitively, modellers know that a nave interpretation of percent correct is notuseful. This paper offers a sophisticated way to interpret a simple, familiar measure.

The temptation among many scientists is to restrict the accuracy assessment to

some subset of pixels, for example to only the pixels that are non-built at the



22/24

beginning of the extrapolation, or to only the pixels that truly change during the

time interval of the extrapolation. We recommend strongly against the exclusion of

any pixels from the accuracy assessment and advise in favour of including the entire

landscape in the accuracy assessment. It is best to consider the entire landscape and

compare the predictive model to a null model of pure persistence. Exclusion of pixels

from the accuracy assessment can make the scientist blind to problems in the model.For example, Geomod predicts only a one-way change from non-built to built,

which is a characteristic that some scientists might consider a weakness. If the

accuracy assessment considers only the pixels that were non-built at the beginning of

the extrapolation, then the accuracy assessment would overlook this potential

problem. Moreover, if the accuracy assessment considers only those pixels that truly

change during the time interval of the extrapolation, then there is an incentive for

the model to predict change everyplace, so as to maximize the predicted change in

the pixels that truly did change. The important message is that if a scientist wants to

gain insight into the dynamics of the entire landscape then (s)he should consider the

entire landscape in the accuracy assessment, and if (s)he would like to know theusefulness of a model, then (s)he should compare its predictions to the predictions of

a null model.

Specifics of the data

Additional investigation is required to tell whether our results are representative of a

wide variety of applications. This paper uses an example from one place in space and

time, specifically central Massachusetts during the end of last century. For this

landscape over this duration, most of the change is a steady one-way increase in the

built category. If the same models are applied to different landscapes at different

times, the conclusions might be quite different, but the appropriate methods for

analysis would be the same. Some landscapes are more dynamic and complex than

others, and the apparent complexity is related in part to the detail of the available

data.

Scientists should be aware that available data is usually much more precise in

spatial resolution and number of categories than is the precision of a models

predictive power, so the modeller should not be tempted to let the availability of

complex data be the major criterion upon which to select a model. In our example,

CA_Markov is poor at predicting the location of conversion of built to non-built.

Geomod was more accurate by following a strategy of ignoring the transition from

built to non-built, even though the reference data was sufficiently detailed to show

this rare phenomenon.

Conclusions

The most important difference between Geomod and CA_Markov is that Geomod

predicts only a one-way transition from one category to one alternative category,

whereas CA_Markov has the ability to predict any transition among any number of

categories. Therefore if the important dynamics of a landscape involve simultaneous

gain or loss of a category or involve substantial interactions among more than two

categories, then it would seem that CA_Markov is the better choice based onqualitative characteristics. Alternatively, it might be advisable to simplify the base

data in order to focus on the major signal of change, which frequently is the

conversion from a single non-disturbed category to a single disturbed category,



23/24

especially in the case of urban landscape modelling. If the goal is to predict a single

dominant process, then Geomod can also be a good choice. Besides this major

conceptual difference, both Geomod and CA_Markov are similar in terms of the

available options that allow for specification of the predicted quantity and location

of categories on a landscape. In our example, the choice of the options accounted for

more variation in accuracy of model runs than the choice of the model.Whatever model is used, the methods of this paper allow a scientist to assess the

accuracy of any model run in a sophisticated manner that gives useful information

for model refinement. The most important aspects of assessing the predictive power

of a model are: 1) to separate the calibration process from the validation process, 2)

to assess the predictive model vis-a-vis a null model, and 3) to perform the accuracy

assessment at multiple resolutions.

Acknowledgements

This research was made possible through the HERO program that NSF supports

through the grant Infrastructure to Develop a Human-Environment RegionalObservatory Network (Award ID 9978052). We acknowledge also programmer

Hao Chen, who was funded by the Center for Integrated Study of the Human

Dimensions of Global Change. This Center has been created through a cooperative

agreement between the National Science Foundation (SRB-9521914) and Carnegie

Mellon University, and has been generously supported by additional grants from

the Electric Power Research Institute, the ExxonMobil Foundation, and the

American Petroleum Institute. The authors thank also Clarklabs, which has made

the methods of this paper available in the GIS software Idrisi.

LiteratureANDERSON, J.R., HARDY, E.E., ROACH, J.T. and WITMER, R.E., 1976, A Land Use And Land

Cover Classification System For Use With Remote Sensor Data. Geological Survey

Professional Paper 964.

BALTZER, H., 2000, Markov chain models for vegetation dynamics. Ecological Modeling, 126,

pp. 139154.

BALTZER, H., BRAUN, P.W. and KOHLER, W., 1998, Cellular automata models for vegetation

dynamics. Ecological Modeling, 107, pp. 113125.

BROWN, D.G., GOOVAERTS, P., BURNUCKI, A. and LI, M.-Y., 2002a, Stochastic Simulation of

Land-Cover Change Using Geostatistics and Generalized Additive Models.

Photogrammetric Engineering & Remote Sensing, 68 (10): pp. 10511061.

BROWN, S., SWINGLAND, I., HANBURY-TENISON, R., PRANCE, G. and MYERS, N., 2002b,Changes in the use and management of forests for abating carbon emissions: issues

and challenges under the Kyoto Protocol. Phil. Trans. R. Soc. Lond. The Royal

Society, 360, pp. 113.

BROWN, S., HALL, M., RUIZ, F., FLAMENCO, A., DEJONG, B., AUKLAND, L.,

MASERA, O. and SHOCH, D., 2002c, http://www.winrock.org/general/Publications/

LandUseAndForestryProjects.pdf.

CLARKE, K.C., 1998, Loose-coupling a cellular automaton model and GIS: long-term urban

growth prediction for San Francisco and Washington/Baltimore. International

Journal of GIS: 12, 17, pp. 699714.

CLARKE, K.C., HOPPEN, S. and GAYDOS, L.J., 1996, A self-modifying cellular automaton

model of historical urbanization in the San Francisco Bay area. Environment andPlanning B, 24, pp. 247261.

COSTANZA, R., 1989, Model goodness of fit: a multiple resolution procedure. Ecological

Modelling, 47, pp. 199215.



24/24

FOODY, G., 2002, Status of land cover classification accuracy assessment. Remote Sensing of

Environment, 80, pp. 185201.

FOODY, G. and ATKINSON, P., (Eds.)., 2002, Uncertainty in Remote Sensing and GIS. (West

Sussex, England: John Wiley & Sons), 307 pages.

HAGEN, A., 2002, Multi-method assessment of map similarity. 5th AGILE Conference on

Geographic Information Science, Palma, Spain. April 2527; 171182.HAGEN, A., 2003, Fuzzy set approach to assessing similarity of categorical maps. International

Journal of Geographic Information Science, 17 (3): pp. 235249.

HALL, C.A.S. (Ed.)., 2000, Quantifying Sustainable Development: The Future of Tropical

Economies. (San Diego, CA: Academic Press), 761 pages.

HALL, C.A.S., TIAN, H., QI, Y., PONTIUS, G. and CORNELL, J., 1995a, Modelling spatial and

temporal patterns of tropical land use change. Journal of Biogeography, 22 (4/5):

pp. 753757.

HALL, C.A.S., TIAN, H., QI, Y., PONTIUS, G., CORNELL, J. and UHLIG, J., 1995b, Spatially

Explicit Models of Land Use Change and Their Application to the Tropics, Carbon

Dioxide Information Analysis Center, Oak Ridge National Laboratory, DOE

Research Summary 31, February.

MARKOV, A.A., 1907, Extension of the limit theorems of probability theory to a sum of

variables connected in a chain. The Notes of the Imperial Academy of Sciences of St.

Petersburg, VIII Series, Physio-Mathematical College XXII.

MERTENS, B. and LAMBIN, E.F., 2000, Land-Cover-Change Trajectories in Southern

Cameroon. Annals of the Association of American Geographers, 90 (3): pp. 467494.

ORESKES, N., SHRADER - FRECHETTE, K. and BELITZ, K., 1994, Verification, Validation, and

Confirmation of Numerical Models in the Earth. Science, 263 (5147): pp. 641646.

PONTIUS JR, R.G., 1994, Modeling tropical land use change and assessing policies to reduce

carbon dioxide release from Africa. Graduate Program in Environmental Science,

Syracuse: SUNY-ESF, pp. 177.

PONTIUS JR, R.G., 2000, Quantification error versus location error in comparison of

categorical maps. Photogrammetric Engineering & Remote Sensing, 66 (8):pp. 10111016.

PONTIUS JR, R.G., 2002, Statistical methods to partition effects of quantity and location

during comparison of categorical maps at multiple resolutions. Photogrammetric

Engineering & Remote Sensing, 68 (10): pp. 10411049.

PONTIUS JR, R.G., CORNELL, J.D. and HALL, C.A.S., 2001, Modeling the spatial pattern of

land-use change with GEOMOD2: application and validation for Costa Rica.

Agriculture, Ecosystems & Environment, pp. 191203.

PONTIUS JR, R.G. and BATCHU, K., 2003, Using the Relative Operating Characteristic to

quantify certainty in prediction of location of land cover change in India. Transactions

in GIS, 7 (4): pp. 467484.

PONTIUS JR, R.G. and PACHECO, P., 2003, Calibration and validation of a model of forestdisturbance in the Western Ghats, India 19201990. GeoJournal, in press.

PONTIUS JR, R.G. and SUEDMEYER, B., 2003, Components of agreement between categorical

maps at multiple resolutions. In R.S. Lunetta and J.G. Lyon, (Eds). Remote Sensing

and GIS Accuracy Assessment. (Boca Raton, FL: CRC Press).

RYKIEL, E.J., 1996, Testing ecological models: the meaning of validation. Ecological

Modeling, 90, pp. 229244.

SCHNEIDER, L.C. and PONTIUS JR, R.G., 2001, Modeling land-use change in the Ipswich

watershed, Massachusetts, USA. Agriculture, Ecosystems & Environment, 85,

pp. 8394.


Documents

Pontius y Malason (2005).pdf