An Expert System for Remote Sensing

IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING, VOL. GE-25, NO. 3, MAY 1987

An Expert System for Remote Sensing

DAVID G. GOODENOUGH, MEMBER, IEEE, MORRIS GOLDBERG, MEMBER, IEEE, GORDON PLUNKETT,AND JOHN ZELEK, MEMBER, IEEE

Abstract-The Canada Centre for Remote Sensing has developed twohierarchical expert systems, the Analyst Advisor and the Map ImageCongruency Evaluation (MICE) advisor. These expert systems are builtupon our Remote-Sensing Shell (RESHELL) written in Logicware'sMPROLOG. A shell is a programming environment that specificallycaters to expert system development. Knowledge is represented in theproduction rules and frames database. Numerical processing takesplace using the extensive FORTRAN code of the Landsat Digital ImageAnalysis System (LDIAS). The LDIAS includes several DEC VAX com-puters, image displays, specialized processors, and DEC Al VAXsta-tions.

The paper describes the architecture of the expert system to com-pare maps and images (MICE) and the expert system to advise on theextraction of resource information from remotely sensed data, the An-alyst Advisor. Details are given concerning the structure of RESHELLand our methods of interfacing symbolic reasoning in PROLOG on theAl VAX stations with numeric processing in FORTRAN on several dif-ferent computers. The first prototype of the Analyst Advisor will bereleased for internal use at CCRS in March 1987.

I. INTRODUCTIONA. HistoricalSINCE 1973, the Canada Centre for Remote Sensing(CCRS) has been conducting research and develop-

ment into information extraction methods and systems.This work was conducted initially on large general-pur-pose computers that were too slow to permit rapid anal-ysis of remotely sensed imagery.

In 1982, we commenced the development of the Land-sat Digital Image Analysis System (LDIAS). This sys-tem, to be completed in 1988, will support the analysisof a full Thematic Mapper scene into 32 classes in 8 hwhile enabling the integration of map-based data. TheLDIAS contains over 1 000 000 lines of FORTRAN-77code. Systems for remote sensing analysis are usuallycomplex. In the case of the LDIAS, there are three VAXcomputers, three Al Vaxstations, four image displays, twomap displays, and several special processors. A humananalyst is an expert on the operations and functionality ofthe LDIAS. There are few LDIAS experts, and it is ben-eficial to record and be able to distribute this knowledgeof the LDIAS operation. An expert system is a method ofcloning this expertise into a computer. To simplify the

Manuscript received October 23, 1986; revised January 6, 1987.D. G. Goodenough and G. Plunkett are with the Department of En-

ergy, Mines, and Resources, Canada Centre for Remote Sensing, Ottawa,Ontario, Canada KlA 0Y7.

M. Goldberg is with the University of Ottawa, Ontario, Canada.J. Zelek is with Intera Technologies Ltd., Ottawa, Ontario, Canada K1Z

8R9.

operation of the LDIAS, we decided to incorporate expertsystems technology and make several expert system "ad-visors." This paper describes our approach to integratingthe FORTRAN-77 code with expert systems written inPROLOG. We have made an Analyst Advisor to guide auser during the analysis of Landsat MSS and TM im-agery. We have also made a Map Image CongruencyEvaluation (MICE) advisor. Both of these advisors utilizea shell developed at CCRS called the Remote Sensing Ex-pert System Shell (RESHELL).

Research in image analysis and interpretation by Brookset al. [1], Hanson and Riseman [2], and Levine and Hong[3] have shown that visual perception can be performedusing rule-based expert systems and that reasonable re-sults can be achieved. Additional examples can be foundin medical image processing, using temporal images suchas the evaluation of heart motion using X-ray image se-quences by Tsotsos [4] and the automatic segmentation ofcoronary vessels by Stansfield [5].

Rule-based machine perception in remote sensing hasbeen investigated by using ancillary or map information.Glicksman [6] and Mackworth researched the use of mul-tiple information sources for image understanding in theMISSEE system. McKeown [7], and McKeown et al. [8]performed map-assisted photointerpretation in the MAPS/SPAM systems. Plunkett et al. [9] examined the spatialcongruency of maps and Landsat images in the MICE sys-tem.The implementation of expert systems is performed

most easily by the use of a shell, which is instantiatedwith domain knowledge, to provide some expertise in aparticular domain. CCRS has developed a shell with theUniversity of Ottawa. This shell is being instantiated toperform various knowledge-based functions related to theprocessing of images and the integration of geographicinformation systems.

B. RESHELL-Remote Sensing Expert System ShellRESHELL [10] is an expert system shell written in

Logicware's version of PROLOG, known as MPROLOG.PROLOG is a computer language that is used for solvingproblems that involve objects and relationships betweenobjects. Computer programming in PROLOG consists ofthe following properties: declaring some facts about ob-jects and their relationships; defining some rules aboutobjects and their relationships; asking questions about ob-jects and their relationships.

Implementing a hierarchical expert system using RE-

0196-2892/87/0500-0349$01.00 © 1987 Canadian Crown Copyright

349


BUILT- IN ARBITRPTORPROCEDURES

Fig. 1. Architecture of reshell.

SHELL requires a partitioning of shared software, andsoftware that is local to instantiations of experts within a

completed system. The primary MPROLOG softwarestructuring mechanism is the module, which translates toa RESHELL-based system being composed of modulescontaining shared (RESHELL core) and nonshared code(code local to some expert).The architecture of RESHELL is given in Fig. 1. The

Analyst Advisor (see Fig. 5 later) is a collection of in-stantiated experts, each built using RESHELL and knowl-edge from an expert. RESHELL is very modular in thesense that it sustains development of a multi-expert sys-tem, with a number of individual experts organized hier-archically utilizing blackboards for a communication me-

dium. Each expert in the hierarchy is responsible to a

single manager, so that control and communication flowsbetween different levels of the hierarchy rather than across

a level. The highest level expert sets broad goals for thenext level of command. The lowest level corresponds tothe image processing algorithms coded in FORTRAN.The two major parts of an expert system are the knowl-edge base and the inference engine. The knowledge baseconsists of facts and rules. Facts are the basic informationregarding a problem whereas rules are applied in solvinga problem. Rules are in the form of production rules. ("ifCONDITION then CONCLUSION.") The inference en-

gine is the scheduler or control mechanism. There are twocontrol strategies that can be followed, namely, bottom-up (backward-chaining) and top-down (forward-chain-ing). Backward chaining refers to the conclusion beingtrue and finding rules leading to the conditions being true,working backward from the final goal to the initial state.In forward chaining, the condition is assumed true and theconclusions deduced that can be used in other rules lead-ing to the final goal.

Other features present in RESHELL include a knowl-edge acquisition system, a justifier to explain the reason-

ing of a program, methods for treating uncertainty inknowledge and data, and a semantic relationships or

frames database, as well as an external communicationsmethod. The frames database stores data in a manner that

preserves descriptive semantic relationships between ob-jects. The interface to the FORTRAN image processingalgorithms consists of an expert, called the LDIAS TaskInterface (LTI), that is responsible for acquiring all thenecessary knowledge to execute the FORTRAN code inbatch mode, when activated to do so by a higher levelexpert. There are two levels of rules-meta rules and ob-ject rules. Object rules are inference rules that manipulateobjects to deduce or prove goals. The object rule inter-preter makes inferences using object rules. Meta rulescontrol the meta level interpreter to select the best appro-priate path for the next stage of a solution by manipulatingsets of object rules.The blackboard stores goals from high-level experts,

object values, and the agenda created by the meta-ruleinterpreter. The blackboard consists of six partitions: goalpartition, agenda partition, database partition, results-by-strategy partition, intermediate results partition, and thefinal results partition. The goal partition contains the goalsfrom the higher level experts. The agenda partition storesthe agenda created by the meta-level interpreter. The da-tabase partition contains the initialization data, data fromhigher level experts, and the frames database knowledge.The results-by-strategy partition is divided into subsec-tions. Where there are different strategies to accomplish agoal, the results would be stored in different subsections.The initialization deduction goals are also stored in theresults-by-strategy partition. The intermediate results par-tition is also divided into subsections. Intermediate resultscoming from the same strategy are stored in the same sub-section. The intermediate results partition also containsany data from a lower level expert. The final results par-tition contains the final results after the process of arbi-tration.The responsibilities of the scheduler include: receiving

messages from the I/O interface; invoking the meta-ruleinterpreter; evaluating action procedures in the agenda ofthe blackboard; calling built-in procedures when re-quired.The data interface is used for communication among

different experts at different levels of the hierarchy. Thearbitrator resolves conflicting results at the completion ofone or more strategies.Each object rule and value is assigned a measure of be-

lief and a measure of disbelief. The measure of belief re-fers to the total amount of confidence that the element istrue or valid. The measure of disbelief represents the totalamount of doubt or uncertainty that the system has aboutthe element.

II. COMPARING MAPS AND IMAGESA. The Map/Image Congruency Problem

In the map-making/updating process, photointerpreterstypically analyze aerial photographs, decide on the clas-sification of the different objects in the photograph, andthen transcribe the classification and location of these ob-jects onto a map or directly into a geographic informationsystem (GIS) [11]. The map is only an approximation to

350

GOODENOUGH et al.: EXPERT SYSTEM FOR REMOTE SENSING

(a)

Fig. 2. (a) A map/image overlay depicting river mismatch. This LandsatMSS image overlayed with the B.C. forest cover map (hydrology level)indicates the image land pixels that are in the river (area A) and the imagewater pixels that are outside the map river (area B). (b) This LandsatMSS image overlayed with the B.C. forest cover map (hydrology level)indicates a spatial mismatch of the map lake and the image lake (areaC). (Fig. 2 continued on page 352.)

the real world; furthermore, this map-making procedureis prone to human error. A further complication is that theworld land mass is a constantly changing entity. For ex-ample, rivers meander, forests bum or are cut, and hous-ing subdivisions and roads are built. Cartographic data onthe other hand, are relatively static and are only updatedperiodically to reflect the changing world.

For some time, the remote-sensing community has beenextolling the virtues of the integration of remote sensingdata with GIS data bases [12]. This data-integration prob-lem has been researched for forestry and some solutionsdeveloped, which are used operationally by the BritishColumbia Ministry of Forests [13]. However, the auto-matic integration of remote-sensing data with geographicinformation systems is not yet possible as human inter-pretation and assistance are still required.

The integration problem occurs when some area in aremote-sensing image, which is usually depicted as apolygon, is to be placed in a GIS. If the image polygonis placed directly into the GIS data base, then data-basecorruption will occur, if there is not perfect spatial jux-taposition of the image polygon and the neighboring mappolygons. The current method of performing this integra-tion is for a human operator to move the image polygoninto the best fitting location in the map and then place itin the GIS data base.

Examples depicting the map/image misregistrationproblem are given in Fig. 2(a) and (b). This figure depictsa geocoded Landsat MSS image overlayed with the hy-drography level of the corresponding map. Area A of Fig.2(a) contains pixels that are clearly land in the image, butare categorized as river in the map, whereas in area B, the

351


(b)

Fig. 2. (Continued.)

opposite phenomenon has occurred: pixels that corre-spond to water in the image are categorized as land in themap. Fig. 2(b) is a second example, where the lake in themap appears to be in a slightly different location than thelake in the image (area C).

B. The Photointerpreter ApproachIn dealing with map/image misregistration, an expert

photointerpreter would be called on to make many judg-ment calls. Both formal and heuristic rules would be em-ployed in making decisions on how to reconcile the twodata sources and finally on how to update the map. Anexpert system paradigm seems to be an appropriate frame-work for attacking this complex problem. The Map/Im-

age Congruency Evaluation (MICE) knowledge-basedsystem (KBS) employs such a paradigm.A question that must be answered before selecting a

strategy is: How does one know that the map and the im-age match? A human would look at the map and then atthe image, find corresponding structures, measure theirposition from some datum, and then report congruency ordiscrepancy for the structure. The strategy selected forthis rule-based system is not unlike the human approach.The MICE system performs the same basic operations.These are: 1) to preprocess the map and the image to thesame spatial datum and symbolic representation; 2) to lo-cate corresponding structures (segments); and 3) to reporton the spatial congruency of the corresponding structures.

There are many types of spatial incongruency. The in-

352


congruencies can be random or systematic, local or global,and large or small. The current strategy is simply to reportthe results and not to try to automatically fix the misre-gistrations. The evaluation is left to the human user.

C. Symbolic Representation ofMaps and ImagesOne of the first questions that had to be addressed in

the design of MICE was how to represent the two dispar-ate data types, map and image data, in a knowledge-basedsystem. This is the iconic to symbolic gap problem thatis being researched for machine perception systems [14].Map data containing dots, lines, and areas are usually de-fined in a coordinate reference system in terms of points,vectors, and polygons. Image data, on the other hand, isstored using a spatially indexed technique called raster orgrid format. It would be extremely unwieldy to attempt tostore and process this data in its native form in a knowl-edge-based system, as the format, data type, and resolu-tion of the data are different. Also, the system does notnecessarily make decisions based on the data, but ratheron various attributes derived from the data. Thus, the uni-form method of data representation selected was to pre-

process the map and image data into segments and to gen-

erate various symbolic segment attributes that can then beused by the rule-based stage.The next question that needs to be answered is: what

attributes of the data are required for congruency evalu-ation? This question is also not easy to answer becausethe image has spectral attributes that are not available inthe map data. The image segments' spectral attributes arerequired, so that the image segments can be spectrallyclassified. The map and image spatial attributes can becalculated relative to the same reference grid, so that theattribute values of the map and image can be comparedand manipulated in a symbolic fashion.The spatial attributes selected for use by MICE, that

are common to both the map and the image, are as fol-lows. (Note that both the map and the image have beenprocessed to the same spatial resolution and thus can becompared on a pixel basis. Note also that this means thatthe attributes are pixel size invariant. Thus, the pixels canbe any size as long as the map data and image data arepreprocessed to the same resolution).

1) Location-The location attribute represents the lo-cation of the pixels in the map or image segment, basedon some reference grid. The value of the location attributeis a list of three-tuple lists that uniquely identify the lo-cation or position of the segment on a line by line basis.The three-tuple list contains: a) the line number contain-ing the pixels; b) the start pixel number; and c) the endpixel number.A list of these three-tuples containing run-length en-

coded line locations, will thus define the location of thesegment. A sample segment with its corresponding loca-tion attribute is depicted in Fig. 3.

2) Size-the size attribute represents the size of thesegment. The value of the size attribute is the total num-

PXL 1LINE 1

(PIXE 8, LMIE 5)

..1 2 u<- WIINDOW' LIMITS'. 3 4 X' \ DT

5 6 7 8 USQUARE PIXELS

9 10 11 12 1 50 MT-RES X 50 MTRES

(PIXEL 10, LINE 8)

LOCATION = [[LItME, START PIX, END PIX].... ]= [[5,8,9],[6,8,9],[7,7,10], [8,7,10]]

SHAPE = (PERIMETER ** 2) / AREA= ((16 * 50) ** 2) / (50 * 50 * 12)= 21.3

SIZE = PIXEL COUNT= 12

WINDOW = [UPPER LETr LINE, UPPER LEFT PIXEL,LOWEER RIGHT LINE, LOWER RIGIH PIXL

= [5, 7, 8, 10 ]

Fig. 3. The spatial attributes of a segment.

ber of pixels in the segment. A sample segment with itscorresponding size attribute is depicted in Fig. 3.

3) Shape-the shape attribute represents the shape ofthe segment. The value of the shape attribute is the perim-eter squared divided by the area. This shape attribute is afairly primitive representation of the shape, but additionalshape attributes can easily be added later. A sample seg-ment with its corresponding shape attribute is depicted inFig. 3.

4) Window-the window attribute represents thesmallest rectangle that can be placed around the entiresegment (bounding rectangle). The value of the windowattribute is a list of four elements that represent the lineand pixel locations of the upper left corner of the window,and the lower right corner. A sample segment with itscorresponding window attribute is depicted in Fig. 3.The image spectral attributes that were selected for

MICE processing are as follows:1) MEAN_CH-X-This attribute equals the mean

grey level value of all the pixels in the segment for chan-nel X of the satellite image.

2) MAX-CH-X-This attribute equals the maxi-mum pixel grey level value in the segment for channel Xof the satellite image.

3) MIN-CH-X-This attribute equals the minimumpixel grey level value in the segment for channel X of thesatellite image.These spectral values may be used to evaluate the spec-

tral classification of the segment corresponding to the at-tribute values. These spectral attributes are by no meansan exhaustive list for classification determination, but theydo provide a basis upon which other attributes can beadded. A simple classification rule is as follows:

if MEAN_CH_4 > MEAN-CH2and MEAN-CH-4 > MEAN_CHA1then CLASS = LAND-COVER.

353


MAP/IMAGECONGRUENCYEVALUATIOt

DETERMINE TIlE DETERMINE THE

NEXT REST NEXT BESTCLASS EXPErT CATEGORY EXPERT

Fig. 4. Hierarchical organization of the MICE KBS.

D. Expert System StructureKnowledge for the MICE KBS is coded in the form of

meta-rules, object-rules, and object values. The rules andvalues are stored in knowledge base files that are inputwhen an instantiated RESHELL expert is invoked. TheMICE KBS uses meta-rules and object-rules that are inputfrom the knowledge base (long-term memory). The objectvalues required for MICE processing are read in from thesymbolic map and image files (short-term memory).The role of the meta-rules is to establish the general

procedure or strategy that is to be applied. The object-level rules, on the other hand, contain the mundane de-tails of the strategy that is being applied. As an example,there could be a meta-rule that states that the first actionis to read in some file. At the object-level, the correspond-ing rules could indicate which portions of the file shouldbe used.RESHELL supports a hierarchy of intercommunicating

expert systems. The advantage of this approach is that theproblem can be decomposed into manageable portions,with limited interaction. For the map/image congruencyevaluation expert, the following decomposition, shown inFig. 4, was chosen. The role of each subexpert is as fol-lows. 1) The map/image congruency evaluation expert isthe high-level expert controlling the input, processing, andoutput of the KBS. 2) The next best class expert returnsthe next best class selected for congruency evaluation. 3)The next best category expert returns the next best cate-gory of the current class, selected for congruency evalu-ation.The knowledge base required by the RESHELL archi-

tecture is such that one expert does not have access to therules in another expert's knowledge base. In other words,the knowledge in the form of rules for each of the threeMICE experts is separate and distinct.Both the best class expert and the best category expert

contain meta-rules and object-rules that define the nextbest category, based on the current class and category.The class and categories were derived from the CanadaCouncil on Surveys and Mapping list of categories ofclasses. The categories of the various classes are rankedas indicated in Table I, where 1 represents the highestranking and 7 the lowest.MICE has been tested comparing the hydrography lev-

els of a 1: 20 000 scale British Columbia Ministry of For-ests forest cover map with the water features found in a

Landsat MSS image (240 x 275 pixels). For this test,there were 25 meta-rules in the MICE expert, 20 object-rules in the class expert, and 30 object-rules in the cate-

TABLE ICLASS CATEGORY RANKING FOR CONGRUENCY EVALUATION

CLASS CATEGORY RANIUNIG

IYDIO3GRAPIIY COASTAL FEATURE 1

INLAND TATER BODY 2

W4ATER COURSE ASSOC. FEATURE 3

GROUND WATr FEATURE 4

WETLAND 5

RELATED HYDROGRAPHIC FEATURE 6

PERMANENTLY FROZEN FEATURE 7

ROAD AND RAIL ROADWAY 1

TEROUGH RAIL LITE 2

UTILITY UTILITY 1

LAND COVER WOODIAND 1

ARABLE CULTIVATED LAND 2

GRASSLAND 3

ICU VEGETATIDS 4

1NO VEGETATICLJ 5

HYPSOGRAPHY HYPSOGRAPY 1

STRUCIURES STRUCTURES 1

BUILDIN4GS BUILDINGS 1

DESIGNIATED AREAS DESIGATED AREAS1

DELIMITERS DELIMlITERS 1

TEXT TEXr 1

gory expert. The rules used are complex with multipleconditions and actions. It takes approximately 3 h on aDEC Al VAXstation to complete this comparison. Theresults were similar to those that would be obtained by aphotointerpreter, in that MICE identified the samematches and mismatches between the image and the map.The human comparison of the map and the image wasperformed in less than 1 h. The main rules coded intoMICE were obtained through discussions with photoin-terpreters.

III. ANALYST ADVISOR FOR INTERPRETATIONA. Problem DescriptionCCRS is presently developing the Landsat Digital Im-

age Analysis System (LDIAS) for the analysis of remotelysensed imagery. The LDIAS provides many algorithmicimage processing techniques for the analysis of satelliteand airborne imagery. An image analysis session on theLDIAS will require the use of remotely sensed imagery,and may require a geocoded database, contextual infor-mation, and image processing tools. Problems arise fromthe difficulty of integration of remote sensing data withgeocoded databases [12] and with the difficulty of han-dling contextual information.The LDIAS hardware system is a complicated inte-

grated system of computer technology. The LDIAS hard-ware includes a VAX 1/785, a VAX 1 1/780, a VAX 11/730, additional specialized processors (a Star TechnologyST-100 array processor, an Intergraph Graphics processor

354


* LTI LOIAS TASK INTERFACE

Fig. 5. Analyst advisor expert system.

and a Canadian Astronautics Limited parallelepiped pro-cessor), two Intergraph map display workstations, andfour image displays (two DIPIX Aries-2 and two GouldDeanza 8500's). All the LDIAS host computers are linkedtogether via an Ethernet network. To assist in artificialintelligence research and development at CCRS, threeDEC Al VAXstations were acquired and connected to theexisting LDIAS network.The LDIAS software has more than a million lines of

highly structured FORTRAN code currently installed. Thecomplexity of the issues addressed by the LDIAS soft-ware is evident by the menu system that drives the soft-ware. There are more than 20 menus having typicallymore than 15 choices. Each choice may lead to lower levelmenus. Thus, a problem arises where the user is per-plexed, on how to utilize all this software and hardwarecapability to their advantage, with respect to the remotesensing application problem at hand. To ease the user'sburden, we provide an assistant who is an expert onLDIAS software and image analysis methods. By devel-oping an expert system to fulfill the functions of the hu-man analyst, we hope to be able to distribute and preservethis expertise.

B. Analyst AdvisorThe Analyst Advisor is an expert system that controls

the LDIAS in order to achieve the user's goal of obtainingresource information. The Analyst Advisor is intended toperform the task of the human image analyst in a user'sapplication session. The analyst is part of a team of two

(analyst plus user) who operates the LDIAS system for aspecific application. The image analyst is knowledgeablein the hardware and software procedures, and thus assistsa user, such as a forester, who is an expert with respectto the goals to be achieved, the accuracies expected, andthe relevant contextual information.The Analyst Advisor expert system (Fig. 5) is hierar-

chical in nature, where the lower level goals are satisfiedby traveling down the tree and transferring control of thespecific task to lower level experts. The structure is thusa hierarchy of various experts that report directly to or viaother higher level experts to the Analyst Advisor. Thehighest level expert and this collection of experts are re-ferred to as the Analyst Advisor.The goal of the Analyst Advisor, which is to extract

resource information from remotely sensed data, can bebroken down into subgoals, such as: 1) determine fromthe user, the application and the desired output productsand accuracies; 2) obtain desired imagery and/or map dataand transfer to disk; 3) analyze data utilizing multilevelexperts; 4) assess the accuracies achieved in meeting theuser's goals and select alternative strategies if necessary;5) produce output products. Presently the Analyst Ad-visor performs these functions. The lowest level expert isthe LDIAS task interface (LTI) expert, which is a methodfor executing an LDIAS FORTRAN program in batchmode utilizing the knowledge acquired by the AnalystAdvisor. Thus, we separate symbolic reasoning from nu-merical processing while preserving our investment in ourexisting image analysis code.

355


RESULTS \1 \1z

LDIAS LDIASTASK , ENVIRONMENT

INfS

Fig. 6. LDIAS task interface general structure.

The Analyst Advisor was developed using RESHELL.RESHELL was designed to allow the implementation ofan expert system that is hierarchical in nature, containingmany experts that communicate with each other. The hi-erarchical approach allowed us to simplify the knowledgeacquisition, the generation of rules, and the maintenanceof the Analyst Advisor.

C. Knowledge Representation in the Analyst Advisor

The sources of knowledge in the Analyst Advisor are

as follows: LDIAS FORTRAN programs, proceduraldeclarative rules, contextual information, and accuracyassessment methods. The LDIAS programs are proce-dural in nature and coded into a hierarchy of FORTRANsubroutines. The LDIAS software holds the basic math-ematical tools for retrieving, manipulating, and storingremote-sensing imagery, map, and contextual data. Tominimize development costs, one would like to utilize thisexisting procedural knowledge coded in FORTRAN to in-teract with the expert system. The approach to this prob-lem was to design an LDIAS Task Interface (LTI). TheLTI provides a remote-sensing expert system with the ca-

pability of calling on LDIAS tasks (in batch mode) as low-level information sources. These capabilities are estab-lished without altering the software for the LDIAS tasks.The LTI interfaces an expert designed in the RESHELLenvironment to a FORTRAN LDIAS task. The LTI pri-marily consists of an operator module, and analyst mod-ule, and a manager module. The LTI captures the promptsof the LDIAS interactive programs in a prompt-descrip-tion form. From the prompt-description form, a set ofPROLOG facts and relationships to describe the promptoperations of the program are generated.To give more insight into the operation of the LDIAS

task interface, the following describes the flow of controland data in Fig. 6.

1) The Manager module formulates a problem to be

solved by a LDIAS task, say task "i ". The problem isformulated by the rules of the manager module that aregiven by the context of the invocation. In doing so, itproduces a set of input context to be incorporated into thebatch command file for task "i ". The input context andthe "LDIAS task identifier" for task "i" are placed inthe Manager's module blackboard.

2) The Manager module initiates the Operator module.The Manager module then waits until it has been signaledby the Operator module that either the task is complete ora failure occurred.

3) The Operator module retrieves the LDIAS task iden-tifier and the input context from the Manager's black-board.

4) The Operator module consults the LDIAS taskprompt knowledge base for knowledge on task "i's" in-put prompt requirements.

5) The Operator module composes a batch commandfile using its knowledge and the input provided by theManager module.

6) The Operator module returns to the Manager'sblackboard any new (output) context that resulted fromthe command composition step. It will also return a fail-ure status to the Manager module if it could not composethe batch command file.

7) The Operator module spawns (initiates) the LDIAStask and provides it with the batch command file.

8) The LDIAS task executes until completion; I/O withthe LDIAS environment is performed as necessary.

9) The Operator module retrieves the LDIAS task com-pletion status from the LDIAS environment and returnsthe status of the task execution to the Manager module.

10) The Manager module formats a question to beposed to the Analyst module and places it in the Manag-er's blackboard.

11) The Manager module initiates the Analyst module.12) The Analyst module retrieves the question and in-

put context from the Manager's module blackboard.13) The Analyst module consults the analyst knowl-

edge base for the knowledge required to answer the Man-ager's module question.

14) The Analyst module retrieves from the LDIAS en-vironment any results it needs in order to answer the Man-ager module's question.

15) The Analyst module generates an answer (includ-ing status) and stores it in the Manager's module black-board.

16) The Manager module proceeds as is required bythe Analyst's response.

Operator knowledge in an LDIAS task primarily con-sists of sequence knowledge and context knowledge. Se-quence knowledge is the knowledge that contains the in-formation on how the software chooses the next interactiveprompt from the present prompt. Context knowledge con-sists of the mapping of context (what is entered at a givenprompt) into the interactive prompts. It should be notedthat one of the features of the LDIAS software is the uti-lization of a man-machine-interface library (MMILIB) for

356


all LDIAS programs. Two features of MMILIB are theability to record the responses of a user during an inter-active session and the ability to execute a program in batchcommand format. The context and sequence knowledgeis elicited from a human LDIAS task expert in an inter-active question/answer format using the Interactive Op-erator Acquisition Module or Interrogator. The Interro-gator generates the necessary knowledge in the form ofPROLOG code.The context-dependent knowledge is determined by

queries issued to the user by the Analyst Advisor. Thecontext dependent knowledge can also be deduced from aset of known facts and other information obtained fromthe user. This knowledge can be passed to an LTI forexecution of an LDIAS task and/or used for deducing theoverall goal of the expert at hand.The procedural and declarative rules are the knowledge

that is coded into production rules ("if-then" rules) forthe various experts. At the top level of the Analyst Ad-visor, the production rules are very dependent on eachother and, therefore, very procedural in nature. Proce-dural rules evolve when there is a concise theory and de-pendent subprocesses. RESHELL, on the other hand, ca-ters to production rules that are declarative in nature. Thusa need arises to embed procedural rules (dependent statecharacteristics) into an environment that has independentstates; namely, a declarative representation. A declarativerepresentation is one in which nothing is said by theserules on how to use the facts at hand. A declarative rep-resentation does not give the programmer the power tospecify the order in which the rules will be executed:RESHELL's control strategy handles the ordering. An in-teresting feature of RESHELL is the meta-level rules. Themeta-level corresponds to knowledge about knowledge.The meta-rules control the execution of the objectrules. Thus, procedural knowledge can be coded intoRESHELL by using a step-variable (the phase) that tracescontrol through the various procedural states of the meta-rules. The step-variable is analogous to a state variablecontrol mechanism: the next transition state is determinedby the results of the present state. It should be noted thatthe step-variable is unlike a PROLOG variable, but shouldbe considered as a symbol that has the present state as-signed as a value.The Analyst Advisor virtually becomes the controller

of the LDIAS software. The LDIAS software encom-passes more than 300 applications-related programs,which makes it very costly to address the accuracy as-sessment of the results from each individual program. Theapproach taken from the accuracy assessment perspectiveis to do a detailed study of a single processing flow, suchas a typical classification study. Presently, all applica-tions supported by the Analyst Advisor include a classi-fication study. The Analyst Advisor incorporates the ac-curacy assessment software by means of the LTI. Theaccuracy assessment software allows the estimation of thecumulative effect of all the errors on the final classifica-tion result. Accuracy assessment also prepares an error

budget to specify the acceptable error tolerance of eachprocessing step in order to meet a specified final accuracy.After the accuracy assessment software has been run, onecan determine how reliable the classification results are.This reliability measure is then transformed into a cer-tainty factor that gives an indication of the measure ofbelief and disbelief of the output knowledge.

D. Implementation ProblemsThere have been several implementation problems in

developing the Analyst Advisor. One of the major stum-bling blocks has been the network control of devices onvarious central processing units (CPU's) other than thatof the expert system. Some devices such as tape drivesand visual display processors are only connected to cer-tain CPU's and certain LDIAS tasks require these de-vices. If an LDIAS task has been spawned from the An-alyst Advisor, it is possible to make this spawning processinvisible to the user if the LDIAS task is to run under thesame controller CPU. However, if the LDIAS task re-quires one or more devices located on another CPU, thisnecessitates spawning a task across the network. Spawn-ing across the network to activate an LDIAS task is pres-ently awkward and slow. Knowledge transfer across thenetwork is also cumbersome. The Analyst Advisor re-quires the output knowledge from the LDIAS task in orderto continue with the analysis session. Also, initiating anLDIAS task requires the input knowledge from anotherCPU in the network to be activated. Various experimentsare now being conducted to determine the best solution tonetwork control involving a separation between numericand symbolic processing. We do have an inefficientmethod operational now.

Other issues being addressed include the handling of apause functionality for RESHELL and the handling of ex-port system explanations in the multiple expert environ-ment. The pause function for an expert system is the ca-pability to temporarily stop the execution of the expertsystem at a particular location, in order to resume exe-cution of the expert system from that same location at alater date. Expert system explanations are a method ofexplaining the reasoning and deductions to the user (if sodesired) that led to the results deduced by the expert sys-tem. The pause function is not only an appealing featurebut also a necessary one. In an Analyst Advisor expertsystem session with a user, there will be times when theuser will not be able to complete his session that day andwill be required to continue at a later date. The user wouldactivate the pause function, which would store the presentstate of the reasoning of the Analyst Advisor and all of itssubexperts, so that when the session is resumed, the An-alyst Advisor would be reinitialized with the deductionsand new knowledge acquired in the previous session. Thepause function capability is going to be implemented atthe level of the expert system environment-RESHELL.Implementing the pause function at the RESHELL leveleliminates the awkward rule structure that would be nec-

357


essary if the pause control was implemented at the levelof the Analyst Advisor.

E. Example Dialogue with Analyst AdvisorEnter user's name:>John SmithThis is the ANALYST ADVISOR.I have limited knowledge about some aspects of the anal-ysis of remotely sensed data. My knowledge is aimed atsupporting the satellite production products of the CanadaCentre for Remote Sensing and the analysis of these prod-ucts on the Landsat Digital Image Analysis System. Theproducts correspond to the landmasses of Canada.

Satellite sensors about which I know are:

Landsat Thematic Mapper (TM)Landsat Multispectral Scanner (MSS)

Note: There is no existing Personal file for John Smith.As a new user, you will have to select the infor-mation required to initiate a session. You will bequeried for the information required.

Enter the project name:>Insect DamageSelect an application area.Enter one of forestry, agriculture, land use, hydrology,geology, user-specified>forestrySelect the region of interest.Enter at least one of British Columbia, Alberta, Sas-katchewan, Manitoba, Ontario, Quebec, New Bruns-wick, Newfoundland, Prince Edward Island, NovaScotia, Yukon, Northwest Territories>British ColumbiaWhat output products would you like to produce?Enter at least one of maps, photographic enhance-ments, area summaries, tapes,statistical tables, geographic information files>maps, geographic information filesWhat is the average percentage of correct classificationyou hope to achieve?Enter a value between 1 and 100{1}>83What is the minimum percentage of correct classsificationyou hope to achieve?Enter a value between 0 and 83{1}>70.5Do you require an existing map?Enter one of yes no>yes

IV. CONCLUSIONS AND RECOMMENDATIONSIt has been shown that the field of remote sensing is a

valid application area for the technology of knowledge-based systems (expert systems). Some of the problems thatexisted before an expert system approach was initiated

could not have been solved as easily with other methods.The expert system development at CCRS is done utilizingan expert system shell (RESHELL) written inMPROLOG.The Map/Image Congruency Evaluation (MICE)

knowledge-based system was developed to study the spa-tial differences between maps and images. The MICE sys-tem addresses the problem of integrating remote sensingdata with geographic information system (GIS) data.MICE was tested comparing a LANDSAT MSS imagewith the hydrography level of a 1: 20 000 scale map andgave results similar to those acheived by a photointer-preter.The Analyst Advisor expert system advises a user on

how to utilize the existing LDIAS hardware and softwarein order to obtain desired resource information. In devel-oping the Analyst Advisor, some key issues were ad-dressed. The LDIAS Task Interface is a method of incor-porating existing FORTRAN coded software asprocedural knowledge into the expert system environmentframework. Thus, the investment in the existing imageanalysis code can be preserved while incorporating sym-bolic reasoning. The expert system allows the handling ofdeclarative as well as procedural knowledge in our envi-ronment.Some problems that have yet to be solved regarding the

Analyst Advisor implementation are as follows: the effi-cient control of devices across a network and the sharingof knowledge in a network, a pause function within theAnalyst Advisor, and the handling of explanations frommultiple experts in a coherent manner. These problemareas are now being addressed. A prototype of the AnalystAdvisor will be released in March 1987 for use withinCCRS and at selected beta-test sites.These developments are very encouraging for the use

of knowledge-based systems for integrating geocodeddatabases with remotely sensed data. Future developmentwith the MICE system will include experiments with datafrom various satellite sensors and topographic maps fromdifferent provincial and federal agencies. The AnalystAdvisor will continue to grow dynamically and be modi-fied so as to include all of the LDIAS software. To thebest of our knowledge, this is the first multiple-expert sys-tem for remote sensing combining computer vision, geo-graphic information systems, and symbolic reasoning.

REFERENCES[1] R. A. Brooks, R. Greiner, and T. 0. Binford, "The acronym model

based vision system," in Proc. IJCAI, pp. 105-113, 1979.[2] A. R. Hanson and E. M. Riseman, "VISIONS: A computer system

for interpreting scenes," in Computer Vision Systems, A. R. Hansonand E. M. Riseman, Eds. New York: Academic, pp. 203-333, 1978.

[3] M. D. Levine and W. Hong, "A knowledge-based approach to com-puter vision systems," Graphics/vision Interface, pp. 260-265, May1986.

[4] J. K. Tsotsos, "Knowledge organization and its role in representationand interpretation for time-varying data: The ALVEN System,"Computational Intell., vol. 1, no. 1, pp. 16-32, Feb. 1985.

[5] S. A. Stansfield, "ANGY-A rule-based expert system for automaticsegmentation of coronary vessels from digital subtracted angio-grams," IEEE Trans. Patt. Anal. Machine Intell., vol. PAMI-8, no.2, pp. 188-199, Mar. 1986.

358


[6] J. Glicksman, "A cooperative scheme for image understanding usingmultiple sources of information," Ph.D. dissertation, Univ. BritishColumbia, Nov. 1982.

[7] D. M. McKeown, "Knowledge-based aerial photo-interpretation,"Photogrammetrica, vol. 39, pp. 91-123, 1984.

[8] D. M. McKeown, W. A. Harvey, and J. McDermott, "Rule-basedinterpretation of aerial imagery," IEEE Trans. Pattern Anal. Ma-chine Intell., vol. PAMI-7, no. 5, pp. 570-585, Sept. 1985.

[9] G. W. Plunkett, D. G. Goodenough, and M. Goldberg, "Map/imagecongruency evaluation knowledge-based system," Graphics/VisionInterface, pp. 273-278, May 1986.

[10] M. Goldberg, D. G. Goodenough, M. Alvo, and G. Karam, "A hi-erarchical expert system for updating forestry maps with Landsatdata," Proc. IEEE, vol. 73, no. 6, pp. 1054-1063, June 1985.

[11] J. M. Zarzycki and M. M. Allam, "Canadian council on surveys andmapping-National standards for the exchange of digital topographicdata," Topographical Surveys Division, Surveys and MappingBranch, Apr. 1982.

[12] D. G. Goodenough, K. B. Fung, F. Hegyi, M. Robson, and N. A.Swanberg, "Integration of geographic information systems withLANDSAT thematic mapper data," presented at IGARSS, Amherst,MA, 1985.

[13] F. Hegyi and P. Sallaway, "Integration of vector and grid data basesin B.C. forest inventory," in Proc. 6th. Int. Symp. Automated Car-tography, B. S. Wellar, Ed., pp. 215-221, Oct. 1983.

[14] S. L. Tanimoto, "Parallel architectures for machine vision," Graph-ics/Vision Interface, p. 349, May 1986.

David G. Goodenough (M'77) received the B.Sc.degree in physics from the University of BritishColumbia and the M.Sc. and Ph.D. degrees in as-tronomy from the University of Toronto.

He is Chief Methodology Research Scientistwith the Canada Centre for Remote Sensing andiS Head of the Methodology Section. His researchhas been focused on information extraction fromremotely sensed data. In particular, he has pub-lished extensively on pattern recognition algo-rithms and image analysis systems. Currently, he

is incorporating artificial intelligence methods into solutions for the inte-gration of geographic information systems and image analysis systems. Heis also an Adjunct Professor of Electrical Engineering at Ottawa Univer-sity.

Dr. Goodenough is a member of the IEEE Geoscience and Remote Sens-ing Society, the IEEE Computer Society, and the AAAI.

Morris Goldberg (S'68-M'73) received his un-dergraduate training at the University of McGilland the Ph.D. in electrical engineering from Im-perial College in 1972.

Since 1974, he has been working in the generalarea of image processing, first for the CanadaCentre for Remote Sensing, and since 1976 at theUniversity of Ottawa, where he is at present anAssociate Professor in the Department of Electri-cal Engineering. In 1982-1983 he was a VisitingProfessor at the ENST in Paris, and in 1986-1987

he was a Consultant for Bell Northern Research. His current research in-terests are in the areas of multimedia communications for medical appli-cations, expert systems for remote sensing, and image compression.

Dr. Goldberg is Treasurer of the Canadian Image Processing and PattemRecognition Society, and Chairman of the IAPR Technical Committee onRemote Sensing.

Gordon Plunkett received the B.Eng. degree inelectrical engineering from Carleton University,Ottawa, Ontario, Canada, in 1974, and the Ma.Sc.degree in engineering from the University of Ot-tawa, Ottawa, Ontario, in 1986.

He has been employed with the Department ofNational Defence and is currently employed as aSenior Physical Scientist with the Department ofEnergy, Mines, and Resources. His current re-search areas are machine perception and imageprocessing.

*

Society and the AAAI.

John Zelek (M'86) received the Ba.Sc. degree insystems design engineering from the University ofWaterloo in 1985. He is currently working towardthe Ma.Sc. degree in electrical engineering at theUniversity of Ottawa.

He is also currently employed by Intera Tech-nologies Ltd. and is working as a Research As-sociate at the Canada Centre for Remote Sensing.His research interests include computer vision andartificial intelligence.

Mr. Zelek is a member of the IEEE Computer

359

Documents

An Expert System for Remote Sensing