Embed Size (px)
Citation preview
1
Situated Image Understanding in a Multi-Agent Framework
N. Bianchi*1, P. Bottoni**, C. Spinu***, C. Garbay***, P. Mussio**
* Department of Physics - University of Milan, via Viotti 5, 20133, Milan, Italy - email [email protected]
** DSI - University of Rome, via Salaria 113, 00198 Rome, Italy - email {bottoni,mussio}@dsi.uniroma1.it
*** Lab. TIMC, Institut Bonniot, 38706 La Tronche, France - email: [email protected]
Running title: Situated image understanding
1
Current address: Nadia Bianchi, ElectroTechnical Laboratory, 1-1-4 Umezono, Tsukuba, Ibaraki, 305 Ja
pan, [email protected]
2
Abstract
The paper addresses the problem of controlling situated image understanding processes. Two complementary control styles are considered and applied cooperatively, a deliberative one and a reactive one. The role of deliberative control is to account for the unpredictability of situations, by dynamically determining which strategies to pursue, based on the results obtained so far and more generally on the state of the understanding process. The role of reactive control is to account for the variability of local properties of the image by tuning operations to subimages, each one being homogeneous with respect to a given operation. A variable organisation of agents is studied to face this variability. The two control modes are integrated into a unified formalism describing segmentation and interpretation activities. A feedback from high level interpretation tasks to low level segmentation tasks thus becomes possible and is exploited to recover wrong segmentations. Preliminary results in the field of liver biopsy image understanding are shown to demonstrate the potential of the approach.
Keywords: planning, image understanding, multi-agent, metareasoning, conditional rewriting systems, concurrent execution, biopsy interpretation
1. Introduction
In many applications [SVW84, ABM89, MH90, RSV93] patterns of different types are to be recognised in digital images, obtained from real world. These images are in general characterised by the variability in shape and colour of the patterns to be recognised and in the noise affecting them. Contextual information and abstraction levels become of major importance for satisfactory recognition.
In these situations, difficulties in the control of image understanding activities [Ros88, Mat89, DGH93] arise because it is not possible to foresee a sequence of operations allowing an acceptable classification of patterns in every possible image; a recognition sequence depends on the characteristics of the image at hand.
In manual or interactive approaches to image understanding, at each step of the process the execution of an operation produces results that the human operator evaluates before choosing the next operations. This choice depends on the context provided by the results already obtained, as well as on the interpreter's skill and experience. Moreover, how to perform a given operation depends on local as well as global properties of the image at hand and each operation is tuned to the characteristics of the image.
In automating this understanding process, the control problem is therefore posed at two levels: the first is to determine which operation to perform next; the second is to determine how to perform the operation.
3
As to the first problem, a successful solution, recently applied to the automatic interpretation of technical drawings [BCM95] and to semiautomatic liver cell counts [BMP94], adopts a deliberative approach, in which deliberation occurs on the state of the process. The state is evaluated by metareasoning mechanisms to determine the next actions to be performed, thus dynamically adapting the plan which prescribes the sequence of operations. The ability to adapt the plan showed to be powerful in managing the context, but not the local properties of the image.
The second problem was recently addressed by a reactive approach to overcome the difficulties which arise because of local property variability [SGC95]. Due to this variability, an image can be partitioned into subimages, each one being homogeneous with respect to a given operation. Each subimage should be treated by a different version of a tool performing the operation, tuned to its specific properties. In this approach, a pre-analysis phase partitions the image into subimages homogeneous with respect to noise and texture and generate a network of image processing agents, each one tuned to a specific subimage. Individual agents in the network work concurrently, each one applying a predefined plan. This approach was powerful in segmenting the image exploiting local variability, while the rigidity of the plan made it difficult the exploitation of context.
In this paper we integrate the two approaches in a multi-agent framework so that the control of image understanding processes is obtained by: deliberating on the state of the process; accounting for concurrent operations; adapting the organisation of the network of agents and the operations the agents perform to the variability of properties within an image.
To reach these goals, we extend some notions presented in [BBS96]: the notions of plan and state of the deliberative approach are generalised to cope with concurrent deliberation and reaction while the concept of homogeneous subimage of the reactive one is generalised to that of homogeneous sub-state.
The integration of the reactive and deliberative techniques allows a computational characterisation of the possible strategies in image understanding and allows their uniform management via metalevel techniques. This results into an original model to control the image processing activities.
The operations to be performed are scheduled following the deliberative approach while activities are concurrently executed exploiting the reactive approach whenever necessary. A feedback from high level interpretation tasks to low level segmentation tasks thus becomes possible and is exploited to recover wrong segmentations.
In this way the pattern recognition ability of the system is increased by a non-monotonic strategy of segmentation and classification. The structure of the system is general: in fact it can be applied in several image understanding experiments. However, in every specific experiment the understanding activity is situated, in that it is driven by a domain knowledge explicitly and
4
separately coded as metarules and rules.
These facts are commented on through an example discussing a real experimental case which illustrates the approach and its pros and cons.
The paper is organised as follows: previous work from the authors is recalled in Section 2, and basic definitions are introduced. The understanding process is formalised in Section 3 as a general control cycle, operating on rules and meta-rules at each stage of the understanding process. Concurrent plans are dynamically generated and executed. This is realised by a variable organisation of agents, generated according to plans, and discussed in Section 4. Section 5 illustrates the potential of the proposed design, discussing the results from a real case in liver biopsy image understanding. Related work is discussed in Section 6 and conclusions are drawn in Section 7.
2. Situated Image UNDERSTANDING and Preliminary Definitions
Image understanding is the process which derives a description from an image. In general, these image descriptions are situated in a specific context and are obtained according to certain conventional rules, established by a community of human experts which have to perform specific tasks.
The community of experts collectively determine the types of patterns significant for their tasks when appearing in a class of images. In [BCL96], such types of patterns are called the characteristic structure types and their instances characteristic structures.
For each type of characteristic structure the community agrees on a name which identifies it. Then the community establishes the attributes necessary to characterise an instance of the type, according to the context of the understanding activity.
Therefore, each characteristic structure is described by the name of its type and the values (called properties) assumed by its attributes. The description of an image enumerates the characteristic structures recognised in the image and univocally identifies them.
The types of patterns become therefore the goals of the understanding and the conventional rules establish how to recognise instances of each type and how to evaluate their properties.
Type definitions and rules constitute a body of knowledge typical of the community. In this sense the understanding process of an image, based on that knowledge, is situated for that domain. A same image, interpreted according to definitions and rules of a different community may result into a different description [MBE94].
2.1 A Scenario: Biopsy Understanding
Physicians constitute a community which frequently uses images as the source of data and the means of communication. The examination of a biopsy of a patient's liver requires the definiti
5
on of the characteristic structure types that are both normal and pathological.
These characteristic structure types are cell types -e.g. hepatocyte- and biological structures -e.g. sinusoid- as well as their components -e.g. hepatocyte nucleus, non hepatocyte
nucleus, membrane. A hepatocyte is a nearly hexagonal structure in which the mean colour is red, and whose texture is finely grained. In general, a hepatocyte contains a round, heavily coloured nucleus. However, being a biopsy the result of a slicing action, the nuclei can be absent or present in part. A sinusoid is an irregular structure whose texture is uniform and mean colour is white. In general it is elongated. A sinusoid may contain some cells of different types, for example Ito cells. For this reason, some structures may appear inside the sinusoid and some of these structures may appear as small nuclei.
For example, in Fig.1 three hepatocytes have been manually outlined, bounded by their membranes. Two of them contain a nucleus, which has been manually circled with a dotted line. Such a description is not universal, on the contrary it is situated in the context of liver studies, and reflects the specific knowledge of the hepatologists.
Figure 1. A liver biopsy image (green component).
In interpreting a biopsy, a physician uses both images and descriptions. It operates on images to improve their readability and to extract characteristic structures and on descriptions to classify characteristic structures. Both activities require the use of local as well as contextual information.
This combined use of images and descriptions appears common to every image understanding activity.
To describe such a process and how to automate it, precise definitions of image and descriptio
6
ns are required
2.2. Preliminary Definitions
A k-component n ∞ m image is specified by a function i: {1,…,n} ∞
{1,…,m} ∅ Vk where n and m are two integers, V is a suitable set and k is an integer.
For example in the image of Fig.1 the elements of V are the squares of size 1 pixel and of different 256 COLOURS and k=1. Here, the alphabet V contains symbols such as: {
}.
A structure is a subset of pixels in the image. A characteristic structure is a structure recognised as an instance of a characteristic structure type. It is described by a finite set of properties. For example, in Fig.2b the characteristic structures nucleus and membrane composing the characteristic structure hepatocyte have been outlined.
A segmented image is an image in which the pixels of each recognised characteristic structure are set to the same colour. For example, Fig.3 shows a segmented version of Fig.1, which is the result of an automatic understanding process.
An image zone is a structure homogeneous with respect to some criterion. For example, Fig.2c shows zones which are homogeneous with respect to noise according to a local variance estimator.
A zone-sensitive segmentation is a segmentation process where actions are selected and adapted, based on local variability.
In the system COALA, implementing the reactive approach quoted in the introduction, zone-sensitive segmentation is performed by a variable network of agents [SGC95]. The definition of the network is data-driven. In a first step a fixed number of agents apply estimation procedures, to locally evaluate the noise and texture present at each point in the image. The image is then partitioned into zones displaying homogeneous noise and texture characteristics. Each zone is then concurrently processed to extract edges by a processing group of connected agents applying a predefined plan, adapted to the zone characteristics. Fig.2d shows the result of this approach on image in Fig.2a.
The description of a characteristic structure is an attributed symbol from an alphabet of attributed symbols W= T ∞ Pa1 ∞ ... ∞ Pam where T is the set of characteristic structure types, ai are the attributes and Pai is the set of properties which can be assumed by ai.
A description of an image i is a string of attributed symbols specified by a function d: {1,...,n}
∅ W. A string is a sequence of symbols separated by Ò¡Ó.
For example the central nucleus of Fig.2b is described by the attributed symbol (Nuc, 1, 453,
93,
…, , , , ,
7
86) where Nuc�T denotes the characteristic structure type, 1 is the value assumed by the attribute Identifier of the characteristic structure, 453 is the value assumed by the attribute Area, 93 by Ellipticity, 86 by Perimeter. The description of the image is the string of attributed symbols describing the hepatocytes, sinusoids, nuclei and membranes recognised in the image.
A state-sensitive interpretation process obtains and refines a description of an image taking into account the results obtained so far and the actions already performed.
In the system IIMS [BMP94], implementing the deliberative approach quoted in the introduction, the state consists of the string d=features § pActions ◊ plan where
a) features is a string of attributed symbols describing the results obtained so far and their relations;
b) pActions is a string of attributed symbols describing the already performed actions and their success; and
c) plan is a string of attributed symbols describing a plan of activity, stating the actions currently foreseen as necessary to go on in the process
d) § and × are two separators.
features is a string on an alphabet W of attributed symbols denoting the different kinds of images, the structural descriptions and the types of characteristic structures to be used. pActions and plan are strings on an alphabet C of attributed symbols, denoting actions which can be performed by the interpreter.
(a) (b) (c)
8
(d) Fig.2: (a) Initial liver biopsy image (a portion of the image of Fig.1); (b) manually highlighted
characteristic structures (c) noise map; (d) final result with COALA
Actions are defined by sets of rules or metarules and the names denote both the actions and the defining sets of (meta)rules. Strings on C are called strategies and a set of strategies is a language, the language of strategies. Metarules dynamically update the plan component of d. Rules update the features component of d.
An image understanding system is one which produces rich descriptions of images [Bra83] and represents them as segmented images.
In our approach, an image understanding system interleaves segmentation and interpretation activities so that feedback from high level interpretation to low level segmentation is effectively exploited.
Figure 3. The final classification of the system: Hepatocytes (blue and green) and Sinusoids (violet, red and black).
3. Situated Strategies for Image Understanding
3.1. Understanding Strategies and Their Specification.
The combined use of zone- and state-sensitive approaches determines the capability of operating both on descriptions and segmented images. The state of the understanding process is thus generalised to s=<P,d> by coupling a set of segmented images P with d.
The set P is composed of the input image together with those images progressively produced during th
9
e understanding process; the input image is considered a segmented image, in which each pixel is assumed to constitute a segment by itself.
As in IIMS, d=features§pActions�plan is a string of attributed symbols in which features describes the already obtained results and is a string on an alphabet W whose attributed symbols describe features and images. We call the string d the description of the state.
Note that the features component of d also contains symbols which describe images in the P component of the state. In particular, an attributed symbol <i, i1,
{i2,i3}> indicates that the P component of state contains an image of type i�T, whose identifier is i1 and that has been obtained by combining via some operation the images with identifiers i2 and i3. Calling Pnames the set of names of the images in P, i1�Pnames is the value of the attribute reference, and {i2,i3}�p(Pnames) is the value of an attribute origin.
pActions describes the already performed actions while plan describes the currently planned strategy. Both pActions and plan are strings on the attributed symbols of an alphabet C of actions. Each attributed symbol s�C is constituted by a name denoting the type of action and the attributes characterising its execution. An important attribute of s�C is table, the definition of the action itself, which is given as the textual representation of a set of (meta)rules.
In general, the set of descriptions and the set of strategies which can be used to interpret images of a certain type according to the knowledge of a certain domain may be infinitely numerable and constitute two languages. As such, they can be defined through finite syntactic tools like grammars [Fu82] and L-Systems [Ros79], in which finite sets of rules are used to generate or recognise the language strings.
The different components of d are updated by rule-based mechanisms. The features component is updated by actions based on already obtained results and described by sets of rules. The pl
an component is updated by actions based on found results and on knowledge of the actions already performed. The actions updating plan component are described by sets of metarules. In either case the pActions component is updated by concatenating at its right end a description of the action which has just been performed.
These sets of rules and metarules are named and these names are the symbols on which the set C is defined.
Rules and metarules share a common format, so that they can be interpreted by the same program. However, rules are used to update the string features defined on W and metarules to update the string plan on C. Moreover, metarules access the features and pActions substrings to evaluate the situation: the use of two different alphabets avoids paradoxes and antinomies [BMP94].
The format common to rules and metarules is the following: a name followed by one syntactic and one semantic parts.
10
The syntactic part is constituted by an antecedent A and a consequent B; and the semantic part by a condition γ and by a set of functions {fi}.
The antecedent A is a set of strings which define the substring of d to be rewritten (called argument) and the context in which it must appear. The argument must always be present. For rules, both argument and context are strings on the alphabet W, and also the consequent B is a string on this same alphabet W. Metarules, instead, are used to update the plan component following ◊. Metarules used in the current framework restrict the argument to be the symbol ◊. Since they do not take into account the already planned parts of the strategy, but they can evaluate already obtained results and already executed actions, their context is a string which must occur in the features or pActions components. Finally, their consequent specifies a subplan to be inserted before the current plan substring: hence, it is a string starting with the symbol ◊ and presenting symbols from only the alphabet C.
In both rules and metarules the condition γ is a predicate to be evaluated on the properties of elements in A and the functions {fi} specify how to compute properties of the rewritten symbols.
3.2. State-sensitive Derivation of a Strategy
A (meta)rule is applied to a string d1 to obtain a new string d2 by the following procedure. First it is checked if the argument and the context are present in d1. If they are present, the condition γ is verified: if it evaluates to true the argument is substituted by the consequent, the functions fi are computed obtaining the new values of attributes of the symbols in d2 and the new string d2 is generated. Finally, pActions is updated by recording a description of the applied action, and the elements on which it was applied. Examples of the rule formalism and of its application are shown in Appendix A.
A novelty of the proposed approach is that actions are performed concurrently whenever possible. In particular, all the rules and metarules in a same set, i.e. defining one action, are applied in parallel, as in L-systems. Moreover, the names of the actions which are to be executed concurrently are grouped - into a string called fragment delimited by the separator # [May92]. Actions whose names are contained in the current fragment are applied concurrently, while an action in the next fragment can be executed only if all the actions in the current one have been completed. In other words, fragments are executed sequentially, while actions in a same fragment are concurrent. Hence, a plan results into a sequence of fragments. Appendix A and the example in Section 5 show the development of a strategy according to this technique.
An image understanding process starts from an initial state s0=<{i0},d0> consisting of an initial image i0 and an associated initial description d0. At the beginning of an understanding activity, no feature has been recognised, no action has been performed. Hence, the components features and pActions of d0 are empty, while its plan component contains the name of a metarule mr0
which identifies the activities which start the process of understanding the image i0 according
11
to the strategies typical of a community of experts. Let us denote by λ the empty string. The initial description is in this case represented as d0=λ§λ×mr0. By the application of mr0, only d0
is transformed, generating a new state s1=<{i0},d1> with d1=λ§mr0
×Fr1#Fr2#…#Frn, where Fr1#Fr2#…#Frn is the consequent of mr0. Fr1…Frn are fragments, strings of names of sets of rules or metarules, which altogether constitute the initial strategy that will be progressively updated according to the obtained results. The name mr0
is moved in the pActions component of the description. Now the fragments are applied one at a time. Whenever a metarule appears in a fragment, its application results into a change of the plan. Whenever a rule appears in a fragment, images or descriptions are generated. On the whole, the application of rules and metarules results in the progressive modification of all the components of the state, as will be exemplified in Section 5. This process is repeated until no rule or metarule can be applied.
3.3. Zone-sensitive Strategies
The notion of zone is crucial in allowing an adaptive behaviour of the system. For this purpose, the concept of image zone, introduced in COALA, is generalised to that of state zone. We define a state zone as a subset of the state which can be treated homogeneously with respect to the execution of a strategy. For example, in liver cell classification, a set of different types of descriptions is obtained (e.g. of candidate hepatocytes and sinusoids, candidate nuclei, edges and white zones). To solve uncertainties on classification, topological and geometrical relations among different candidates are explored. For example the structure marked "b" in Fig.4 is classified as a candidate hepatocyte at an intermediate stage of the understanding process (see Section 5). At a subsequent stage, the presence inside it of a nucleus of a Non Hepatocyte cell determines its reclassification as a sinusoid. In this case the strategy NucPresStr (for Nuclei Presence Strategy) is applied to a state zone formed by the two sets of descriptions DescCN and De
scCHS, storing respectively the structures classified as candidate or certain Nuclei (CN), or as Hepatocytes and Sinusoids (CHS). Zones as defined in COALA are special cases of state zones constituted by subimages. This fact is exploited in the strategy EdgStr, which applies the procedure described in [SGC95]. In the following, by zone we mean state zone.
12
Figure 4. Classification of Candidate Hepatocytes (green) and Sinusoids (red).
Any image understanding process involves two phases:
1) Identification of zones in the current state, on which to perform the next steps of the understanding process, and assignment of actions to each zone (plan deliberation phase).
2) Execution of actions on identified zones and collection of results (plan application phase).
Both phases are performed according to strategies which are not predictable in advance and in which the plan deliberation phase may in turn occur according to the following two categories:
1) Situation assessment: According to the recognised situation, the system will proceed by performing some specific action.
2) Fine-tuning: the system has arrived to a state where it must perform an action of a given type. It can, however, fine-tune the parameters of the action to the specific characteristics of the data at hand.
The development of such a model for an understanding process is realised by the use of sets of metarules which generate substrategies - i.e. metarule names- specialised to the different zones.
Situation assessment activities are performed by sets of metarules of type MrStr. Fine-tuning a
13
ctivities are performed by sets of metarules of type MrSeg. A formal specification of these types of metarules is given in Appendix B.
The two types of metarule sets are organised and act on the plan in different ways. Metarules in a set of type MrStr
are mutually exclusive. Each metarule analyses the current state to define a new specialised strategy. Only one metarule succeeds, so that the strategy will be adapted to the analysed situation. On the contrary, a set of type MrSeg
is formed by only one metarule which computes a collection of zones and generates a specialised collection of (meta)-rules, one for each identified zone, based on the analysis of the current state.
Metarules of type MrSeg also
allow the management of distributed segmentation process involving subimages as zones and will start specialised rules on different zones.
Situation assessment and fine tuning can be combined to generate complex strategies. Metarules of type MrStr and MrSeg can be further refined, so as to generate several fragments. A typical case is when they place in the plan not only actions to be performed on zones, but also metarules which determine the reasoning on data and already obtained results to deduce the next steps in the understanding strategy. Examples of such metarules are shown in Section 5.1.
4. Multi-Agent Execution of Situated Strategies
4.1. Agents and Agent Types
Strategies are realised by a dynamical organisation of agents. An agent is a software entity, defined by a state and a behaviour, whose role is to permit the application of rules or metarules.
Agents are dynamically created whenever needed to execute sets of (meta)rules. We distinguish three different types of agents, called Managers, Schedulers and Executors, according to their function in the organisation.
A Manager agent has the role of applying a metarule and deliberating a subplan. As a result, it generates the agents necessary to manage the subplan. Scheduler agents support a manager in the case that more than one (meta)rule appear in a set. They take the list of (meta)rules in such sets and generate a suitable agent for each (meta)rule.
An Executor agent has the role of executing an operation on a (sub)image and/or a (sub)description, as specified by a rule. Hence, it reacts to the environment situation, evaluating the antecedent and the condition of the rule and deciding whether or not to rewrite the argument.
4.2. Management of the image understanding activity
To apply a metarule, a Manager agent is created with a copy of an interpreter I
14
and with a local state variable, whose initial value is a subset of the current state of the image understanding process.
The interpreter implements the procedure IntImage described in Table 1, where halt-
condition is a predicate which defines termination: in the experiment described in Section 5 the halt-condition, derived from [BMP94] is the disjunction of two predicates: 1) the next foreseen action has already been applied twice with equal results; or 2) no rule or metarule has to be applied (plan is empty).
Table 1. A procedure describing the image understanding process Procedure IntImage(image, description) /* Initialisation */ state♦ <image,description> /* initial deliberation determines a new plan (state) */ state♦ DeliberatePlan(state) /* generates detailed initial plan */ /*execution cycle */ /* Application of plan determines detection of features and updating of plan*/ while (halt-condition(state) is false) state ♦ ApplyPlan(state) endwhile Return(state)
DeliberatePlan and ApplyPlan in IntImage define the strategy to be used by updating the plan component of description. The understanding activity is started by a call to a predefined Manager to operate on the image i0 and on d0=λ§λ×mr0, where mr0 is the name of the set of metarules to be applied. This Manager first establishes the initial state of the process, then deliberates the plan to be executed, interpreting mr0 by a call to DeliberatePlan which updates plan by inserting the names of sets of (meta)actions to be performed. This realises the rewriting process for the metarule mr0
and the updating of plan is specified by the consequent of mr0. DeliberatePlan also updates pActi
ons with the name of mr0.
15
Table 2. The procedure for plan application. Procedure ApplyPlan(state) /* Extracts the fragment to be executed */ plan ♦ PlanComponentOf(state) fragment♦ FirstFragmentFrom(plan) Remove(fragment, plan) foreach symbol � fragment /* Instantiates a new agent */ child♦CreateAgent(symbol,state) children♦ children ≈ {child} endfor ActivateAgentsOf(children) while NotEmpty(children) WaitForA(completionMessage) /* Completion messages specify the results of the activity of a child, i.e. newly created features and performed actions */ /* On arrival of a completion message */ UpdateFeatures(FeaturesOf(completionMessage)) UpdatepActions(PActionsOf(completionMessage)) Delete(SenderOf(completionMessage), children) endwhile Return(state)
Next, Manager checks the halt condition: if the condition is not satisfied Manager applies the current plan by calling ApplyPlan. ApplyPlan extracts the first fragment from the plan and generates an agent for each name in the fragment. CreateAgent
is in charge of this activity: it examines the symbol, determines whether it is the name of a set of rules or metarules and the number of rules and metarules in the set. We consider here a version of CreateAgent
in which no cooperation among agents is foreseen. If the symbol is the name of a rule set and only one rule is in the set, CreateAgent
creates an Executor, passing as arguments the text of the rule to be executed, the features state component to be used and updated and the set P of images. In the case that more than one rule exist, a Scheduler is created with the same arguments and in charge of the correct generation of one Executor for each rule. If the symbol is the name of a metarule set and only one metarule is in the set, CreateAgent
creates a Manager, passing as arguments the text of the metarule to be executed, the state components features and pActions, and the set P of images. In the case that more than one rule exist, a Scheduler is created with the same arguments and in charge of the correct generation of one Manager for each metarule.
If a generated agent is a Manager or a Scheduler, it will generate other agents. Managers and Schedulers also record their children activities. On their side, children agents maintain their parents address so as to communicate back their results to them via completion messages. A parent agent collects its children completion messages and uses them to produce a description of
16
the results of its global activity. In its turn it will send such a description as a completion message to its parent. Tab.2 shows the pseudo-code of ApplyPlan.
Fig.5: The network of agents concurrently realising the substrategy defined by mr.
Fig.5 shows an instantaneous configuration of the system, during the execution of a strategy generated by a set of metarules mr. In mr only one metarule is present, whose consequent is r0°r1°mr2. To evaluate mr, a Manager (mr) has been created with a state defined as s=<{i0,…,in},
someFeatures§somePastActions◊mr>. The successful application of the metarule in mr resulted in the generation of the fragment r0°r1°mr2 in the plan substring. To execute this fragment, Manager (mr) generated three agents, each one in charge of one action. The set of rules r0 is a singleton. Hence an Executor agent (r0) has been created. The set of rules r1 consists of two rules: its execution implies the generation of a Scheduler Agent (r1). The set of metarules mr2 requires a new manager (mr2), starting in turn the execution of IntImage({i0,…,in}, someFeatures§
somePastActions ◊ mr2).
In this situation all the generated agents are active concurrently. As soon as an agent completes its activity, it reports the obtained results to its parent and cease to exist. At last, Manager (mr) receives all the reports, combines them in its final state value and reports this value to its parent agent..
5. An Experiment In Biopsy UNDERSTANDING.
The way in which agents are created, come into activity, apply rules, updating and using their states and disappear is illustrated by commenting on a real experiment in liver biopsies understanding. We also discuss the efficacy of the method by commenting on the results of this experiment.
5.1 Outline of the Experiment
Liver biopsies as the one in Fig.1 are interpreted by histologists looking for liver structures such as hepatocyte cells and sinusoids. These structures are then described - area and colour for
Manager (mr)
ExecutorAgent(r0)
CentralizedControl
DecentralizedControl Scheduler
Agent (r1) ManagerAgent(mr2)
Executor Agent
Executor Agent
… …
17
example are evaluated - for pathology detection [Bia95].
A semiautomatic sequential biopsy understanding process -proposed in [MPB91] and refined in [Bia95] by the CSMT group- is here restated and extended in terms of the new approach.
Biopsies in this experiment are three colour RGB images from a Sony camera DCX_M2PH and form a set I={p | p:{1,…512} ∞ {1,…,384} ∅
{0,…,255}3}. Fig.1 shows the green band of a biopsy.
The understanding process starts with an interactive call to the system by the user. The system requests the user to specify the image to be interpreted and the type of understanding process. In our case, the user replies with the name of the liver biopsy (ÒbiopsyÓ in our example) and specifying that a hepatological interpretation is required. The system reacts by instantiating a manager MA0, establishing the initial state s0=<{biopsy},d0> with d0=λ§λ�CellStr, where CellStr is the name of a set of one metarule defining the strategy proposed by the CSMT histologists for biopsy understanding. Then the system gives the control to MA0 which executes a call IntImage(biopsy,
λ§λ�CellStr), MA0 first executes the deliberation phase applying CellStr which results into adding the names of a set of rules Init and of two sets of metarules NucleiStr and EvStr1 to the plan.
The three actions are to be executed sequentially, so they are organised into three different fragments. MA0 state is now
s01=<{biopsy}, λ§CellStr � Init # NucleiStr #EvStr1>
Next, MA0
start applying the plan and generates a new Executor to evaluate Init. This Executor, applying the rule in Init, generates a symbol of type RGB whose identifier attribute takes the value biopsy, for further reference. During this activity MA0 and Executor are both alive. On Init completion, Executor ceases to exist and MA0 records the name RGB in the features component and the name Init in the pActions component. MA0 state becomes:
s2=< {biopsy}, RGB § CellStr ° Init � NucleiStr #EvStr1>
MA0 prosecutes its application phase by generating MA1 which rewrites (in its own deliberation phase) NucleiStr into two new fragments: NucSegmStr # NucDescStr. MA1 state is now:
sI1=<{biopsy}, RGB §CellStr ° Init ° NucleiStr � NucSegmStr#NucDescStr#EvStr1>
Now MA0 and MA1
are active, MA0 waiting for results from MA1, and MA1 executing its current plan. A plan activation phase starts for MA1. It generates a manager MA2 to execute the strategy NucSegStr to single out the Candidate Nuclei. This strategy applies image processing operators to the biopsy a
18
nd obtains a segmented image iCN (see Fig.6) in which black pixels are candidates to belong to a nucleus. In particular, it performs an adaptive thresholding on the red component of the biopsy.
Fig.6: Segmentation of the cell nuclei.
The threshold is set at a value corresponding to the flex which marks the beginning of highest rise in the component histogram. MA1 then generates MA3 to obtain the description DescCN of iCN. The MA3 strategy classifies and describes each set of black pixels according to its compactness, shape and geometrical characteristics. These strategies involve thresholding and neural networks and are described in [BD95]. On completion of these activities, MA0
prosecutes its plan application by generating a new manager MA4
to apply EvStr1. EvStr1 evaluates the quality of the image from the number and type of nuclei identified and decides if it is worthwhile to continue the understanding process. After the deliberation phase, the state of MA4 is
sIV1=<{iCN, biopsy}, RGB°CN°DescCN §CellStr ° Init ° # NucleiStr°NucSegmStr#NucDescStr# EvStr1� HepSinStr°WhiteZoneStr°EdgStr#EvStr2>
MA0 is waiting for MA4 to complete its activity, after having removed the fragment EvStr1 from its plan. MA0 state therefore is:
s03=<{iCN, biopsy}, RGB°CN°DescCN §CellStr ° Init ° # NucleiStr°NucSegmStr#NucDescStr� λ>
Note that the image component of this state is now a set of two images, the original biopsy and the segmented image storing some results in pictorial form, while the associated description stores some specific properties and relations in symbolic form.
19
In the MA4 application phase, the three strategies HepSinStr, WhiteZoneStr and EdgStr of the same fragment are concurrently executed. Three managers are generated, each one executing one of the strategies on a copy of the image biopsy. The development of the strategies defined by the first two metarules involves specialised actions, defined by metarules of the type MrStr, while EdgStr involves the use of metarules of type MrSeg, as described in [SGC95]. In particular, White Zones are obtained by thresholding the intensity image at a value corresponding to the last valley in the component histogram, so that high response zones are identified. The image CHS, instead, is segmented with a threshold defined by the formula th=f+(m2-
m1)/2, where m1 and m2 are the values of the first two peaks in the green component histogram and f is the value of the flex at the beginning of the first rise. The formulae for segmentation were obtained experimentally and allow an adaptive thresholding related to the amount of cellular matter in the biopsy.
The three strategies exploit local properties (area, compactness, ellipticity, ...) of image structures, without considering their relations with other kinds of structure, but taking into account their shape. They produce images and descriptions of candidate Hepatocytes and Sinusoids (iCHS and DescCHS -see Fig.4), White Zones (iCW and DescCW -see Fig.7) and edges (candidate Membranes) (iCM -see Fig.8- and DescCM). For example, in Fig.4 structure "a" is classified as a Candidate Sinusoid due to its elongated shape, while structure "b" is classified as a candidate hepatocyte due to its nearly round shape [Bia95].
Fig. 7. Classification of White Zones, obtained by the materialisation of DescCW. The zone marked with b will be used in the final classification of Sinusoids.
After their application the MA4 state is:
20
sIV2=<{iCHS,iCW,iCM,iCN,biopsy},RGB°CN°DescCN°CHS°CW°CM°DescCHS°DescCW°DescCM § CellStr ° Init ° # NucleiStr°NucSegmStr#NucDescStr# EvStr1 # HepSinStr°WhiteZoneStr°EdgStr �
EvStr2> MA4 is still applying the plan generated by EvStr1. Now MA4 instantiates a new agent for the evaluation of EvStr2, which has the aim of checking the classification proposed for the candidate structures by verifying their mutual geometrical and topological relations, as outlined in Section 3.3.
Eight metarules have been defined to manage this process (see Appendix C). These metarules manage eight mutually exclusive situations, so that only one of these metarules can be applied on a given image. Hence EvStr2
is a set of metarules of type MrStr. MA4 generates a scheduler SC1 to manage the eight rules. SC1 generates eight managers of which seven fail in their deliberation phase and one survives.
In the case at hand, nuclei, white zones and edges have been found and only the manager MA5
evaluating the first metarule of EvStr2
as shown in Appendix C survives, and after deliberation, it is in the state:
sV1=<{iCHS,iCW,iCM,iCN,biopsy},RGB°CN°DescCN°CHS°CW°CM°DescCHS°DescCW°DescCM § CellStr ° Init ° # NucleiStr°NucSegmStr#NucDescStr# EvStr1 # …#EvStr2
� NucPresStr°WhitePresStr°# FusionStr>
As outlined in Section3.3, NucPresStr and WhitePresStr exploit the context provided by previously identified Nuclei and White zones to refine the classification of candidate Hepatocytes and Sinusoids stored in DescCHS. In particular, NucPresStr activates rules for merging information provided by classification of structures in iCN as Nuclei of Hepatocytes and of structures in iCHS as Sinusoids. The latter are reclassified as Hepatocytes if they contain a Nucleus. Analogously, Nuclei of non Hepatocytes modify the classification of Hepatocytes into Sinusoids; Possible Nuclei within Sinusoids are turned to Nuclei of non Hepatocytes and Possible Nuclei within Hepatocytes are turned to Nuclei of non Hepatocytes. Similarly, rules exist which combine information on White Zones and Hepatocytes to reclassify some of these as Sinusoids.
On the whole, this refinement implies both the reclassification and the merging of structures as shown in Fig.3. For example, Fig.3 shows that the structure "a" of Fig.4 is reclassified as a Hepatocyte because of the presence of a Nucleus of Hepatocyte inside it. On the contrary, structure "b" is reclassified as a Sinusoid because it contains a Non-Hepatocyte Nucleus and is a white zone, as shown in Fig.7.
FusionStr is the last set of metarules in the subplan established by EvStr2. It merges the obtained descriptions, producing a global description (DescFus). At this point, MA5 gives back its results to SC1 and ceases to exist.SC1 transmits these results to MA4 and having no further action to perform, ceases to exist. MA4 has completed its activity, its plan is empty, so that the halt-condition in IntImage succeeds and MA4 returns its results to MA0 and ceases to exist. MA0 is in t
21
he state:
s04=<{iCHS,iCW,iCM,iCN,biopsy}, RGB°CN°DescCN°CES°CW°CM°DescCHS°DescCW°DescCM°DescFus §
CellStr ° Init ° # NucleiStr°NucSegmStr#NucDescStr# EvStr1 … # EvStr2… FusionStr� >
In this case FusionStr does not prescribe any further action. Now, since all the managers generated from MA0 have completed their actions, the halt condition is true (plan is empty). MA0 ceases to exist, outputting the set of obtained images and descriptions.
The example illustrates three control mechanisms which establish the strategy to be executed only when sufficient data are present and tailor it to these data.
- EvStr1 defines the necessary conditions to proceed from Phase 1 (segmentation and description of Nuclei) to Phase 2 (segmentation and description of candidate Hepatocytes and Sinusoids, white zones and edges).
- EvStr2 defines the conditions under which to refine the current state by the contextual evaluation of the already obtained results
- FusionStr defines how to synthesise the various results obtained during phase 1 and phase 2 to produce a final global understanding of the image.
5.2 Discussion of Results
The strategy described was proved on 20 images, obtained via a new staining technique. The idea was to automate part of the current procedure of interactive understanding with the aim of future screening.
These images are characterised by a fine grained texture and discontinuities in the edges of hepatocytes and sinusoids. This leads to the splitting of some structures into structures too small to be classified and into the merging of some separate structures (In Fig.4 structures a and b). This makes segmentation of the image difficult.
For example, the current automatic understanding process of Fig.1 singles out 42 objects, while the histologist, who collaborated to the experiment and from which the rules were obtained, singles out 44 objects. However, of these 31 objects, the procedure correctly classified 27 Hepatocytes and 9 Sinusoids. Of these, 26 Hepatocytes and 5 Sinusoids were correctly classified at the end of the first fragment of EvStr1 (Fig.4). EvStr2 recovered the classification of another hepatocyte and four sinusoids. The sinusoids are the structures b, c, d and e in Fig.3 and the hepatocyte is the structure a.
The structures which were missed correspond to image zones were the texture is fine grain an
22
d no program is able to recover reasonable structures. Note that in Fig.3 some structures were completed by FusionStr by filling the holes.
A further recovery is expected by the use of a new set of metarules MemCutStr which has been designed to exploit the edges produced by EdgStr (see Fig.8) to split structures whose area is above a certain threshold and which have to be examined to check if they represent entities which were fused due to discontinuities in the boundary. A preliminary exploration has shown how this strategy can exploit long edges in DescCM to separate fused structures in DescCHS. As an example, Fig.9 shows the edges found according to EdgStr in the zone around the neck of the structure "a" of Fig.4.
Figure 9. Edges in a window around the neck of structure a in Figure 2 (Larger size than original)
According to our biologist the 20 biopsies on which the present strategy was experimented contained 784 identifiable objects (642 hepatocytes and 142 sinusoids). Note that a different histologist, after discussing our criteria on the image of Fig.1, independently examined the other 19 images. This second histologist recognised 673 objects in the 19 images (against 740 by the first histologist). Of these, they agreed on 448 Hepatocytes (95.5%) and on 119 Sinusoids (90.1%).
On the 20 images the system singled out 706 objects, of which it agreed with histologist 1 on 467 hepatocytes (89.1%) and 163 sinusoids (89.5%). As to sinusoids, they are often split by the system due to the presence of other structures. So we found an excess of 21 sinusoids (14.7%).
It should be noticed that the ratio of agreement on hepatocytes and sinusoids is rather constant for all the 20 images, with a standard deviation of 3.29% for the recognition of Hepatocytes and of 3.50% for the recognition of Sinusoids.
The advantage of adaptation and dynamicity in the definition of strategies is highlighted by considering that, on the average, about 10% of the correctly classified Hepatocytes and 25% of the correctly classified Sinusoids are classified during the execution of the substrategy generated by EvStr2.
These results are interesting because they show that the weak points are in segmentation, even if our approach allows some recovery. The present procedure can be exploited in a semi-inter
23
active tool in which the histologist can perfect the segmentation interactively.
6. Related Work and Discussion
Our approach has the ability to dynamically adapt its plan according to progressively obtained results. This ability overcomes the rigidity deriving from the execution of a precomputed plan, which leaves to a post-mortem analysis the possibility of plan revision, as happens in VISIPLAN [GK95]. VISIPLAN is a system for the automatic generation of image analysis plans for images of a given class. It follows a hierarchical process, based on progressive recognition of the objects needed to reach a certain goal. The computed plans are then executed on images of the same class. VISIPLAN does not show reactive capabilities and plan revision can only be performed by a human expert, evaluating the obtained final results. VISIPLAN also exploits mutual constraints among types of characteristic structures, a technique also used to integrate different visual modules [Alo87].
A similar technique is used in our approach where the results produced by one modules are used to constrain the activity of other modules.
The planning of the understanding process also remain of limited flexibility in VISIONS [HR87], a system where a blackboard supports communication between the various knowledge sources ensuring the analysis of the image. The process is goal-directed and proceeds through the exploration of a hierarchy of schemas encapsulating the figurative and operative knowledge necessary to handle objects. The operative knowledge is represented in a procedural way.
In our approach moreover, the interleaving of segmentation and description phases allows a progressive refinement of a classification, involving resegmentation, and does not separate the segmentation phase from the high-level description one, as happens for example in [CDP93] or in [MH90]. In SIGMA [MH90], three expert modules are in charge of low level analysis, model selection and geometric reasoning; they communicate through requests. Knowledge is represented by means of frames and rules. The approach is goal-directed, the low level expert being driven by high level goals. However, there is no real reasoning operating at this level, but rather a selection and tuning of operators acting globally on the image, given a goal. A similar line is followed by Nagao and Matsuyama [NM80], whose system is blackboard-based. It obeys a fixed data-driven analysis strategy (pre-processing - region segmentation and classification - object detection). Split-and merge is used in case of failure of the interpretation process. There is no real integration between the two levels, and no real reasoning operating at the segmentation level.
In our approach the integration of data from different modules and sources is guided by high-level knowledge. This is due to the fact that reactive behaviour is constrained by an overall assessment of the situation and of the current goal, thus differentiating from purely reactive systems based on stimulus-response patterns, such as [AC87]. This makes our approach differ fro
24
m the one described in [NL84]. Nazif & LevineÕs system [NL84] shows a flexible integration of different levels of analysis, from segmentation to interpretation. The modules in these levels communicate by means of a short term memory storing current results and a long term memory storing knowledge about the objects, their relations, as well as condition-action pairs describing how to react to certain situations. However, problem solving strategy is data-driven (from an initial segmentation to the final interpretation), and feedback between interpretation and segmentation remains questionable.
Flexibility in plan management helps avoiding dead-ends deriving from the failures in reaching subgoals. Reactive features allow combination of local and centralised control. A similar aim is pursued for autonomous motion in [PRK90], where, however, plans are expressed in the form of overall goals. In our approach, instead, plans also provide constraints on the activity of executor agents, in the form of initial conditions on the executability of a strategy and of its specialisation.
As regards cooperation, the proposed model can be classified in the language of Zhang as realising a tree model of cooperation because one agent can delegate the realisation of parts of a strategy to others which later have to report to it [Zha92]. However, our problem is not that of generating plans to be subsequently applied, related to a defined search space [CT91], but rather that to dynamically adapt schemata of plans to situations reached during the understanding process, so that the search space can be enriched during the process.
In this way, our approach differ from the one in OCAPI [CT93], where the planning of programs is performed in a deliberative way exploiting knowledge on the available programs to select the appropriate ones and to adjust their parameters. The obtained results are analysed to assess their quality and the plan is defined in a hierarchical way. In OCAPI in fact, quality assessment is used to launch repairing mechanism whose role is to develop a new plan, as an answer to a failure in the execution of the plan.
Many of these systems show powerful capabilities as regards hypothetical reasoning, by considering the result of the understanding activity as uncertain and imprecise. For example, state and metareasoning are used in an approximate processing blackboard system by [DGH93]. Here, different levels of control are used and metalevel mechanisms are exploited to manage scheduling of actions and their specialisation. The authors exploit their architecture to produce approximate solutions for a vision problem, adequate to the constraints posed by real-time situations. The study of these problems can be a field of future study for our approach.
6.1 Generality of the proposed approach.
A feature of our approach also lies in the ability to formalise complex reasoning to generate situated understanding strategies. In fact, the types of sets of metarules MrStr and MrSeg, together with the possibility of placing in a plan names of metarule sets, define generic reasoning typ
25
es which, when combined, allow the generation of a variety of reasoning processes.
From the point of view of the theory of concurrent processes, the type MrStr distinguishes processes whose development may be expressed as a choice between several possible processes (i.e. the operator + in CCS [Mil89]), while the type MrSeg distinguishes processes whose development is expressed as the parallel composition of several processes (i.e. the operator | in CCS). Rules are instead used to denote immediate actions that allow the progress of the process (i.e. prefixes of processes in CCS). In CCS the algebra of processes is completed by the recursive operator. In our approach, the possibility of recursion is allowed by the possibility of partitioning into fragments plans expressed by a metarule. In particular, the last fragment may be occupied by the name of a set of metarules defining a continuation strategy. In case this name is the same as the one of the set of metarules from which it was placed in the plan, then a recursive strategy is possible. This recursive mechanism can be applied both in the case of assessment and of fine-tuning.
In more general terms, it is possible to see that MrStr defines "case-
of" reasoning: it specifies which strategy to develop, depending on the type of situation that is faced (for example a set of large objects, or small objects, or a combination of both). The choice is exclusive and each possible type is represented explicitly. The definition of such sets of metarules requires the ability to type a concrete situation with respect to a set of possible realisations; this activity specialises the type of situation to a set of different cases (the set of objects under consideration, already identified as such, may occur as including large, small, or both types of objects). MrStr allows the adaptation of the goal to be reached to the situation at hand.
On the contrary, MrSeg defines "for-
each" reasoning: it specifies a generic strategy to be applied (and specialised) for each element in a set. MrSeg requires the ability to cluster a (complex) situation into a set of elements (or zones) verifying a given predicate: for example, divide an image into textured and homogeneous zones, or into zones with impulsive, additive or multiplicative noise, and apply a different edge extraction action according to the type of noise, as realised by the strategy EdgeStr in Section 5, or divide a set of nuclei into large and small nuclei. It accounts for the variability of appearance of objects, in tuning the actions to a general goal.
The types of reasoning allowed by MrStr and MrSeg apply at both segmentation and interpretation levels, so that the "situations" at hand may be sets of physical primitives (points, sets of points) or sets of descriptions (nuclei, white zones, hepatocytesÉ). Therefore, goal driven and data driven strategies are interleaved in a natural way, as well as high and low analysis levels.
The introduction of continuation mechanisms, i.e. placing a set of metarules in the last fragment, defines the system as continuously working, unless some stopping condition is met, that comes from some evaluation somewhere. The end of the system life is not explicitly specified. T
26
he definition of the halt-condition as described in Section 4, however, guarantees that loops are avoided, so that the understanding process terminates.
As regards image understanding, continuation has well identified meanings:
¥ to fuse the results obtained during a given analysis phase, to obtain a synthetic representation of the phase result;
¥ to evaluate the consistency of the reached understanding of the image to proceed to eventual revision / feedback;
¥ to evaluate if the conditions are met to proceed with the next phase of analysis;
These three continuation modes, illustrated in Section 5, appear more generally relevant to modelling general problem solving tasks in AI.
From the point of view of the theory of sequential computations, it is possible to see that sets of metarules of the MrStr type allow the expression of the alternative construct (i.e. if-then-
else), while sets of metarules of the MrSeg type allow the expression of the iteration construct (i.e. for). It is well known that these two control constructs, together with the sequential construct allow the expression of any computation [BJ66]. The sequential control mechanism is defined in our approach by the existence of fragments.
7. Conclusions
The paper has presented an approach to image understanding which interleaves image segmentation and interpretation activities in a non-monotonic way. To this end it exploits a model of control based on a tight cooperation between deliberative and reactive control abilities.
The role of deliberative control is to account for the unpredictability of situations, by dynamically determining which operation to perform next, based on the results obtained so far and more generally on the state of the understanding process.
The role of reactive control is to account for the variability of local properties of the image by tuning the operation to subimages, each one being homogeneous with respect to a given operation.
A variable organisation of agents faces this variability. Agents operate within two different spaces: a goal space in which the understanding process state is evaluated and the plan deliberated, and a data space in which problem complexity is evaluated, and actions distributed and finely tuned.
This organisation allows the realisation of an actual strategy to result from a compromise between constraints set by the overall understanding goals and those set by the characteristics of
27
data at hand. The two control modes are integrated into a unified formalism describing segmentation and interpretation activities.
A feedback from high level interpretation tasks to low level segmentation tasks becomes possible and is exploited to recover wrong segmentations. Preliminary experiments in liver biopsy image understanding demonstrate the potential of the approach. Hypothetical reasoning has not been treated as such, but it has been shown how the proposed framework accommodates for revision of hypothetical classifications. Further experiments are foreseen to improve the proposed design. Future work would also include the study of a stronger inter-agent cooperation.
8. Acknowledgements
Work supported by the Galileo/Galil�e Project for France-Italy cooperation. Thanks to Centro Studi Medicina Teoretica for providing images, to Dr. E.Arosio of CSMT for coordinating and executing the experiment of biopsy manual classification, together with Dr. F.Grizzi.
9. References [ABM89]
A.Accomazzi, G.Bordogna, P.Mussio, A.Rampini, "An approach to heuristic exploitation of astronomers' knowledge in automatic interpretation of optical pictures", in Knowledge-based systems in astronomy, F. Murtagh ed., Springer-Verlag, pp.191-212, 1989
[AC87] P.E.Agre, D.Chapman, "Pengi: an implementation of a theory of activity", Proc.AAAI-87, pp.268-272, 1987
[Alo87] N.Aloimonos, Integration of Visual Modules, Academic Press, 1987 [BBS96]
N.Bianchi, P.Bottoni, C.Spinu, C.Garbay, P.Mussio, "A Dynamical Organisation for Situated Image Interpretation", Proc. 13th Int. Conf. on Pattern Recognition, vol. I, IEEE Computer Society Press, 1996, pp.228-232
[BCM95] P.Bottoni, U.Cugini, P.Mussio, C.Papetti, M.Protti, "A system for form feature based interpretation of technical drawings", Machine Vision and Application, vol.8, n.5, pp.326-335, 1995
[BCL96] P.Bottoni, M.F.Costabile, S.Levialdi, P.Mussio, "Visual Conditional Attributed Rewriting Systems in Visual Language Specification", Proceedings IEEE Symposium on Visual Languages, pp.156-163, 1996
[BD95] N.Bianchi, C.Diamantini, "Integration of Neural Networks and Rule Based Systems in the interpretation of liver biopsy images", in P.Barahona, M.Stefanelli, J.Wyatt eds., Artificial Intelligence in Medicine, Springer, pp.367-378, 1995
[Bia95] N.Bianchi, L'immagine Attiva come Strumento Informatico per la Ricerca Biomedica, PhD. Thesis in Science of Images in Biology and Medicine. (In Italian).
28
[BJ66] C.Boehm, G.Jacopini, "Flow diagrams, Turing Machines and Languages with only Two Formation Rules", Comm. of the ACM, vol.9, n.5, pp.366-371,1966.
[BMP94] P.Bottoni, P.Mussio, M.Protti "Metareasoning in the determination of image interpretation strategies", Pattern Recognition Letters, Vol. 15, pp.177-190, 1994
[Bra83] M.Brady, "Computational approach to image understanding", ACM Computing Surveys, vol.14, n.1, pp.3-73, 1983
[CDP93] G.Coppini, M.Demi, R.Poli, G.Valli, "An Artificial Vision System for X-Ray Images of Human Coronary Trees", IEEE PAMI-15, n.2, pp.156-162, 1993
[CT91] K.Currie, A.Tate, "O-plan: the open planning architecture", Artificial Intelligence, vol.52, n.1, pp.49-86, 1991
[CT93] V.Cl�ment, M.Thonnat, "Integration of image processing procedures, OCAPI: A knowledge-based approach", CVGIP: Image Understanding, vol.57, n.2, pp., 1993
[DGH93] K.Decker, A.Garvey, M.Humphrey, V.Lesser, "A Real-time Control Architecture for an Approximate Processing Blackboard System", International Journal of Pattern Recognition and Artificial Intelligence, vol.7, n.2, pp.265-284, 1995
[Fu82] K.S.Fu, Syntactic Pattern Recognition and Applications, Prentice-Hall, 1982 [GK95]
L.Gong, C.A.Kulikowski, "Composition of Image Analysis Processes Through Object-Centered Hierarchical Planning", IEEE PAMI, vol.17, pp.997-1099, 1995
[HR87] A.Hanson, E.Riseman, "The Visions Image-Understanding System". In C.M. Brown (ed.), Advances in Computer Vision, pp.1-114, Erlbaum Assoc., 1987.
[Mat89] T.Matsuyama, "Expert systems for image processing: knowledge-based composition of image analysis processes", Computer Vision , Graphics, and Image Processing, vol.48, pp.22-49, 1989
[May92] B.Mayoh, "Templates, fragments and skins", in G.Rozenberg, A.Salomaa, Lindenmayer Systems, Springer, 1992, pp.497-514
[MH90] T.Matsuyama, V.S.-S.Hwang, SIGMA: a knowledge-based aerial image understanding system, Plenum Press, 1990
[Mil89] R.Milner, Communication and Concurrency, Prentice-Hall, 1989 [MBE94]
P.Mussio, M.Bruno, F.Esposito, "Image Interpretation and Ambiguities", in Human and Machine Vision: Analogies and Divergencies, V.Cantoni ed., pp.319-338, Plenum 1994
[MPB91] P.Mussio, M.Pietrogrande, P.Bottoni, M.Dell'Oca, E.Arosio, E.Sartirana, M.R.Finanzon, N.Dioguardi, "Automatic cell count in digital images of liver tissue sections", Proc. 4th IEEE Symp. on CBMS, pp.153-160, 1991
29
[NL84] A.M.Nazif, M.D.Levine, "Low Level Image Segmentation: an Expert System". IEEE PAMI, 6 (5):555-577, 1984.
[NM80] M.Nagao, T.Matsuyama, A Structural Analysis of Complex Aerial Photographs, Plenum, 1980.
[PRK90] D.W.Payton, J.K.Rosenblatt, D.M.Keirsey, "Plan Guided Reaction", IEEE SMC, vol.20, n.6, 1370-1382, 1990
[Ros79] A.Rosenfeld, Picture Languages, Academic Press, 1979 [Ros88]
A.Rosenfeld, "Computer Vision: Basic Principles" Proceedings of the IEEE, vol.76, n.8, pp.863-868, 1988
[SGC95] C.Spinu, C.Garbay, J.M.Chassery, "A cooperative and adaptive approach to medical image segmentation", in P.Barahona, M.Stefanelli, J.Wyatt eds., Artificial Intelligence in Medicine, Springer, pp.379-390, 1995
[RSV93] F.Roli, S.B.Serpico, G.Vernazza, "Image recognition by integration of connectionist and symbolic approaches" International Journal of Pattern Recognition and Artificial Intelligence, vol.9, n.3, pp.495-514, 1995
[SVW84] H.Simon, K.Voss, K.Wenzelides, Automated Microscopic Image Analysis-Application in Experimental and Human Pathology, Experimental Pathology-Supplement 9, VEB Gustav Fischer Verlag Jena, 1984.
[Zha92] C.Zhang, "Cooperation under uncertainty in distributed expert systems", Artificial Intelligence, vol.56 , n.1, pp.21-69, 1992
APPENDIX A: application of a metarule
We show here the particular case of a typical metarule, which has the following standard format: a name, followed by the specification of the syntactic part of the rule and by the semantic part of the rule, as described in the following:
mrj: Syntactic part <<someContext, �,>; � r0 ° r1 ° mr2 # mr4>
Semantic part γ: SomeCondition(someContext)
F: {table(r0):=f1(someContext); table(r1):=f2(someContext);
table(mr2):= f3(someContext); table(mr4):= f4(someContext)}
where <someContext, �> is the antecedent. The antecedent is formed by a left context (someContext) and a string which must be rewritten into the consequent (the string of the only symbol �). Note that for metarule
30
s only a left context is indicated in the antecedent and the right context is always empty. The antecedent must occur in the description d and the condition γ must evaluate to true for the metarule to be applied. If this is the case, the consequent of the rule, i.e. the string � r0 ° r1 ° mr2 #
mr4 substitutes the symbol � in d. ° is the separator between names in the same fragment. The actual content of rules and metarules is specialised by the functions fi to the current situation. Fig.A1 shows how a plan originally constituted by the only name mri is successively rewritten generating a strategy. When the system applies mrj, it checks for the presence of someContext and � in d. If this is the case, functions fi are executed to generate the content of r0, r1, mr2 and mr4. Then, the plan is updated, rewriting � into �r0°r1°mr2#mr4.
r0, r1 and mr2 will be concurrently applied in the next step, belonging to a same fragment. Subsequently, as the last fragment is evaluated, metarule mr4 is used to establish how to rewrite the description d resulting from the execution of the two actions r0 and r1 and of the substrategy defined by mr2.
Fig.A1: Three successive steps in the unfolding of a strategy.
APPENDIX B: types of metarules
mrStr: Syntactic part (for i = 1,…,ns) <<someContexti, �, >; � subStri1 ° subStri2 ° … ° subStrin #>
Semantic part (for i = 1,…,ns, j=1,…,in) γi: SomeConditioni(someContexti)
F: {zij:=Zonej(someContexti); table(subStrij):=Specialise(zij, templSubStrij)}
where conditions γi are mutually exclusive, Zonej
r1 mr2 mr4# #
r2 r3
mr7 mr8#
… …
r0
mrj…someContext…$somePastActions ◊
…someContext…$somePastActions ◊
…someContext…$somePastActions ◊
…someContext…newResults$somePastActions r0 r1 mr2 ◊
31
is a function computing the zone relative to subStrij and Specialise is a function which generates a set of (meta-)rules specialised to the current zone zij, by instantiating a template templSubSt
rij. A template specifies a set of values of the attribute table up to the value of some parameters, which have to be specialised by a metarule.
mrSeg: Syntactic part <<someContext, �, > ; � segRules #>
Semantic part γ: SomeCondition(someContext)
F: { zones:= SomeFunction(someContext)
table(segRules) = Ø
foreach zone � zones
spec:= SomeFunction(zone)
table(segRules):=AddToTable(table(segRules),Specialise(templSegRules, spec))
endfor }
Thus the attribute table of segRules would result in a set of segmentation rules like:
segRules: Syntactic part (for i = 1, …, |zones|) <<SomeImage> ; i-thResult>
Semantic part (for i = 1, …, |zones|) γ: i-thCondition(SomeImage)
F: attr(i-thResult):= i-thFunction(value(SomeImage))
where attr is some designated attribute (for example the grey level map of an image), i-thFunction and i-thCondition derive from the evaluation of Specialise and i-thResult designates some new element of features (typically a new segmented image).
Formally, this is expressed by a set of metarules of the form:
mrRec Syntactic part: <<someContext, �, >; � ° … # ° mrRec>
Semantic part γ: SomeCondition(someContext)
F: SomeOperation(someContext)
APPENDIX C: Examples of metarules of type mrStr and of type mrSeg
EvStr2 is an example of a set of metarules of the type mrStr. In particular it denotes the following set
32
of metarules:
EvStr2: Syntactic part: <<DescCN°DescCW°DescCM , � ,> ; � °NucPresStr°WhitePresStr°MemCutStr# FusionStr>
<<DescCN°DescCW°DescCM , � ,> ; � °NucPresStr°WhitePresStr°# FusionStr>
<<DescCN°DescCW°DescCM , � ,> ; � °NucPresStr°MemCutStr# FusionStr>
<<DescCN°DescCW°DescCM , � ,> ; � °WhitePresStr°MemCutStr# FusionStr>
<<DescCN°DescCW°DescCM , � ,> ; � ° EvStr3>
<<DescCN°DescCW°DescCM , � ,> ; � ° EvStr4>
<<DescCN°DescCW°DescCM , � ,> ; � ° EvStr5>
<<DescCN°DescCW°DescCM , � ,> ; � ° EvStr6>
Semantic part: γ1: (NotEmpty(DescCN) � NotEmpty(DescCW) � NotEmpty(DescCM)
γ2: (NotEmpty(DescCN) � NotEmpty(DescCW) � Empty(DescCM)
γ3: (NotEmpty(DescCN) � Empty(DescCW) � NotEmpty(DescCM)
γ4: (Empty(DescCN) � NotEmpty(DescCW) � NotEmpty(DescCM)
γ5: (Empty(DescCN � NotEmpty(DescCW) � Empty(DescCM)
γ6: (NotEmpty(DescCN) � Empty(DescCW) � Empty(DescCM)
γ7: (Empty(DescCN) � Empty(DescCW) � NotEmpty(DescCM)
γ8: (Empty(DescCN) � Empty(DescCW) � Empty(DescCM)
F1: …
(We do not report the semantic rules which specify the construction of the different sets of metarules. A description is in [Bia95].)
The metarule set MemCutStr is an instance of type mrSeg, restricting the use of edges to zones of uncertainty on the classification. In particular it can be expressed as:
MemCutStr: Syntactic part <<DescCHS, �, >; � SplitRules# LocalFusion>
Semantic part γ: ExistBigAreas(DescCHS,threshold)
F: {zones=BigAreaZones(DescCHS,threshold);
table(splitRules)=Ø;
foreach zone � zones
table(splitRules)=AddTable(table(splitHepRules), Specialise(templSplitRules, zone);
endfor }
The set of rules resulting from specialisation of the templates is:
33
SplitHepRules: Syntactic part (for i=1, ..., |zones|) <<DescCHS°DescCM,λ,>; DescCHSiref>
Semantic part (for i=1, ..., |zones|) γi: ExistLongEdge(DescCM, zonei)
Fi: DescCHS= Split(DescCHS, DescCM, zonei)
