Inductive Logic Programming

t.k.prasad@wright.edu

http://www.knoesis.org/tkprasad/

Includes slides by Luis Tari

CS774 1L16ILP

Logic Programming Consider the following example of a logic program:

parent_of(charles,george).

parent_of(george,diana).

parent_of(bob,harry).

parent_of(harry,elizabeth).

grandparent_of(X,Y) :- parent_of(X,Z), parent_of(Z,Y).

• Query: grandparent_of(X,Y)?• Answers:

• grandparent_of(charles,diana).• grandparent_of(bob,elizabeth).

• From the program, we can ask queries about grandparents.

CS774 2L16ILP

(Machine) Learning The process by which relatively permanent

changes occur in behavioral potential as a result of experience. (Anderson)

Learning is constructing or modifying representations of what is being experienced. (Michalski)

A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P, if its performance at tasks in T, as measured by P, improves with experience E. (Mitchell)

CS774 3L16ILP

Forms of Reasoning Deduction: From causes to effect (Prediction)

fact a, rule a => b INFER b (*First-order logic*)

Abduction: From effects to possible causes

(Explanation) rule a => b, observe b

AN EXPLANATION a

Induction: From correlated observations to rules

(Learning) observe correlation between a1, b1, ... an, bn

LEARN a -> b

CS774 4L16ILP

What is ILP?

Inductive Logic Programming (ILP) Automated learning of logic rules from examples

and background knowledge E.g., learn the rule for grandparents, given

background knowledge of parents and examples of grandparents

ILP can be used for classification and prediction

CS774 5L16ILP

Why ILP ? – multiple relations

Genealogy example: Given known relations…

father(Old,Young) and mother(Old,Young) male(Somebody) and female(Somebody)

…learn new relations parent(X,Y) :- father(X,Y). parent(X,Y) :- mother(X,Y). brother(X,Y) :-

male(X),father(Z,X),father(Z,Y).

Most ML techniques cannot use more than one relatione.g., decision trees, neural networks, …

CS774 6L16ILP

ILP – formal definitions

Given a logic program B representing background

knowledge a set of positive examples E+

a set of negative examples E-

Find hypothesis H such that:1. B U H e for every e E+.

2. B U H f for every f E-.

3. B U H is consistent.

Assume that B e for some e E+.

CS774 7L16ILP

ILP – logical foundation

Prolog = Programming with Logicis used to represent: Background knowledge (of the domain): facts Examples (of the relation to be learned): facts Theories (as a result of learning): rules

Supports two forms of logical reasoning Deduction Induction

CS774 8L16ILP

Logical reasoning: deduction

From rules to facts…

B T |- E

mother(penelope,victoria).mother(penelope,arthur).father(christopher,victoria).father(christopher,arthur).

parent(X,Y) :- father(X,Y).parent(X,Y) :- mother(X,Y).

parent(penelope,victoria).parent(penelope,arthur).parent(christopher,victoria).parent(christopher,arthur).

CS774 9L16ILP

Logical reasoning: induction

From facts to rules…

B E |- T

mother(penelope,victoria).mother(penelope,arthur).father(christopher,victoria).father(christopher,arthur).

parent(X,Y) :- father(X,Y).parent(X,Y) :- mother(X,Y).

parent(penelope,victoria).parent(penelope,arthur).parent(christopher,victoria).parent(christopher,arthur).

CS774 10L16ILP

Example

Background knowledge B: parent_of(charles,george). parent_of(george,diana). parent_of(bob,harry). parent_of(harry,elizabeth).

Positive examples E+:

grandparent_of(charles,diana). grandparent_of(bob,elizabeth).

Generate hypothesis H: grandparent_of(X,Y) :- parent_of(X,Z),

parent_of(Z,Y).

CS774 11L16ILP

L16ILP 12

Example: Same Generation

CS774

Why ILP ? - Structured data

Seed example of East-West trains (Michalski)

What makes a train to go eastward ?CS774 13L16ILP

Why ILP ? – multiple relations

This is related to structured data

Train Car

t1 c11

t1 c12

t1 c13

t1 c14

t2 c21

… …

Car Length Shape Axes Roof …

c11 short rectangle 2 none …

c12 long rectangle 3 none …

c13 short rectangle 2 peaked …

c14 long rectangle 2 none …

c21 short rectangle 2 flat …

… … … … … …

has_car car_properties

CS774 14L16ILP

Induction of a classifier: exampleExample of East-West trains B: relations has_car and car_properties

(length, roof, shape, etc.)

e.g., has_car(t1,c11) E: the trains t1 to t10 C: east, west

• Possible T: east(T) :-

has_car(T,C), length(C,short), roof(C,_).

CS774 15L16ILP

ILP systems Two of the most popular ILP systems:

Progol FOIL

Progol [Muggleton95] Developed by S. Muggleton et. al. Learns first-order Horn clauses (no negation in

head and body literals of hypotheses) FOIL [Quinlan93]

Developed by J. Quinlan et. al. Learns first-order rules (no negation in head

literals of the hypotheses)CS774 16L16ILP

Rule Learning (Intuition) How to come up with a rule for grandparent_of(X,Y)?

1. Take the example grandparent_of(bob,elizabeth).

2. Find the subset of background knowledge relevant to this example: parent_of(bob,harry), parent_of(harry,elizabeth).

3. Form a rule from these factsgrandparent_of(bob,elizabeth) :-

parent_of(bob,harry), parent_of(harry,elizabeth).

4. Generalize the rulegrandparent_of(X,Y) :- parent_of(X,Z), parent_of(Z,Y).

5. Check if this rule is valid w.r.t the positive and the negative examples

CS774 17L16ILP

Top-down induction of logic programs

Employs refinement operators Typical refinement operators on a clause:

Apply a substitution to clause Add a literal to the body of clause

Refinement graph: Nodes correspond to clauses Arcs correspond to refinements

CS774 18L16ILP

Part of refinement graph

has_a_daughter(X).

has_a_daughter(X) :- has_a_daughter(X) :- has_a_daughter(X) :-

male(Y). female(Y). parent(Y,Z).

has_a_daughter(X) :- has_a_daughter(X) :- has_a_daughter(X) :-

male(X). female(Y), parent(X,Z).

parent(S,T).

... ... ... ... ... has_a_daughter(X) :-

parent(X,Z),

female(U).

has_a_daughter(X) :-

parent(X,Z),

female(Z).

CS774 19L16ILP

Progol Algorithm Outline

1. From a subset of positive examples, construct the most specific rule rs.

2. Based on rs, find a generalized form rg of rs so that score(rg) has the highest value among all candidates.

3. Remove all positive examples that are covered by rg.

4. Go to step 1 if there are still positive examples that are not yet covered.

CS774 20L16ILP

Scoring hypotheses

score(r) is a measure of how well a rule r explains all the examples with preference given to shorter rules. pr = number of +ve examples correctly deducible

from r nr = number of -ve examples correctly deducible

from r cr = number of body literals in rule r

score(r) = pr – (nr + cr)

CS774 21L16ILP

Applications of ILP

Constructing Biological Knowledge Bases by Extracting Information from Text Sources (M. Craven & J. Kumlien) [Craven99]

The automatic discovery of structural principles describing protein fold space (A. Cootes, S.H. Muggleton, and M.J.E. Sternberg) [Cootes03]

More from UT-ML group (Ray Mooney) http://www.cs.utexas.edu/~ml/publication/ilp.html

CS774 22L16ILP

Example of relation from text

We want to extract the following relation:

Sample sentence from biomedical articles:

CS774 23L16ILP

References [Quinlan93] J. R. Quinlan, R. M. Cameron-Jones.

FOIL: A Midterm Report. Proceedings of Machine Learning: ECML-93

[Muggleton95] S. Muggleton. Inverse Entailment and Progol. New Generation Computing Journal, 13:245-286, 1995.

[Craven99] M. Craven & J. Kumlien (1999). Constructing Biological Knowledge Bases by Extracting Information from Text Sources. ISMB 99.

[Cootes03] A. Cootes, S.H. Muggleton, and M.J.E. Sternberg. The automatic discovery of structural principles describing protein fold space. Journal of Molecular Biology, 330(4):839-850, 2003.

CS774 24L16ILP