Upload
amber-york
View
223
Download
1
Embed Size (px)
Citation preview
23/4/19 223/4/19 EIE426-AICV 2
Contents
Machine learning concepts and procedures
Learning by recording cases
Learning by building identification trees
Simplification of decision rules
23/4/19 3
Machine Learning
Learning is based on coupling new information to previously acquired knowledge. Usually, a great deal of reasoning is involved.
(1) Learning by analyzing differences
(2) Learning by managing multiple models
(3) Learning by explaining experience
(4) Learning by correcting mistakes Learning is based on digging useful regularity out of data
(1) Learning by recording cases
(2) Learning by building identification trees
(3) Learning by training neural nets
(4) Learning by simulation evolution
Two kinds of learning:
EIE426-AICV
23/4/19 4
Learning by Recording Cases
The consistency heuristic: whenever you want to guess a property of something, given nothing
else to go on but a set of reference cases, find the most similar case, as measured by known properties, for which the property is known. Guess that the unknown property is the same as that known property.
This technique is good for problem domains in which good models are impossible to build.
The learning will do nothing to the information in the recorded cases until that information is used.
EIE426-AICV
23/4/19 7
Finding Nearest Neighbors
The straightforward way: calculate the distance to each other object and find the minimum among those distances. For n other objects, there are n distances to compute and (n-1) distance comparisons to do.
EIE426-AICV
Decision Trees
A decision tree is a representation
that is a semantic tree
in which
Each leaf node is connected to a set of possible answers.
Each non-leaf node is connected to a test that splits its set of
possible answers into subsets corresponding to different test
results.
Each branch carries a particular test result’s subset to another
node.
23/4/19 EIE426-AICV 8
K-D Tree
A k-d tree is a representationThat is a decision treeIn which
The set of possible answers consists of points, one of which may
be the nearest neighbor to a given point.
Each test specifies a coordinate, a threshold, and a neutral zone
around the threshold containing no points.
Each test divides a set of points into two sets, according to on
which side of the threshold each point lies.
23/4/19 EIE426-AICV 10
23/4/19 11
00
2
2
4
4
6
6
Height
Width
Red
Orange
Yellow
Purple
RedViolet
Blue
Green
2.00
U4.00
U
2.00
U
EIE426-AICV
K-D Tree(cont.)
23/4/19 12
Height > 3.5
Height > 5.5?
Orange
No
No
Width > 3.5
Yes
Width > 3.0
Height > 1.5? Height > 1.5?
Violet Red Green Blue
No
No
NoNo
Yes
Yes Yes
Red
Yes
Height > 5.5?
Purple Yellow
No
Yes
Yes
EIE426-AICV
K-D Tree(cont.)
K-D Tree (cont.)
To divide the cases into sets,
If there is only one case, stop.
If this is the first division of cases, pick the vertical axis for comparison;
otherwise, pick the axis that is different from the axis at the next higher level.
Considering only the axis of comparison, find the average position of the two
middle objects. Call this average position the threshold, and construct a
decision-tree test that compares unknowns in the axis of comparison against
the threshold. Also note the position of the two middle objects in the axis of
comparison. Call these positions the upper and lower boundaries.
Divide up all the objects into two subsets, according to on which side of the
average position they lie.
Divide up the objects in each subset, forming a subtree for each, using this
procedure.
23/4/19 EIE426-AICV 13
23/4/19 14
To find the nearest neighbor using the K-D procedure,
Determine whether there is only one element in the set under
consideration.
If there is only one, report it.
Otherwise, compare the unknown, in the axis of comparison,
against the current node’s threshold. The result determines the
likely set.
Find the nearest neighbor in the likely set using this procedure.
Determine whether the distance to the nearest neighbor in the
likely set is less than or equal to the distance to the other set’s
boundary in the axis of comparison:
If it is, then report the nearest neighbor in the likely set.
If it is not, check the unlikely set using this procedure; return
the nearer of the nearest neighbors in the likely set and in
the unlikely set.
EIE426-AICV
K-D Tree(cont.)
23/4/19 15
Learning by Building Identification Trees
Identification-tree building is the most widely used learning method. Thousands of practical identification trees, for applications ranging from medical diagnosis to process control, has been built using the method.
EIE426-AICV
23/4/19 16
From Data to Identification Trees
Name Hair Height Weight Lotion ResultSarah blonde average light no sunburnedDana blonde tall average yes noneAlex brown short average yes noneAnnie blonde short average no sunburnedEmily red average heavy no sunburnedPete brown tall heavy no noneJohn brown average heavy no noneKatie blonde short light yes none
EIE426-AICV
23/4/19 17
An identification tree is a representation
That is a decision tree
In Which Each set of possible conclusions is established implicitly by a
list of samples of known class.
In the table, there are 3 x 3 x 3 x 2 = 54 possible combinations. The probability of an exact match with someone already observed is 8/54. It can be impractical to classify an unknown object by looking for an exact match.
EIE426-AICV
From Data to Identification Trees (cont.)
23/4/19 18
Height
TallAverage
Short
Dana PeteWeight
Sarah
LightAverage
Heavy
Hair color
Blonde RedBrown
Alex
Weight
Light AverageHeavy
Katie Annie
Hair
BlondeRed Brown
Emily John
EIE426-AICV
Identification Tree
23/4/19 19
The world is inherently simple. Therefore the smallest identification tree that is consistent with the samples is the one that is most likely to identify unknown objects correctly.
Which is the right identification tree?
How can you construct the smallest identification tree?
EIE426-AICV
Identification Tree (cont.)
23/4/19 21EIE426-AICV
Tests Should Minimize Disorder (cont.)
Hair Color: Blonde4 Samples: Sarah, Dana, Annie, Katie
23/4/19 22
Information Theory Supplies a Disorder Formula
Where
nb is the number of samples in branch b,
nt is the total number of samples in all branches,
nbc is the number of samples in branch b of class c.
c b
bc
b
bc
n
n
n
n)(logDisorder 2
b c b
bc
b
bc
t
b
n
n
n
n
n
n)(log)(disorder Average 2
EIE426-AICV
Disorder Formula
23/4/19 EIE426-AICV 23
For two classes, A and B:
If they are perfectly balanced, that is, nbc = 0.5 (c=1,2), then
1 2
1log
2
1
2
1log
2
1
)(logDisorder
22
2
c b
bc
b
bc
n
n
n
n
If there are only A’s or only B’s (perfect homogeneity), then
0
0log01log1
)(logDisorder
22
2
c b
bc
b
bc
n
n
n
n
23/4/19 24
As it moves from perfect homogeneity to perfect balance, disorder varies smoothly between zero and one.
EIE426-AICV
Disorder Measure
23/4/19 25
Test Disorder
Hair 0.5
Height 0.69
Weight 0.94
Lotion 0.61
The first test:
Thus, the hair-color test is the winner.
5.008
30
8
1
4
2log
4
2
4
2log
4
2
8
4
color)-(hairDisorder Average
22
EIE426-AICV
Disorder Measure (cont.)
23/4/19 26
Test Disorder
Height 0.5
Weight 1
Lotion 0
Once the hair test is selected, the choice of another test to separate out the sunburned people from among Sarah, Dana, Annie, and Katie is decided by the following calculations:
Thus, the lotion-used test is the clear winner.
EIE426-AICV
Disorder Measure (cont.)
Identification Tree AlgorithmTo generate an identification tree using SPROUTER,
Until each leaf node is populated by as homogeneous a sample
set as possible:
Select a leaf node with an inhomogeneous sample set.
Replace that leaf node by a test node that divides the
inhomogeneous sample set into minimally inhomogeneous
subsets, according to some measure of disorder.
23/4/19 EIE426-AICV 27
23/4/19 28
From Trees to Rules
If the person’s hair color is blonde and
the person uses lotion,
then nothing happens.
If the person’s hair color is blonde and
the person uses no lotion,
then the person turns red.
If the person’s hair color is red,
then the person turns red.
If the person’s hair color is brown,
then nothing happens.
EIE426-AICV
23/4/19 29
Unnecessary Rule Antecedents Should be Eliminated
If the person’s hair color is blonde and
the person uses lotion.
then nothing happens.
If the person uses lotion,then nothing happens.
EIE426-AICV
23/4/19 30
Contingency Table
No change SunburnedPerson is blonde 2 0Person is not blonde 1 0
The first antecedent can be eliminated.
No change SunburnedPerson uses lotion 2 0Person uses no lotion 0 2
The second antecedent cannot be eliminated.
EIE426-AICV
Keep the 2nd antecedent
Samples:DanaAlexKatie
Keep the 1stantecedent
Samples:SarahDanaAnnieKatie
23/4/19 31
If the person’s hair color is blondethe person does not use lotion
then the person turns red
No change Sunburned Person is blonde 0
2Person is not blonde 2 1
The first antecedent cannot be eliminated.
No change SunburnedPerson uses no lotion 0 2Person uses lotion 2 0
The second antecedent cannot be eliminated either.
EIE426-AICV
Contingency Table (cont.) Keep the 2nd antecedent
Samples:Sarah, AnnieEmily, PeteJohn
Keep the 1stantecedent
Samples:SarahDanaAnnieKatie
23/4/19 32
If the person’s hair color is red,then the person turns red.
No change SunburnedPerson is red haired 0 1Person is not red haired 5 2
The antecedent cannot be eliminated.
If the person’s hair color is brown,then nothing happens.
No change SunburnedPerson is brown haired 3 0Person is not brown haired 2 3
The antecedent cannot be eliminated.
EIE426-AICV
Contingency Table (cont.) No antecedent
All 8 samples are considered.
No antecedent
All 8 samples are considered.
23/4/19 33
Unnecessary Rules Should be Eliminated
If the person’s hair color is blonde and
the person uses no lotion,
then the person turns red. ----- Rule 1
If the person uses lotion,
then nothing happens. ----- Rule 2
If the person’s hair color is red,
then the person turns red. ----- Rule 3
If the person’s hair color is brown,
then nothing happens. ----- Rule 4
EIE426-AICV
Default Rules and Tie BreakerDefault rule:
If no other rule applies,
then the person turns red, ----- Rule 5
or
If no other rule applies,
then nothing happens. ----- Rule 6
23/4/19 EIE426-AICV 34
Choose the default rule to minimize the total number of rules.
Tie breaker 1: Choose the default rule that covers the most common consequent in the sample set.
Rule 6 is used together with Rules 1 and 3.
Tie breaker 2: Choose the default rule that produces the simplest rules.
Rule 5 is used together with Rules 2 and 4.
Rule Generation Algorithm
To generate rules from an identification tree using PRUNER,
Create one rule for each root-to-leaf path in the identification
tree.
Simplify each rule by discarding antecedents that have no effect
on the conclusion reached by the rule.
Replace those rules that share the most common consequent by
a default rule that is triggered when on other rule is triggered
(eliminating as many other rules as possible). In the event of a
tie, use some heuristic tie breaker to choose a default rule.
23/4/19 EIE426-AICV 35