Training Examples

NoStrongHighMildRainD14YesWeakNormalHotOvercastD13YesStrongHighMildOvercastD12YesStrongNormalMildSunnyD11YesWeakNormalMildRainD10YesWeakNormalCoolSunnyD9NoWeakHighMildSunnyD8YesWeakNormalCoolOvercastD7NoStrongNormalCoolRainD6YesWeakNormalCoolRainD5YesWeakHighMildRain D4 YesWeakHighHotOvercastD3NoStrongHighHotSunnyD2NoWeakHighHotSunnyD1

Play GolfWindHumidityTemp.OutlookDay

Entropy and Information Gain

• Information answers questions• The more clueless I am about the answer initially,

the more information is contained in the final answer.

• Scale: – 1 = completely clueless – the answer to Boolean

question with prior <0.5,0.5>– 0 bit = complete knowledge – the answer to Boolean

question with prior <1.0,0.0>– ? = answer to Boolean question with prior <0.75,0.25>– The concept of Entropy

Entropy

• S is a sample of training examples

• p+ is the proportion of positive examples

• p- is the proportion of negative examples

• Entropy measures the impurity of S

Entropy(S) = -p+ log2 p+ - p- log2 p-

Information Gain• Gain(S,A): expected reduction in entropy due to

sorting S on attribute A

Gain(S,A)=Entropy(S) - vvalues(A) |Sv|/|S| Entropy(Sv)

Information Gain• Gain(S,A): expected reduction in entropy due to

sorting S on attribute A

Gain(S,A)=Entropy(S) - vvalues(A) |Sv|/|S| Entropy(Sv)

Training Examples

NoStrongHighMildRainD14YesWeakNormalHotOvercastD13YesStrongHighMildOvercastD12YesStrongNormalMildSunnyD11YesWeakNormalMildRainD10YesWeakNormalCoolSunnyD9NoWeakHighMildSunnyD8YesWeakNormalCoolOvercastD7NoStrongNormalCoolRainD6YesWeakNormalCoolRainD5YesWeakHighMildRain D4 YesWeakHighHotOvercastD3NoStrongHighHotSunnyD2NoWeakHighHotSunnyD1

Play GolfWindHumidityTemp.OutlookDay

Selecting the First Attribute

Humidity

High Normal

[3+, 4-] [6+, 1-]

S=[9+,5-]E=0.940

Gain(S,Humidity)=0.940-(7/14)*0.985 – (7/14)*0.592=0.151

E=0.985 E=0.592

Wind

Weak Strong

[6+, 2-] [3+, 3-]

S=[9+,5-]E=0.940

E=0.811 E=1.0

Gain(S,Wind)=0.940-(8/14)*0.811 – (6/14)*1.0=0.048

Humidity provides greater info. gain than Wind, w.r.t target classification.

Selecting the First Attribute

Outlook

Sunny Rain

[2+, 3-] [3+, 2-]

S=[9+,5-]E=0.940

Gain(S,Outlook)=0.940-(5/14)*0.971 -(4/14)*0.0 – (5/14)*0.971=0.247

E=0.971

E=0.971

Overcast

[4+, 0]

E=0.0

Selecting the First AttributeThe information gain values for the 4 attributes are:• Gain(S,Outlook) =0.247• Gain(S,Humidity) =0.151• Gain(S,Wind) =0.048• Gain(S,Temperature) =0.029

where S denotes the collection of training examples

Selecting the Next AttributeOutlook

Sunny Overcast Rain

Yes

[D1,D2,…,D14] [9+,5-]

Ssunny=[D1,D2,D8,D9,D11] [2+,3-]

? ?

[D3,D7,D12,D13] [4+,0-]

[D4,D5,D6,D10,D14] [3+,2-]

Gain(Ssunny , Humidity)=0.970-(3/5)0.0 – 2/5(0.0) = 0.970Gain(Ssunny , Temp.)=0.970-(2/5)0.0 –2/5(1.0)-(1/5)0.0 = 0.570Gain(Ssunny , Wind)=0.970= -(2/5)1.0 – 3/5(0.918) = 0.019

ID3 AlgorithmOutlook

Sunny Overcast Rain

Humidity

High Normal

Wind

Strong Weak

No Yes

Yes

YesNo

[D3,D7,D12,D13]

[D8,D9,D11] [D6,D14][D1,D2] [D4,D5,D10]

Which attribute should we start with?

ID# Texture Temp Size Classification

1 Smooth Cold Large Yes

2 Smooth Cold Small No

3 Smooth Cool Large Yes

4 Smooth Cool Small Yes

5 Smooth Hot Small Yes

6 Wavy Cold Medium No

7 Wavy Hot Large Yes

8 Rough Cold Large No

9 Rough Cool Large Yes

10 Rough Hot Small No

11 Rough Warm Medium Yes

Which node is the best?

• Texture (smooth,wavy,rough)5/11 * ( -4/5*log4/5 – 1/5*log1/5) +

2/11 * (-1/2*log1/2 – ½ *log1/2) +

4/11 * (-2/4*log2/4 – 2/4*log2/4)

= 5/11*(.722) + 2/11*1 + 4/11*1

= .874

Which node is the best?

• Temperature(cold,cool,hot,warm)4/11* ( -1/4*log1/4 – 3/4*log3/4) +

3/11 * (-3/3*log3/3 – 0/3 *log0/3) +

3/11 * (-2/3*log2/3 – 1/3 *log1/3) +

1/11 * (-1/1*log1/1 – 0/1*log0/1)

= 4/11*(.811) + 0 + 3/11*(.918) + 0

= .545

Which node is the best?

• Size (large,medium,small)5/11 * ( -4/5*log4/5 – 1/5*log1/5) +

2/11 * (-1/2*log1/2 – ½ *log1/2) +

4/11 * (-2/4*log2/4 – 2/4*log2/4)

= 5/11*(.722) + 2/11*1 + 4/11*1

= .874

Learning over time

• How do you evolve knowledge over time when you learn little bit by little bit?– Abstract version – the “Frinkle”

The Question

• The Question– How can we build this kind of representation

over time?

• The Answer– Rely on the concepts of false positives and false

negatives

The idea

• False Positive– An example which is predicted to be positive but whose

known outcome is negative– The problem is that our hypothesis is too general.– The solution is to add another condition to our hypothesis.

• False Negative– An example which is predicted to be negative but whose

known outcome is positive– The problem is that our hypothesis is too restrictive.– The solution is to remove a condition to our hypothesis [or

to add disjunction]

Creating a model one “case” at a time

ID# Texture Temp Size Classification

1 Smooth Cold Large Yes

2 Smooth Cold Small No

3 Smooth Cool Large Yes

4 Smooth Cool Small Yes

5 Smooth Hot Small Yes

6 Wavy Cold Medium No

7 Wavy Hot Large Yes

8 Rough Cold Large No

9 Rough Cool Large Yes

10 Rough Hot Small No

11 Rough Warm Medium Yes