Upload
others
View
24
Download
0
Embed Size (px)
Citation preview
Introduction to Machine Learning
Introduction to Machine Learning Amo G. Tong 1
Lecture 8Supervised Learning
• Artificial Neural Networks
• Backpropagation Algorithm
• Some materials are courtesy of Vibhave Gogate, Eric Xing and Tom Mitchell.
• All pictures belong to their creators.
Introduction to Machine Learning Amo G. Tong 2
Introduction to Machine Learning Amo G. Tong 3
Supervised Learning
• < 𝑥, 𝑦 >
• Estimate Pr[𝑦|𝑥] by Bayesian Theory: Pr 𝑦 𝑥 =Pr 𝑥 𝑦 Pr 𝑦
Pr[𝑥]
• Naïve Bayes.
• Estimate Pr[𝑦|𝑥] directed by assuming a certain form
• Logistic regression.
• Assume the form of 𝑦 = 𝑓(𝑥)
• Error-driven.
Pr 𝑥 𝑦 and Pr 𝑦 are hard to compute.No good form for Pr[𝑦|𝑥]No prior Knowledge on 𝑓(𝑥).
You can try Neural Networks.
Introduction to Machine Learning Amo G. Tong 4
Neural Networks
• Input 𝑥, output 𝑦
• Given 𝑥, how to compute 𝑦?
• 𝑥 and 𝑦 are generally real vectors.
perceptronSigmoid unit
Introduction to Machine Learning Amo G. Tong 5
Neural Networks
• Input 𝑥, output 𝑦
• Given 𝑥, how to compute 𝑦?
• 𝑥 and 𝑦 are generally real vectors.
Introduction to Machine Learning Amo G. Tong 6
Neural Networks
• Input 𝑥, output 𝑦
• Given 𝑥, how to compute 𝑦?
• 𝑥 and 𝑦 are generally real vectors.
• Many neuron-like units (perceptron, sigmoid)• Weighted interconnections between units.• Parallel distributed process.
Introduction to Machine Learning Amo G. Tong 7
Neural Networks
• Input 𝑥, output 𝑦
• Given 𝑥, how to compute 𝑦?
• 𝑥 and 𝑦 are generally real vectors.
• Design a neural network:
• What is the function within each unit?
• How are the units connected?
Introduction to Machine Learning Amo G. Tong 8
Neural Networks
• Input 𝑥, output 𝑦
• Given 𝑥, how to compute 𝑦?
• 𝑥 and 𝑦 are generally real vectors.
• Units:
• Input Unit
• Processing Units
• Receive input from other units
• Output result to other units
• Edges
• weights
Introduction to Machine Learning Amo G. Tong 9
Examples
• Linear function 𝑦 = 𝜔1𝑥1 +⋯+𝜔𝑛𝑥𝑛
• Input (𝑥1, … , 𝑥𝑛) output 𝑦.
𝑥1
sum𝑥2
𝑥𝑛
…
𝜔1
𝜔2
𝜔𝑛
Introduction to Machine Learning Amo G. Tong 10
Examples
• Quadratic function 𝑦 = (𝜔11𝑥1 + 𝜔12𝑥2)(𝜔21𝑥1 + 𝜔22𝑥2)
• Input (𝑥1, … , 𝑥𝑛) output 𝑦.
𝑥1sum
𝑥2
multiply
𝜔11
𝜔12
1
sum
𝜔21
𝜔22
1
Introduction to Machine Learning Amo G. Tong 11
Train a Neural Network
• Define the error 𝐸
• Update the weight 𝜔 of each edge to minimize 𝐸
• 𝜔 ← 𝜔 − 𝜂𝜕 𝐸
𝜕 𝜔(delta rule)
• Batch Mode
• E: the total error among training data.
• Consider all the training data each time.
• Incremental Mode
• E: the error for one training instance.
• Consider training instances one by one.
Calculate 𝜕 𝐸
𝜕 𝜔
Introduction to Machine Learning Amo G. Tong 12
Train a Neural Network
• Define the error 𝐸
• Update the weight 𝜔 of each edge to minimize 𝐸
• 𝜔 ← 𝜔 − 𝜂𝜕 𝐸
𝜕 𝜔(delta rule)
• Incremental Mode
• E: the error for one training instance.
• Consider training instances one by one.
Calculate 𝜕 𝐸
𝜕 𝜔
Introduction to Machine Learning Amo G. Tong 13
Neural Networks
• One layer of sigmoid with one output
• Two layers of single functions
• Two layers of multiple sigmoid units with one output.
• Two layers of multiple sigmoid units with multiple outputs.
• Goal: how to find the parameters that can minimize the error.
Introduction to Machine Learning Amo G. Tong 14
Neural Networks
• One layer of sigmoid with one output
𝑥1
sigmoid𝑥2
𝑥𝑛
…
𝜔1
𝜔2
𝜔𝑛
𝜕𝐸𝑥𝜕 𝜔𝑖
= −(𝑡𝑥 − 𝑜𝑥)𝜕𝑜𝑥𝜕 𝜔𝑖
= −(𝑡𝑥 − 𝑜𝑥) ⋅ 𝑥𝑖⋅ 𝑜𝑥(1 − 𝑜𝑥)
𝑜 𝑥1, … , 𝑥𝑛 =1
1 + 𝑒−𝝎⋅𝒙
𝑜𝑥
Introduction to Machine Learning Amo G. Tong 15
Neural Networks
• Multilayers
• Update 𝜔1 or 𝜔2 first?
𝑥 𝑓 𝑔𝜔1 𝜔2
𝑓 𝜔1𝑥 𝑔 𝜔2𝑓 𝜔1𝑥
𝜕𝐸
𝜕 𝜔1= −(𝑡𝑥 − 𝑔)
𝜕𝑔
𝜕 𝜔1𝜕𝑔
𝜕 𝜔1=
𝜕𝑔
𝜕𝜔2𝑓
𝜕𝜔2𝑓
𝜕𝜔1
=𝜕𝑔
𝜕𝜔2𝑓𝜔2
𝜕𝑓
𝜕𝜔1
=𝜕𝑔
𝜕𝜔2𝑓𝜔2
𝜕𝑓
𝜕𝜔1𝑥
𝜕𝜔1𝑥
𝜕𝜔1
=𝜕𝑔
𝜕𝜔2𝑓𝜔2
𝜕𝑓
𝜕𝜔1𝑥𝑥
𝜕𝐸
𝜕 𝜔2= −(𝑡𝑥 − 𝑔)
𝜕𝑔
𝜕 𝜔2𝜕𝑔
𝜕 𝜔2=
𝜕𝑔
𝜕𝜔2𝑓
𝜕𝜔2𝑓
𝜕𝜔2=
𝜕𝑔
𝜕𝜔2𝑓𝑓
𝐸 =1
2|𝑡𝑥 − 𝑔 ቚ
2(𝑥, 𝑡𝑥)
Introduction to Machine Learning Amo G. Tong 16
Neural Networks
• Two layers of multiple sigmoid units with one output.
𝑥1 sigmoid
𝑥2
𝑥𝑖
…
sigmoidsigmoid
sigmoid
…
today’s weather tomorrow’s temp (normalized)
𝑥
𝑡𝑥 = 𝑓(𝑥)
𝑓(𝑥, 𝜔)
𝜔: weight on each edge
𝑜𝑥
For each 𝑥 in 𝐷
𝐸𝑥 =1
2𝑡𝑥 − 𝑜𝑥
2
Introduction to Machine Learning Amo G. Tong 17
Neural Networks
• Two layers of multiple sigmoid units with one output.
𝑥1 sigmoid
𝑥2
𝑥𝑖
…
sigmoidsigmoid
sigmoid
…
𝑜𝑥
𝜔𝑗
ℎ𝑥1
ℎ𝑥2
ℎ𝑥𝑗
For each 𝑥 in 𝐷
𝐸𝑥 =1
2𝑡𝑥 − 𝑜𝑥
2
𝜕𝐸𝑥
𝜕 𝜔𝑗= − 𝑡𝑥 − 𝑜𝑥
𝜕𝑜𝑥
𝜕 𝜔𝑗= − 𝑡𝑥 − 𝑜𝑥 ⋅ ℎ𝑥
𝑗⋅ 𝑜𝑥(1 − 𝑜𝑥)
Introduction to Machine Learning Amo G. Tong 18
Neural Networks
• Two layers of multiple sigmoid units with one output.
𝑥1 sigmoid
𝑥2
𝑥𝑖
…
sigmoidsigmoid
sigmoid
…
𝑜𝑥
𝜔𝑖,𝑗
ℎ𝑥1
ℎ𝑥2
ℎ𝑥𝑗
For each 𝑥 in 𝐷
𝐸𝑥 =1
2𝑡𝑥 − 𝑜𝑥
2
𝜕𝐸𝑥
𝜕 𝜔𝑖,𝑗= − 𝑡𝑥 − 𝑜𝑥
𝜕𝑜𝑥
𝜕𝜔𝑖,𝑗= − 𝑡𝑥 − 𝑜𝑥
𝜕𝑜𝑥
𝜕 ℎ𝑥𝑗
𝜕ℎ𝑥𝑗
𝜕𝜔𝑖,𝑗
𝑜𝑥 =1
1 + 𝑒−∑𝜔𝑗⋅ℎ𝑥𝑗
𝜔𝑗
Introduction to Machine Learning Amo G. Tong 19
Neural Networks
• Two layers of multiple sigmoid units with one output.
𝑥1 sigmoid
𝑥2
𝑥𝑖
…
sigmoidsigmoid
sigmoid
…
𝑜𝑥
𝜔𝑖,𝑗
ℎ𝑥1
ℎ𝑥2
ℎ𝑥𝑗
For each 𝑥 in 𝐷
𝐸𝑥 =1
2𝑡𝑥 − 𝑜𝑥
2
𝜕𝑜𝑥
𝜕 ℎ𝑥𝑗= 𝜔𝑗 ⋅ 𝑜𝑥(1 − 𝑜𝑥)
𝑜𝑥 =1
1 + 𝑒−∑𝜔𝑗⋅ℎ𝑥𝑗
𝜕ℎ𝑥𝑗
𝜕𝜔𝑖,𝑗= 𝑥𝑖 ⋅ ℎ𝑥
𝑗(1 − ℎ𝑥
𝑗)
𝜔𝑗
Introduction to Machine Learning Amo G. Tong 20
Neural Networks
• Two layers of multiple sigmoid units with one output.
𝑥𝑖 sigmoidsigmoid𝑜𝑥𝜔𝑖,𝑗
ℎ𝑥𝑗
𝜕𝐸𝑥
𝜕 𝜔𝑖,𝑗= − 𝑡𝑥 − 𝑜𝑥 𝜔𝑗 ⋅ 𝑜𝑥(1 − 𝑜𝑥) 𝑥𝑖 ⋅ ℎ𝑥
𝑗(1 − ℎ𝑥
𝑗)
𝜔𝑗
𝜕𝐸𝑥
𝜕 𝜔𝑗= − 𝑡𝑥 − 𝑜𝑥 ⋅ 𝑜𝑥(1 − 𝑜𝑥) ⋅ ℎ𝑥
𝑗
𝛿𝑥𝑜 = 𝑡𝑥 − 𝑜𝑥 𝑜𝑥(1 − 𝑜𝑥)
𝛿𝑥ℎ,𝑗
= ℎ𝑥𝑗1 − ℎ𝑥
𝑗𝜔𝑗𝛿𝑥
𝑜
𝜕𝐸𝑥𝜕 𝜔𝑗
= −ℎ𝑥𝑗𝛿𝑥𝑜
𝜕𝐸𝑥𝜕 𝜔𝑖,𝑗
= −𝑥𝑖𝛿𝑥ℎ,𝑗
𝜔 ← 𝜔 + Δ𝜔
Δ𝜔 = −𝜂𝜕 𝐸
𝜕 𝜔Δ𝜔j = 𝜂 ℎ𝑥
𝑗𝛿𝑥𝑜
Δ𝜔𝑖,j = 𝜂 𝑥𝑖𝛿𝑥ℎ,𝑗
Introduction to Machine Learning Amo G. Tong 21
Neural Networks
• Two layers of multiple sigmoid units with one output.
𝑥1 sigmoid
𝑥2
𝑥𝑖
…
sigmoidsigmoid
sigmoid
…
𝑜𝑥
𝜔𝑖,𝑗
ℎ𝑥1
ℎ𝑥2
ℎ𝑥𝑗
𝜔𝑗
𝜔 ← 𝜔 + Δ𝜔
Δ𝜔j = 𝜂 ℎ𝑥𝑗𝛿𝑥𝑜
Δ𝜔𝑖,j = 𝜂 𝑥𝑖𝛿𝑥ℎ,𝑗
ℎ𝑥𝑗
and 𝑥𝑖, input of the edge
𝛿𝑥𝑜 and 𝛿𝑥
𝑗, error term
associated with the units
𝛿 = −𝜕𝐸
𝜕 𝑢𝑛𝑖𝑡
𝑎 𝑏𝜔 Input from 𝑎
Error term 𝜕 𝐸
𝜕 𝑏on 𝑏
Δ𝜔 = 𝜂𝑎𝜕 𝐸
𝜕 𝑏
Introduction to Machine Learning Amo G. Tong 22
Neural Networks
• Two layers of multiple sigmoid units with one output.
𝑥1 sigmoid
𝑥2
𝑥𝑖
…
sigmoidsigmoid
sigmoid
…
𝑜𝑥
𝜔𝑖,𝑗
ℎ𝑥1
ℎ𝑥2
ℎ𝑥𝑗
𝜔𝑗
𝜔 ← 𝜔 + Δ𝜔
Δ𝜔j = 𝜂 ℎ𝑥𝑗𝛿𝑥𝑜
Δ𝜔𝑖,j = 𝜂 𝑥𝑖𝛿𝑥ℎ,𝑗
𝛿𝑥𝑜 = 𝑡𝑥 − 𝑜𝑥 𝑜𝑥(1 − 𝑜𝑥)
𝛿𝑥ℎ,𝑗
= ℎ𝑥𝑗1 − ℎ𝑥
𝑗𝜔𝑗𝛿𝑥
𝑜
For each 𝑥 in 𝐷
Forward, calculate ℎ𝑥𝑗
and 𝑜𝑥Backward, calculate 𝛿𝑥
𝑜 and 𝛿𝑥ℎ,𝑗
Update 𝜔𝑗 and 𝜔𝑖,𝑗
Backpropagation Framework
Introduction to Machine Learning Amo G. Tong 23
Neural Networks
• Two layers of multiple sigmoid units with multiple outputs.
𝑥1 sigmoid
𝑥2
𝑥𝑖
…
sigmoid
sigmoid
sigmoid
sigmoid
sigmoid
… …
𝑜𝑥1
𝑜𝑥2
𝑜𝑥𝑘
today’s weather tomorrow’s weather (normalized)
𝑥 𝑡𝑥 = 𝑓(𝑥)𝑓(𝑥, 𝜔)
𝜔: weight on each edge
ℎ𝑥1
ℎ𝑥2
ℎ𝑥𝑗
Introduction to Machine Learning Amo G. Tong 24
Neural Networks
• Two layers of multiple sigmoid units with multiple outputs.
𝑥1 sigmoid
𝑥2
𝑥𝑖
…
sigmoid
sigmoid
sigmoid
sigmoid
sigmoid
… …
𝑜𝑥1
𝑜𝑥2
𝑜𝑥𝑘
ℎ𝑥1
ℎ𝑥2
ℎ𝑥𝑗
𝜔𝑖,𝑗
For each 𝑥 in 𝐷
𝐸𝑥 =1
2
𝑘
𝑡𝑥𝑘 − 𝑜𝑥
𝑘 2 𝑎 𝑏𝜔 Input from 𝑎
Error term 𝜕 𝐸
𝜕 𝑏on 𝑏
Δ𝜔 = 𝜂𝑎𝜕 𝐸
𝜕 𝑏
𝜔𝑗,𝑘
Introduction to Machine Learning Amo G. Tong 25
Neural Networks
• Two layers of multiple sigmoid units with multiple outputs.
𝑥1 sigmoid
𝑥2
𝑥𝑖
…
sigmoid
sigmoid
sigmoid
sigmoid
sigmoid
… …
𝑜𝑥1
𝑜𝑥2
𝑜𝑥𝑘
ℎ𝑥1
ℎ𝑥2
ℎ𝑥𝑗
𝜔𝑖,𝑗 𝜔𝑗,𝑘
For each 𝑥 in 𝐷
𝐸𝑥 =1
2
𝑘
𝑡𝑥𝑘 − 𝑜𝑥
𝑘 2
One ouput: 𝛿𝑥𝑜 = 𝑡𝑥 − 𝑜𝑥 𝑜𝑥(1 − 𝑜𝑥)
Multiple output: 𝛿𝑥𝑜,𝑘 = 𝑡𝑥
𝑘 − 𝑜𝑥𝑘 𝑜𝑥
𝑘(1 − 𝑜𝑥𝑘)
Δ𝜔𝑗,𝑘 = 𝜂 ℎ𝑥𝑗𝛿𝑥𝑜,𝑘
Introduction to Machine Learning Amo G. Tong 26
Neural Networks
• Two layers of multiple sigmoid units with multiple outputs.
sigmoid sigmoid
sigmoid
sigmoid
sigmoid
sigmoid
… …
𝑜𝑥1
𝑜𝑥2
𝑜𝑥𝑘
ℎ𝑥1
ℎ𝑥2
ℎ𝑥𝑗
One ouput: 𝛿𝑥ℎ,𝑗
= ℎ𝑥𝑗1 − ℎ𝑥
𝑗𝜔𝑗 𝛿𝑥
𝑜
Multiple output: 𝛿𝑥ℎ,𝑗
= ℎ𝑥𝑗1 − ℎ𝑥
𝑗 ∑𝑘𝜔𝑗,𝑘𝛿𝑥𝑜,𝑘
If not fully connected, sum over the connected units.
Introduction to Machine Learning Amo G. Tong 27
Neural Networks
• Two layers of multiple sigmoid units with multiple outputs.
𝑥1 sigmoid
𝑥2
𝑥𝑖
…
sigmoid
sigmoid
sigmoid
sigmoid
sigmoid
… …
𝑜𝑥1
𝑜𝑥2
𝑜𝑥𝑘
ℎ𝑥1
ℎ𝑥2
ℎ𝑥𝑗
𝜔𝑖,𝑗
Δ𝜔𝑖,𝑗 = 𝜂 𝑥𝑖 𝛿𝑥ℎ,𝑗
One ouput: 𝛿𝑥ℎ,𝑗
= ℎ𝑥𝑗1 − ℎ𝑥
𝑗𝜔𝑗 𝛿𝑥
𝑜
Multiple output: 𝛿𝑥ℎ,𝑗
= ℎ𝑥𝑗1 − ℎ𝑥
𝑗 ∑𝑘𝜔𝑗,𝑘𝛿𝑥𝑜,𝑘
𝜔𝑗,𝑘
For each 𝑥 in 𝐷
𝐸𝑥 =1
2
𝑘
𝑡𝑥𝑘 − 𝑜𝑥
𝑘 2
Introduction to Machine Learning Amo G. Tong 28
Neural Networks
• Two layers of multiple sigmoid units with multiple outputs.
𝜔 ← 𝜔 + Δ𝜔
Δ𝜔𝑗,𝑘 = 𝜂 ℎ𝑥𝑗𝛿𝑥𝑜,𝑘
Δ𝜔𝑖,𝑗 = 𝜂 𝑥𝑖 𝛿𝑥ℎ,𝑗
𝛿𝑥𝑜,𝑘 = 𝑡𝑥
𝑘 − 𝑜𝑥𝑘 𝑜𝑥
𝑘 1 − 𝑜𝑥𝑘
𝛿𝑥ℎ,𝑗
= ℎ𝑥𝑗1 − ℎ𝑥
𝑗 ∑𝑘𝜔𝑗,𝑘𝛿𝑥𝑜,𝑘
For each 𝑥 in 𝐷
Forward, calculate ℎ𝑥𝑗
and 𝑜𝑥𝑘
Backward, calculate 𝛿𝑥𝑜,𝑘 and 𝛿𝑥
ℎ,𝑗
Update 𝜔𝑗,𝑘 and 𝜔𝑖,𝑗
Backpropagation Framework
𝑥1 sigmoid
𝑥2
𝑥𝑖
…
sigmoid
sigmoid
sigmoid
sigmoid
sigmoid
… …
𝑜𝑥1
𝑜𝑥2
𝑜𝑥𝑘
ℎ𝑥1
ℎ𝑥2
ℎ𝑥𝑗
𝜔𝑖,𝑗 𝜔𝑗,𝑘
Introduction to Machine Learning Amo G. Tong 29
Neural Networks
• Two layers of multiple sigmoid units with multiple outputs.
• Initialize the weights with random values.
• Do until converge
• For each 𝑥 ∈ 𝐷
• (Forward) Calculate ℎ𝑥𝑗
and 𝑜𝑥𝑘
• (Backward) calculate 𝛿𝑥𝑜,𝑘 and 𝛿𝑥
ℎ,𝑗
• Update 𝜔𝑗,𝑘 and 𝜔𝑖,𝑗
𝛿𝑥𝑜,𝑘 = 𝑡𝑥
𝑘 − 𝑜𝑥𝑘 𝑜𝑥
𝑘 1 − 𝑜𝑥𝑘
𝛿𝑥ℎ,𝑗
= ℎ𝑥𝑗1 − ℎ𝑥
𝑗 ∑𝑘𝜔𝑗,𝑘𝛿𝑥𝑜,𝑘
𝜔 ← 𝜔 + Δ𝜔
Δ𝜔𝑗,𝑘 = 𝜂 ℎ𝑥𝑗𝛿𝑥𝑜,𝑘
Δ𝜔𝑖,𝑗 = 𝜂 𝑥𝑖 𝛿𝑥ℎ,𝑗
Introduction to Machine Learning Amo G. Tong 30
Neural Networks
• Train a neural network with:
• One sigmoid units.
• Two layers of multiple sigmoid units with one output.
• Two layers of multiple sigmoid units with multiple outputs.
• Backpropagation framework: any units, any acyclic graph.
• Update the weight from right to left.
Introduction to Machine Learning Amo G. Tong 31
Neural Networks
• Backpropagation framework
• Good news: using the network is fast.
• Bad news: training the network is slow
• More bad news: it may converge to local minima.
• More good news: it performs well in practice.
• To avoid local minima
• Add a momentum
• Δ𝜔𝑛∗ = Δ𝜔𝑛 + 𝛼Δ𝜔𝑛−1
∗
• Train with different initializations.
Introduction to Machine Learning Amo G. Tong 32
Neural Networks
• Representational Power of Neural Networks.
• Boolean functions: every Boolean function can be represented exactly by some network with two layers of units.
• Continuous functions: every bounded continuous function can be approximated with arbitrarily small error by a network with two layers of units.(sigmoid + linear will do)
• Arbitrary function: any function can be approximated to arbitrary accuracy by a network with three layers of units. (sigmoid + sigmoid+ linear will do)
• Learn an identity function with eight training examples.
Introduction to Machine Learning Amo G. Tong 33
An example (Mitchell)
• Learn an identity function with eight training examples.
Introduction to Machine Learning Amo G. Tong 34
An example (Mitchell)
• Learn an identity function with eight training examples.
Introduction to Machine Learning Amo G. Tong 35
An example (Mitchell)
1 0 0
0 0 1
0 1 0
1 1 1
0 0 0
0 1 1
1 0 1
1 1 0
• Overfitting is everywhere…
• Overfitting may happen, when
• The learned model is too complicated
• Fitting data too well when data has noise
• When training set is small
• Avoid large parameters.
Introduction to Machine Learning Amo G. Tong 36
Overfitting
• Overfitting is everywhere…
Introduction to Machine Learning Amo G. Tong 37
Overfitting
• Overfitting is everywhere…
Introduction to Machine Learning Amo G. Tong 38
Overfitting
• Overfitting is everywhere…
• Weight Decay:
• Decrease each weight by some small factor during each iteration?
• Penalize large weights:
• 𝐸𝑟𝑟𝑜𝑟 = 𝐸𝑟𝑟𝑜𝑟 + 𝛾 ∑𝜔𝑖2
• Using Validation data.
Introduction to Machine Learning Amo G. Tong 39
Overfitting
• Overfitting is everywhere…
• Weight Decay:
• Decrease each weight by some small factor during each iteration?
• Penalize large weights:
• 𝐸𝑟𝑟𝑜𝑟 = 𝐸𝑟𝑟𝑜𝑟 + 𝛾 ∑𝜔𝑖2
• Using Validation data.
Introduction to Machine Learning Amo G. Tong 40
Overfitting
• What is an artificial neural network?
• Process the input by the connected units.
• How to training a neural network?
• Error-driven.
• Backpropagation algorithm.
• Two layers of sigmoid units.
Introduction to Machine Learning Amo G. Tong 41
Summary