6.036/6.862: Introduction to Machine Learning

6.036/6.862: Introduction to Machine Learning

Lecture: starts Tuesdays 9:35am (Boston time zone) Course website: introml.odl.mit.edu Who’s talking? Prof. Tamara Broderick Questions? Ask on Discourse: discourse.odl.mit.edu Materials: Will all be available at course website

1

Today’s PlanI. (More) logistics II. Machine learning setup III. Linear classifiers


Lecture: starts Tuesdays 9:35am (Boston time zone) Course website: introml.odl.mit.edu Who’s talking? Prof. Tamara Broderick Questions? Ask on Discourse: discourse.odl.mit.edu Materials: Will all be available at course website

1

Today’s PlanI. (More) logistics II. Machine learning setup III. Linear classifiers (set “Lecture 1” category)

2

Is Introduction to Machine Learning (6.036/6.862) right for you?

2


Computer Science Prerequisites

2


Computer Science Prerequisites• Python programming

2


Computer Science Prerequisites• Python programming • Algorithms (read & understand

pseudocode)

2



pseudocode) Math Prerequisites

2



pseudocode) Math Prerequisites• Matrix manipulations (inverse,

transpose, multiplication, etc.)

2




transpose, multiplication, etc.) • Points and planes in dimension > 2

2




transpose, multiplication, etc.) • Points and planes in dimension > 2 • Gradients

2




transpose, multiplication, etc.) • Points and planes in dimension > 2 • Gradients • Basic discrete probability (random

variables, independence, conditioning, etc.)

2






2






3


3

6.036/6.862: Introduction to Machine Learning, Staff

3

JehangirAmjad

TamaraBroderick

DuaneBoning

IkeChuang

IddoDrori

PhillipIsola

DavidSontag

Instructors:


3

JehangirAmjad

TamaraBroderick

DuaneBoning

IkeChuang

IddoDrori

PhillipIsola

DavidSontag

Instructors:

Teaching Assistants:

MattBeveridge

SatvatJagwani

DheekshitaKumar

CalebNoble


JustinLim

Hye youngShin

PeterTran

JulieVaughn

AudreyWang

CrystalWang

QuentinWellens

JuliaWu

EmilyZhang

3

JehangirAmjad

TamaraBroderick

DuaneBoning

IkeChuang

IddoDrori

PhillipIsola

DavidSontag

Instructors:

Teaching Assistants:

And Lab Assistants!

MattBeveridge

SatvatJagwani

DheekshitaKumar

CalebNoble


JustinLim

Hye youngShin

PeterTran

JulieVaughn

AudreyWang

CrystalWang

QuentinWellens

JuliaWu

EmilyZhang

4

6.036/6.862: Introduction to Machine Learning, Weekly Plan

4


4


Complete/update by noon today!

4


• Lecture + course notes


4


• Lecture + course notes • Exercises

• Due 9am before lecture


4



• Due 9am before lecture • Lab (synchronous, required!) Complete/

update by noon today!

4



• Due 9am before lecture • Lab (synchronous, required!)

• (New!) MLyPod: 10 studentsComplete/update by noon today!

4




• (New!) MLyPod: 10 students • Work in groups of 2 to 3;

check off with staff


4






check off with staff • Homework

4






4






4






4





check off with staff • Homework • Nanoquiz (no midterm/final)

• Timed

4






• Timed • Office hours

4






• Timed • Office hours• 6.862: project (canvas.mit.edu)

5

Machine learning (ML): why & what

5 [https://www.techrepublic.com/article/machine-learning-algorithm-confirms-50-new-exoplanets-in-historic-first/]


5 [https://www.thelancet.com/journals/lanchi/article/PIIS2352-4642(20)30239-X/fulltext]


5 [https://www.reuters.com/investigates/special-report/scotus/]


5 [https://www.forbes.com/sites/cognitiveworld/2020/08/01/using-machine-learning-to-automate-data-coding-at-the-bureau-of-labor-statistics-bls]


5 [https://www.forbes.com/sites/louiscolumbus/2020/08/12/5-ways-machine-learning-can-thwart-phishing-attacks]


5 [https://www.abc.net.au/news/science/2020-04-14/clearview-ai-facial-recognition-tech-australian-federal-police/12146894]


5 [https://www.businessinsider.in/finance/banks/news/ icici-bank-is-looking-to-pump-out-more-kisan-credit-cards-with-satellite-data-and-machine-learning-at-its-side/articleshow/77741696.cms]



5


5

• What is ML?


5

• What is ML? A set of methods for making decisions from data. (See the rest of the course!)


5


• Why study ML? To apply; to understand; to evaluate


5


• Why study ML? To apply; to understand; to evaluate • Notes: ML is not magic. ML is built on math.


5


• Why study ML? To apply; to understand; to evaluate • Notes: ML is not magic. ML is built on math.

[https://www.natureasia.com/en/nmiddleeast/article/10.1038/nmiddleeast.2020.46]

6

Getting started

6

Getting startedWhat do we have?

6

Getting startedWhat do we have? (Training) data

6


• n training data points

6


• n training data points • For data point i 2 {1, . . . , n}

<latexit sha1_base64="8hJFN8cgfSsm9/Zjj/qy/I7Ngvk=">AAAB/XicbVDLSsNAFJ3UV62v+Ni5GSyCi1ISKai7ohuXFewDmlAmk2k7dDIJMzdCDcVfceNCEbf+hzv/xmmbhbYeuHA4517uvSdIBNfgON9WYWV1bX2juFna2t7Z3bP3D1o6ThVlTRqLWHUCopngkjWBg2CdRDESBYK1g9HN1G8/MKV5LO9hnDA/IgPJ+5wSMFLPPuLY4xJ7mVvxRBiDrkhv0rPLTtWZAS8TNydllKPRs7+8MKZpxCRQQbTuuk4CfkYUcCrYpOSlmiWEjsiAdQ2VJGLaz2bXT/CpUULcj5UpCXim/p7ISKT1OApMZ0RgqBe9qfif102hf+lnXCYpMEnni/qpwBDjaRQ45IpREGNDCFXc3IrpkChCwQRWMiG4iy8vk9Z51a1Vr+5q5fp1HkcRHaMTdIZcdIHq6BY1UBNR9Iie0St6s56sF+vd+pi3Fqx85hD9gfX5A04blIQ=</latexit>

6


• n training data points • For data point

• Feature vector i 2 {1, . . . , n}


6





x(i) = (x(i)1 , . . . , x(i)

d )> 2 Rd

<latexit sha1_base64="E5PXGaP/huhuW7UrjJGwGk73mVA=">AAACJ3icbZDNSgMxFIUz/lv/qi7dBIvQQikzIqgLpejGZRWrhU5bMplUQzPJkNwRy9C3ceOruBFURJe+iWkdQa0HAh/n3kvuPUEsuAHXfXcmJqemZ2bn5nMLi0vLK/nVtQujEk1ZnSqhdCMghgkuWR04CNaINSNRINhl0Dse1i9vmDZcyXPox6wVkSvJu5wSsFYnf3jbTou8NMAHuJhhxytjX4QKTBl/W2Gp7YOKsc8l9iMC10GQng3aYSdfcCvuSHgcvAwKKFOtk3/yQ0WTiEmgghjT9NwYWinRwKlgg5yfGBYT2iNXrGlRkoiZVjq6c4C3rBPirtL2ScAj9+dESiJj+lFgO4c7mr+1oflfrZlAd6+VchknwCT9+qibCAwKD0PDIdeMguhbIFRzuyum10QTCjbanA3B+3vyOFxsV7ydyv7pTqF6lMUxhzbQJioiD+2iKjpBNVRHFN2hB/SMXpx759F5dd6+WiecbGYd/ZLz8QkGnaRM</latexit>

6





x(i) = (x(i)1 , . . . , x(i)

d )> 2 Rd


x1

<latexit sha1_base64="Zgso4r/5+qHuvA0cy8G5JHMuw8k=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEUG9FLx4r2g9oQ9lsN+3SzSbsTsQS+hO8eFDEq7/Im//GbZuDtj4YeLw3w8y8IJHCoOt+O4WV1bX1jeJmaWt7Z3evvH/QNHGqGW+wWMa6HVDDpVC8gQIlbyea0yiQvBWMbqZ+65FrI2L1gOOE+xEdKBEKRtFK9089r1euuFV3BrJMvJxUIEe9V/7q9mOWRlwhk9SYjucm6GdUo2CST0rd1PCEshEd8I6likbc+Nns1Ak5sUqfhLG2pZDM1N8TGY2MGUeB7YwoDs2iNxX/8zophpd+JlSSIldsvihMJcGYTP8mfaE5Qzm2hDIt7K2EDammDG06JRuCt/jyMmmeVb3z6tXdeaV2ncdRhCM4hlPw4AJqcAt1aACDATzDK7w50nlx3p2PeWvByWcO4Q+czx8PkI2s</latexit>

x2

<latexit sha1_base64="O76kjDC2rk5pOTxbQq6SNK3hW1I=">AAAB6nicbVDLSgNBEOyNrxhfUY9eBoPgKeyGgHoLevEY0TwgWcLspDcZMju7zMyKIeQTvHhQxKtf5M2/cZLsQRMLGoqqbrq7gkRwbVz328mtrW9sbuW3Czu7e/sHxcOjpo5TxbDBYhGrdkA1Ci6xYbgR2E4U0igQ2ApGNzO/9YhK81g+mHGCfkQHkoecUWOl+6depVcsuWV3DrJKvIyUIEO9V/zq9mOWRigNE1Trjucmxp9QZTgTOC10U40JZSM6wI6lkkao/cn81Ck5s0qfhLGyJQ2Zq78nJjTSehwFtjOiZqiXvZn4n9dJTXjpT7hMUoOSLRaFqSAmJrO/SZ8rZEaMLaFMcXsrYUOqKDM2nYINwVt+eZU0K2WvWr66q5Zq11kceTiBUzgHDy6gBrdQhwYwGMAzvMKbI5wX5935WLTmnGzmGP7A+fwBERSNrQ==</latexit>

6





x(i) = (x(i)1 , . . . , x(i)

d )> 2 Rd


x1


x2


x(1)

<latexit sha1_base64="zGoqMzp4iDsG8397zyI3iJbhjNQ=">AAAB7nicbVDLSgNBEOz1GeMr6tHLYBDiJexKQL0FvXiMYB6QxDA76U2GzM4uM7NiWPIRXjwo4tXv8ebfOEn2oIkFDUVVN91dfiy4Nq777aysrq1vbOa28ts7u3v7hYPDho4SxbDOIhGplk81Ci6xbrgR2IoV0tAX2PRHN1O/+YhK80jem3GM3ZAOJA84o8ZKzaeHtOSdTXqFolt2ZyDLxMtIETLUeoWvTj9iSYjSMEG1bntubLopVYYzgZN8J9EYUzaiA2xbKmmIupvOzp2QU6v0SRApW9KQmfp7IqWh1uPQt50hNUO96E3F/7x2YoLLbsplnBiUbL4oSAQxEZn+TvpcITNibAllittbCRtSRZmxCeVtCN7iy8ukcV72KuWru0qxep3FkYNjOIESeHABVbiFGtSBwQie4RXenNh5cd6dj3nripPNHMEfOJ8/msCPHA==</latexit>

x

6





x(i) = (x(i)1 , . . . , x(i)

d )> 2 Rd


x1


x2


x(1)


x(2)

<latexit sha1_base64="cyjrfHFUII6CpU7ZkCOHgU9e85g=">AAAB7nicbVBNSwMxEJ3Ur1q/qh69BItQL2W3FNRb0YvHCvYD2rVk02wbms0uSVYsS3+EFw+KePX3ePPfmLZ70NYHA4/3ZpiZ58eCa+M43yi3tr6xuZXfLuzs7u0fFA+PWjpKFGVNGolIdXyimeCSNQ03gnVixUjoC9b2xzczv/3IlOaRvDeTmHkhGUoecEqMldpPD2m5ej7tF0tOxZkDrxI3IyXI0OgXv3qDiCYhk4YKonXXdWLjpUQZTgWbFnqJZjGhYzJkXUslCZn20vm5U3xmlQEOImVLGjxXf0+kJNR6Evq2MyRmpJe9mfif101McOmlXMaJYZIuFgWJwCbCs9/xgCtGjZhYQqji9lZMR0QRamxCBRuCu/zyKmlVK26tcnVXK9WvszjycAKnUAYXLqAOt9CAJlAYwzO8whuK0Qt6Rx+L1hzKZo7hD9DnD5xGjx0=</latexit>

x x

6





x(i) = (x(i)1 , . . . , x(i)

d )> 2 Rd


x1


x2


x(1)


x(2)


x x

x(3)

<latexit sha1_base64="vXawohlkiYMW3s8tEmLQzGlI/jI=">AAAB7nicbVBNSwMxEJ2tX7V+VT16CRahXsquFtRb0YvHCvYD2rVk02wbmmSXJCuWpT/CiwdFvPp7vPlvTNs9aOuDgcd7M8zMC2LOtHHdbye3srq2vpHfLGxt7+zuFfcPmjpKFKENEvFItQOsKWeSNgwznLZjRbEIOG0Fo5up33qkSrNI3ptxTH2BB5KFjGBjpdbTQ1o+P530iiW34s6AlomXkRJkqPeKX91+RBJBpSEca93x3Nj4KVaGEU4nhW6iaYzJCA9ox1KJBdV+Ojt3gk6s0kdhpGxJg2bq74kUC63HIrCdApuhXvSm4n9eJzHhpZ8yGSeGSjJfFCYcmQhNf0d9pigxfGwJJorZWxEZYoWJsQkVbAje4svLpHlW8aqVq7tqqXadxZGHIziGMnhwATW4hTo0gMAInuEV3pzYeXHenY95a87JZg7hD5zPH53Mjx4=</latexit>

x

6



• Feature vector

• Label

i 2 {1, . . . , n}


x(i) = (x(i)1 , . . . , x(i)

d )> 2 Rd


y(i) 2 {�1,+1}

<latexit sha1_base64="713ykEbssIo/Rc2amVUhyV5Oe0I=">AAAB/nicbVDLSsNAFL2pr1pfUXHlZrAIFbUkUlB3RTcuK9gHNLFMppN26GQSZiZCCQV/xY0LRdz6He78G6ePhVoPXDiccy/33hMknCntOF9WbmFxaXklv1pYW9/Y3LK3dxoqTiWhdRLzWLYCrChngtY105y2EklxFHDaDAbXY7/5QKVisbjTw4T6Ee4JFjKCtZE69t7wPiuxoxHymEBeduqeHLveqGMXnbIzAZon7owUYYZax/70ujFJIyo04Viptusk2s+w1IxwOip4qaIJJgPco21DBY6o8rPJ+SN0aJQuCmNpSmg0UX9OZDhSahgFpjPCuq/+emPxP6+d6vDCz5hIUk0FmS4KU450jMZZoC6TlGg+NAQTycytiPSxxESbxAomBPfvy/OkcVZ2K+XL20qxejWLIw/7cAAlcOEcqnADNagDgQye4AVerUfr2Xqz3qetOWs2swu/YH18A8uWlBs=</latexit>

x1


x2


x(1)


x(2)


x x

x(3)


x

6



• Feature vector

• Label

i 2 {1, . . . , n}


x(i) = (x(i)1 , . . . , x(i)

d )> 2 Rd


y(i) 2 {�1,+1}


x1


x2


–x(1)


x(2)


x

x(3)


x

6



• Feature vector

• Label

i 2 {1, . . . , n}


x(i) = (x(i)1 , . . . , x(i)

d )> 2 Rd


y(i) 2 {�1,+1}


x1


x2


– –x(1)


x(2)


x(3)


x

6



• Feature vector

• Label

i 2 {1, . . . , n}


x(i) = (x(i)1 , . . . , x(i)

d )> 2 Rd


y(i) 2 {�1,+1}


x1


x2


– –x(1)


x(2)


+

x(3)


6



• Feature vector

• Label

i 2 {1, . . . , n}


x(i) = (x(i)1 , . . . , x(i)

d )> 2 Rd


y(i) 2 {�1,+1}


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


6



• Feature vector

• Label • Training data

i 2 {1, . . . , n}


x(i) = (x(i)1 , . . . , x(i)

d )> 2 Rd


y(i) 2 {�1,+1}


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


6



• Feature vector


i 2 {1, . . . , n}


x(i) = (x(i)1 , . . . , x(i)

d )> 2 Rd


y(i) 2 {�1,+1}


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

Dn = {(x(1), y(1)), . . . , (x(n), y(n))}

<latexit sha1_base64="OVqEx11BAxLVTMEmIIhu5cyWyT4=">AAACK3icbZDLSsNAFIYn9VbrLerSzWARWiglkYK6EEp14bKCvUATy2Q6bYdOJmFmIpaQ93Hjq7jQhRfc+h5O0wraemDg4//P4cz5vZBRqSzr3cgsLa+srmXXcxubW9s75u5eUwaRwKSBAxaItockYZSThqKKkXYoCPI9Rlre6GLit+6IkDTgN2ocEtdHA077FCOlpa5Zc3ykhhix+DLpcngOnbhwfxsX7GJSguMpFEvQYb1AyRJMPf7jaSg6SdfMW2UrLbgI9gzyYFb1rvns9AIc+YQrzJCUHdsKlRsjoShmJMk5kSQhwiM0IB2NHPlEunF6awKPtNKD/UDoxxVM1d8TMfKlHPue7pxcJue9ifif14lU/9SNKQ8jRTieLupHDKoAToKDPSoIVmysAWFB9V8hHiKBsNLx5nQI9vzJi9A8LtuV8tl1JV+tzeLIggNwCArABiegCq5AHTQABg/gCbyCN+PReDE+jM9pa8aYzeyDP2V8fQN9wKTH</latexit>

x(3)


6



• Feature vector


What do we want?

i 2 {1, . . . , n}


x(i) = (x(i)1 , . . . , x(i)

d )> 2 Rd


y(i) 2 {�1,+1}


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

Dn = {(x(1), y(1)), . . . , (x(n), y(n))}


x(3)


6



• Feature vector


What do we want? A good way to label new points

i 2 {1, . . . , n}


x(i) = (x(i)1 , . . . , x(i)

d )> 2 Rd


y(i) 2 {�1,+1}


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

Dn = {(x(1), y(1)), . . . , (x(n), y(n))}


x(3)


6



• Feature vector



i 2 {1, . . . , n}


x(i) = (x(i)1 , . . . , x(i)

d )> 2 Rd


y(i) 2 {�1,+1}


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+

x

x(1)


x(2)


–––

Dn = {(x(1), y(1)), . . . , (x(n), y(n))}


x(3)


6



• Feature vector



i 2 {1, . . . , n}


x(i) = (x(i)1 , . . . , x(i)

d )> 2 Rd


y(i) 2 {�1,+1}


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+

x

x(1)


x(2)


–––

Dn = {(x(1), y(1)), . . . , (x(n), y(n))}


x(3)


6



• Feature vector



i 2 {1, . . . , n}


x(i) = (x(i)1 , . . . , x(i)

d )> 2 Rd


y(i) 2 {�1,+1}


Dn = {(x(1), y(1)), . . . , (x(n), y(n))}


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+

x

x(1)


x(2)


–––

x(3)


6



• Feature vector



i 2 {1, . . . , n}


x(i) = (x(i)1 , . . . , x(i)

d )> 2 Rd


y(i) 2 {�1,+1}


Dn = {(x(1), y(1)), . . . , (x(n), y(n))}


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+

x

x(1)


x(2)


–––

x(3)


6



• Feature vector


What do we want? A good way to label new points • How to label?

i 2 {1, . . . , n}


x(i) = (x(i)1 , . . . , x(i)

d )> 2 Rd


y(i) 2 {�1,+1}


Dn = {(x(1), y(1)), . . . , (x(n), y(n))}


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+

x

x(1)


x(2)


–––

x(3)


6



• Feature vector


What do we want? A good way to label new points • How to label? Hypothesis

i 2 {1, . . . , n}


x(i) = (x(i)1 , . . . , x(i)

d )> 2 Rd


y(i) 2 {�1,+1}


Dn = {(x(1), y(1)), . . . , (x(n), y(n))}


h : Rd ! {�1,+1}

<latexit sha1_base64="Qglo07NTvoHTsQPcH9BIBVMHQAI=">AAACCHicbVDLSsNAFJ3UV62vqEsXDhZBUEsiBR+rohuXVewDmlgmk0k7dDIJMxOhhCzd+CtuXCji1k9w5984bbPQ1gMDh3PuZe45XsyoVJb1bRTm5hcWl4rLpZXVtfUNc3OrKaNEYNLAEYtE20OSMMpJQ1HFSDsWBIUeIy1vcDXyWw9ESBrxOzWMiRuiHqcBxUhpqWvu9i+gEyLV97z0NrtP/Qw6KoJOemwfHdpO1jXLVsUaA84SOydlkKPeNb8cP8JJSLjCDEnZsa1YuSkSimJGspKTSBIjPEA90tGUo5BINx0HyeC+VnwYREI/ruBY/b2RolDKYejpydHNctobif95nUQFZ25KeZwowvHkoyBhUEcdtQJ9KghWbKgJwoLqWyHuI4Gw0t2VdAn2dORZ0jyp2NXK+U21XLvM6yiCHbAHDoANTkENXIM6aAAMHsEzeAVvxpPxYrwbH5PRgpHvbIM/MD5/AM4nmJI=</latexit>

x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+

x

x(1)


x(2)


–––

x(3)


6



• Feature vector



i 2 {1, . . . , n}


x(i) = (x(i)1 , . . . , x(i)

d )> 2 Rd


y(i) 2 {�1,+1}


x

<latexit sha1_base64="g3vERbzU3kfXibT9Kek4KMf3Qbg=">AAAB6HicbVDLSgNBEOyNrxhfUY9eBoPgKexKQL0FvXhMwDwgWcLspDcZMzu7zMyKIeQLvHhQxKuf5M2/cZLsQRMLGoqqbrq7gkRwbVz328mtrW9sbuW3Czu7e/sHxcOjpo5TxbDBYhGrdkA1Ci6xYbgR2E4U0igQ2ApGtzO/9YhK81jem3GCfkQHkoecUWOl+lOvWHLL7hxklXgZKUGGWq/41e3HLI1QGiao1h3PTYw/ocpwJnBa6KYaE8pGdIAdSyWNUPuT+aFTcmaVPgljZUsaMld/T0xopPU4CmxnRM1QL3sz8T+vk5rwyp9wmaQGJVssClNBTExmX5M+V8iMGFtCmeL2VsKGVFFmbDYFG4K3/PIqaV6UvUr5ul4pVW+yOPJwAqdwDh5cQhXuoAYNYIDwDK/w5jw4L86787FozTnZzDH8gfP5A+oZjQg=</latexit>

h

<latexit sha1_base64="0LoNUPId1hhmBuXcotBZd/4zZho=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0mkoN6KXjy2YGuhDWWznbRrN5uwuxFK6C/w4kERr/4kb/4bt20O2vpg4PHeDDPzgkRwbVz32ymsrW9sbhW3Szu7e/sH5cOjto5TxbDFYhGrTkA1Ci6xZbgR2EkU0igQ+BCMb2f+wxMqzWN5byYJ+hEdSh5yRo2VmqN+ueJW3TnIKvFyUoEcjX75qzeIWRqhNExQrbuemxg/o8pwJnBa6qUaE8rGdIhdSyWNUPvZ/NApObPKgISxsiUNmau/JzIaaT2JAtsZUTPSy95M/M/rpia88jMuk9SgZItFYSqIicnsazLgCpkRE0soU9zeStiIKsqMzaZkQ/CWX14l7YuqV6teN2uV+k0eRxFO4BTOwYNLqMMdNKAFDBCe4RXenEfnxXl3PhatBSefOYY/cD5/ANHZjPg=</latexit>

y

<latexit sha1_base64="t5lWBlxnhPc8vatgOCKMO4kpbDU=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0mkoN6KXjy2YGuhDWWznbRrN5uwuxFK6C/w4kERr/4kb/4bt20O2vpg4PHeDDPzgkRwbVz32ymsrW9sbhW3Szu7e/sH5cOjto5TxbDFYhGrTkA1Ci6xZbgR2EkU0igQ+BCMb2f+wxMqzWN5byYJ+hEdSh5yRo2VmpN+ueJW3TnIKvFyUoEcjX75qzeIWRqhNExQrbuemxg/o8pwJnBa6qUaE8rGdIhdSyWNUPvZ/NApObPKgISxsiUNmau/JzIaaT2JAtsZUTPSy95M/M/rpia88jMuk9SgZItFYSqIicnsazLgCpkRE0soU9zeStiIKsqMzaZkQ/CWX14l7YuqV6teN2uV+k0eRxFO4BTOwYNLqMMdNKAFDBCe4RXenEfnxXl3PhatBSefOYY/cD5/AOudjQk=</latexit>

Dn = {(x(1), y(1)), . . . , (x(n), y(n))}


h : Rd ! {�1,+1}


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+

x

x(1)


x(2)


–––

x(3)


6



• Feature vector



i 2 {1, . . . , n}


x(i) = (x(i)1 , . . . , x(i)

d )> 2 Rd


y(i) 2 {�1,+1}


x


h


y


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+

xx(1)


x(2)


–––

Dn = {(x(1), y(1)), . . . , (x(n), y(n))}


h : Rd ! {�1,+1}


x(3)


6



• Feature vector



i 2 {1, . . . , n}


x(i) = (x(i)1 , . . . , x(i)

d )> 2 Rd


y(i) 2 {�1,+1}


x


h


y


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+

x

x(1)


x(2)


–––

Dn = {(x(1), y(1)), . . . , (x(n), y(n))}


h : Rd ! {�1,+1}


x(3)


6



• Feature vector



i 2 {1, . . . , n}


x(i) = (x(i)1 , . . . , x(i)

d )> 2 Rd


y(i) 2 {�1,+1}


x


h


y


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+

x

x(1)


x(2)


–––

Dn = {(x(1), y(1)), . . . , (x(n), y(n))}


h : Rd ! {�1,+1}


x(3)


6



• Feature vector



i 2 {1, . . . , n}


x(i) = (x(i)1 , . . . , x(i)

d )> 2 Rd


y(i) 2 {�1,+1}


x


h


y


• Example h: For any x, h(x) = +1

x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+

x

x(1)


x(2)


–––

Dn = {(x(1), y(1)), . . . , (x(n), y(n))}


h : Rd ! {�1,+1}


x(3)


6



• Feature vector



i 2 {1, . . . , n}


x(i) = (x(i)1 , . . . , x(i)

d )> 2 Rd


y(i) 2 {�1,+1}


x


h


y


• Example h: For any x, h(x) = +1 • Is this a hypothesis?

Dn = {(x(1), y(1)), . . . , (x(n), y(n))}


h : Rd ! {�1,+1}


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+

x

x(1)


x(2)


–––

x(3)


6



• Feature vector



i 2 {1, . . . , n}


x(i) = (x(i)1 , . . . , x(i)

d )> 2 Rd


y(i) 2 {�1,+1}


x


h


y


• Example h: For any x, h(x) = +1 • Is this a good hypothesis?

Dn = {(x(1), y(1)), . . . , (x(n), y(n))}


h : Rd ! {�1,+1}


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+

x

x(1)


x(2)


–––

x(3)


7

Linear classifiers

x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


7

Linear classifiers• Hypothesis class : set of H

<latexit sha1_base64="oYhv6ScZyUxRyTAPyqiZC2s1B+I=">AAAB8nicbVDLSgMxFM3UV62vqks3wSK4KjNSUHdFN11WsA+YDiWTZtrQTDIkd4Qy9DPcuFDErV/jzr8x085CWw8EDufcS849YSK4Adf9dkobm1vbO+Xdyt7+weFR9fika1SqKetQJZTuh8QwwSXrAAfB+olmJA4F64XT+9zvPTFtuJKPMEtYEJOx5BGnBKzkD2ICE0pE1poPqzW37i6A14lXkBoq0B5WvwYjRdOYSaCCGON7bgJBRjRwKti8MkgNSwidkjHzLZUkZibIFpHn+MIqIxwpbZ8EvFB/b2QkNmYWh3Yyj2hWvVz8z/NTiG6CjMskBSbp8qMoFRgUzu/HI64ZBTGzhFDNbVZMJ0QTCralii3BWz15nXSv6l6jfvvQqDXvijrK6Aydo0vkoWvURC3URh1EkULP6BW9OeC8OO/Ox3K05BQ7p+gPnM8fftGRag==</latexit>

h 2 H

<latexit sha1_base64="tgvEbOGEhvSa1jRIEcbjIsjq67c=">AAAB+nicbVDLSsNAFL2pr1pfqS7dDBbBVUlEUHdFN11WsA9oQplMJ+3QySTMTJQS+yluXCji1i9x5984abPQ1gMDh3Pu5Z45QcKZ0o7zbZXW1jc2t8rblZ3dvf0Du3rYUXEqCW2TmMeyF2BFORO0rZnmtJdIiqOA024wuc397gOVisXiXk8T6kd4JFjICNZGGtjVMfKYQF6E9ZhgnjVnA7vm1J050CpxC1KDAq2B/eUNY5JGVGjCsVJ910m0n2GpGeF0VvFSRRNMJnhE+4YKHFHlZ/PoM3RqlCEKY2me0Giu/t7IcKTUNArMZB5RLXu5+J/XT3V45WdMJKmmgiwOhSlHOkZ5D2jIJCWaTw3BRDKTFZExlpho01bFlOAuf3mVdM7r7kX9+u6i1rgp6ijDMZzAGbhwCQ1oQgvaQOARnuEV3qwn68V6tz4WoyWr2DmCP7A+fwC+H5Oy</latexit>

x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


7

Linear classifiers• Hypothesis class : set of

• Example : All hypotheses that label +1 on one side of a line and -1 on the other side

H


h 2 H


H


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


7



H


h 2 H


H


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


7



H


h 2 H


H


7



H


h 2 H


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


H


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


7



H


h 2 H


H


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


7



H


h 2 H


H


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


7



H


h 2 H


H


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


7



H


h 2 H


H


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


7



H


h 2 H


H


7



H


h 2 H


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


H


7



H


h 2 H


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


H


7



H


h 2 H


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


H


7



H


h 2 H


Math facts!

H


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


7



H


h 2 H


Math facts!

H


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


7



H


h 2 H


Math facts!

H


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


x1


x2


7



H


h 2 H


Math facts!

H


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


+x


x1


x2


7



H


h 2 H


Math facts!

H


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


+x


x1


x2


7



H


h 2 H


Math facts!

H


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


+x


✓

<latexit sha1_base64="W2PKfIkuMomCS5JZF2x4iAdaa48=">AAAB7XicbVBNS8NAEJ3Ur1q/qh69LBbBU0mkoN6KXjxWsB/QhrLZbtq1m03YnQgl9D948aCIV/+PN/+N2zYHbX0w8Hhvhpl5QSKFQdf9dgpr6xubW8Xt0s7u3v5B+fCoZeJUM95ksYx1J6CGS6F4EwVK3kk0p1EgeTsY38789hPXRsTqAScJ9yM6VCIUjKKVWj0ccaT9csWtunOQVeLlpAI5Gv3yV28QszTiCpmkxnQ9N0E/oxoFk3xa6qWGJ5SN6ZB3LVU04sbP5tdOyZlVBiSMtS2FZK7+nshoZMwkCmxnRHFklr2Z+J/XTTG88jOhkhS5YotFYSoJxmT2OhkIzRnKiSWUaWFvJWxENWVoAyrZELzll1dJ66Lq1arX97VK/SaPowgncArn4MEl1OEOGtAEBo/wDK/w5sTOi/PufCxaC04+cwx/4Hz+AKfXjzQ=</latexit>

x1


x2


7



H


h 2 H


Math facts!

H


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


+x


✓


✓>x

<latexit sha1_base64="0YtwuWmPnt1gPViPqfkhOnsT5vE=">AAAB9HicbVBNSwMxEM3Wr1q/qh69BIvgqeyKoN6KXjxWsB/QXUs2zbah2WRNZotl6e/w4kERr/4Yb/4b03YP2vpg4PHeDDPzwkRwA6777RRWVtfWN4qbpa3tnd298v5B06hUU9agSijdDolhgkvWAA6CtRPNSBwK1gqHN1O/NWLacCXvYZywICZ9ySNOCVgp8GHAgDz4oBL81C1X3Ko7A14mXk4qKEe9W/7ye4qmMZNABTGm47kJBBnRwKlgk5KfGpYQOiR91rFUkpiZIJsdPcEnVunhSGlbEvBM/T2RkdiYcRzazpjAwCx6U/E/r5NCdBlkXCYpMEnni6JUYFB4mgDucc0oiLElhGpub8V0QDShYHMq2RC8xZeXSfOs6p1Xr+7OK7XrPI4iOkLH6BR56ALV0C2qowai6BE9o1f05oycF+fd+Zi3Fpx85hD9gfP5A8Ydkh8=</latexit>

x1


x2


7



H


h 2 H


Math facts!

H


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


+x


✓


dx1✓>x


x1


x2


7



H


h 2 H


Math facts!

H


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


+x


✓


1xd, dx1✓>x


x1


x2


7



H


h 2 H


Math facts!

H


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


+x


✓


1xd, dx1✓>x


x1


x2


7



H


h 2 H


Math facts!

H


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


+x


✓


1xd, dx1✓>x


x1


x2


7



H


h 2 H


Math facts!

H


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


+x


✓


1xd, dx1✓>x


x1


x2


7



H


h 2 H


+x


✓


Math facts!

H


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


1xd, dx1✓>x


x1


x2


7



H


h 2 H


+x


✓


{

Math facts!

H


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


1xd, dx1✓>x


x1


x2


7



H


h 2 H


+x


✓


{

Math facts!

H


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


1xd, dx1✓>x/k✓k

<latexit sha1_base64="6ugvwT+TJwGsgV3ofp7NId/FmqQ=">AAACBXicbZC7TsMwFIadcivlFmCEwaJCYioJqgRsFSyMRaIXqQmV4zqtVceJ7BNEVbqw8CosDCDEyjuw8Ta4bQZo+SVLn/5zjo7PHySCa3Ccbyu3sLi0vJJfLaytb2xu2ds7dR2nirIajUWsmgHRTHDJasBBsGaiGIkCwRpB/3Jcb9wxpXksb2CQMD8iXclDTgkYq23ve9BjQG49iBN8j4+x94CnlqG2XXRKzkR4HtwMiihTtW1/eZ2YphGTQAXRuuU6CfhDooBTwUYFL9UsIbRPuqxlUJKIaX84uWKED43TwWGszJOAJ+7viSGJtB5EgemMCPT0bG1s/ldrpRCe+UMukxSYpNNFYSowxHgcCe5wxSiIgQFCFTd/xbRHFKFggiuYENzZk+ehflJyy6Xz63KxcpHFkUd76AAdIRedogq6QlVUQxQ9omf0it6sJ+vFerc+pq05K5vZRX9kff4AIlCXtw==</latexit>

x1


x2


7



H


h 2 H


+x


✓


{

Math facts!

H


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


1xd, dx1✓>x/k✓k


x1


x2


7



H


h 2 H


+x


{

Math facts!

H


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


1xd, dx1✓>x/k✓k


✓

<latexit sha1_base64="W2PKfIkuMomCS5JZF2x4iAdaa48=">AAAB7XicbVBNS8NAEJ3Ur1q/qh69LBbBU0mkoN6KXjxWsB/QhrLZbtq1m03YnQgl9D948aCIV/+PN/+N2zYHbX0w8Hhvhpl5QSKFQdf9dgpr6xubW8Xt0s7u3v5B+fCoZeJUM95ksYx1J6CGS6F4EwVK3kk0p1EgeTsY38789hPXRsTqAScJ9yM6VCIUjKKVWj0ccaT9csWtunOQVeLlpAI5Gv3yV28QszTiCpmkxnQ9N0E/ox oFk3xa6qWGJ5SN6ZB3LVU04sbP5tdOyZlVBiSMtS2FZK7+nshoZMwkCmxnRHFklr2Z+J/XTTG88jOhkhS5YotFYSoJxmT2OhkIzRnKiSWUaWFvJWxENWVoAyrZELzll1dJ66Lq1arX97VK/SaPowgncArn4MEl1OEOGtAEBo/wDK/w5sTOi/PufCxaC04+cwx/4Hz+AKfXjzQ=</latexit>

x1


x2


7



H


h 2 H


+x


✓


{

Math facts!

H


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


1xd, dx1✓>x/k✓k


x1


x2


7



H


h 2 H


+x


✓


Math facts!

H


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


1xd, dx1✓>x/k✓k


x1


x2


7



H


h 2 H


Math facts!

H


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


+x


✓


1xd, dx1✓>x/k✓k


x1


x2


7



H


h 2 H


Math facts!

H


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


✓


x


+

1xd, dx1✓>x/k✓k


x1


x2


7



H


h 2 H


Math facts!

H


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


✓


+

+

1xd, dx1✓>x/k✓k


x1


x2


7



H


h 2 H


Math facts!

H


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


✓


+

++

++

1xd, dx1✓>x/k✓k


x1


x2


7



H


h 2 H


Math facts!

H


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


✓


+

++

++

x1


x2


7



H


h 2 H


Math facts!

H


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


✓


+

++

++

x1


x2


7



H


h 2 H


Math facts!

H


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


✓


x1


x2


7



H


h 2 H


Math facts!

H


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


✓


x :✓> x/

k✓k=0

<latexit sha1_base64="ikP/PKwPMN9MVTUQuWh++2pkrVc=">AAACDHicbVDLSgMxFM3UV62vqks3wSK4qjNS8AFC0Y3LCvYBnbFk0kwbmnmQ3JGWsR/gxl9x40IRt36AO//GTDsLbT0QOJxzLjf3uJHgCkzz28gtLC4tr+RXC2vrG5tbxe2dhgpjSVmdhiKULZcoJnjA6sBBsFYkGfFdwZru4Cr1m/dMKh4GtzCKmOOTXsA9TgloqVMsDc+xDX0G5M6GMMJDfITth0xK2QU2dcosmxPgeWJlpIQy1DrFL7sb0thnAVBBlGpbZgROQiRwKti4YMeKRYQOSI+1NQ2Iz5STTI4Z4wOtdLEXSv0CwBP190RCfKVGvquTPoG+mvVS8T+vHYN36iQ8iGJgAZ0u8mKBIcRpM7jLJaMgRpoQKrn+K6Z9IgkF3V9Bl2DNnjxPGsdlq1I+u6mUqpdZHXm0h/bRIbLQCaqia1RDdUTRI3pGr+jNeDJejHfjYxrNGdnMLvoD4/MHr7aZfA==</latexit>

x1


x2


7



H


h 2 H


Math facts!

H


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


✓


{a

<latexit sha1_base64="4w2P+y77hdxOxTkPAvaBFLGE+PA=">AAAB6HicdVDLSsNAFJ3UV62vqks3g0VwFSYxxWZXdOOyBfuANpTJdNKOnUzCzEQooV/gxoUibv0kd/6N04egogcuHM65l3vvCVPOlEbowyqsrW9sbhW3Szu7e/sH5cOjtkoySWiLJDyR3RArypmgLc00p91UUhyHnHbCyfXc79xTqVgibvU0pUGMR4JFjGBtpCYelCvI9n3keVWI7CpyXbdmCLpwa74DHRstUAErNAbl9/4wIVlMhSYcK9VzUKqDHEvNCKezUj9TNMVkgke0Z6jAMVVBvjh0Bs+MMoRRIk0JDRfq94kcx0pN49B0xliP1W9vLv7l9TId1YKciTTTVJDloijjUCdw/jUcMkmJ5lNDMJHM3ArJGEtMtMmmZEL4+hT+T9qu7Xi23/Qq9atVHEVwAk7BOXDAJaiDG9AALUAABQ/gCTxbd9aj9WK9LlsL1mrmGPyA9fYJMTGNOg==</latexit>

x :✓> x/

k✓k=0


x1


x2


7



H


h 2 H


Math facts!

H


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


✓


{a


x :✓> x/

k✓k=0


x1


x2


7



H


h 2 H


Math facts!

H


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


✓


{a


x :✓> x/

k✓k=0


x :✓> x/

k✓k=a

<latexit sha1_base64="S6UepToT8fXAlZTB6p/gaVqrd6E=">AAACDHicbVDLSgMxFM3UV62vqks3wSK4qjNS8AFC0Y3LCvYBnbFk0kwbmnmQ3JGWsR/gxl9x40IRt36AO//GTDsLbT0QOJxzLjf3uJHgCkzz28gtLC4tr+RXC2vrG5tbxe2dhgpjSVmdhiKULZcoJnjA6sBBsFYkGfFdwZru4Cr1m/dMKh4GtzCKmOOTXsA9TgloqVMsDc+xDX0G5M6GMMJDfITth0xK2QUmOmWWzQnwPLEyUkIZap3il90NaeyzAKggSrUtMwInIRI4FWxcsGPFIkIHpMfamgbEZ8pJJseM8YFWutgLpX4B4In6eyIhvlIj39VJn0BfzXqp+J/XjsE7dRIeRDGwgE4XebHAEOK0GdzlklEQI00IlVz/FdM+kYSC7q+gS7BmT54njeOyVSmf3VRK1cusjjzaQ/voEFnoBFXRNaqhOqLoET2jV/RmPBkvxrvxMY3mjGxmF/2B8fkD+fqZrQ==</latexit>

x1


x2


7



H


h 2 H


Math facts!

H


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


✓


{a


{b

<latexit sha1_base64="+bM9eYcSG5ADnqDF7lPNM0DzkNo=">AAAB6HicdZDLSgMxFIbP1Futt6pLN8EiuBoyUmtnV3TjsgV7gXYomTTTRjMXkoxQhj6BGxeKuPWR3Pk2ZtoKKvpD4Oc755Bzfj8RXGmMP6zCyura+kZxs7S1vbO7V94/6Kg4lZS1aSxi2fOJYoJHrK25FqyXSEZCX7Cuf3eV17v3TCoeRzd6mjAvJOOIB5wSbVDLH5Yr2D7HjlurIWxj7FTrjjGuWzcQOYbkqsBSzWH5fTCKaRqySFNBlOo7ONFeRqTmVLBZaZAqlhB6R8asb2xEQqa8bL7oDJ0YMkJBLM2LNJrT7xMZCZWahr7pDImeqN+1HP5V66c6qHsZj5JUs4guPgpSgXSM8qvRiEtGtZgaQ6jkZldEJ0QSqk02JRPC16Xof9M5s52q7baqlcblMo4iHMExnIIDF9CAa2hCGygweIAneLZurUfrxXpdtBas5cwh/JD19gkyTY07</latexit>

x :✓> x/

k✓k=0


x :✓> x/

k✓k=a


x1


x2


7



H


h 2 H


Math facts!

H


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


✓


{a


{b


x :✓> x/

k✓k=0


x :✓> x/

k✓k=a


x :✓> x/

k✓k=�b

<latexit sha1_base64="fN/Kuv+CifcArGvoMT2NBFxRO+Y=">AAACDXicbZC7SgNBFIZn4y3G26qlzWAUbIy7EvACQtDGMoK5QDaG2ckkGTJ7YeasJKx5ARtfxcZCEVt7O9/G2WQLTfxh4OM/53Dm/G4ouALL+jYyc/MLi0vZ5dzK6tr6hrm5VVVBJCmr0EAEsu4SxQT3WQU4CFYPJSOeK1jN7V8l9do9k4oH/i0MQ9b0SNfnHU4JaKtl7g3OsQM9BuTOgSDEA3yEnYfUSugCH7otM28VrLHwLNgp5FGqcsv8ctoBjTzmAxVEqYZthdCMiQROBRvlnEixkNA+6bKGRp94TDXj8TUjvK+dNu4EUj8f8Nj9PRETT6mh5+pOj0BPTdcS879aI4LOaTPmfhgB8+lkUScSGAKcRIPbXDIKYqiBUMn1XzHtEUko6ABzOgR7+uRZqB4X7GLh7KaYL12mcWTRDtpFB8hGJ6iErlEZVRBFj+gZvaI348l4Md6Nj0lrxkhnttEfGZ8/cGyZ5Q==</latexit>

x1


x2


7



H


h 2 H


Math facts!

H


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


✓


{b


x :✓> x/

k✓k=�b


x1


x2


7



H


h 2 H


Math facts!

H


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


✓


{b


x :✓> x/

k✓k=�b


x1


x2


7



H


h 2 H


Math facts!

H


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


✓


{b


x :✓> x/

k✓k=�b


x :✓> x/

k✓k>�b

<latexit sha1_base64="ovTgODpoWoqgPxKyqHqecIYK6no=">AAACDXicbZC7SgNBFIZn4y3G26qlzWAUbIy7EvBSSNDGMoK5QDaG2ckkGTJ7YeasJKx5ARtfxcZCEVt7O9/G2WQLTfxh4OM/53Dm/G4ouALL+jYyc/MLi0vZ5dzK6tr6hrm5VVVBJCmr0EAEsu4SxQT3WQU4CFYPJSOeK1jN7V8l9do9k4oH/i0MQ9b0SNfnHU4JaKtl7g3OsQM9BuTOgSDEA3yEnYfUSugCH7otM28VrLHwLNgp5FGqcsv8ctoBjTzmAxVEqYZthdCMiQROBRvlnEixkNA+6bKGRp94TDXj8TUjvK+dNu4EUj8f8Nj9PRETT6mh5+pOj0BPTdcS879aI4LOaTPmfhgB8+lkUScSGAKcRIPbXDIKYqiBUMn1XzHtEUko6ABzOgR7+uRZqB4X7GLh7KaYL12mcWTRDtpFB8hGJ6iErlEZVRBFj+gZvaI348l4Md6Nj0lrxkhnttEfGZ8/cfOZ5g==</latexit>

x1


x2


7



H


h 2 H


Math facts!

H


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


✓


{b


x :✓> x/

k✓k=�b


x :✓> x/

k✓k>�b


x1


x2


7



H


h 2 H


Math facts!

H


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


✓


{b


x :✓> x/

k✓k=�b


x :✓> x/

k✓k>�b


x :✓> x/

k✓k<�b

<latexit sha1_base64="aF+JQpnQku0T3ScAai/FaF338cc=">AAACDXicbZC7SgNBFIZn4y3G26qlzWAUbIy7EvCCRdDGMoK5QDaG2ckkGTJ7YeasJKx5ARtfxcZCEVt7O9/G2WQLTfxh4OM/53Dm/G4ouALL+jYyc/MLi0vZ5dzK6tr6hrm5VVVBJCmr0EAEsu4SxQT3WQU4CFYPJSOeK1jN7V8l9do9k4oH/i0MQ9b0SNfnHU4JaKtl7g3OsQM9BuTOgSDEA3yEnYfUSugCH7otM28VrLHwLNgp5FGqcsv8ctoBjTzmAxVEqYZthdCMiQROBRvlnEixkNA+6bKGRp94TDXj8TUjvK+dNu4EUj8f8Nj9PRETT6mh5+pOj0BPTdcS879aI4LOaTPmfhgB8+lkUScSGAKcRIPbXDIKYqiBUMn1XzHtEUko6ABzOgR7+uRZqB4X7GLh7KaYL12mcWTRDtpFB8hGJ6iErlEZVRBFj+gZvaI348l4Md6Nj0lrxkhnttEfGZ8/buWZ5A==</latexit>

x1


x2


7



H


h 2 H


Math facts!

H


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


✓


{b


x :✓> x/

k✓k=�b


x :✓> x/

k✓k>�b


x1


x2


7



H


h 2 H


Math facts!

H


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


✓


{b


x :✓> x/

k✓k=�b


x :✓> x/

k✓k>�b


x :✓> x+

bk✓k =

0

<latexit sha1_base64="aPBEUlLKxshDruuYKi1Ph3lWvSY=">AAACEHicbZDLSsNAFIYnXmu9RV26GSyiIJRECl5AKLpxWcFeoIllMp22QycXZk6kJfYR3Pgqblwo4talO9/GSZuFtv4w8PGfczhzfi8SXIFlfRtz8wuLS8u5lfzq2vrGprm1XVNhLCmr0lCEsuERxQQPWBU4CNaIJCO+J1jd61+l9fo9k4qHwS0MI+b6pBvwDqcEtNUyDwbn2IEeA3KXOBBGIzzAR9jDzkNmp3SBrZZZsIrWWHgW7AwKKFOlZX457ZDGPguACqJU07YicBMigVPBRnknViwitE+6rKkxID5TbjI+aIT3tdPGnVDqFwAeu78nEuIrNfQ93ekT6KnpWmr+V2vG0Dl1Ex5EMbCAThZ1YoEhxGk6uM0loyCGGgiVXP8V0x6RhILOMK9DsKdPnoXacdEuFc9uSoXyZRZHDu2iPXSIbHSCyugaVVAVUfSIntErejOejBfj3fiYtM4Z2cwO+iPj8weya5sa</latexit>

x1


x2


7



H


h 2 H


Math facts!

H


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


✓


{b


x :✓> x/

k✓k=�b


x :✓> x/

k✓k>�b


x :✓> x+

bk✓k =

0

<latexit sha1_base64="aPBEUlLKxshDruuYKi1Ph3lWvSY=">AAACEHicbZDLSsNAFIYnXmu9RV26GSyiIJRECl5AKLpxWcFeoIllMp22QycXZk6kJfYR3Pgqblwo4talO9/GSZuFtv4w8PGfczhzfi8SXIFlfRtz8wuLS8u5lfzq2vrGprm1XVNhLCmr0lCEsuERxQQPWBU4CNaIJCO+J1jd61+l9fo9k4qHwS0MI+b6pBvwDqcEtNUyDwbn2IEeA3KXOBBGIzzAR9jDzkNmp3SBrZZZsIrWWHgW7AwKKFOlZX457ZDGPguACqJU07YicBMigVPBRnknViwitE+6rKkxID5TbjI+aIT3tdPGnVDqFwAeu78nEuIrNfQ93ekT6KnpWmr+V2vG0Dl1Ex5EMbCAThZ1YoEhxGk6uM0loyCGGgiVXP8V0x6RhILOMK9DsKdPnoXacdEuFc9uSoXyZRZHDu2iPXSIbHSCyugaVVAVUfSIntErejOejBfj3fiYtM4Z2cwO+iPj8weya5sa</latexit>

x :✓> x+

✓0=0

<latexit sha1_base64="R3ZMxDHQDlZqqzhmRrqwP8uSW24=">AAACCnicbVDJSgNBEO2JW4zbqEcvrUEQhDAjARcQgl48RjALZMahp9NJmvQsdNdIwpCzF3/FiwdFvPoF3vwbO8kcNPFBweO9Kqrq+bHgCizr28gtLC4tr+RXC2vrG5tb5vZOXUWJpKxGIxHJpk8UEzxkNeAgWDOWjAS+YA2/fz32Gw9MKh6FdzCMmRuQbsg7nBLQkmfuDy6wAz0G5D51IIpHeICPM8Wz8CW2PLNolawJ8DyxM1JEGaqe+eW0I5oELAQqiFIt24rBTYkETgUbFZxEsZjQPumylqYhCZhy08krI3yolTbuRFJXCHii/p5ISaDUMPB1Z0Cgp2a9sfif10qgc+amPIwTYCGdLuokAkOEx7ngNpeMghhqQqjk+lZMe0QSCjq9gg7Bnn15ntRPSna5dH5bLlausjjyaA8doCNko1NUQTeoimqIokf0jF7Rm/FkvBjvxse0NWdkM7voD4zPH7AKmPs=</latexit>

x1


x2


7



H


h 2 H


Math facts!

H


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


✓


{b


x :✓> x/

k✓k=�b


x :✓> x/

k✓k>�b


x :✓> x+

✓0=0


x1


x2


7



H


h 2 H


Math facts!

H


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


✓


{b


• Linear classifier:

x :✓> x/

k✓k=�b


x :✓> x/

k✓k>�b


x :✓> x+

✓0=0


x1


x2


7



H


h 2 H


Math facts!

H


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


✓


{b


• Linear classifier:h(x) = sign(✓>x+ ✓0)

<latexit sha1_base64="4uzB9A7HQ1dOL6uY8fODVh6Lx6o=">AAACF3icbVBNSwMxEM36bf2qevQSLEKLUHaloB4E0YvHCrYK3Vqy6bQNzWaXZFZalv4LL/4VLx4U8ao3/43px0FbHwy8vDdDZl4QS2HQdb+dufmFxaXlldXM2vrG5lZ2e6dqokRzqPBIRvouYAakUFBBgRLuYg0sDCTcBt3LoX/7ANqISN1gP4Z6yNpKtARnaKVGttjJ9wr0jPoIPdRhakRbDfI+dgDZvY9RTHv0kI7fDbfQyObcojsCnSXehOTIBOVG9stvRjwJQSGXzJia58ZYT5lGwSUMMn5iIGa8y9pQs1SxEEw9Hd01oAdWadJWpG0ppCP190TKQmP6YWA7Q4YdM+0Nxf+8WoKtk3oqVJwgKD7+qJVIihEdhkSbQgNH2beEcS3srpR3mGYcbZQZG4I3ffIsqR4VvVLx9LqUO7+YxLFC9sg+yROPHJNzckXKpEI4eSTP5JW8OU/Oi/PufIxb55zJzC75A+fzByWPnq4=</latexit>

x :✓> x/

k✓k=�b


x :✓> x/

k✓k>�b


x :✓> x+

✓0=0


x1


x2


7



H


h 2 H


Math facts!

H


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


✓


{b




=

⇢+1 if ✓>x+ ✓0 > 0�1 if ✓>x+ ✓0 < 0

<latexit sha1_base64="W3UJhhjSFFhW1P2tUwctJladKCs=">AAACeniclVFNaxsxENVu+pE4beOkxxIYYtIPTMxuMaSFtpjm0mMCdRKwXKOVZ20RrXaRZovNsj+if623/pJecqjs7KFNcukDwdObeTPSTFJo5SiKfgXhxoOHjx5vbrW2nzx9ttPe3Tt3eWklDmWuc3uZCIdaGRySIo2XhUWRJRovkquTVfziO1qncvOVlgWOMzEzKlVSkJcm7R8fgWtMiVfAE5wpUwlrxbKudA3dGDjhgmxWgUqh9rc5kvhWccqLGhbQbZRJBJ8gAs5bR/9h+bCyoJk2HYFbNZtTb9LuRL1oDbhL4oZ0WIPTSfsnn+ayzNCQ1MK5URwVNPZVSUmNdYuXDgshr8QMR54akaEbV+vR1XDolSmkufXHEKzVvx2VyJxbZonPzATN3e3YSrwvNiopfTeulClKQiNvGqWlBsphtQeYKouS9NITIa3ybwU5F1ZI8ttq+SHEt798l5y/7cX93vuzfmfwuRnHJnvBDthrFrNjNmBf2CkbMsl+B/vBy+BVcB0ehG/C7k1qGDSe5+wfhP0/nh69yw==</latexit>

x :✓> x/

k✓k=�b


x :✓> x/

k✓k>�b


x :✓> x+

✓0=0


x1


x2


7



H


h 2 H


Math facts!

H


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


✓


{b




=

⇢+1 if ✓>x+ ✓0 > 0�1 if ✓>x+ ✓0 < 0

<latexit sha1_base64="W3UJhhjSFFhW1P2tUwctJladKCs=">AAACeniclVFNaxsxENVu+pE4beOkxxIYYtIPTMxuMaSFtpjm0mMCdRKwXKOVZ20RrXaRZovNsj+if623/pJecqjs7KFNcukDwdObeTPSTFJo5SiKfgXhxoOHjx5vbrW2nzx9ttPe3Tt3eWklDmWuc3uZCIdaGRySIo2XhUWRJRovkquTVfziO1qncvOVlgWOMzEzKlVSkJcm7R8fgWtMiVfAE5wpUwlrxbKudA3dGDjhgmxWgUqh9rc5kvhWccqLGhbQbZRJBJ8gAs5bR/9h+bCyoJk2HYFbNZtTb9LuRL1oDbhL4oZ0WIPTSfsnn+ayzNCQ1MK5URwVNPZVSUmNdYuXDgshr8QMR54akaEbV+vR1XDolSmkufXHEKzVvx2VyJxbZonPzATN3e3YSrwvNiopfTeulClKQiNvGqWlBsphtQeYKouS9NITIa3ybwU5F1ZI8ttq+SHEt798l5y/7cX93vuzfmfwuRnHJnvBDthrFrNjNmBf2CkbMsl+B/vBy+BVcB0ehG/C7k1qGDSe5+wfhP0/nh69yw==</latexit>

x :✓> x/

k✓k=�b


x :✓> x/

k✓k>�b


x :✓> x+

✓0=0


x1


x2


7



H


h 2 H


Math facts!

H


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


✓


{b




=

⇢+1 if ✓>x+ ✓0 > 0�1 if ✓>x+ ✓0 0

<latexit sha1_base64="asNGSVAT/MAX9Si4UI95qS5DjgI=">AAACfHiclVFNTxsxEPVuP6CBtmk59sCoAYkqItqtEIUDCMGFI5UaQIrTyOvMJhZe78qeRUSr/RX8M278FC5VnbCHFnrpSJae38ybZ88khVaOoug+CF+8fPV6aflNa2X17bv37Q8fz11eWol9mevcXibCoVYG+6RI42VhUWSJxovk6mSev7hG61RuftCswGEmJkalSgry1Kh9ewBcY0q8Ap7gRJlKWCtmdaVr6MbACW/IZhWoFGp/myKJnxWnvKjhBroNM4rgECLgvLX9HxLvOxehGTeewK2aTKk3aneiXrQIeA7iBnRYE2ej9h0f57LM0JDUwrlBHBU09F1JSY11i5cOCyGvxAQHHhqRoRtWi+HVsOmZMaS59ccQLNg/FZXInJtlia/MBE3d09yc/FduUFK6N6yUKUpCIx+N0lID5TDfBIyVRUl65oGQVvm3gpwKKyT5fbX8EOKnX34Ozr/24p3e/vedztFxM45l9ol9ZlssZt/YETtlZ6zPJHsI1oOt4EvwK9wIu+H2Y2kYNJo19leEu78Bwiy+0A==</latexit>

x :✓> x/

k✓k=�b


x :✓> x/

k✓k>�b


x :✓> x+

✓0=0


x1


x2


7



H


h 2 H


Math facts!

H


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


✓


{b




=

⇢+1 if ✓>x+ ✓0 > 0�1 if ✓>x+ ✓0 0


x :✓> x/

k✓k=�b


x :✓> x/

k✓k>�b


x :✓> x+

✓0=0


x1


x2


7



H


h 2 H


Math facts!

H


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


✓


{b




=

⇢+1 if ✓>x+ ✓0 > 0�1 if ✓>x+ ✓0 0


x :✓> x/

k✓k=�b


x :✓> x/

k✓k>�b


x :✓> x+

✓0=0


x1


x2


7



H


h 2 H


Math facts!

H


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


✓


{b



=

⇢+1 if ✓>x+ ✓0 > 0�1 if ✓>x+ ✓0 0


h(x; ✓, ✓0) = sign(✓>x+ ✓0)

<latexit sha1_base64="5utf0Sqg7GOicEztyYZb4aYDjWI=">AAACKXicbVDLSgMxFM34tr6qLt0Ei1BRyowIKiIU3bisYKvQ1pJJb9vQTGZI7kjL0N9x46+4UVDUrT9i2g74PBByOOdcknv8SAqDrvvmTExOTc/Mzs1nFhaXlleyq2sVE8aaQ5mHMtTXPjMghYIyCpRwHWlggS/hyu+eDf2rW9BGhOoS+xHUA9ZWoiU4Qys1ssVOvndMa9gBZLvp3XC36Ynl0EMdJEa01SA/dm5qGEa0R3e+ko1szi24I9C/xEtJjqQoNbJPtWbI4wAUcsmMqXpuhPWEaRRcwiBTiw1EjHdZG6qWKhaAqSejTQd0yypN2gq1PQrpSP0+kbDAmH7g22TAsGN+e0PxP68aY+uwnggVxQiKjx9qxZJiSIe10abQwFH2LWFcC/tXyjtMM4623Iwtwfu98l9S2St4+4Wji/1c8TStY45skE2SJx45IEVyTkqkTDi5Iw/kmbw4986j8+q8j6MTTjqzTn7A+fgEUralfA==</latexit>

x :✓> x/

k✓k=�b


x :✓> x/

k✓k>�b


x :✓> x+

✓0=0


x1


x2


7



H


h 2 H


Math facts!

H


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


✓


{b



=

⇢+1 if ✓>x+ ✓0 > 0�1 if ✓>x+ ✓0 0


h(x; ✓, ✓0) = sign(✓>x+ ✓0)


x :✓> x/

k✓k=�b


x :✓> x/

k✓k>�b


x :✓> x+

✓0=0


x1


x2


7



H


h 2 H


Math facts!

H


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


✓


{b



=

⇢+1 if ✓>x+ ✓0 > 0�1 if ✓>x+ ✓0 0


h(x; ✓, ✓0) = sign(✓>x+ ✓0)


parameters

x :✓> x/

k✓k=�b


x :✓> x/

k✓k>�b


x :✓> x+

✓0=0


x1


x2


7



H


h 2 H


Math facts!

H


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


✓


{b



=

⇢+1 if ✓>x+ ✓0 > 0�1 if ✓>x+ ✓0 0


h(x; ✓, ✓0) = sign(✓>x+ ✓0)


parametersH = set of all such h

<latexit sha1_base64="y+stEct9Xxiq+NlhDxAYXfRywT4=">AAACE3icbVC7SgNBFJ31GeMramkzmAhiEXYloBZC0CZlBPOAJITZyd1kyOyDmbtiWPIPNv6KjYUitjZ2/o2zSQpNPDBwOOde7pzjRlJotO1va2l5ZXVtPbOR3dza3tnN7e3XdRgrDjUeylA1XaZBigBqKFBCM1LAfFdCwx3epH7jHpQWYXCHowg6PusHwhOcoZG6udO2z3DAmUwqY3pF2wgPqPxEA9LQo0xKqmM+oIVBYdzN5e2iPQFdJM6M5MkM1W7uq90LeexDgFwyrVuOHWEnYQoFlzDOtmMNEeND1oeWoQHzQXeSSaYxPTZKj3qhMi9AOlF/byTM13rku2YyTaDnvVT8z2vF6F10EhFEMULAp4e8WFIMaVoQ7QkFHOXIEMaVMH+lfMAU42hqzJoSnPnIi6R+VnRKxcvbUr58PasjQw7JETkhDjknZVIhVVIjnDySZ/JK3qwn68V6tz6mo0vWbOeA/IH1+QPa8Z2D</latexit>

x :✓> x/

k✓k=�b


x :✓> x/

k✓k>�b


x :✓> x+

✓0=0


x1


x2


8

How good is a classifier?

x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


• Should predict well on future data

8


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


• Should predict well on future data • How good is a classifier at a

single point?

8


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)



single point?

8


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


x


single point? Loss

8

L(g, a)

<latexit sha1_base64="pRoPrYTcPfAsF74TXT/JbgQpf5k=">AAAB7nicbVDLSgNBEOz1GeMr6tHLYBAiSNiVgHoLevHgIYJ5QLKE2UlvMmR2dpmZFULIR3jxoIhXv8ebf+Mk2YMmFjQUVd10dwWJ4Nq47rezsrq2vrGZ28pv7+zu7RcODhs6ThXDOotFrFoB1Si4xLrhRmArUUijQGAzGN5O/eYTKs1j+WhGCfoR7UseckaNlZr3pf45oWfdQtEtuzOQZeJlpAgZat3CV6cXszRCaZigWrc9NzH+mCrDmcBJvpNqTCgb0j62LZU0Qu2PZ+dOyKlVeiSMlS1pyEz9PTGmkdajKLCdETUDvehNxf+8dmrCK3/MZZIalGy+KEwFMTGZ/k56XCEzYmQJZYrbWwkbUEWZsQnlbQje4svLpHFR9irl64dKsXqTxZGDYziBEnhwCVW4gxrUgcEQnuEV3pzEeXHenY9564qTzRzBHzifP6dxjn0=</latexit>


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


x


single point? Loss

8

L(g, a)



x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


x

g: guess, a: actual


single point? Loss • Example: 0-1 loss

8

L(g, a)



x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


x

g: guess, a: actual



8

L(g, a)


L(g, a) =

⇢0 if g = a1 else

<latexit sha1_base64="cJSPUyx6dttJxW97TJokkxJUHKE=">AAACSnicbVBNaxRBFOzZRI3r16pHL48sQgRZZiSQeBCCXjx4iOAmge1ledP7ZrZJT8/Q/UZchvl9Xjzl5o/IJQdFvNi7GdAkFjQUVfVed1daGe05jr9HvY3NW7fvbN3t37v/4OGjweMnR76snaKxKk3pTlL0ZLSlMWs2dFI5wiI1dJyevlv5x5/JeV3aT7ysaFpgbnWmFXKQZgP8sJO/BHwBb0Aaylg2IFPKtW3QOVy2jWkhBsn0hV3RgM6ghTyEEaSE5K9BxlOwJNl5NwnS6XzBo9lgGI/iNeAmSToyFB0OZ4MzOS9VXZBlZdD7SRJXPA1bWStDbV/WnipUp5jTJFCLBflps66ihedBmUNWunAsw1r9d6LBwvtlkYZkgbzw172V+D9vUnO2P220rWomqy4vymoDXMKqV5hrR4rNMhBUToe3glqgQ8Wh/X4oIbn+5Zvk6NUo2R29/rg7PHjb1bElnoltsSMSsScOxHtxKMZCia/iXPwQP6Nv0UX0K/p9Ge1F3cxTcQW9zT/FwrEi</latexit>


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


x

g: guess, a: actual



8

L(g, a)


L(g, a) =

⇢0 if g = a1 else


• Example: asymmetric loss


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


x

g: guess, a: actual



8

L(g, a)


L(g, a) =

⇢0 if g = a1 else



L(g, a) =

8<

:

1 if g = 1, a = �1100 if g = �1, a = 10 else

<latexit sha1_base64="XCevrajL+qcOKd/Dewn/wEVPuag=">AAACdnicbZFBaxNBFMdnV1trrDXqSQR5NBRbaMKuBKoHoejFg4cIpi1kQpidvN0MmZ1dZt4Ww7IfwS/nzc/hxaOTdA817YNh/rz/782beZOUWjmKot9B+ODhzu6jvcedJ/tPD551n7+4cEVlJY5loQt7lQiHWhkckyKNV6VFkScaL5Pl57V/eY3WqcJ8p1WJ01xkRqVKCvKpWffn1+PsFMQJfASuMSVeA08wU6YW1opVU+sGYuCEP8jmNagUGsg8HPsiv/XXJoc4iu5h+i20QW4BqB16hKOZt12AW5UtaDDr9qJBtAm4K+JW9Fgbo1n3F58XssrRkNTCuUkclTT1p5KSGpsOrxyWQi5FhhMvjcjRTevN2Bo48pk5pIX1yxBssrcrapE7t8oTT+aCFm7bWyfv8yYVpe+ntTJlRWjkTaO00kAFrP8A5sqiJL3yQkir/F1BLoQVkvxPdfwQ4u0n3xUX7wbxcPDh27B3/qkdxx57zQ7ZMYvZGTtnX9iIjZlkf4JXwWHQC/6Gb8Kj8O0NGgZtzUv2X4TRP8jXt8U=</latexit>


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


x

g: guess, a: actual



8

L(g, a)


L(g, a) =

⇢0 if g = a1 else



• Test error (n’ new points):

L(g, a) =

8<

:

1 if g = 1, a = �1100 if g = �1, a = 10 else



x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


x

g: guess, a: actual



8

L(g, a)


L(g, a) =

⇢0 if g = a1 else




L(g, a) =

8<

:

1 if g = 1, a = �1100 if g = �1, a = 10 else


E(h) = 1

n0

n+n0X

i=n+1

L(h(x(i)), y(i))

<latexit sha1_base64="m8ItS+ygWb5M7h1+iuXfxKdM0KM=">AAACMXicbVDNSgMxGMz6W+tf1aOXYBG3KGVXCuqhUBTBgwcFW4VuLdk024Zms0uSFUvYV/Lim4gXD4p49SXM1j1odSAwzHwfX2b8mFGpHOfFmpqemZ2bLywUF5eWV1ZLa+stGSUCkyaOWCRufCQJo5w0FVWM3MSCoNBn5NofnmT+9R0Rkkb8So1i0glRn9OAYqSM1C2deSFSA4yYPk3tQQXWoRcIhLWbar6TQk8mYVfTOt9101vNdzPt3B7Y97fappW0sgdHOeuWyk7VGQP+JW5OyiDHRbf05PUinISEK8yQlG3XiVVHI6EoZiQteokkMcJD1CdtQzkKiezoceIUbhulB4NImMcVHKs/NzQKpRyFvpnM8slJLxP/89qJCg47mvI4UYTj70NBwqCKYFYf7FFBsGIjQxAW1PwV4gEyjSlTctGU4E5G/kta+1W3Vj26rJUbx3kdBbAJtoANXHAAGuAMXIAmwOABPINX8GY9Wi/Wu/XxPTpl5Tsb4Beszy9NnqfN</latexit>


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


x

g: guess, a: actual



8

L(g, a)


L(g, a) =

⇢0 if g = a1 else




L(g, a) =

8<

:

1 if g = 1, a = �1100 if g = �1, a = 10 else


E(h) = 1

n0

n+n0X

i=n+1

L(h(x(i)), y(i))


• Training error:


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


x

g: guess, a: actual



8

L(g, a)


L(g, a) =

⇢0 if g = a1 else




L(g, a) =

8<

:

1 if g = 1, a = �1100 if g = �1, a = 10 else


E(h) = 1

n0

n+n0X

i=n+1

L(h(x(i)), y(i))


• Training error: En(h) =1

n

nX

i=1

L(h(x(i)), y(i))

<latexit sha1_base64="3zNoi/fy75VtUErOFgvHrzVzVq8=">AAACLXicbVBLSwMxGMz6rPVV9eglWIQtSNmVgnooFB/gwYOCtUK3Ltk024Zms0uSFUvYP+TFvyKCB0W8+jfM1j34GghMZr6PZCZIGJXKcV6sqemZ2bn50kJ5cWl5ZbWytn4l41Rg0sYxi8V1gCRhlJO2ooqR60QQFAWMdILRUe53bomQNOaXapyQXoQGnIYUI2Ukv3LsRUgNMWL6JPO5PazBJvRCgbB2M80z6Mk08jVtutlNfj2zh/bdjbZpLavtwHHB/ErVqTsTwL/ELUgVFDj3K09eP8ZpRLjCDEnZdZ1E9TQSimJGsrKXSpIgPEID0jWUo4jInp6kzeC2UfowjIU5XMGJ+n1Do0jKcRSYyTyb/O3l4n9eN1Xhfk9TnqSKcPz1UJgyqGKYVwf7VBCs2NgQhAU1f4V4iExXyhRcNiW4vyP/JVe7dbdRP7hoVFuHRR0lsAm2gA1csAda4BScgzbA4B48ghfwaj1Yz9ab9f41OmUVOxvgB6yPT103pvI=</latexit>


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


x

g: guess, a: actual



8

L(g, a)


L(g, a) =

⇢0 if g = a1 else




L(g, a) =

8<

:

1 if g = 1, a = �1100 if g = �1, a = 10 else


E(h) = 1

n0

n+n0X

i=n+1

L(h(x(i)), y(i))


• Training error:

• Prefer to if h̃

<latexit sha1_base64="sbB1zTKgA0EBF8La76CIVwOiVGM=">AAAB8HicbVBNS8NAEN3Ur1q/qh69LBbBU0mkoN6KXjxWsB/ShrLZTNulu0nYnQgl9Fd48aCIV3+ON/+N2zYHbX0w8Hhvhpl5QSKFQdf9dgpr6xubW8Xt0s7u3v5B+fCoZeJUc2jyWMa6EzADUkTQRIESOokGpgIJ7WB8O/PbT6CNiKMHnCTgKzaMxEBwhlZ67KGQIWSjab9ccavuHHSVeDmpkByNfvmrF8Y8VRAhl8yYrucm6GdMo+ASpqVeaiBhfMyG0LU0YgqMn80PntIzq4R0EGtbEdK5+nsiY8qYiQpsp2I4MsveTPzP66Y4uPIzESUpQsQXiwappBjT2fc0FBo4yokljGthb6V8xDTjaDMq2RC85ZdXSeui6tWq1/e1Sv0mj6NITsgpOSceuSR1ckcapEk4UeSZvJI3RzsvzrvzsWgtOPnMMfkD5/MHLu2Qrg==</latexit>

h


En(h) =1

n

nX

i=1

L(h(x(i)), y(i))

<latexit sha1_base64="3zNoi/fy75VtUErOFgvHrzVzVq8=">AAACLXicbVBLSwMxGMz6rPVV9eglWIQtSNmVgnooFB/gwYOCtUK3Ltk024Zms0uSFUvYP+TFvyKCB0W8+jfM1j34GghMZr6PZCZIGJXKcV6sqemZ2bn50kJ5cWl5ZbWytn4l41Rg0sYxi8V1gCRhlJO2ooqR60QQFAWMdILRUe53bomQNOaXapyQXoQGnIYUI2Ukv3LsRUgNMWL6JPO5PazBJvRCgbB2M80z6Mk08jVtutlNfj2zh/bdjbZpLavtwHHB/ErVqTsTwL/ELUgVFDj3K09eP8ZpRLjCDEnZdZ1E9TQSimJGsrKXSpIgPEID0jWUo4jInp6kzeC2UfowjIU5XMGJ+n1Do0jKcRSYyTyb/O3l4n9eN1Xhfk9TnqSKcPz1UJgyqGKYVwf7VBCs2NgQhAU1f4V4iExXyhRcNiW4vyP/JVe7dbdRP7hoVFuHRR0lsAm2gA1csAda4BScgzbA4B48ghfwaj1Yz9ab9f41OmUVOxvgB6yPT103pvI=</latexit>

En(h) < En(h̃)

<latexit sha1_base64="wyowSN7UmYLt9Uiw1bnJKcDlVPk=">AAACFHicbVDLSsNAFJ3UV62vqEs3g0VoEUoiBRVcFEVwWcE+oAllMp00QyeTMDMRSuhHuPFX3LhQxK0Ld/6NkzYL23rgwuGce7n3Hi9mVCrL+jEKK6tr6xvFzdLW9s7unrl/0JZRIjBp4YhFoushSRjlpKWoYqQbC4JCj5GON7rJ/M4jEZJG/EGNY+KGaMipTzFSWuqbp06IVIARS28nfV4JqvAKzkuOomxA0mBS7Ztlq2ZNAZeJnZMyyNHsm9/OIMJJSLjCDEnZs61YuSkSimJGJiUnkSRGeISGpKcpRyGRbjp9agJPtDKAfiR0cQWn6t+JFIVSjkNPd2b3ykUvE//zeonyL9yU8jhRhOPZIj9hUEUwSwgOqCBYsbEmCAuqb4U4QAJhpXMs6RDsxZeXSfusZtdrl/f1cuM6j6MIjsAxqAAbnIMGuANN0AIYPIEX8AbejWfj1fgwPmetBSOfOQRzML5+AQzknjk=</latexit>


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


x

g: guess, a: actual

9

• Have data; have hypothesis class • Want to choose a good classifier

• Recall:

Learning a classifier

x1


x2


x


h


y


9


• Recall:


x1


x2


x


h


y


9


• Recall:


x1


x2


x


h


y


x

9


• Recall:


x1


x2


x


h


y


9


x1


x2


x


h


y



• Recall: • New:

9


x1


x2


x


h


y




h


Dn

<latexit sha1_base64="7Uc1mRxWy2VAobYcZ46dW/jF8w8=">AAAB9HicbVDLSgMxFL2pr1pfVZdugkVwVWakoO6KunBZwT6gHUomzbShmcyYZApl6He4caGIWz/GnX9jpp2Fth4IHM65l3ty/FhwbRznGxXW1jc2t4rbpZ3dvf2D8uFRS0eJoqxJIxGpjk80E1yypuFGsE6sGAl9wdr++Dbz2xOmNI/ko5nGzAvJUPKAU2Ks5PVCYkaUiPRu1pf9csWpOnPgVeLmpAI5Gv3yV28Q0SRk0lBBtO66Tmy8lCjDqWCzUi/RLCZ0TIasa6kkIdNeOg89w2dWGeAgUvZJg+fq742UhFpPQ99OZiH1speJ/3ndxARXXsplnBgm6eJQkAhsIpw1gAdcMWrE1BJCFbdZMR0RRaixPZVsCe7yl1dJ66Lq1qrXD7VK/SavowgncArn4MIl1OEeGtAECk/wDK/whiboBb2jj8VoAeU7x/AH6PMHAsSSRw==</latexit>

learning algorithm

9


x


h


y


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)




h


Dn


learning algorithm

9


x


h


y


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)




h


Dn


learning algorithm

9


x


h


y


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


x1


x2


+–––

–

–

–+

+––

–

++

–+

–

–

+

–– –

+

–––



h


Dn


learning algorithm

9


x


h


y


x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


x1


x2


+–––

–

–

–+

+––

–

++

–+

–

–

+

–– –

+

–––



h


Dn


learning algorithm

10




x


h


y


h


Dn


learning algorithm

x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)





x


h


y


h


Dn


learning algorithm

x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


• Example:

10




x


h


y


h


Dn


learning algorithm

x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


• Example:

10




x


h


y


h


Dn


learning algorithm

x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


• Example:for j = 1, …, 1 trillion

10




x


h


y


h


Dn


learning algorithm

x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


• Example:for j = 1, …, 1 trillion Randomly sample (✓(j), ✓(j)0 )

<latexit sha1_base64="/WnBIhUy+K6S5kMys2ts25oeKdk=">AAACCHicbVC7SgNBFJ2Nrxhfq5YWDgYhAQm7ElC7oI1lBPOAZF1mJ5NkzOyDmbtCWFLa+Cs2ForY+gl2/o2T7BaaeGDg3HPu5c49XiS4Asv6NnJLyyura/n1wsbm1vaOubvXVGEsKWvQUISy7RHFBA9YAzgI1o4kI74nWMsbXU391gOTiofBLYwj5vhkEPA+pwS05JqHpS4MGZC7pHRfnpzgtHKttC5j1yxaFWsGvEjsjBRRhrprfnV7IY19FgAVRKmObUXgJEQCp4JNCt1YsYjQERmwjqYB8ZlyktkhE3yslR7uh1K/APBM/T2REF+pse/pTp/AUM17U/E/rxND/9xJeBDFwAKaLurHAkOIp6ngHpeMghhrQqjk+q+YDokkFHR2BR2CPX/yImmeVuxq5eKmWqxdZnHk0QE6QiVkozNUQ9eojhqIokf0jF7Rm/FkvBjvxkfamjOymX30B8bnD0slmD8=</latexit>

10




x


h


y


h


Dn


learning algorithm

x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)


• Example:for j = 1, …, 1 trillion Randomly sample Set

(✓(j), ✓(j)0 )


h(j)(x) = h(x; ✓(j), ✓(j)0 )

<latexit sha1_base64="keB7458i3rGAYsDmCjPQ8C6RtXU=">AAACGHicbVDLSgMxFM3UV62vUZdugkVoQeqMFFREKLpxWcE+oK0lk6ad2MyD5I60DP0MN/6KGxeKuO3OvzFtZ6GtBwLnnnMvN/c4oeAKLOvbSC0tr6yupdczG5tb2zvm7l5VBZGkrEIDEci6QxQT3GcV4CBYPZSMeI5gNad/M/FrT0wqHvj3MAxZyyM9n3c5JaCltnniPsS5x/woN8jjK+zmBpe4CS4DMpOPk6ptzep828xaBWsKvEjshGRRgnLbHDc7AY085gMVRKmGbYXQiokETgUbZZqRYiGhfdJjDU194jHViqeHjfCRVjq4G0j9fMBT9fdETDylhp6jOz0Crpr3JuJ/XiOC7nkr5n4YAfPpbFE3EhgCPEkJd7hkFMRQE0Il13/F1CWSUNBZZnQI9vzJi6R6WrCLhYu7YrZ0ncSRRgfoEOWQjc5QCd2iMqogip7RK3pHH8aL8WZ8Gl+z1pSRzOyjPzDGP7E+nbk=</latexit>

10




x


h


y


h


Dn


learning algorithm

x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)



(✓(j), ✓(j)0 )


h(j)(x) = h(x; ✓(j), ✓(j)0 )


10

Ex_learning_alg




x


h


y


h


Dn


learning algorithm

x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)



(✓(j), ✓(j)0 )


h(j)(x) = h(x; ✓(j), ✓(j)0 )


10

Ex_learning_alg( )




x


h


y


h


Dn


learning algorithm

x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)



(✓(j), ✓(j)0 )


h(j)(x) = h(x; ✓(j), ✓(j)0 )


Dn


10

Ex_learning_alg( ; < 1 trillion)




x


h


y


h


Dn


learning algorithm

x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)



(✓(j), ✓(j)0 )


h(j)(x) = h(x; ✓(j), ✓(j)0 )


k

<latexit sha1_base64="ESqfkpoOzQTKxIV+zY20hF3GUbw=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0mkoN6KXjy2YGuhDWWznbRrN5uwuxFK6C/w4kERr/4kb/4bt20O2vpg4PHeDDPzgkRwbVz32ymsrW9sbhW3Szu7e/sH5cOjto5TxbDFYhGrTkA1Ci6xZbgR2EkU0igQ+BCMb2f+wxMqzWN5byYJ+hEdSh5yRo2VmuN+ueJW3TnIKvFyUoEcjX75qzeIWRqhNExQrbuemxg/o8pwJnBa6qUaE8rGdIhdSyWNUPvZ/NApObPKgISxsiUNmau/JzIaaT2JAtsZUTPSy95M/M/rpia88jMuk9SgZItFYSqIicnsazLgCpkRE0soU9zeStiIKsqMzaZkQ/CWX14l7YuqV6teN2uV+k0eRxFO4BTOwYNLqMMdNKAFDBCe4RXenEfnxXl3PhatBSefOYY/cD5/ANZljPs=</latexit>

Dn


10

Ex_learning_alg( ; < 1 trillion)




x


h


y


h


Dn


learning algorithm

x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)



(✓(j), ✓(j)0 )


h(j)(x) = h(x; ✓(j), ✓(j)0 )


k


Dn


hyperparameter

10

Ex_learning_alg( ; < 1 trillion) Set




x


h


y


h


Dn


learning algorithm

x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)



(✓(j), ✓(j)0 )


h(j)(x) = h(x; ✓(j), ✓(j)0 )


k


Dn


j⇤ = argminj2{1,...,k}En(h(j))

<latexit sha1_base64="uDyqnxwu54Ntrg6zX7ZLen/NAU0=">AAACLnicbVBNSwMxEM36bf2qevQSLEIrpeyKoB4EUQSPFawK3bZk07RNm2SXZFYsy/4iL/4VPQgq4tWfYfpxUOuDgcd7M8zMCyLBDbjuqzM1PTM7N7+wmFlaXlldy65vXJsw1pRVaChCfRsQwwRXrAIcBLuNNCMyEOwm6J0N/Js7pg0P1RX0I1aTpK14i1MCVmpkz7v1XXyMfWD3oGVCdFtylTaSLva5wn7iFX3RDMEUe36aYl8S6FAiknPbotJ8p57ku4W00Mjm3JI7BJ4k3pjk0BjlRvbZb4Y0lkwBFcSYqudGULPrgVPB0owfGxYR2iNtVrVUEclMLRm+m+IdqzRxK9S2FOCh+nMiIdKYvgxs5+Be89cbiP951Rhah7WEqygGpuhoUSsWGEI8yA43uWYURN8SQjW3t2LaIZpQsAlnbAje35cnyfVeydsvHV3u505Ox3EsoC20jfLIQwfoBF2gMqogih7QE3pD786j8+J8OJ+j1ilnPLOJfsH5+gZmSajK</latexit>

hyperparameter

10




x


h


y


h


Dn


learning algorithm

x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)



Ex_learning_alg( ; < 1 trillion) Set Return

(✓(j), ✓(j)0 )


h(j)(x) = h(x; ✓(j), ✓(j)0 )


k


Dn




h(j⇤)

<latexit sha1_base64="6XLMSHq7IrETni4CCNBGbzWszq4=">AAAB8HicbVDLSgNBEOyNrxhfUY9eBoMQPYRdCai3oBePEcxDkk2YncwmY2Zml5lZISz5Ci8eFPHq53jzb5w8DppY0FBUddPdFcScaeO6305mZXVtfSO7mdva3tndy+8f1HWUKEJrJOKRagZYU84krRlmOG3GimIRcNoIhjcTv/FElWaRvDejmPoC9yULGcHGSg+DTlp87Jydjrv5gltyp0DLxJuTAsxR7ea/2r2IJIJKQzjWuuW5sfFTrAwjnI5z7UTTGJMh7tOWpRILqv10evAYnVilh8JI2ZIGTdXfEykWWo9EYDsFNgO96E3E/7xWYsJLP2UyTgyVZLYoTDgyEZp8j3pMUWL4yBJMFLO3IjLAChNjM8rZELzFl5dJ/bzklUtXd+VC5XoeRxaO4BiK4MEFVOAWqlADAgKe4RXeHOW8OO/Ox6w148xnDuEPnM8f9huP4Q==</latexit>

hyperparameter

10




x


h


y


h


Dn


learning algorithm

x1


x2


+

–––

–

–

–

++ –

–

–

++

–

+–

–

+––

–+ x(1)


x(2)


–––

x(3)



Ex_learning_alg( ; < 1 trillion) Set Return

• How does training error of Ex_learning_alg( ;1) compare to the training error of Ex_learning_alg( ;2)?

(✓(j), ✓(j)0 )


h(j)(x) = h(x; ✓(j), ✓(j)0 )


k


Dn




h(j⇤)

<latexit sha1_base64="6XLMSHq7IrETni4CCNBGbzWszq4=">AAAB8HicbVDLSgNBEOyNrxhfUY9eBoMQPYRdCai3oBePEcxDkk2YncwmY2Zml5lZISz5Ci8eFPHq53jzb5w8DppY0FBUddPdFcScaeO6305mZXVtfSO7mdva3tndy+8f1HWUKEJrJOKRagZYU84krRlmOG3GimIRcNoIhjcTv/FElWaRvDejmPoC9yULGcHGSg+DTlp87Jydjrv5gltyp0DLxJuTAsxR7ea/2r2IJIJKQzjWuuW5sfFTrAwjnI5z7UTTGJMh7tOWpRILqv10evAYnVilh8JI2ZIGTdXfEykWWo9EYDsFNgO96E3E/7xWYsJLP2UyTgyVZLYoTDgyEZp8j3pMUWL4yBJMFLO3IjLAChNjM8rZELzFl5dJ/bzklUtXd+VC5XoeRxaO4BiK4MEFVOAWqlADAgKe4RXeHOW8OO/Ox6w148xnDuEPnM8f9huP4Q==</latexit>

Dn


Dn


hyperparameter

10

Documents

6.036/6.862: Introduction to Machine Learning