UVA CS 4501: Machine Learning Lecture 12: Probability Review · Machine Learning Lecture 12: Probability Review Dr. Yanjun Qi University of Virginia Department of Computer Science

UVACS4501:MachineLearning

Lecture12:ProbabilityReview

Dr.YanjunQi

UniversityofVirginia

DepartmentofComputerScience

Today:ProbabilityReview

•  Thebigpicture•  EventsandEventspaces•  Randomvariables•  Jointprobability,MarginalizaHon,condiHoning,chainrule,BayesRule,lawoftotalprobability,etc.

•  StructuralproperHes,e.g.,Independence,condiHonalindependence

•  MaximumLikelihoodEsHmaHon10/31/18 2

Dr.YanjunQi/UVACS6316/f16

TheBigPicture

Modeli.e.DatageneraHng

process

ObservedData

Probability

EsEmaEon/learning/Inference/Datamining

10/31/18 3


Probability

•  CounHng•  Basicsofprobability•  CondiHonalprobability•  Randomvariables•  DiscreteandconHnuousdistribuHons•  ExpectaHonandvariance•  Tailboundsandcentrallimittheorem•  ……

StaHsHcs

•  MaximumlikelihoodesHmaHon•  BayesianesHmaHon•  HypothesistesHng•  Linearregression•  [Machinelearning]•  ……

Probabilityasfrequency

•  ConsiderthefollowingquesHons:– 1.WhatistheprobabilitythatwhenIflipacoinitis“heads”?

– 2.why?– 3.WhatistheprobabilityofBlueRidgeMountainstohaveanerupHngvolcanointhenearfuture?

10/31/18 6

Message:Thefrequen*stviewisveryuseful,butitseemsthatwecanalsousedomainknowledgetocomeupwithprobabili*es.


Wecancountè~1/2

ècouldnotcount

AdaptfromProf.NandodeFreitas’sreviewslides

Probabilityasameasureofuncertainty

10/31/18 7

•  Imaginewearethrowingdartsatawallofsize1x1andthatalldartsareguaranteedtofallwithinthis1x1wall.

•  Whatistheprobabilitythatadartwillhittheshadedarea?



Probabilityasameasureofuncertainty

•  Probabilityisameasureofcertaintyofaneventtakingplace.

•  i.e.intheexample,weweremeasuringthechancesofhi?ngtheshadedarea.

10/31/18 8

prob = #RedBoxes#Boxes



Probability

10/31/18 10

O:ElementaryEvent“Throw2”

Odie={1,2,3,4,5,6}

TheelementsofOarecalledelementaryevents.


Probability

•  Probabilityallowsustomeasuremanyevents.•  TheeventsaresubsetsofthesamplespaceO.Forexample,foradiewemayconsiderthefollowingevents:e.g.,

•  Assignprobabili7estotheseevents:e.g.,

10/31/18 11

EVEN={2,4,6}GREATER={5,6}

P(EVEN)=1/2



SamplespaceandEvents

•  O : SampleSpace,•  resultofanexperiment/setofalloutcomes

•  IfyoutossacointwiceO = {HH,HT,TH,TT}

•  Event:asubsetofO•  Firsttossishead={HH,HT}

•  S:eventspace,asetofevents:•  ContainstheemptyeventandO10/31/18 12


10/31/18


13

AxiomsforProbability

•  Definedover(O,S) s.t.•  1>=P(a)>=0forallainS•  P(O)=1

•  IfA, Baredisjoint,then•  P(AUB)=p(A)+p(B)

10/31/18 14


AxiomsforProbability

10/31/18


15

B1

B2B3B4

B5

B6B7

P Bi( )∑• P(O)=

ORoperaHonforProbability

•  Wecandeduceotheraxiomsfromtheaboveones• Ex:P(AUB)fornon-disjointevents

10/31/18 16


P(UnionofAsetandBset)

A

10/31/18 17


A

B

10/31/18 18


P(IntersecHonofAandB)

A

B

10/31/18 19


FromEventstoRandomVariable

•  Concisewayofspecifyingaqributesofoutcomes•  Modelingstudents(GradeandIntelligence):

•  O = allpossiblestudents(samplespace)•  Whatareevents(subsetofsamplespace)

•  Grade_A=allstudentswithgradeA•  Grade_B=allstudentswithgradeB•  HardWorking_Yes=…whoworkshard

•  Verycumbersome

•  Need“funcHons”thatmapsfromO toanaqributespaceT.•  P(H=YES)=P({studentϵO : H(student)=YES})10/31/18 21


RandomVariables(RV)O

Yes

No

A

B A+

H:hardworking

G:Grade

P(H=Yes)=P({allstudentswhoisworkinghardonthecourse})

10/31/18 22


• “funcHons”thatmapsfromO toanaqributespaceT.

NotaHons

•  P(A)isshorthandforP(A=true)•  P(~A)isshorthandforP(A=false)•  SamenotaHonappliestootherbinaryRVs:P(Gender=M),P(Gender=F)

•  SamenotaHonappliestomul*valuedRVs:P(Major=history),P(Age=19),P(Q=c)

•  Note:uppercaseleqers/namesforvariables,lowercaseleqers/namesforvalues

10/31/18 23


DiscreteRandomVariables

•  Randomvariables(RVs)whichmaytakeononlyacountablenumberofdisEnctvalues

•  XisaRVwitharitykifitcantakeonexactlyonevalueoutof{x1,…,xk}

10/31/18 24


ProbabilityofDiscreteRV

•  ProbabilitymassfuncHon(pmf):P(X=xi)•  Easyfactsaboutpmf

§  ΣiP(X=xi)=1§  P(X=xi∩X=xj)=0ifi≠j§  P(X=xiUX=xj)=P(X=xi)+P(X=xj)ifi≠j§  P(X=x1UX=x2U…UX=xk)=1

10/31/18


25

e.g.CoinFlips

•  Youflipacoin– Headwithprobabilityp,e.g.=0.5

•  Youflipacoinfork,e.g.,=100Hmes– Howmanyheadswouldyouexpect

10/31/18


26

e.g.CoinFlipscont.

•  Youflipacoin– Headwithprobabilityp– Binaryrandomvariable– Bernoullitrialwithsuccessprobabilityp

•  YouflipacoinforkHmes– Howmanyheadswouldyouexpect– NumberofheadsXisadiscreterandomvariable– BinomialdistribuHonwithparameterskandp

10/31/18


27

DiscreteRandomVariables

•  Randomvariables(RVs)whichmaytakeononlyacountablenumberofdisEnctvalues– E.g.thetotalnumberofheadsXyougetifyouflip100coins

•  XisaRVwitharitykifitcantakeonexactlyonevalueoutof– E.g.thepossiblevaluesthatXcantakeonare0,1,2,…,100

x1,…,xk{ }

10/31/18


28

e.g.,twoCommonDistribuHons

•  Uniform– Xtakesvalues1,2,…,N–  – E.g.pickingballsofdifferentcolorsfromabox

•  Binomial– Xtakesvalues0,1,…,k

–  – E.g.coinflipskHmes

X ∼U 1,..., N⎡⎣ ⎤⎦

( )P X 1i N= =

X ∼ Bin k, p( )

P X = i( ) = k

i⎛

⎝⎜⎞

⎠⎟pi 1− p( )k−i

10/31/18


29



•  StructuralproperHes•  Independence,condiHonalindependence

10/31/18 30


CondiHonal/Joint/MarginalProbability

A

B

10/31/18 31


fromProf.NandodeFreitas’sreview

IfhardtodirectlyesHmatefromdata,mostlikelywecanesHmate

•  1.Jointprobability– UseChainRule

•  2.Marginalprobability– Usethetotallawofprobability

•  3.CondiHonalprobability– UsetheBayesRule

10/31/18


32

(1).TocalculateJointProbability:UseChainRule

•  Twowaystousechainrulesonjointprobability

10/31/18 33

P(A,B)=p(B|A)p(A)P(A,B)=p(A|B)p(B)


(2).TocalculateMarginalProbability:UseRuleoftotalprobability(Eventversion)

A

B1

B2B3B4

B5

B6B7

p A( ) = P Bi( )P A | Bi( )∑ WHY???

10/31/18 34


(2).TocalculateMarginalProbability:UseRuleoftotalprobability(RVversion)

•  GiventwodiscreteRVsXandY,whichtakevaluesinand,Wehave

x1,…,xk{ } y1,…, ym{ }

( ) ( )( ) ( )

P X P X Y

P X Y P Y

i i jj

i j jj

x x y

x y y

= = = ∩ =

= = = =

∑∑

10/31/18


35

(3).TocalculateCondiHonalProbability:UseBayesRule

•  istheprobabilityof,giventheoccurrenceof

( )P X Yx y= =

( ) ( )( )P X Y

P X YP Yx y

x yy

= ∩ == = =

=

X x=Y y=

10/31/18


36

BayesRule

•  XandYarediscreteRVs…

( ) ( ) ( )( ) ( )P Y X P X

P X YP Y X P X

j i ii j

j k kk

y x xx y

y x x

= = == = =

= = =∑

( ) ( )( )P X Y

P X YP Yx y

x yy

= ∩ == = =

=

10/31/18


37 x1,…,xk{ }

10/31/18 38


BayesRule

P(A = a1 | B) =P(B | A = a1)P(A = a1)P(B | A = ai )P(A = ai )

i∑

10/31/18 39


10/31/18


40

OneExample

P (B1 = r|B2 = r)

P (B2 = r)

OneExample:Joint

10/31/18 41



OneExample:Joint

10/31/18 42



OneExample:Joint

10/31/18 43



OneExample:Marginal

10/31/18 44


OneExample:Marginal

10/31/18 45


OneExample:CondiHonal

10/31/18 46


SimplifyNotaHon:CondiHonalProbability

( ) ( )( )P X Y

P X YP Yx y

x yy

= ∩ == = =

=

( ))(),(|

ypyxpyxP =

Butwewillalwayswriteitthisway:

events

X=x

Y=y

10/31/18 47


P(X=xtrue)->P(X=x)->P(x)

SimplifyNotaHon:CondiHonal

•  BayesRule

•  YoucancondiHononmorevariables

( ))|(

),|()|(,|zyP

zxyPzxPzyxP =

10/31/18 48


P x | y( ) = P(x)P(y | x)P(y)

SimplifyNotaHon:Marginal

•  Weknowp(X,Y),whatisP(Y=y)orP(X=x)?

•  Wecanusethelawoftotalprobability( ) ( )

( ) ( )∑

∑=

=

y

y

yxPyP

yxPxp

|

,

10/31/18 49


y1,…, ym{ }

( ) ( )

( ) ( )∑

∑=

=

yz

zy

zyxPzyP

zyxPxp

,

,

,|,

,,

SimplifyNotaHon:AnExample

•  WeknowthatP(rain)=0.5•  Ifwealsoknowthatthegrassiswet,thenhowthisaffectsourbeliefaboutwhetheritrainsornot?

€

P rain |wet( ) = P(rain)P(wet | rain)P(wet)

10/31/18 50




€


€

P x | y( ) = P(x)P(y | x)P(y)

10/31/18 51




€


10/31/18 52


IndependentRVs

•  IntuiHon:XandYareindependentmeansthatneithermakesitmoreorlessprobablethat

•  DefiniHon:XandYareindependentiff( ) ( ) ( )P X Y P X P Yx y x y= ∩ = = = =

X x=Y y=

10/31/18


54

MoreonIndependence

•  E.g.nomaqerhowmanyheadsyouget,yourfriendwillnotbeaffected,andviceversa

( ) ( )P X Y P Xx y x= = = =( ) ( )P Y X P Yy x y= = = =

( ) ( ) ( )P X Y P X P Yx y x y= ∩ = = = =

10/31/18


55

MoreonIndependence

•  XisindependentofYmeansthatknowingYdoesnotchangeourbeliefaboutX.Thefollowingformsareequivalent:•  P(X=x,Y=y)=P(X=x)P(Y=y)•  P(X=x|Y=y)=P(X=x)

•  Theaboveshouldholdforallxi,yj•  Itissymmetricandwriqenas

10/31/18


!X ⊥Y56

CondiHonallyIndependentRVs

•  IntuiHon:XandYarecondiHonallyindependentgivenZmeansthatonceZisknown,thevalueofXdoesnotaddanyaddiEonalinformaHonaboutY

•  DefiniHon:XandYarecondiHonallyindependentgivenZiff

( ) ( ) ( )P X Y Z P X Z P Y Zx y z x z y z= ∩ = = = = = = =10/31/18


Ifholdingforallxi,yj,zk !!X ⊥Y |Z 57

MoreonCondiHonalIndependence

( ) ( ) ( )P X Y Z P X Z P Y Zx y z x z y z= ∩ = = = = = = =

( ) ( )P X Y ,Z P X Zx y z x z= = = = = =

( ) ( )P Y X ,Z P Y Zy x z y z= = = = = =10/31/18


58

TodayRecap:ProbabilityReview



•  MaximumLikelihoodEsHmaHon(nextclass)10/31/18 59


independenceandcondiHonalindependence

•  IndependencedoesnotimplycondiHonalindependence.

•  CondiHonalindependencedoesnotimplyindependence.

10/31/18


60

References

q Prof.AndrewMoore’sreviewtutorialq Prof.NandodeFreitas’sreviewslidesq Prof.CarlosGuestrinrecitaHonslides

10/31/18 61


Documents

UVA CS 4501: Machine Learning Lecture 12: Probability Review · Machine Learning Lecture 12: Probability Review Dr. Yanjun Qi University of Virginia Department of Computer Science