情報工学実験4：データマイニング班 - University of the Ryukyustnal/2016/info4/dm/2016info4...2015年度：情報工学実験4：データマイニング班 2 1 sample

情報工学実験4：データマイニング班(week3)線形回帰モデルと最急降下法

1. 復習2. scikit-learn入門3. モデルとは？（問題設定、アルゴリズム、モデル）4. 線形回帰モデル5. 仮説、損失関数、目的関数6. 最小二乗法7. 最急降下法8. 参考サイト

2016年度：情報工学実験4：データマイニング班 1

実験ページ:http://ie.u-ryukyu.ac.jp/~tnal/2015/info4/dm/

Example:Iris flowerdatasethttp://en.wikipedia.org/wiki/Iris_flower_data_set

• Classification– InClassification,thesamplesbelongtotwoormoreclassesandwewanttolearnfromalreadylabeleddatahowtopredicttheclassofunlabeleddata.

– E.g.,distinguishesthespeciesfromeachother.– Dataset=samples vs.features andclasses


1sample

- Inputdata,X- 4featuresorattributes

- Teachdata- supervisorysignal- outputdata,Y- target- 1classin3classes

(1)WhatisexperienceE?(2)WhatistaskT?

(3)HowtomeasuretheperformanceP?

review

Example:boston housepricesdatasethttp://archive.ics.uci.edu/ml/datasets/Housing

• Regression– Ifthedesiredoutputconsistsofoneormorecontinuousvariables,thenthetaskiscalledregression.

– E.g.,concernshousingvaluesinsuburbsofBoston.– Dataset=samples vs.features andcontinuousvariables


1sample

13features Continuousvariable



CRIM ZN INDUS （中略） LSTAT MEDV

6.32E-03 1.80E+01 2.31E+00 4.98E+00 24.00

2.73E-02 0.00E+00 7.07E+00 9.14E+00 21.60

2.73E-02 0.00E+00 7.07E+00 4.03E+00 34.70

review

Example:Iris flowerdatasethttp://en.wikipedia.org/wiki/Iris_flower_data_set

• Classification– InClassification,thesamplesbelongtotwoormoreclassesandwewanttolearnfromalreadylabeleddatahowtopredicttheclassofunlabeleddata.

– E.g.,distinguishesthespeciesfromeachother.




review

Terminology• supervised,unsupervised

learning• classification,regression,

clustering• sample• features,attributes

– numericalvalue– categoricalvalue– trueorfalse

• supervisorysignal,teacher,class,label,outputdata,targetvariable

• input,output• trainingdata/trainingset• testdata/testset

– opentest– closetest

• model• parameters• learn,fit• predict,estimate• evaluation


review





Anintroductiontomachinelearningwithscikit-learn(1/3)

• Loadingandanexampledataset– python–version

• Python 3.5.2::Anaconda4.1.1(x86_64)– python– >>>fromsklearn importdatasets– >>>iris=datasets.load_iris()#datasets.load[tab]– >>>print(iris.DESCR)– >>>print(iris.data)– >>>print(iris.target)– >>>print(iris.target_names)


http://scikit-learn.org/stable/tutorial/basic/tutorial.html

hgclonessh://info3dm@shark//home/info3dm/HG/tnallesssklearn_intro.py


• Learningandpredicting– >>>fromsklearn importsvm– >>>clf =svm.SVC(gamma=0.001,C=100.)– >>>clf.fit(iris.data[:-1],iris.target[:-1])– >>>clf.predict(iris.data[-1]]– >>>print(iris.target[-1])




• Modelpersistence– #save– >>>importpickle– >>>file=open("PredictiveModel.dump","wb")– >>>pickle.dump(clf,file)– >>>file.close()– #load– >>>file=open("PredictiveModel.dump","rb")– >>>clf2=pickle.loads(file)– >>>file.close()– >>>clf2.predict(iris.data[-1]]– >>>print(iris.target[-1])







Problems,Models,Algorithms

• Problems– Classification– Regression– Clustering

• Algorithms– OrdinaryLeastSquares– GradientDescent– BackPropagation

• Models– LinearRegressionModel– GeneralizedLinearModels

– NeuralNetwork– DecisionTree– (otherkindsofmodels)

• Bag-of-wordsdocumentmodel


Whatisthat?

Models

• Representbyanyformulaswith(sometimesone)parameters fortherelationshipbetweeninputX’sandoutputY’s.– Inmachinelearning,theformulascalledas“hypothesis”.

– E.g.,h =a*x+b• a,b:parameters

– Parameterizedmodel.– Predictivemodel.(e.g.,a=1,b=2)


-5 -2.5 0 2.5 5

-2.5

2.5

Problem<->Algorithm+Model


Dataset

LearningAlgorithm

hInputX’s Estimatedvalue’s

hθ (x)=θ0x0 +θ1x1 = θi xi∑ = θiΦi (x)∑hθ (x)=θ0 +θ1x

LinearRegressionModel

• Howdoweprepareamodel?• Howdoweevaluatethegoodness?• Howdowechoosetheappropriateparameters?





LinearRegressionModel• Trainingdatasets– (x,y)=(4,7),(8,10),(13,11),(17,14)

• Hypothesis

• Parameters– θ0,θ1

• Costfunction

• Objectivefunction (measurementofthegoodness)


J(θ0,θ1) =12m

(hθ (x(i) )− y(i) )2

i=1

m

∑

minθJ(θ0,θ1)

hθ (x) =θ0 +θ1xAssumption1Linearfunction

Assumption2Squarederror

Hypothesisvs.Costfunction(θ1=1)

Hypothesis: Costfunction:


h(x) = x

θ0=0,θ1=1,(x,y)=(4,7),(8,10),(13,11)

J(θ0,θ1) =12m

(hθ (x(i) )− y(i) )2

i=1

m

∑

0

5

10

15

20

0 5 10 15 20

y

x

"train.txt"x

0

5

10

15

20

0 5 10 15 20

y

x

"train.txt"0.5*x

Hypothesisvs.Costfunction(θ1=0.5)

Hypothesis: Costfunction:


h(x) = 0.5* x

θ0=0,θ1=0.5,(x,y)=(4,7),(8,10),(13,11)

J(θ1) =12m

(hθ (x(i) )− y(i) )2

i=1

m

∑

Hypothesisvs.Costfunction(θ1=others)

• θ1=0:– H(x)=0*x=0– J(0)=1/6{(0-4)^2+(0-10)^2+(0-13)^2}

• =1/6{16+100+169}=47.5• θ1=2:– H(x)=2*x– J(2)=1/6{(8-7)^2+(16-10)^2+(26-11)^2}

• =1/6{1+25+225}=41.83• θ1=1.5:– H(x)=1.5*x– J(1.5)=1/6{(6-7)^2+(12-10)^2+(19.5-11)^2}

• =1/6{1+4+72.25}=12.87


θ0=0,θ1=others,(x,y)=(4,7),(8,10),(13,11)

Objectivefunction: minimizeJ(θ1)


0

5

10

15

20

-1 -0.5 0 0.5 1 1.5 2

Cos

t fun

ctio

n: J

(thet

a 1)

theta 1

"h1-sample5.dat"

Convexfunction

• Howdoweobservetheshapeoffunction?• HowdoweobservethebehaviorofGD?





OrdinaryLeastSquares


h(x) =θ0 +θ1x (x,y)=(4,7),(8,10),(13,11),(17,14)

7 =θ0 + 4θ10 =θ0 + 4θ1 − 7e1 :=θ0 + 4θ1 − 7e12 = (θ0 + 4θ1 − 7)

2

E = ei2∑ ≥ 0

0

5

10

15

20

0 5 10 15 20

y

x

"train.txt"0.495*x+5.28

problems?

Ref.,http://gihyo.jp/dev/serial/01/machine-learning/0008

E = ?





Gradientdescentalgorithm


θi :=θi −α∂∂θi

J(θ0,θ1)Repeatuntilconvergence{

}

(1)Startwithanyparameters.(2)Updatetheparameterssimultaneously,untilconvergence.

Simpleexample

J(θ ) =θ 2,α = 0.1

• e.g.,α=0.1,θ=3,J(θ)=9• 1stupdate

• New_θ =?• J(θ)=?

• 2ndupdate• New_θ =?• J(θ)=?

• 3rdupdate• New_?• J(θ)=?

• 4thupdate• New_θ =?• J(θ)=?

new_θ =θ −α ddθ

J(θ ) = ?

Cont.)thebehaviorofGD


0

5

10

15

20

25

-4 -2 0 2 4

J(th

eta

theta

"xx.dat"x*x

GradientdescentforLinearRegression


θi :=θi −α∂∂θi

J(θ0,θ1)

J(θ0,θ1) = 538θ12 +84θ0θ1 + 4θ0

2 − 978θ1 −84θ0 + 466• e.g.,α=0.001,θ0=0,θ1=0,J(θ)=466• 1stupdate

• New_θ0=?• New_θ1=?• J(θ)=?

• 2ndupdate• New_θ0=?• New_θ1=?• J(θ)=?

• 3rdupdate• New_θ0=?• New_θ1=?• J(θ)=?

• 4thupdate• New_θ0=?• New_θ1=?• J(θ)=?

0

5

10

15

20

0 5 10 15 20

y

x

"train.txt"0

0.0084+0.0978*x0.01597176+0.18500616*x

0.022804930848+0.2627653344*x

∂∂θ0

J(θ0,θ1) = ?

∂∂θ1

J(θ0,θ1) = ?

(x,y)=(4,7),(8,10),(13,11),(17,14)hθ (x) =θ0 +θ1x

(optional)zig-zagging behavior


Ref.,http://en.wikipedia.org/wiki/Gradient_descent

References• MachineLearning|Coursera,https://class.coursera.org/ml-007• Gradientdescent– Wikipedia,

http://en.wikipedia.org/wiki/Gradient_descent• 数理計画法第12回,

http://www.dais.is.tohoku.ac.jp/~shioura/teaching/mp11/mp11-12.pdf

• 機械学習はじめよう第8回線形回帰［前編］,http://gihyo.jp/dev/serial/01/machine-learning/0008

• 機械学習はじめよう第9回線形回帰［後編］,http://gihyo.jp/dev/serial/01/machine-learning/0009

• PRMLの線形回帰モデル（線形基底関数モデル）,http://www.slideshare.net/yasunoriozaki12/prml-29439402

• Anintroductiontomachinelearningwithscikit-learn,http://scikit-learn.org/stable/tutorial/basic/tutorial.html


Documents

情報工学実験4：データマイニング班 - University of the Ryukyustnal/2016/info4/dm/2016info4...2015年度：情報工学実験4：データマイニング班 2 1 sample