DeepLearningProgramming Lecture1:Welcometo ...€¦ · 1Introduction Aboutme - MynameisHanheLin.IgotmyPhDdegreeinUniversityofOtago,NewZealand. - FromOctober2016,Iamapostdocresearcherinworkgroupofmultimediasignal

Deep Learning Programming course!Lecture 1: Welcome toDeep Learning Programming

Dr. Hanhe LinUniversität Konstanz, 16.04.2018

1 Introduction

About me

- My name is Hanhe Lin. I got my PhD degree in University of Otago, New Zealand.- From October 2016, I am a postdoc researcher in workgroup of multimedia signal

processing (MMSP). Bachelor and Master are welcome!- Contact me:

- Office: Z704- Email: [email protected] Homepage: LINK

2 / 37 16.04.2018 Deep Learning Programming - Lecture 1 Dr. Hanhe Lin

https://www.mmsp.uni-konstanz.de/

https://www.mmsp.uni-konstanz.de/people/overview/research-staff/hanhe-lin/

1 Introduction

About you

- What is your study background?- Have you learned some statistics/machine learning/computer vision courses before?- Any programming experience (C/C++, Java, R, Python...)?- Any special interest in Deep Learning?- . . .


1 Introduction

Course schedule

- One lecture and one exercise per week- Lecture

- Lecturer: Dr. Hanhe Lin- Time: Monday 13:30 – 15:00- Place: Z613

- Exercise- Tutor: Oliver Wiedemann- Assignment (programming exclusively) will be released on every Monday, you

must submit your result by Sunday- Time: Wednesday 13:30 – 15:00- Place: Z613

- I will attend an conference between 28/05 and 01/06, we should discuss an alternativedate for the missing lecture and exercise.


1 Introduction

Course content

- Lecture session:- Softmax regression- Support Vector Machine- Deep neural network- Convolutional neural network- Recurrent neural network- Hypterparameter tuning, regularization, optimization- Transfer learning- . . .

- Exercise session:- You will implement the learned model


1 Introduction

Course works and credits

- Credit will be given based on solving assignments and completing a final project. Youare required to use your laptop/desktop to solve assignments in exercise session.

- In the last few weeks, you are required to finish a project with a given dataset for aspecific target.

- You will be given training set and validation set to build and train a deep learningmodel

- You credits will be ranked by applied your model to a test set (not available toyou)


1 Introduction

Project plan

Phase 1: find your group members, up to 4 members in each groupPhase 2: framework design and implementation, present your workPhase 3: submit your train model and write technical report, one report per group, up to

4 pages by a given template


1 Introduction

Credits

- Exercise: 20%- Presentation: 20%- Technical report: 20%- Evaluation: 40%


1 Introduction

Presentation

- It will be arranged in exercise session in the last few weeks- Each group has 35 minutes to present their work and 10 minutes for Q&A


1 Introduction

Learning objective

You will:- have an insight of the fundamentals of Deep Learning- have ability to design your own deep learning algorithm to solve some specific

problems- know how to adopt the existing model to your work via transfer learning and fine-tuning


1 Introduction

Note

On Wednesday you will install Jupyter Notebook and Tensorflow, please bring your laptop.


1 Introduction

Any question?


2 What is Deep Learning

Introduction

Deep learning- a data-driven approach to Artificial Intelligence (AI)- a specific kind of machine learning- an interdisciplinary research, e.g., statistics, biology, ...- new capability for computers- ubiquitous

Examples: face detection and recognition, image analysis, self-driving car, . . .


https://cloud.google.com/vision/


Machine Learning definition

- Field of study that gives computers the ability to learn without being explicitlyprogrammed (Arthur Samuel, 1959)

- Well-posed Learning Problem: A computer program is said to learn from experience Ewith respect to some task T and some performance measure P, if its performance onT, as measured by P, improves with experience E (Tom Mitchell, 1998)



Question

Suppose your email program watches which emails you do or do not mark as spam, andbased on that learns how to better filter spam. What is its experience E, task T, andperformance P?

- Classifying emails as spam or not spam- Watching you label emails as spam or not spam- The number of emails correctly classified as spam/not spam



Question

Suppose your email program watches which emails you do or do not mark as spam, andbased on that learns how to better filter spam. What is its experience E, task T, andperformance P?

- Classifying emails as spam or not spam T- Watching you label emails as spam or not spam E- The number of emails correctly classified as spam/not spam P



Supervised and unsupervised learning

Supervised learning- Right answers, or ground-truth, are given- Two kinds in terms of outputs:

- Regression: predict continuous valued output- Classification: predict discrete valued output

Unsupervised learning- We cannot give right answer to all data as the data increase exponentially with the

development of technology- Unsupervised learning is proposed to handle the unlabeled data



Question

You’re running a company, and you want to develop learning algorithms to address eachof two problems:

- Problem 1: you have a large inventory of identical items. You want to predict howmany of these items will sell over the next 3 months

- Problem 2: you’d like software to examine individual customer accounts, and for eachaccount decide if it has been hacked/compromised



Question

You’re running a company, and you want to develop learning algorithms to address eachof two problems:

- Problem 1: you have a large inventory of identical items. You want to predict howmany of these items will sell over the next 3 months Regression

- Problem 2: you’d like software to examine individual customer accounts, and for eachaccount decide if it has been hacked/compromised Classification



Question

Which would you address using an unsupervised learning algorithm?

- Given email labeled as spam/not spam, learn a spam filter- Given a set of news articles found on the web, group them into set of articles about the

same story- Given a database of customer data, automatically discover market segments and

group customers into different market segments- Given a dataset of patients diagnosed as either having diabetes or not, learn to

classify new patients as having diabetes or not



Question

Which would you address using an unsupervised learning algorithm?

- Given email labeled as spam/not spam, learn a spam filter- Given a set of news articles found on the web, group them into set of articles about the

same story ✓- Given a database of customer data, automatically discover market segments and

group customers into different market segments ✓- Given a dataset of patients diagnosed as either having diabetes or not, learn to

classify new patients as having diabetes or not



Why is deep learning taking off?

Neural network has a very long history in machine learning, it has attracted moreattentions recently due to:

- Increasing dataset size- Increasing computational power- Increasing performance



Increasing dataset size



Increasing computational power



Increasing performance


3 Logistic regression

Binary classification

- Examples- Email: spam/not spam?- Disease: cancer/not cancer- Object: car/not car?

- y ∈ {0, 1}, where 1 is positive class, and 0 is negative class, e.g., spam – 1 vs notspam – 0

- Intuitively, negative class conveys absence of something



Notations

- m: number of training example- x : “input” variables/features- y : “output” variables/features- (x (i), y (i)): i th training example



Hypothesis

- if hθ(x) ≥ 0.5, predict y = 1- if hθ(x) < 0.5, predict y = 0- 1 ≥ hθ(x) ≥ 0



Logistic regression model

- sigmoid/logistic function: σ(z) = 11+e−z

- hθ(x) = σ(θTx) = 1

1+e−θT x

, where θ = [θ0 θ1 . . . θn]



Interpretation of hypothesis output

- Estimated probability that y = 1, given x , “parameterized” by θ:

hθ(x) = P (y = 1|x ; θ)

- Example: given tumor size, if we have hθ(x) = 0.7, which means: tell patient that 70%chance of tumor being malignant

- Since we only have two classes, we have:

P (y = 0|x ; θ) + P (y = 1|x ; θ) = 1⇒P (y = 0|x ; θ) = 1− P (y = 1|x ; θ)



Logistic regression

hθ(x) = σ(θTx) =

1

1 + e−θT x

- When hθ(x) ≥ 0.5⇒ θTx ≥ 0, predict y = 1- When hθ(x) < 0.5⇒ θTx < 0, predict y = 0



Loss function

- Loss function, cost function, error function are interchangeable.- Idea: choose θ so that hθ(x) is close to y for training examples (x, y)- The representation of loss function:

J(θ) =1

m

m∑i=1

Loss(hθ(x(i)), y (i))



Loss function of logistic regression

-

Loss(hθ(x(i)), y (i)) =

{−log(hθ(x)) if y = 1−log(1− hθ(x)) if y = 0

- Intuition: if hθ(x) = 0, but y = 1, the learningalgorithm will be penalized by a very large cost

- We can compact two cases into one equation:

Loss(hθ(x(i)), y (i)) = −y log(hθ(x))−(1−y)log(1−hθ(x))



Logistic regression summary

- Hypothesis: hθ(x) = 1

1+e−θT x

- Parameter: θ- Loss function: Loss(hθ(x (i)), y (i)) = −y log(hθ(x))− (1− y)log(1− hθ(x))- Goal: minθJ(θ), but how?



Gradient descent

- Given loss function Loss(hθ(x (i)), y (i)) = −y log(hθ(x))− (1− y)log(1− hθ(x)), ourobjective is minθJ(θ).

- Repeat until converge {

θj := θj − α∂

∂θjJ(θ) j = 0, 1, 2, . . . , n

⇒θj := θj − α1

m

m∑i=1

(hθ(x(i))− y (i))x (i)j

}



Advanced optimization

- Other advanced algorithms:- Conjugate gradient- Broyden–Fletcher–Goldfarb–Shanno (BFGS)- . . .

- Advantages:- No need to pick learning rate manually- Often faster than gradient descent

- Disadvantages:- More complex to implement

- We will discuss some approaches in our later course



Now you have learned your firstmachine learning model!

Questions?


Documents

DeepLearningProgramming Lecture1:Welcometo ...€¦ · 1Introduction Aboutme - MynameisHanheLin.IgotmyPhDdegreeinUniversityofOtago,NewZealand. - FromOctober2016,Iamapostdocresearcherinworkgroupofmultimediasignal