01 artificial neuron - · PDF file 1/1/2018  · ARTIFICIAL NEURON 2 Topics: connection weights, bias, activation function • Neuron pre-activation (or input activation): • Neuron

Neural networks Feedforward neural network - artificial neuron

ARTIFICIAL NEURON 2

Topics: connection weights, bias, activation function • Neuron pre-activation (or input activation):

• Neuron (output) activation

• are the connection weights • is the neuron bias • is called the activation function

... 1

Feedforward neural network

Hugo Larochelle D

´

epartement d’informatique

Universit

´

e de Sherbrooke

[email protected]

September 6, 2012

Abstract

Math for my slides “Feedforward neural network”.

• a(x) = b+ P

i wixi = b+w > x

• h(x) = g(a(x)) = g(b+ P

i wixi)

• x 1

xd

• w

• {

• g(·) b

• h(x) = g(a(x))

• a(x) = b(1) +W(1)x ⇣ a(x)i = b

(1)

i

P j W

(1)

i,j xj

⌘

• o(x) = g(out)(b(2) +w(2)>x)

1

Feedforward neural network

Hugo Larochelle D

´

epartement d’informatique

Universit

´

e de Sherbrooke

[email protected]

September 6, 2012

Abstract

Math for my slides “Feedforward neural network”.

• a(x) = b+ P

i wixi = b+w > x

• h(x) = g(a(x)) = g(b+ P

i wixi)

• x 1

xd

• w

• {

• g(·) b

• h(x) = g(a(x))

• a(x) = b(1) +W(1)x ⇣ a(x)i = b

(1)

i

P j W

(1)

i,j xj

⌘

• o(x) = g(out)(b(2) +w(2)>x)

1

Feedforward neural network

Hugo Larochelle D

´

epartement d’informatique

Universit

´

e de Sherbrooke

[email protected]

September 6, 2012

Abstract

Math for my slides “Feedforward neural network”.

• a(x) = b+ P

i wixi = b+w > x

• h(x) = g(a(x)) = g(b+ P

i wixi)

• x 1

xd b w1 wd

• w

• {

• g(·) b

• h(x) = g(a(x))

• a(x) = b(1) +W(1)x ⇣ a(x)i = b

(1)

i

P j W

(1)

i,j xj

⌘

• o(x) = g(out)(b(2) +w(2)>x)

1

Feedforward neural network

Hugo Larochelle D

´

epartement d’informatique

Universit

´

e de Sherbrooke

[email protected]

September 6, 2012

Abstract

Math for my slides “Feedforward neural network”.

• a(x) = b+ P

i wixi = b+w > x

• h(x) = g(a(x)) = g(b+ P

i wixi)

• x 1

xd b w1 wd

• w

• {

• g(·) b

• h(x) = g(a(x))

• a(x) = b(1) +W(1)x ⇣ a(x)i = b

(1)

i

P j W

(1)

i,j xj

⌘

• o(x) = g(out)(b(2) +w(2)>x)

1

Feedforward neural network

Hugo Larochelle D

´

epartement d’informatique

Universit

´

e de Sherbrooke

[email protected]

September 6, 2012

Abstract

Math for my slides “Feedforward neural network”.

• a(x) = b+ P

i wixi = b+w > x

• h(x) = g(a(x)) = g(b+ P

i wixi)

• x 1

xd b w1 wd

• w

• {

• g(·) b

• h(x) = g(a(x))

• a(x) = b(1) +W(1)x ⇣ a(x)i = b

(1)

i

P j W

(1)

i,j xj

⌘

• o(x) = g(out)(b(2) +w(2)>x)

1

Feedforward neural network

Hugo Larochelle D

´

epartement d’informatique

Universit

´

e de Sherbrooke

[email protected]

September 6, 2012

Abstract

Math for my slides “Feedforward neural network”.

• a(x) = b+ P

i wixi = b+w > x

• h(x) = g(a(x)) = g(b+ P

i wixi)

• x 1

xd b w1 wd

• w

• {

• g(·) b

• h(x) = g(a(x))

• a(x) = b(1) +W(1)x ⇣ a(x)i = b

(1)

i

P j W

(1)

i,j xj

⌘

• o(x) = g(out)(b(2) +w(2)>x)

1

Feedforward neural network

Hugo Larochelle D

´

epartement d’informatique

Universit

´

e de Sherbrooke

[email protected]

September 6, 2012

Abstract

Math for my slides “Feedforward neural network”.

• a(x) = b+ P

i wixi = b+w > x

• h(x) = g(a(x)) = g(b+ P

i wixi)

• x 1

xd b w1 wd

• w

• {

• g(·) b

• h(x) = g(a(x))

• a(x) = b(1) +W(1)x ⇣ a(x)i = b

(1)

i

P j W

(1)

i,j xj

⌘

• o(x) = g(out)(b(2) +w(2)>x)

1

Feedforward neural network

Hugo Larochelle D

´

epartement d’informatique

Universit

´

e de Sherbrooke

[email protected]

September 6, 2012

Abstract

Math for my slides “Feedforward neural network”.

• a(x) = b+ P

i wixi = b+w > x

• h(x) = g(a(x)) = g(b+ P

i wixi)

• x 1

xd b w1 wd

• w

• {

• g(·) b

• h(x) = g(a(x))

• a(x) = b(1) +W(1)x ⇣ a(x)i = b

(1)

i

P j W

(1)

i,j xj

⌘

• o(x) = g(out)(b(2) +w(2)>x)

1

Feedforward neural network

Hugo Larochelle D

´

epartement d’informatique

Universit

´

e de Sherbrooke

[email protected]

September 6, 2012

Abstract

Math for my slides “Feedforward neural network”.

• a(x) = b+ P

i wixi = b+w > x

• h(x) = g(a(x)) = g(b+ P

i wixi)

• x 1

xd b w1 wd

• w

• {

• g(·) b

• h(x) = g(a(x))

• a(x) = b(1) +W(1)x ⇣ a(x)i = b

(1)

i

P j W

(1)

i,j xj

⌘

• o(x) = g(out)(b(2) +w(2)>x)

1

Feedforward neural network

Hugo Larochelle D

´

epartement d’informatique

Universit

´

e de Sherbrooke

[email protected]

September 6, 2012

Abstract

Math for my slides “Feedforward neural network”.

• a(x) = b+ P

i wixi = b+w > x

• h(x) = g(a(x)) = g(b+ P

i wixi)

• x 1

xd b w1 wd

• w

• {

• g(·) b

• h(x) = g(a(x))

• a(x) = b(1) +W(1)x ⇣ a(x)i = b

(1)

i

P j W

(1)

i,j xj

⌘

• o(x) = g(out)(b(2) +w(2)>x)

1

ARTIFICIAL NEURON 3

Topics: connection weights, bias, activation function

2Réseaux de neurones

-1 1

-1

1

1 1

1 1

.5

-1.5

.7 -.4-1

x1 x2

x1

x2

z=+1

z=-1

z=-1

0

1 -1

0

1

-1

0

1

-1

0

1 -1

0

1

-1

0

1

-1

0

1 -1

0

1

-1

0

1

-1

R2

R2

R1

y1 y2

z

zk

wkj

wji

x1

x2

x1

x2

x1

x2 y1 y2

sortie k

entrée i

cachée j biais

(from Pascal Vincent’s slides)

Feedforward neural network

Hugo Larochelle D

´

epartement d’informatique

Universit

´

e de Sherbrooke

[email protected]

September 6, 2012

Abstract

Math for my slides “Feedforward neural network”.

• a(x) = bi P

i wixi = bi +w > x

• h(x) = g(a(x)) = g(bi P

i wixi)

• w

1

Feedforward neural network

Hugo Larochelle D

´

epartement d’informatique

Universit

´

e de Sherbrooke

[email protected]

September 6, 2012

Abstract

Math for my slides “Feedforward neural network”.

• a(x) = bi P

i wixi = bi +w > x

• h(x) = g(a(x)) = g(bi P

i wixi)

• w

• {

1

range determined 
 by

bias only changes the position of the riff

Feedforward neural network

Hugo Larochelle D

´

epartement d’informatique

Universit

´

e de Sherbrooke

[email protected]

September 6, 2012

Abstract

Math for my slides “Feedforward neural network”.

• a(x) = b+ P

i wixi = b+w > x

• h(x) = g(a(x)) = g(b+ P

i wixi)

• w

• {

• g(·) b

1

Feedforward neural network

Hugo Larochelle D

´

epartement d’informatique

Universit

´

e de Sherbrooke

[email protected]

September 6, 2012

Abstract

Math for my slides “Feedforward neural network”.

• a(x) = b+ P

i wixi = b+w > x

• h(x) = g(a(x)) = g(b+ P

i wixi)

• w