View
222
Download
1
Tags:
Embed Size (px)
Citation preview
Arizona State University DMML
Gaussian Processes
• Extending role of kernels to probabilistic discriminative models leads to framework of Gaussian processes
• Linear regression model
– Evaluate posterior distribution over W
• Gaussian Processes: Define probability distribution over functions directly
Arizona State University DMML
Linear regression
x - input vectorw – M Dimensional weight vector
Prior distribution of w given by the Gaussian form
Prior distribution over w induces a probability distribution over function y(x)
Arizona State University DMML
Linear regressionY is a linear combination of Gaussian distributed variables
given by elements of W,
where is the design matrix with elements
We need only mean and covariance to find the joint distribution of Y
where K is the Gram matrix with elements
Arizona State University DMML
Gaussian Processes
• Defn. : Probability distributions over functions y(x) such that the set of values of y(x) evaluated at an arbitrary set of points jointly have a gaussian distribution
– Mean is assumed zero– Covariance of y(x) evaluated at any two values of x is
given by the kernel function
Arizona State University DMML
Gaussian Processes for regression
To apply Gaussian process models for regression we need to take account of noise on observed target values
Consider noise processes with gaussian distribution
with
To find marginal distribution over ‘t’ we need to integrate over ‘Y’
where covariance matrix C
has elements
Arizona State University DMML
Gaussian Processes for regression
Joint distribution over is given by
Conditional distribution of is a Gaussian distribution with mean and covariance given by
where and is N*N covariance matrix
Arizona State University DMML
Learning the hyperparameters
• Rather than fixing the covariance function we can use a parametric family of functions and then infer the parameter values from the data
• Evaluation of likelihood function where denotes the hyperparameters of Gaussian process model
• Simplest approach is to make a point estimate of by maximizing the log likelihood function
Arizona State University DMML
Gaussian Process for classification
• We can adapt gaussian processes to classification problems by transforming the output using an appropriate nonlinear activation function– Define Gaussian process over a function a(x),
and transform using Logistic sigmoid function ,we obtain a non-Gaussian
stochastic process over functions
Arizona State University DMML
The left plot shows a sample from the Gaussian process prior over functions a(x). The right plot shows the result of transforming this sample using a logistic sigmoid function.
Probability distribution function over target variable is given by Bernoulli distribution on one dimensional input space
Arizona State University DMML
Gaussian Process for classification• To determine the predictive distribution
we introduce a Gaussian process prior over vector , the Gaussian prior takes the form
The predictive distribution is given by
where
Arizona State University DMML
Gaussian Process for classification• The integral is analytically intractable so may be
approximated using sampling methods.
• Alternatively techniques based on analytical approximation can be used– Variational Inference– Expectation propagation– Laplace approximation
Arizona State University DMML
Illustration of Gaussian process for classification
Optimal decision boundary – Green
Decision boundary from Gaussian Process classifier - Black
Arizona State University DMML
Connection to Neural Networks• For a broad class of prior distributions over w,
the distribution of functions generated by a neural network will tend to a Gaussian process as M -> Infinity
• In this Gaussian process limit the ouput variables of the neural network become independent.