Upload
ronda
View
44
Download
0
Tags:
Embed Size (px)
DESCRIPTION
On Natural Scenes Analysis, Sparsity and Coding Efficiency. Vivienne Ming. Mind, Brain & Computation Stanford University. Redwood Center for Theoretical Neuroscience University of California, Berkeley. Adapted by J. McClelland for PDP class, March 1, 2013. Two Proposals. - PowerPoint PPT Presentation
Citation preview
On Natural Scenes Analysis, Sparsity
and Coding Efficiency
Redwood Center forTheoretical NeuroscienceUniversity of California, Berkeley
Mind, Brain & ComputationStanford University
Vivienne Ming
Adapted by J. McClelland for PDP class, March 1, 2013
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Two Proposals Natural Scene Analysis
Neural/cognitive computation can only be fully understood in “naturalistic” contexts
Efficient (Sparse) Coding TheoryNeural computation should follow
information theoretic principles
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Classical Physiology
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Classical Physiology
+
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Classical Physiology
+
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Reverse Correlation
Jones and Palmer (1987)
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Limits of Classical Physiology
Assumes units (neurons) are linear so known nonlinearities are "added on" to the models
Contrast sensitivity “Non-classical receptive fields” Two-tone inhibition ETC.
Assumes that units operate independently activity of one cell doesn't depend on the activity of others i.e., characterizing cell-by-cell equivalent to characterizing the whole
population of evolution and development, drifting gratings
and white noise are very "unnatural“ Is it possible that our sensory systems are functionally adapted to the
statistics of “natural” (evolutionarily relevant) signals? Would this adaptation affect our characterization of cells?
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Response to Natural MovieClassical Receptive
Field Response
Response in“Context”
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Limits of Classical Physiology
Assumes units (neurons) are linear so known nonlinearities are "added on" to the models
Contrast sensitivity “Non-classical receptive fields” Two-tone inhibition ETC.
Assumes that units operate independently activity of one cell doesn't depend on the activity of others i.e., characterizing cell-by-cell equivalent to characterizing the whole
population
Finally, in terms of evolution and development, drifting gratings and white noise seem very "unnatural“ Is it possible that our sensory systems are functionally adapted to the
statistics of “natural” (evolutionarily relevant) signals? Would this adaptation affect our characterization of cells? How can we test this?
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Efficient Coding Theory Barlow (1961); Attneave (1954)
Natural images are redundantStatistical dependencies amongst pixel
values in space and time An efficient visual system should
reduce redundancyRemoving statistical dependencies
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Information TheoryShannon (1949)
Optimally efficient codes reflect the statistics of target signals
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Natural Scenes Analysis:First-Order StatisticsN
aïve
Mod
els
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Natural Scenes Analysis:First-Order Statistics
Histogram Equalization
Intensity Histogram
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Natural Scenes Analysis:Second-Order Statistics
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Natural Scenes Analysis:Second-Order Statistics
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Natural Scenes Analysis:Second-Order Statistics
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Spatial CorrelationsCompare intensityat this pixel
To the intensityat this neighbor
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Spatial Correlations
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
The Ubiquitous .
Flat (White) PowerSpectrum
f1
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Example: synthetic 1/f signals
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Natural Scenes Analysis:Principal Components Analysis
PCA Rotation Whitening
Information theory saysthis is an ideal code.
No redundancy
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
PCA vs. Center Surround
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Natural Scenes Analysis:Higher-Order Statistics
PCA Rotation Whitening
Principle dimensions of variation don’t align with data’s intrinsic structure
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Natural Scenes Analysis:Higher-Order Statistics
Need a more powerful learning algorithmIndependent Component Analysis (ICA)
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Which are the independent components in the scene
below?
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
M
mmm tstx
1
)()(
+_______
+=
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
The Modelx = s + n
Overcomplete: #(s) >> #(x) Factorial: p(s) = i p(si) Sparse: p(si) = exp(g(si))
Where g(.) is some non-Gaussian distribution e.g., Laplacian: g(s) = −|s| e.g., Cauchy: g(s) = −log(2 + s2)
The noise is assumed to be additive Gaussian n ~ N(0, 2I)
Goal: find dictionary of functions, , such that coefficients, s, are as sparse and statistically independent as possible
Information Theorydemands sparseness
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Learning log likelihood L() = <log p(x|)> Learning rule:
Basically the delta rule:
D = (x − s)sT
Impose constraint to encourage the variances of each s to be approximately equal to prevent trivial solutions
Usually whiten the inputs before learning Forces network to find structure beyond second-order Increases stability
DL
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Sparsity
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
?
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Efficient Auditory CodingSmith & Lewicki (2006)
Extend Olshausen (2002) to deal with time-varying signalse.g., sounds or movies
Train the network on “Natural” soundsEnvironmental TransientsEnvironmental AmbientsAnimal Vocalizations
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Cat ANF Revcor Filters
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Efficient Kernels
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Population Coding
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Population Coding
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Population Coding
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Population Coding
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Population Coding
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Speech
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Speech
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Speech
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Efficient Coding Literature Empirical
Weliky, Fiser, Hunt & Wagner (2003) Vinje & Gallant (2002) DeWeese, Wehr & Zador (2003) Laurent (2002) Theunissen (2003)
Theoretical Field (1987) van Hateren (1992) Simoncelli & Olshausen (2001) Olshausen & Field (1996) Bell & Sejnowski (1997) Hyvarinen & Hoyer (2000) Smith & Lewicki (2006) Doi & Lewicki (2006)
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Hierarchical Structure? Can we identify interesting structure in the
world by looking at higher order statistics of the activations of the linear features discovered by the first-order model? Karklin and Lewicki (2005) looked for patterns
at the level of the variances of the linear features.
Karklin and Lewicki (2009) looked for patterns at the level of the covariances of the linear features.
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Looking at Hierarchical Structure
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Looking at Hierarchical Structure
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Looking at Hierarchical Structure
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Looking at Hierarchical Structure
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Looking at Hierarchical Structure
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Generalizing the standard ICA model
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Generalizing the standard ICA model
Instead of:
we now have units u and v such that
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Independent density components
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Karklin & Lewicki (2009)
The model tries to find the values of the yj’s that lead to a combined covariance matrixC that matches the covariance of the data represented by activities across first-level filters.
The learning process involves a search for vectors bk and weights wjk that allow themodel to fit the data while keeping the yj’s sparse and independent.
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Responses of Cell to Gratings
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.
Efficient Coding SummaryStatistic Computatio
nAlgorithmExample
Biological Example Reference
1st-orderContrast
gain control
Histogram equalizatio
n
Retina or H1
adaptationFairhall et al.
(2001)
2nd-order Whitening PCARetinal/
Thalamic coding
Atick (1992)
Higher-order
Sparse Coding
ICA / Sparsenet V1 coding Olshausen &
Field (1996)
Time-varying
Shift-invariance
Efficient Spike
CodingCochlear coding Smith & Lew
icki 2006
Hierarchical
Conditional Independenc
eHierarchical coding ?
Karklin & Lewicki ’05,’09