Modeling Science - Columbia Universityblei/talks/Blei_Science_2008.pdfsequencing phylogenetic control model map living infectious parallel information diversity malaria methods genetics

Modeling Science

David M. Blei

Department of Computer SciencePrinceton University

April 17, 2008

Joint work with John Lafferty (CMU)

D. Blei Modeling Science 1 / 53

Modeling ScienceScience, August 13, 1886

water acid disease

milk water blood

food solution cholera

dry experiments bacteria

fed liquid found

cows chemical bacillus

houses action experiments

butter copper organisms

fat crystals bacilli

found carbon cases

made alcohol diseases

contained made germs

wells obtained animal

produced substances koch

poisonous nitrogen made

5

Science, June 24, 1994

evolution rna disease

evolutionary mrna host

species site bacteria

organisms splicing diseases

biology rnas new

phylogenetic nuclear bacterial

life sequence resistance

origin introns control

diversity messenger strains

groups cleavage infectious

molecular two malaria

animals splice parasites

two sequences parasite

new polymerase tuberculosis

living intron health

6• On-line archives of document collections require better

organization. Manual organization is not practical.• Our goal: To discover the hidden thematic structure with

hierarchical probabilistic models called topic models.• Use this structure for browsing, search, and similarity.


Modeling ScienceScience, August 13, 1886

water acid disease

milk water blood

food solution cholera

dry experiments bacteria

fed liquid found

cows chemical bacillus

houses action experiments

butter copper organisms

fat crystals bacilli

found carbon cases

made alcohol diseases

contained made germs

wells obtained animal

produced substances koch

poisonous nitrogen made

5

Science, June 24, 1994

evolution rna disease

evolutionary mrna host

species site bacteria

organisms splicing diseases

biology rnas new

phylogenetic nuclear bacterial

life sequence resistance

origin introns control

diversity messenger strains

groups cleavage infectious

molecular two malaria

animals splice parasites

two sequences parasite

new polymerase tuberculosis

living intron health

6• Our data are the pages Science from 1880-2002 (from JSTOR)• No reliable punctuation, meta-data, or references.• Note: this is just a subset of JSTOR’s archive.


Discover topics from a corpus

“Genetics” “Evolution” “Disease” “Computers”

human evolution disease computergenome evolutionary host models

dna species bacteria informationgenetic organisms diseases datagenes life resistance computers

sequence origin bacterial systemgene biology new network

molecular groups strains systemssequencing phylogenetic control model

map living infectious parallelinformation diversity malaria methods

genetics group parasite networksmapping new parasites softwareproject two united new

sequences common tuberculosis simulations


Model the evolution of topics over time

1880 1900 1920 1940 1960 1980 2000

o o o o o o o ooooooooo o o o o o o o o o

o o o o oo o o

o oo o

o oooo

o

oooo

oo o

o o oo o

oooo o

o

o

ooo

o

o

o

oo o o

o o o

1880 1900 1920 1940 1960 1980 2000

o o ooo o

oooo o o o

oo o o o o o

o o o o o

o o oooo

ooooo o

o

o

oooooo o

ooo o

o o o o o o o o o oo o

o ooo

o

o

oo o o o o o

RELATIVITY

LASERFORCE

NERVE

OXYGEN

NEURON

"Theoretical Physics" "Neuroscience"


Model connections between topics

wild typemutant

mutationsmutantsmutation

plantsplantgenegenes

arabidopsis

p53cell cycleactivitycyclin

regulation

amino acidscdna

sequenceisolatedprotein

genedisease

mutationsfamiliesmutation

rnadna

rna polymerasecleavage

site

cellscell

expressioncell lines

bone marrow

united stateswomen

universitiesstudents

education

sciencescientists

saysresearchpeople

researchfundingsupport

nihprogram

surfacetip

imagesampledevice

laseropticallight

electronsquantum

materialsorganicpolymerpolymersmolecules

volcanicdepositsmagmaeruption

volcanism

mantlecrust

upper mantlemeteorites

ratios

earthquakeearthquakes

faultimages

dataancientfoundimpact

million years agoafrica

climateocean

icechanges

climate change

cellsproteins

researchersproteinfound

patientsdisease

treatmentdrugsclinical

geneticpopulationpopulationsdifferencesvariation

fossil recordbirds

fossilsdinosaurs

fossil

sequencesequences

genomedna

sequencing

bacteriabacterial

hostresistanceparasite

developmentembryos

drosophilagenes

expression

speciesforestforests

populationsecosystems

synapsesltp

glutamatesynapticneurons

neuronsstimulusmotorvisual

cortical

ozoneatmospheric

measurementsstratosphere

concentrations

sunsolar wind

earthplanetsplanet

co2carbon

carbon dioxidemethane

water

receptorreceptors

ligandligands

apoptosis

proteinsproteinbindingdomaindomains

activatedtyrosine phosphorylation

activationphosphorylation

kinase

magneticmagnetic field

spinsuperconductivitysuperconducting

physicistsparticlesphysicsparticle

experimentsurfaceliquid

surfacesfluid

model reactionreactionsmoleculemolecules

transition state

enzymeenzymes

ironactive sitereduction

pressurehigh pressure

pressurescore

inner core

brainmemorysubjects

lefttask

computerproblem

informationcomputersproblems

starsastronomers

universegalaxiesgalaxy

virushiv

aidsinfectionviruses

miceantigent cells

antigensimmune response


Outline

1 Introduction

2 Latent Dirichlet allocation

3 Dynamic topic models

4 Correlated topic models


Outline

1 Introduction





Probabilistic modeling

1 Treat data as observations that arise from a generativeprobabilistic process that includes hidden variables

• For documents, the hidden variables reflect the thematicstructure of the collection.

2 Infer the hidden structure using posterior inference• What are the topics that describe this collection?

3 Situate new data into the estimated model.• How does this query or new document fit into the estimated

topic structure?


Intuition behind LDA

Simple intuition: Documents exhibit multiple topics.


Generative process

• Cast these intuitions into a generative probabilistic process• Each document is a random mixture of corpus-wide topics• Each word is drawn from one of those topics


Generative process

• In reality, we only observe the documents• Our goal is to infer the underlying topic structure

• What are the topics?• How are the documents divided according to those topics?


Graphical models (Aside)

· · ·

Y

X1 X2 XN

Xn

Y

N

≡

• Nodes are random variables• Edges denote possible dependence• Observed variables are shaded• Plates denote replicated structure


Graphical models (Aside)

· · ·

Y

X1 X2 XN

Xn

Y

N

≡

• Structure of the graph defines the pattern of conditionaldependence between the ensemble of random variables

• E.g., this graph corresponds to

p(y , x1, . . . , xN) = p(y)

N∏n=1

p(xn | y)


Latent Dirichlet allocation

θd Zd,n Wd,nN

D Kβk

α η

Dirichletparameter

Per-documenttopic proportions

Per-wordtopic assignment

Observedword Topics

Topichyperparameter

Each piece of the structure is a random variable.



θd Zd,n Wd,nN

D Kβk

α η

1 Draw each topic βi ∼ Dir(η), for i ∈ {1, . . . , K }.2 For each document:

1 Draw topic proportions θd ∼ Dir(α).2 For each word:

1 Draw Zd,n ∼ Mult(θd ).2 Draw Wd,n ∼ Mult(βzd,n).



θd Zd,n Wd,nN

D Kβk

α η

• From a collection of documents, infer• Per-word topic assignment zd,n• Per-document topic proportions θd• Per-corpus topic distributions βk

• Use posterior expectations to perform the task at hand, e.g.,information retrieval, document similarity, etc.



θd Zd,n Wd,nN

D Kβk

α η

• Computing the posterior is intractable:

p(θ | α)∏N

n=1 p(zn | θ)p(wn | zn, β1:K )∫θ

p(θ | α)∏N

n=1∑K

z=1 p(zn | θ)p(wn | zn, β1:K )

• Several approximation techniques have been developed.



θd Zd,n Wd,nN

D Kβk

α η

• Mean field variational methods (Blei et al., 2001, 2003)• Expectation propagation (Minka and Lafferty, 2002)• Collapsed Gibbs sampling (Griffiths and Steyvers, 2002)• Collapsed variational inference (Teh et al., 2006)


Example inference

• Data: The OCR’ed collection of Science from 1990–2000• 17K documents• 11M words• 20K unique terms (stop words and rare words removed)

• Model: 100-topic LDA model using variational inference.


Example inference

1 8 16 26 36 46 56 66 76 86 96

Topics

Probability

0.0

0.1

0.2

0.3

0.4


Example topics

“Genetics” “Evolution” “Disease” “Computers”

human evolution disease computergenome evolutionary host models

dna species bacteria informationgenetic organisms diseases datagenes life resistance computers

sequence origin bacterial systemgene biology new network

molecular groups strains systemssequencing phylogenetic control model

map living infectious parallelinformation diversity malaria methods

genetics group parasite networksmapping new parasites softwareproject two united new

sequences common tuberculosis simulations


LDA summary

• LDA is a powerful model for• Visualizing the hidden thematic structure in large corpora• Generalizing new data to fit into that structure

• LDA is a mixed membership model (Erosheva, 2004) that buildson the work of Deerwester et al. (1990) and Hofmann (1999).

• For document collections and other grouped data, this might bemore appropriate than a simple finite mixture


LDA summary

• Modular : It can be embedded in more complicated models.• E.g., syntax and semantics; authorship; word sense

• General : The data generating distribution can be changed.• E.g., images; social networks; population genetics data

• Variational inference is fast; lets us to analyze large data sets.

• See Blei et al., 2003 for details and a quantitative comparison.• Code to play with LDA is freely available on my web-site,

http://www.cs.princeton.edu/∼blei.


LDA summary

• But, LDA makes certain assumptions about the data.• When are they appropriate?


Outline

1 Introduction





LDA and exchangeability

θd Zd,n Wd,nN

D Kβk

α η

• LDA assumes that documents are exchangeable.• I.e., their joint probability is invariant to permutation.• This is too restrictive.


Documents are not exchangeable

"Infrared Reflectance in Leaf-Sitting Neotropical Frogs" (1977)"Instantaneous Photography" (1890)

• Documents about the same topic are not exchangeable.• Topics evolve over time.


Dynamic topic model

• Divide corpus into sequential slices (e.g., by year).• Assume each slice’s documents exchangeable.

• Drawn from an LDA model.• Allow topic distributions evolve from slice to slice.


Dynamic topic models

D

θd

Zd,n

Wd,n

N

K

α

D

θd

Zd,n

Wd,n

N

α

D

θd

Zd,n

Wd,n

N

α

βk,1 βk,2 βk,T

. . .


Modeling evolving topics

βk,1 βk,2 βk,T

. . .

• Use a logistic normal distribution to model evolving topics(Aitchison, 1980)

• A state-space model on the natural parameter of the topicmultinomial (West and Harrison, 1997)

βt,k | βt−1,k ∼ N (βt−1,k , Iσ 2)

p(w | βt,k ) = exp{βt,k − log(1 +

∑V−1v=1 exp{βt,k ,v })

}


Posterior inference

• Our goal is to compute the posterior distribution,

p(β1:T ,1:K , θ1:T ,1:D, z1:T ,1:D | w1:T ,1:D).

• Exact inference is impossible• Per-document mixed-membership model• Non-conjugacy between p(w | βt,k ) and p(βt,k )

• MCMC is not practical for the amount of data.• Solution: Variational inference


Science data

TECHVIEW: DNA S E Q U E N C I NG

Sequencing the Genome, Fast

James C. Mullikin and Amanda A. McMurray

Genome sequencing projects reveal the genetic makeup of an organism by reading off the sequence of theDNA bases, which encodes all of the infor-mation necessary for the life of the organ-ism. The base sequence contains four nu-cleotides-adenine, thymidine, guanosine,and cytosine-which are linked togetherinto long double-helical chains. Over thelast two decades, automated DNA se-quencers have made the process of obtain-ing the base-by-base sequence of DNA...

• Analyze JSTOR’s entire collection from Science (1880-2002)• Restrict to 30K terms that occur more than ten times• The data are 76M words in 130K documents


Analyzing a document

Original article Topic proportions


Analyzing a document

sequencegenomegenessequenceshumangenednasequencingchromosomeregionsanalysisdatagenomicnumber

devicesdevicematerialscurrenthighgatelightsiliconmaterialtechnologyelectricalfiberpowerbased

datainformationnetworkwebcomputerlanguagenetworkstimesoftwaresystemwordsalgorithmnumberinternet

Original article Most likely words from top topics


Analyzing a topic

1880electric

machinepowerenginesteam

twomachines

ironbattery

wire

1890electricpower

companysteam

electricalmachine

twosystemmotorengine

1900apparatus

steampowerengine

engineeringwater

constructionengineer

roomfeet

1910air

waterengineeringapparatus

roomlaboratoryengineer

madegastube

1920apparatus

tubeair

pressurewaterglassgas

madelaboratorymercury

1930tube

apparatusglass

airmercury

laboratorypressure

madegas

small

1940air

tubeapparatus

glasslaboratory

rubberpressure

smallmercury

gas

1950tube

apparatusglass

airchamber

instrumentsmall

laboratorypressurerubber

1960tube

systemtemperature

airheat

chamberpowerhigh

instrumentcontrol

1970air

heatpowersystem

temperaturechamber

highflowtube

design

1980high

powerdesignheat

systemsystemsdevices

instrumentscontrollarge

1990materials

highpowercurrent

applicationstechnology

devicesdesigndeviceheat

2000devicesdevice

materialscurrent

gatehighlight

siliconmaterial

technology


Visualizing trends within a topic

1880 1900 1920 1940 1960 1980 2000

o o o o o o o ooooooooo o o o o o o o o o

o o o o oo o o

o oo o

o oooo

o

oooo

oo o

o o oo o

oooo o

o

o

ooo

o

o

o

oo o o

o o o

1880 1900 1920 1940 1960 1980 2000

o o ooo o

oooo o o o

oo o o o o o

o o o o o

o o oooo

ooooo o

o

o

oooooo o

ooo o

o o o o o o o o o oo o

o ooo

o

o

oo o o o o o

RELATIVITY

LASERFORCE

NERVE

OXYGEN

NEURON

"Theoretical Physics" "Neuroscience"


Time-corrected document similarity

• Consider the expected Hellinger distance between the topicproportions of two documents,

dij = E

[K∑

k=1

(√

θi,k −√

θj,k )2| wi , wj

]

• Uses the latent structure to define similarity• Time has been factored out because the topics associated to the

components are different from year to year.• Similarity based only on topic proportions



The Brain of the Orang (1880)



Representation of the Visual Field on the Medial Wall ofOccipital-Parietal Cortex in the Owl Monkey (1976)


Browser of Science


Quantitative comparison

• Compute the probability of each year’s documents conditional onall the previous year’s documents,

p(wt | w1, . . . , wt−1)

• Compare exchangeable and dynamic topic models


Quantitative comparison

1920 1940 1960 1980 2000

1015

2025

Year

Per

−w

ord

nega

tive

log

likel

ihoo

d

● ●●

●

● ●●

● ●

●

●

●

● ● ●

● ●●

●●

●

●

●

●

● ●

●

●

●

●

●●

● ●

● ●● ●

●

●

LDADTM


Outline

1 Introduction





The hidden assumptions of the Dirichlet distribution

• The Dirichlet is an exponential family distribution on the simplex,positive vectors that sum to one.

• However, the near independence of components makes it a poorchoice for modeling topic proportions.

• An article about fossil fuels is more likely to also be aboutgeology than about genetics.


The logistic normal distribution

• The logistic normal is a distribution on the simplex that canmodel dependence between components.

• The natural parameters of the multinomial are drawn from amultivariate Gaussian distribution.

X ∼ NK−1(µ, 6)

θi = exp{xi − log(1 +∑K−1

j=1 exp{xj})}


Correlated topic model (CTM)

Zd,n Wd,nN

D

K

Σ

µ

ηd

βk

• Draw topic proportions from a logistic normal, where topicoccurrences can exhibit correlation.

• Use for:• Providing a “map” of topics and how they are related• Better prediction via correlated topics


wild typemutant

mutationsmutantsmutation

plantsplantgenegenes

arabidopsis

p53cell cycleactivitycyclin

regulation

amino acidscdna

sequenceisolatedprotein

genedisease

mutationsfamiliesmutation

rnadna

rna polymerasecleavage

site

cellscell

expressioncell lines

bone marrow

united stateswomen

universitiesstudents

education

sciencescientists

saysresearchpeople

researchfundingsupport

nihprogram

surfacetip

imagesampledevice

laseropticallight

electronsquantum

materialsorganicpolymerpolymersmolecules

volcanicdepositsmagmaeruption

volcanism

mantlecrust

upper mantlemeteorites

ratios

earthquakeearthquakes

faultimages

dataancientfoundimpact

million years agoafrica

climateocean

icechanges

climate change

cellsproteins

researchersproteinfound

patientsdisease

treatmentdrugsclinical

geneticpopulationpopulationsdifferencesvariation

fossil recordbirds

fossilsdinosaurs

fossil

sequencesequences

genomedna

sequencing

bacteriabacterial

hostresistanceparasite

developmentembryos

drosophilagenes

expression

speciesforestforests

populationsecosystems

synapsesltp

glutamatesynapticneurons

neuronsstimulusmotorvisual

cortical

ozoneatmospheric

measurementsstratosphere

concentrations

sunsolar wind

earthplanetsplanet

co2carbon

carbon dioxidemethane

water

receptorreceptors

ligandligands

apoptosis

proteinsproteinbindingdomaindomains

activatedtyrosine phosphorylation

activationphosphorylation

kinase

magneticmagnetic field

spinsuperconductivitysuperconducting

physicistsparticlesphysicsparticle

experimentsurfaceliquid

surfacesfluid

model reactionreactionsmoleculemolecules

transition state

enzymeenzymes

ironactive sitereduction

pressurehigh pressure

pressurescore

inner core

brainmemorysubjects

lefttask

computerproblem

informationcomputersproblems

starsastronomers

universegalaxiesgalaxy

virushiv

aidsinfectionviruses

miceantigent cells

antigensimmune response


Summary

• Topic models provide useful descriptive statistics for analyzingand understanding the latent structure of large text collections.

• Probabilistic graphical models are a useful way to expressassumptions about the hidden structure of complicated data.

• Variational methods allow us to perform posterior inference toautomatically infer that structure from large data sets.

• Current research• Choosing the number of topics• Continuous time dynamic topic models• Topic models for prediction• Inferring the impact of a document


“We should seek out unfamiliar summaries of observational material,and establish their useful properties... And still more novelty cancome from finding, and evading, still deeper lying constraints.”(John Tukey, The Future of Data Analysis, 1962)


Supervised topic models (with Jon McAuliffe)

• Most topic models are unsupervised. They are fit by maximizingthe likelihood of a collection of documents.

• Consider documents paired with response variables.For example:

• Movie reviews paired with a number of stars• Web pages paired with a number of “diggs”

• We develop supervised topic models, models of documents andresponses that are fit to find topics predictive of the response.


Supervised LDA

θd Zd,n Wd,nN

D

Kβk

α

Yd η, σ2

1 Draw topic proportions θ | α ∼ Dir(α).2 For each word

1 Draw topic assignment zn | θ ∼ Mult(θ).2 Draw word wn | zn, β1:K ∼ Mult(βzn).

3 Draw response variable y | z1:N, η, σ 2∼ N

(η>z̄, σ 2

), where

z̄ = (1/N)∑N

n=1 zn.


Comments

• SLDA is used as follows.• Fit coefficients and topics from a collection of

document-response pairs.• Use the fitted model to predict the responses of previously

unseen documents,

E[Y | w1:N, α, β1:K , η, σ 2] = η>E[Z̄ | w1:N, α, β1:K ].

• The process enforces that the document is generated first,followed by the response. The response is generated from theparticular topics that were realized in generating the document.


Example: Movie reviews

bothmotionsimpleperfectfascinatingpowercomplex

howevercinematographyscreenplayperformancespictureseffectivepicture

histheircharactermanywhileperformancebetween

−30 −20 −10 0 10 20

● ●● ● ●● ● ● ● ●

morehasthanfilmsdirectorwillcharacters

onefromtherewhichwhomuchwhat

awfulfeaturingroutinedryofferedcharlieparis

notaboutmovieallwouldtheyits

havelikeyouwasjustsomeout

badguyswatchableitsnotonemovie

leastproblemunfortunatelysupposedworseflatdull

• We fit a 10-topic sLDA model to movie review data (Pang andLee, 2005).

• The documents are the words of the reviews.• The responses are the number of stars associated with

each review (modeled as continuous).• Each component of coefficient vector η is associated with a topic.


Simulations

●

● ●

●

●●

●

●

●

●●

●●

●

2 4 10 20 30

0.00

0.02

0.04

0.06

0.08

0.10

0.12

Number of topics

Pred

ictive

R2

● ● ●●

●

●

●

●

●● ●

●

●

●

2 4 10 20 30

−8.6

−8.5

−8.4

−8.3

−8.2

−8.1

−8.0

Number of topics

Per−

word

hel

d ou

t log

likel

ihoo

d

●● ●

●

●

●

●

●

●

●

●

●

● ●

●

●

●

●

●

●

5 10 15 20 25 30 35 40 45 50

−6.4

2−6

.41

−6.4

0−6

.39

−6.3

8−6

.37

Number of topics

Per−

word

hel

d ou

t log

likel

ihoo

d

●

●

● ●● ● ● ●

●●

●

●

●● ●

●● ● ●

●

5 10 15 20 25 30 35 40 45 50

0.0

0.1

0.2

0.3

0.4

0.5

Number of topics

Pred

ictive

R2

sLDALDA

Movie corpus

Digg corpus


Diversion: Variational inference

• Let x1:N be observations and z1:M be latent variables• Our goal is to compute the posterior distribution

p(z1:M | x1:N) =p(z1:M , x1:N)∫

p(z1:M , x1:N)dz1:M

• For many interesting distributions, the marginal likelihood of theobservations is difficult to efficiently compute


Variational inference

• Use Jensen’s inequality to bound the log prob of theobservations:

log p(x1:N) ≥ Eqν [log p(z1:M , x1:N)] − Eqν [log qν(z1:M)].

• We have introduced a distribution of the latent variables with freevariational parameters ν.

• We optimize those parameters to tighten this bound.• This is the same as finding the member of the family qν that is

closest in KL divergence to p(z1:M | x1:N).


Mean-field variational inference

• Complexity of optimization is determined by factorization of qν

• In mean field variational inference qν is fully factored

qν(z1:M) =

M∏m=1

qνm(zm).

• The latent variables are independent.• Each is governed by its own variational parameter νm.

• In the true posterior they can exhibit dependence(often, this is what makes exact inference difficult).


MFVI and conditional exponential families

• Suppose the distribution of each latent variable conditional onthe observations and other latent variables is in the exponentialfamily:

p(zm | z−m, x) = hm(zm) exp{gm(z−m, x)T zm − am(gi(z−m, x))}

• Assume qν is fully factorized and each factor is in the sameexponential family:

qνm(zm) = hm(zm) exp{νTmzm − am(νm)}


MFVI and conditional exponential families

• Variational inference is the following coordinate ascent algorithm

νm = Eqν [gm(Z−m, x)]

• Notice the relationship to Gibbs sampling


Variational family for the DTM

βk,1 βk,2 βk,T

. . .

β̂k,1 β̂k,2 β̂k,T

• Distribution of θ and z is fully-factorized (Blei et al., 2003)• Distribution of {β1,k , . . . , βT ,k } is a variational Kalman filter• Gaussian state-space model with free observations β̂k ,t .• Fit observations such that the corresponding posterior over the

chain is close to the true posterior.


Variational family for the DTM

βk,1 βk,2 βk,T

. . .

β̂k,1 β̂k,2 β̂k,T

• Given a document collection, use coordinate ascent on all thevariational parameters until the KL converges.

• Yields a distribution close to the true posterior of interest• Take expectations w/r/t the simpler variational distribution
