Embed Size (px)
Citation preview
Text Categorization
PengBo10/31/2010
本次课大纲
Text Categorization Problem definition Build a Classifier
Naïve Bayes Classifier K-Nearest Neighbor Classifier
Evaluation
Definition
Given: 实例 instance, xX, where X is the instance
language or instance space. Issue: how to represent text documents.
固定的类别集合 categories: C = {c1, c2,…, cn}
Determine: The category of x : c(x)C,
where c(x) is a categorization function 分类函数 We want to know how to build categorization
functions (“classifiers 分类器” ).
Text Categorization Examples
Assign labels to each document or web-page: Labels are most often topics such as Yahoo-categories
e.g., "finance," "sports," "news>world>asia>business" Labels may be genres
e.g., "editorials" "movie-reviews" "news“ Labels may be opinion
e.g., “like”, “hate”, “neutral” Labels may be domain-specific binary
e.g., "interesting-to-me" : "not-interesting-to-me” e.g., “spam” : “not-spam” e.g., “contains adult language” :“doesn’t”
Classification Methods
人工分类 Manual classification Used by Yahoo!, Looksmart, about.com, ODP, Medline Accurate but expensive to scale
自动文本分类 Automatic document classification 基于规则： Hand-coded rule-based systems
Spam mail filter,… 有监督的学习： Supervised learning of a document-
label assignment function No free lunch: requires 人工标注的训练集 hand-
classified training data Note that many commercial systems use a
mixture of methods
Think about it…
How to represent text documents and categories? Vectors & Regions String & Language (models)
How to build categorization functions ？ Closeness/Similarity to regions Probability to generate the
string/language model
K-Nearest Neighbors
Classes in a Vector Space
Government
Science
Arts
Classification Using Vector Spaces
Each training doc a point (vector) labeled by its topic (= class)
Hypothesis: docs of the same class form a contiguous region of space
We define surfaces to delineate classes in space
Test Document = Government
Government
Science
Arts
Similarityhypothesistrue ingeneral?
k Nearest Neighbor Classification
To classify document d into class c Define k-neighborhood N as k nearest
neighbors of d Count number of documents i in N that belong
to c Estimate P(c|d) as i/k Choose as class argmaxc P(c|d) [ = majority
class]
Example: k=6 (6NN)
Government
Science
Arts
P(science| )?
Nearest-Neighbor Learning Algorithm
Learning is just storing the representations of the training examples in D.
Testing instance x: Compute similarity between x and all examples
in D. Assign x the category of the most similar
example in D. Does not explicitly compute a generalization
or category prototypes. Also called:
Case-based learning Memory-based learning Lazy learning
Why K?
Using only the closest example to determine the categorization is subject to errors due to: A single atypical example. Noise (i.e. error) in the category label of a single
training example. More robust alternative is to find the k
most-similar examples and return the majority category of these k examples.
Value of k is typically odd to avoid ties; 3 and 5 are most common.
kNN decision boundaries
Government
Science
Arts
Boundaries are in principle arbitrary surfaces – but usually polyhedra
Similarity Metrics
Nearest neighbor method depends on a similarity (or distance) metric. Simplest for continuous m-dimensional instance
space is Euclidean distance. Simplest for m-dimensional binary instance
space is Hamming distance (number of feature values that differ).
For text, cosine similarity of tf.idf weighted vectors is typically most effective.
Illustration of 3 Nearest Neighbor for Text Vector Space
Nearest Neighbor with Inverted Index
Naively finding nearest neighbors requires a linear search through |D| documents in collection
But determining k nearest neighbors is the same as determining the top-k best retrievals using the test document as a query to a database of training documents.
Use standard vector space inverted index methods to find the k nearest neighbors.
Testing Time: O(B|Vt|) where B is the average number of training documents in which a test-document word appears. Typically B << |D|
kNN: Discussion
No training necessary No feature selection necessary Scales well with large number of classes
Don’t need to train n classifiers for n classes Classes can influence each other
Small changes to one class can have ripple effect
Scores can be hard to convert to probabilities
Naïve Bayes
Bayes Classifiers
Task: Classify a new instance D based on a tuple of attribute values into one of the classes cj C nxxxD ,,, 21
),,,|(argmax 21 njCc
MAP xxxcPcj
),,,(
)()|,,,(argmax
21
21
n
jjn
Cc xxxP
cPcxxxP
j
)()|,,,(argmax 21 jjnCc
cPcxxxPj
Naïve Bayes Assumption
P(cj) Can be estimated from the frequency of classes
in the training examples. P(x1,x2,…,xn|cj)
O(|X|n•|C|) parameters Could only be estimated if a very, very large
number of training examples was available.
Flu
X1 X2 X5X3 X4
feversinus coughrunnynose muscle-ache
Conditional Independence Assumption
Assume that the probability of observing the conjunction of attributes is equal to the product of the individual probabilities P(xi|cj).
features detect term presence and are independent of each other given the class:)|()|()|()|,,( 52151 CXPCXPCXPCXXP
Learning the Model
First attempt: maximum likelihood estimates simply use the frequencies in the data
)(
),()|(ˆ
j
jiiji cCN
cCxXNcxP
C
X1 X2 X5X3 X4 X6
N
cCNcP jj
)()(ˆ
What if we have seen no training cases where patient had no flu and muscle aches?
Zero probabilities cannot be conditioned away, no matter the other evidence!
Problem with Max Likelihood
0)(
),()|(ˆ 5
5
nfCN
nfCtXNnfCtXP
i ic cxPcP )|(ˆ)(ˆmaxarg
Flu
X1 X2 X5X3 X4
feversinus coughrunnynose muscle-ache
)|()|()|()|,,( 52151 CXPCXPCXPCXXP
Smoothing to Eliminate Zeros
kcCN
cCxXNcxP
j
jiiji
)(
1),()|(ˆ
# of values of Xi
Add one smooth (Laplace smoothing) As a uniform prior (each attribute occurs once for
each class) that is then updated as evidence from the training data comes in.
Document Generative Model
“Love is patient, love is kind. “ Basic : bag of words a binary independence model
Multivariate binomial generation feature Xi is term Value Xi = 1 or 0, indicate term Xi present in doc
or not a multinomial unigram language model
Multinomial generation feature Xi is term position Value of Xi = term at position i position independent
Love is
patient
kind
Bernoulli Naive Bayes Classifiers
Multivariate binomial Model One feature Xw for each word in
dictionary Value Xw = true in document d if w
appears in d Naive Bayes assumption:
Given the document’s topic, appearance of one word in the document tells us nothing about chances that another word appears
Love is
patient
kind
Multinomial Naive Bayes Classifiers II
Multinomial = Class conditional unigram One feature Xi for each word pos in
document feature’s values are all words in
dictionary Value of Xi is the word in position i Naïve Bayes assumption:
Given the document’s topic, word in one position in the document tells us nothing about words in other positions
)|text""()|our""()(argmax
)|()(argmax
1j
j
jnjjCc
ijij
CcNB
cxPcxPcP
cxPcPc
Still too many possibilities Assume that classification is independent of
the positions of the words Second assumption:
Word appearance does not depend on position
Just have one multinomial feature predicting for all words
Use same parameters for each position
Multinomial Naive Bayes Classifiers
)|()|( cwXPcwXP ji for all positions i,j, word w, and class c
Parameter estimation
fraction of documents of topic cj
in which word w appears
Binomial model:
Multinomial model:
)|(ˆjw ctXP
fraction of times in which word w appears
across all documents of topic cj
)|(ˆji cwXP
Naive Bayes algorithm (Multinomial model)
Naive Bayes algorithm (Bernoulli model)
NB Example
c(5)=?
NB Example
c(5)=?
Multinomial NB Classifier
Feature likelihood estimate
Posterior
Result: c(5) = China
NB Example
c(5)=?
Bernoulli NB Classifier
Feature likelihood estimate
Posterior
Result: c(5) <> China
Classification
Multinomial vs Multivariate binomial?
Multinomial is in general better
Classification Evaluation
Let’s think about it…
How to do evaluation experiments for classifiers? Dataset Measures Generalization Performance
Training setTraining set
Test setTest set
RecallRecall PrecisionPrecision
F1F1 AccuracyAccuracy
Most (over)used data set 21578 documents 9603 training, 3299 test articles (ModApte
split) 118 categories
An article can be in more than one category Learn 118 binary category distinctions
Average document: about 90 types, 200 tokens
Average number of classes assigned 1.24 for docs with at least one category
Only about 10 out of 118 categories are largeCommon categories(#train, #test)
Classic Reuters Data Set
• Earn (2877, 1087) • Acquisitions (1650, 179)• Money-fx (538, 179)• Grain (433, 149)• Crude (389, 189)
• Trade (369,119)• Interest (347, 131)• Ship (197, 89)• Wheat (212, 71)• Corn (182, 56)
Measuring ClassificationFigures of Merit
Accuracy of classification Main evaluation criterion in academia
Speed of training statistical classifier Some methods are very cheap; some very
costly Speed of classification (docs/hour)
No big differences for most algorithms Exceptions: kNN, complex preprocessing
requirements Effort in creating training set/hand-built
classifier human hours/topic
Measuring ClassificationFigures of Merit
In the real world, economic measures: Your choices are:
Do no classification That has a cost (hard to compute)
Do it all manually Has an easy to compute cost if doing it like that now
Do it all with an automatic classifier Mistakes have a cost
Do it with a combination of automatic classification and manual review of uncertain/difficult/”new” cases
Commonly the last method is most cost efficient and is adopted
Per class evaluation measures
Recall: Fraction of docs in class i classified correctly:
Precision: Fraction of docs assigned class i that are actually about class i:
“Correct rate”: (1- error rate) Fraction of docs classified correctly:
j
ij
ii
c
c
j
ji
ii
c
c
jij
i
iii
c
c
A B C
A
B
C
Actual Class
Predictedclass
Measuring Classification
Overall error rate Not a good measure for small classes. Why?
Precision/recall for classification decisions F1 measure: 1/F1 = ½ (1/P + 1/R) Correct estimate of size of category
Why is this different? Stability over time / category drift Utility
Costs of false positives / false negatives may be different
For example, cost = tp-0.5fp
Generalization Performance
Generalization Performance Results can vary based on sampling error
due to different training and test sets. Average results over multiple training and
test sets (splits of the overall data) for the best results.
Ideally, test and training sets are independent on each trial.
But this would require too much labeled data.But this would require too much labeled data.
Good practice department
N-Fold Cross-Validation Partition data into N equal-sized disjoint
segments. Run N trials, each time using a different
segment of the data for testing, and training on the remaining N1 segments.
This way, at least test-sets are independent.
Report average classification accuracy over the N trials.
Typically, N = 10.
Good practice department II
Learning Curves Would like to know how performance varies with
the number of training instances. Learning curves plot classification accuracy on
independent test data (Y axis) versus number of training examples (X axis).
One can do both the above and produce learning curves averaged over multiple trials from cross-validation
how to combine multiple measures?
If we have more than one class, how do we combine multiple performance measures into one quantity?
Macroaveraging: Compute performance for each class, then
average. Microaveraging:
Collect decisions for all classes, compute contingency table, evaluate.
Micro- vs. Macro-Averaging: Example
Truth: yes
Truth: no
Classifier: yes
10 10
Classifier: no
10 970
Truth: yes
Truth: no
Classifier: yes
90 10
Classifier: no
10 890
Truth: yes
Truth: no
Classifier: yes
100 20
Classifier: no
20 1860
Class 1 Class 2 Micro.Av. Table
Macroaveraged precision: (0.5 + 0.9)/2 = 0.7 Microaveraged precision: 100/120 = .83 Why this difference?
Exercise
Federalist papers 1787-1788 年间由
Hamilton, Jay and Madison 用笔名发表的 77 篇短文，来劝说NY 支持 US Constitution
其中 12 篇 papers 的作者存在争议
谁是作者？ 谁是作者？
53
Author identification
In 1964 Mosteller and Wallace* solved the problem Mosteller, Frederick and Wallace, David L.
1964. Inference and Disputed Authorship: The Federalist.
It’s a Text Catergorization Problem They identified 70 function words as good
candidates for authorship analysis Using statistical inference they concluded
the author was Madison
Feature Selection
Classifier
Function words for Author Identification
Function Words for Author Identification
Summary
Definition The category of x:
c(x)C K-Nearest Neighbor Naïve Bayes
Bayesian Methods Bernoulli NB classifier Multinomial NB classifier
Categorization Evaluation
Training data/Test data Over-fitting & Generalize
)|text""()|our""()(argmax
)|()(argmax
1j
j
jnjjCc
ijij
CcNB
cxPcxPcP
cxPcPc
Thank You!
Q&A
Readings
[1] IIR Ch13, Ch14.2 [2] Y. Yang and X. Liu, "A re-examination of
text categorization methods," presented at Proceedings of ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR'99), 1999.
Bernoulli trial
Bernoulli trial is an experiment whose outcome is random and can be either of two possible outcomes, "success" and "failure".
Binomial Distribution
binomial distribution is the discrete probability distribution of the number of successes in a sequence of n independent yes/no experiments, each of which yields success with probability p.
Multinomial Distribution
multinomial distribution is a generalization of the binomial distribution.
each trial results in one of some fixed finite number k of possible outcomes, with probabilities p1, ..., pk,
there are n independent trials.
We can use a random variable Xi to indicate the number of times outcome number i was observed over the n trials.
Bayes’ Rule
priorprior
posteriorposterior likelihoodlikelihood
Use Bayes Rule to Gamble
someone draws an envelope at random and offers to sell it to you. How much should you pay?
before deciding, you are allowed to see one bead drawn from the envelope. Suppose it’s red: How much should you pay?
Prosecutor's fallacy
You win the lottery jackpot. You are then charged with having cheated, for instance with having bribed lottery officials.
At the trial, the prosecutor points out that winning the lottery without cheating is extremely unlikely, and that therefore your being innocent must be comparably unlikely.
Maximum a posteriori Hypothesis
)|(argmax DhPhHh
MAP
)(
)()|(argmax
DP
hPhDP
Hh
)()|(argmax hPhDPHh
As P(D) isconstant
Maximum likelihood Hypothesis
If all hypotheses are a priori equally likely, we only
need to consider the P(D|h) term:)|(argmax hDPh
HhML
Likelihood
a likelihood function is a conditional probability function considered as a function of its second argument with its first argument held fixed
Given a parameterized family of probability density functions
Where θ is the parameter, the likelihood function is
where x is the observed outcome of an experiment.
when f(x | θ) is viewed as a function of x with θ fixed, it is a probability density function,
when viewed as a function of θ with x fixed, it is a likelihood function.
)|( xfx
)|()|( xfxL
Reuters Text Categorization data set (Reuters-21578) document
<REUTERS TOPICS="YES" LEWISSPLIT="TRAIN" CGISPLIT="TRAINING-SET" OLDID="12981" NEWID="798">
<DATE> 2-MAR-1987 16:51:43.42</DATE>
<TOPICS><D>livestock</D><D>hog</D></TOPICS>
<TITLE>AMERICAN PORK CONGRESS KICKS OFF TOMORROW</TITLE>
<DATELINE> CHICAGO, March 2 - </DATELINE><BODY>The American Pork Congress kicks off
tomorrow, March 3, in Indianapolis with 160 of the nations pork producers from 44 member states determining industry positions on a number of issues, according to the National Pork Producers Council, NPPC.
Delegates to the three day Congress will be considering 26 resolutions concerning various issues, including the future direction of farm policy and the tax law as it applies to the agriculture sector. The delegates will also debate whether to endorse concepts of a national PRV (pseudorabies virus) control and eradication program, the NPPC said.
A large trade show, in conjunction with the congress, will feature the latest in technology in all areas of the industry, the NPPC added. Reuter
</BODY></TEXT></REUTERS>
New Reuters: RCV1: 810,000 docs
Top topics in Reuters RCV1
北大天网 : 中文网页分类 通过动员不同专业的几十个学生，人工选取形成了一个基
于层次模型的大规模中文网页样本集。 包括 12,336 个训练网页实例和 3,269 个测试网页实例，分
布在 12 个大类 , 共计 733 个类别中，每个类别平均有 17个训练实例和 4.5 个测试实例
天网免费提供网页样本集给有兴趣的同行，燕穹产品号：YQ-WEBENCH-V0.8
中文信息检索论坛 www.cwirf.org 全国搜索引擎和网上信息挖掘学术研讨会 SEWM 上进行分
类评测
北大天网 : 中文网页分类
类别编号
类别名称 类别数 训练样本数
测试样本数
1 人文与艺术 24 419 110
2 新闻与媒体 7 125 19
3 商业与经济 48 839 214
4 娱乐与休闲 88 1510 374
5 计算机与因特网 58 925 238
6 教育 18 286 85
7 各国风情 53 891 235
8 自然科学 113 1892 514
9 政府与政治 18 288 84
10 社会科学 104 1765 479
11 医疗与健康 136 2295 616
12 社会与文化 66 1101 301
共计 733 12336 3269
Concept Drift
Categories change over time Example: “president of the united states”
1999: clinton is great feature 2002: clinton is bad feature
One measure of a text classification system is how well it protects against concept drift.
Feature selection: can be bad in protecting against concept drift
Think about it…
Describe the process, can you tell what is it?
How do you do it? Why we talk about it here, or what does it
mean to Information Overloading?
鹰 肉鸡
Recall: Vector Space Representation
Each document is a vector, one component for each term (= word).
Normalize to unit length. High-dimensional vector space:
Terms are axes 10,000+ dimensions, or even 100,000+ Docs are vectors in this space
