Upload
butest
View
234
Download
1
Tags:
Embed Size (px)
Citation preview
Libraries and IntelligenceNSF/NIJ Symposium on Intelligence and Security
Informatics. Tucson, AR.
Paul Kantor
June 2, 2003 Research supported in part by the National Science Foundation under Grant EIA-0087022and by the Advanced Research Development Activity under Contract 2002-H790400-000. Theviews expressed in this presentation are those of the author, and do not necessarily represent the views of the sponsoring agency.
Relation to General Intelligence and Security Informatics
• Signal information
• Map and image information
• Sound/voice information
• Geographic information
• Structured (Database) information
• Free form textual information in machine readable form
Relation to Librarianship
• Much of the needed “technology” for managing information related to homeland security is of the same type that librarians have provided “by hand”.
• But ..– Millions of documents– dozens of languages– many media
Librarianship
• Cataloging– organizing information according to what it is about– Classification – Machine Learning– Use training examples– Adapt as more data is received– Filter huge streams of potentially relevant data
• Monitoring Message Streams
Librarianship
• Reference– Understand what the user wants– Understand both relevance and quality/genre– Learn from a dialog with the user
• Intelligent Question Answering
Two Projects
• Filtering/Monitoring Message Streams National Science Foundation (NSF) -- acting for the National Security Agency HITIQA - High quality interactive Question Answering
• Advanced Research Development Activity (ARDA) of the Intelligence Community
Motivation:
monitoring of global satellite communications (though this may produce voice rather than text)
sniffing and monitoring email traffic
OBJECTIVE:
Monitor streams of textualized communication to detect pattern changes and "significant" events
© Paul Kantor 2002
1. Accumulated documents
2. Unexpected event
3. Initial Profile
4. Guided Retrieval
5.Clustering
6. Revision and Iteration
Retrospective/Supervised/Tracking
1. Accumulated documents
4. Anticipated event
3. Initial Profile
5.. Guided Retrieval
2.Clustering
Prospective/Unsupervised/Detection
Rutgers DIMACS: Automatic Event Finding in Streams of Messages
7. Track New documents
Analysts
Analysts
MMS TeamStatisticians, computer scientists, experts in info. Retrieval &
library science, etc
Prof. Fred Roberts – decision rules
Prof. David Madigan – statistics
Dr. David Lewis –text classification
Prof. Paul Kantor – info science
Prof. Ilya Muchnik – statistics
Prof. Muthu Muthukrishnan –algorithms
Dr. Martin Strauss, AT&T Labs –algorithms
Dr. Rafail Ostrovsky, Telcordia Technologies, -algorithms
Prof. Endre Boros, --Boolean optimization.
Dr. Vladimir Menkov programming;
Dr. Alex Genkin programming;
Mr. Andrei Anghelescu; graduate asisstant
Mr. Dmitiry Fradkin; graduate assistant
• Given stream of text in any language.
• Decide whether "new events" are present in the flow of messages.
• Event: new topic or topic with unusual level of activity.
• Retrospective or “Supervised” Event Identification: Classification into pre-existing classes.
TECHNICAL PROBLEM:
More Complex Problem: Prospective Detection or “Unsupervised” Learning
1) Classes change - new classes or change meaning
2) A difficult problem in statistics
3) Recent new CS approaches
4) Algorithm detects a new class
5) Human analyst labels it; determines its significance
COMPONENTS OF AUTOMATIC MESSAGE PROCESSING
(1). Compression of Text -- to meet storage and processing limitations;
(2). Representation of Text -- put in form amenable to computation and statistical analysis;
(3). Matching Scheme -- computing similarity between documents;
(4). Learning Method -- build on judged examples to determine characteristics of document cluster (“event”)
(5). Fusion Scheme -- combine methods (scores) to yield improved detection/clustering.
Random Projections
Boolean Random
Projections Robust Feature
Selection
Compr
essio
n
Repre
sent
atio
n
Bag of Words
Bag of Bits
Mat
chin
g
Learn
ing
Fusio
n
tf-idf
kNN
Boolean
r-NN
Rocchio separator
Combinatorial Clustering
Naïve Bayes
Sparse Bayes
Discriminant Analysis
Support Vector
Machines
Non-linear Classifiers
Project Components: Rutgers DIMACS MMS
• Existing methods use some or all 5 automatic processing components, but don’t exploit the full power of the components and/or an understanding of how to apply them to text data.
• Lewis' methods used an off-the-shelf support vector machine supervised learner, but tuned it for frequency properties of the data.Very good TREC 2002 results on batch learning.
• Chinese Academy of Sciences used most basic linear classifier (Roccho model) and achieved the best adaptive learning)
Proposed Advances
• We can trace a path (called a homotopy) in method space, from a poor Rocchio model to the CAS one -- find some better results along the way.
• Next steps are:
more sophisticated statistical methods
sophisticated data compression in a pre-processing stage
Proposed Advances II
• Representations: Boolean representations; weighting schemes
• Matching Schemes: Boolean matching; nonlinear transforms of individual feature values
• Learning Methods: new kernel-based methods (nonlinear classification); more complex Bayes classifiers to assign objects to highest probability class
• Fusion Methods: combining scores based on ranks, linear functions, or nonparametric schemes
MORE SOPHISTICATED STATISTICAL APPROACHES:
•.
• Identify best combination of newer methods through careful exploration of variety of tools.
• Address issues of effectiveness (how well task is done) and efficiency (in computational time and space)
• Use combination of new or modified algorithms and improved statistical methods built on the algorithmic primitives.
• Systematic Experimentation on components and on fusion schemes
THE APPROACH•.
Mercer KernelsMercer’s Theorem gives necessary and sufficient conditions for a continuous symmetric function K to admit this representation:
“Mercer Kernels”
This kernel defines a set of functions HK,
elements of which have an expansion as:
This set of functions is a “reproducing kernel hilbert space”
)()()()(),(1
zxzxzxK iii
i
N
iiii
ii xxKxcxf
11
),()()(
K “pos. semi-definite”
Prepared by David L. Madigan
Support Vector MachineTwo-class classifier with the form:
parameters chosen to minimize:
Many of the fitted ’s are usually zero; x’s corresponding the the non-zero ’s are the “support vectors.”
N
iii xxKxf
10 ),()(
Kxfy TN
iii
1,
)(10
min
complexity penalty
Gram matrix
tuning constant
Prepared by David L. Madigan
Regularized Linear Feature Space Model
Choose a model of the form:
to minimize:
Solution is finite dimensional:
bxwxf ii
i
)()(1
2
1
))(,( fxfyLN
iii
)()(),( zxzxK
just need to know K, not !
prediction is sign(f(x))
bxxyxfN
iiii
1
)()()(
A kernel is a function K, such that for all x,z X
where is a mapping from X to an inner product feature space F
Pre
pare
d by
Dav
id L
. Mad
igan
Mixture Models
• Pr(d|Rel)=af(d)+(1-a)g(d)
• f, g may be centered at different points in document space. So distinct conceptual representations are accommodated easily.
• Examples: multinomial distributions.
Example Results on Fusion
• http://dimacspc6.rutgers.edu/~dfradkin/fusion/centroid/try.pdf
• http://dimacspc6.rutgers.edu/~dfradkin/applet/topicShowApplet.jsp
• 60,000 documents.
Feature space
Random Subspace
Score space
Learning takes place in two spaces: For matching and filtering, we learn rules in the primary space of document features. For fusion processes we learn rules in a secondary space of “pseudo-features” which are assigned by entire systems, to incoming documents.
RelevantRelevant
REFERENCE ASPECT
Effective Communication with the Analyst User
HITIQA: High-Quality
Interactive Question Answering
University at Albany, SUNYRutgers University
HITIQA Team
• SUNY Albany:– Prof. Tomek Strzalkowski, PI/PM– Prof. Rong Tang– Prof. Boris Yamrom, consultant– Ms. Sharon Small, Research Scientist– Mr. Ting Liu, Graduate Student– Mr. Nobuyuki Shimizu, Graduate Student– Mr. Tom Palen, summer intern– Mr. Peter LaMonica, summer intern/AFRL
• Rutgers:– Prof. Paul Kantor, co-PI– Prof. K.B. Ng– Prof. Nina Wacholder– Mr. Robert Rittman, Graduate Student– Ms. Ying Sun, Graduate Student– Mr. Peng Song, Graduate student
HITIQA Concept
Question: What recent disasters occurred in tunnels used for transportation?
Possible Category Axes SeenV
ehic
le t
yp
eLosses/Cost
loca
tion
other
auto
train
USER PROFILE; TASK CONTEXT
QUESTION NL PROCESSING
Clarification Dialogue:S: Are you interested in train accidents,automobile accidents or others?U: Any that involved lost life or a majordisruption in communication. Must identifyloses.
Semantics: What the question“means”:• to the system• to the userS
EM
AN
TIC
PR
OC
FUSE &SUMMARIZE
Answer &Justification
AN
SW
ER
GE
NE
R.
SEARCH &CATEGORIZE
KB
TEMPLATE SELECTION
Focused Information Need
QUALITY ASSESSMENT
Key Research Issues
• Question Semantics – how the system “understands” user requests
• Human-Computer Dialogue – how the user and the system negotiate this
understanding
• Information Quality Metrics – how some information is better than other
• Information Fusion – how to assemble the answer that fits user
needs.
Document Retrieval
Document Retrieval
BuildFrames
BuildFrames
ProcessFrames
ProcessFrames
DialogueManager
DialogueManager
QuestionProcessor
QuestionProcessor
Wordnet
Completed Work
question
Segment/Filter
Segment/Filter
ClusterSegments
ClusterSegments
Query Refinement
Query Refinement
Current Focus
DB
Gate
AnswerGenerator
AnswerGenerator
answer
Visualization
Data-Driven NL Semantics
What does the question mean to the user?– The speech act– The focus– User’s task,
intention, goal– User’s background
knowledge
What does the question mean to the system?– Available
information– Information that
can be retrieved– The dimensions of
the retrieved information
Answer Space Topology
KERNELQUESTION
MATCH
KERNELQUESTION
MATCH
NEARMISSES,
ALTERNATIVE INTERPRETATIONS
ALL RETRIEVED
FRAMES
Quality Judgments
• Focus Group:– Sessions conducted: March-April, 2002– Results: Nine quality aspects generated
• Expert Sessions:– Sessions Conducted: May-June, 2002– Results: 100 documents scored twice along 9 quality aspects
• Student Sessions:– Training and Testing Sessions: June-July, 2002
• 10 documents judged by experts used for training/testing
– Actual Judgment Sessions: June-August, 2002• Qualified students evaluated 10 documents per session
– Results: 900 documents scored twice along 9 quality aspects
Factor Analysis of 9 Quality Features
Appearance
Content
Modeling Quality of Text• Kitchen sink approach
– 160 “independent” variables– Part-of-speech, vocabulary – stylistics, named entities, …
• Statistical pruning– Statistically significant variables– May be nonsensical to human
• Human pruning– Only “sensible” variables retained for each quality
• Pruning improves performance– Kitchen sink overfits– Statistics and Human close in performance– More work needed to understand the relationship
Quality Prediction by Linear Combination of Textual Features (from 5 to 17 variables). Split Half for Training and Testing.
Quality Factors Prediction Rate
Depth 67%Author Credential 55%
Accuracy 69%Source 57%
Objectivity 64%Grammar 79%
One Side vs Multi View 70%
Verbosity 63%Readability 76%
Performance of models
Quality Aspect: Depth ROC Using Stepwise Discriminant Function
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 False Alarm Rate
Det
ectio
n R
ate
Training
Testing
Perfect Knowledge
No Knowledge
In Summary
• The two conceptual foundations of librarianship: cataloging and reference, translate to two important problems in managing streams of textual messages:
• Both involve pattern recognition or machine learning.
Two Roles for Learning
• Cataloging: learning which features of a message mean that it is significant to the problem at hand
• Reference: learning which features of a message mean that it is “salient” to a specific user of the system.
Appendix:The following slides were not presented at the conference.
Communicating Credibility
• A system that is correct 75% or 80% of the time will be wrong one time in every four or 5.
• Unless it can “shade” its judgments or recommendations, the analyst will lose confidence in it.
• Credibility must be high enough to avoid extensive rework.
Data Fusion
• Use multiple methods to assess the relevance of documents or passages, – For a given question, dialogue, or cluster– Each method assigns a “score”
• Candidates → points in a “score space”• Seek patterns to localize the most relevant
documents or passages in this “score space”• Developed interactive data analysis tool
Background on Fusion Problem
• There are systems S, T, U, …
• There are problems to be solved P,Q,R…
• This defines several fusion problemsLocal fusion: for a given problem P, and a pair of
systems S,T, what is the best fusion rule:
Let s(d) ,t(d) be the scores assigned to document d by systems S and T. Fusion tries to find the “best” combining function f(s,t)
Non-linear “iso-relevance”
Local Fusion Rule
• A local fusion rule fP(s,t) depends on the specific problem P.– This is relevant if P represents a static problem
or profile, which will be considered on many occasions
• A global fusion rule f(s,t) does not depend on a specific problem P, – and can be safely used on a variety of problems.
• Completely rigorous For each topic:
• 1) Randomly split the documents into two parts: training and testing
• 2) Do the logistic regression on training part and get the fusion scores for both training and testing documents
• 3) Calculate p_100 on testing documents.
• 4) Excellent results (one random sample for each)
• 5) Test SMART and InQuery on the same random testing set
Local Fusion Results are Good
Summary of Local FusionF & SM F & IN F & BEST
win 11 7 5tie 4 7 9lose 0 1 1
Smart InQuery Fusion318 0 3 0318 1 5 2318 1 5 4318 0 3 1318 1 4 2
PROBLEM CASE
We ran 5 split half runs on the odd case (318) and the results persist.
Is Local Sensible?
• Local fusion depends on getting information about a particular topic, and doing the best possible fusion.
• Not available in an AdHoc (e.g. Google) setting
• Potentially available in an intelligence applications - -filtering; standing profile
Fusion of InQuery and Smart: Topic 450* Easy case – almost any linear rule works well. Either system works well
Fusion of InQuery and Smart: Topic 450* Easy case – almost any linear rule works well. Either system works well Fusion of InQuery and Smart: Topic 392
* Easy case – SMART works well. InQuery works poorly
Fusion of InQuery and Smart: Topic 392* Easy case – SMART works well. InQuery works poorly
Fusion of InQuery and Smart: Topic 432* Another hard case – relevant documents not compactly grouped in the score space. Not many relevant documents found at all.
Fusion of InQuery and Smart: Topic 432* Another hard case – relevant documents not compactly grouped in the score space. Not many relevant documents found at all.
Fusion of InQuery and Smart: Topic 318* Interesting case – no linear rule works well. Relevant documentsembedded. Requires non-linear methods – Quadratic; SVM; other
Fusion of InQuery and Smart: Topic 421* Really challenging case: Quite a few relevant documents. Very diffuse in score space. Neither system works well. Possibly Boolean AND
Fusion of InQuery and Smart: Topic 421* Really challenging case: Quite a few relevant documents. Very diffuse in score space. Neither system works well. Possibly Boolean AND
Fusion of InQuery and Smart: Topic 359* A disaster.
Fusion of InQuery and Smart: Topic 359* A disaster.
Fusion of InQuery and Smart: Topic 374* Possible Boolean AND. Neither works well alone
Fusion of InQuery and Smart: Topic 374* Possible Boolean AND. Neither works well alone
Fusion of InQuery and Smart: Topic 415* Part of the relevant material is easily found. Part is embedded
Fusion of InQuery and Smart: Topic 415* Part of the relevant material is easily found. Part is embedded
Our Approach to Retrieval Fusion
SMART
InQuery
FUSION PROCESS
Request
DOCUMENTS SETS
Result Set
Delivered SET
Result Set
ADOPT: Fusion
System
Monitor Fusion Set
and Receive
Feedback
USE: Better System
Adaptive “Local” Fusion