View
215
Download
0
Tags:
Embed Size (px)
Citation preview
KNOWLEDGE ON DEMAND:Knowledge and
Expert Discovery
Dr. Mark T. MayburyExecutive Director
Information Technology Division
Knowledge Management Conference Baden Baden, Germany
15 March 2001Organization: G060Project: 05AAV061-C1 MITREhttp://www.mitre.org/resources/centers/it
204/18/23 20:08
MITRE
Knowledge on Demand
•Knowledge Management Strategy
•Knowledge Extraction and Discovery- TIDES, GeoNODE, QANDA, SIAM
•Expert and Expert Community Discovery- ExpertFinder, XperNET
•Facilitating Group Knowledge Creation- KEAN, OWL, SCOUT
•Facilitating Knowledge Communication/Exchange- CVW, TrIM
•Conclusion- Knowledge Management Lessons Learned- Grand Challenges
304/18/23 20:08
MITRE
Why KM?Change is Accelerating
0
2
4
6
8
10
12
14
16
18
0 6 12 18 24 30 36
OpticalNetwork Speeddoubles every 8monthsStoragecapacitydoubles every12 monthsComputingpower doublesevery 18months
dot COM storage requirements double every 90 days
404/18/23 20:08
MITRE
What is KM? An Enterprise Perspective
The strategies, processes, and technologies employed to enable anenterprise to acquire, create, share, and make actionable theknowledge needed to achieve mission objectives
KM Process
Influences
EnablingTechnologiesand Processes
CoPsLeadership
StrategyReward &Recognition
Best PracticeDBs
604/18/23 20:08
MITRE
KM Enablers
Str
ateg
y
Mea
sure
men
tP
olic
y
Con
tent
Pro
cess
Tech
nolo
gyC
ultu
re
State 2Harvesting the Benefits
KM Targets
Enterprise Processes
Knowledge Discovery
Tool/Process integration
Knowledge Creation and Re-use Impact
State 1Fostering Knowledge Development
Common KM Understanding
Center Pilots
Consolidated Resource View
Greater Tool Standardization
Knowledge-Sharing
Knowledge-Enabled Outcome States
State 0Where We Were
Ad Hoc Processes
Local Initiatives
Disparate Views of Resources
Low Tool Standardization
Collaboration Valued
State “V”Ultimate
Vision
Embedded KM
Known Knowledge Value
Pervasive Infrastructure
Innovative Outcomes
804/18/23 20:08
MITRE
APQC Model of Stages of KM Implementation
Develop interest and enthusiasm No formal businesscase; belief in the
valueDefine KM in termspeople understand
Capitalize on intranet Understand organizational readiness
Select pilots oridentify grassroot efforts
Businessobjectives arespecific to pilots
Form a cross-functional KMtask force
Scale up; buildcapability
Business caseand measuresbecome moreformal
KM coordinationteam
Identify roles and resources for the KM function Establish awards and recognition
KM embedded inbusiness model
Organizationalalignment
Project work withactivity andknowledge basesupport
Standards
Pilot Path
StrategicPilots
OpportunisticPilots
Improve
Expand
Disengage
Way ofdoingbusiness
Decision
Support pilots Business case ispotential gainfrom pilots
Share pilotlessons learnedDevelop methodologies
that can be replicated
KM Portfolios of KM Best Practice Companies (APQC, 2000)
World Bank
Seimens
HP Consulting
MITRE
Chevron
Resource Communications Collaboration Work Application
Resource Tool (Pull) - Yellow Pages, Best Practice DBs, Search Engines
Communications - e-mail, Web pages
Collaboration - Access to knowledgeable human resources
Work Application - Project Management, Problem Solutions,Customer Service
X X X
X X X X
X X X X
X X X X
X X X X
Xerox X X X X
Elements Central to KM Approach: Intranet, CoP/Networks, Best Practice Publication
1004/18/23 20:08
MITRE
KNOWLEDGEINFORMATION
INFRASTRUCTURE(KII)
Process
Expertise & Knowledge Discovery
Knowledge Creation
KnowledgeRequirement
Customer(s)
Knowledge TeamFormation
Knowledge Delivery
1104/18/23 20:08
MITRE
Knowledge on Demand
•Knowledge Management Strategy
•Knowledge Extraction and Discovery- TIDES, GeoNODE, QANDA, SIAM
•Expert and Expert Community Discovery- ExpertFinder, XperNET
•Facilitating Group Knowledge Creation- KEAN, OWL, SCOUT
•Facilitating Knowledge Communication/Exchange- CVW, TrIM
•Conclusion- Knowledge Management Lessons Learned- Grand Challenges
1204/18/23 20:08
MITRE
Today
Documents, Not answers
Documents, Not answers
Multilingual,Multimedia,Multiparty Resources
Multilingual,Multimedia,Multiparty Resources
Vision: Ask Questions, Get Answers
Tomorrow
Question: Where are the leaders of the ELN?
Question: Where are the leaders of the ELN?
Answer: Francisco Galan and Felipe Torres are in the penitentiary at Itagui, Columbia
Answer: Francisco Galan and Felipe Torres are in the penitentiary at Itagui, Columbia
Answers & Drill down
Answers & Drill down
Knowledge Discovery Tools
Collect
Extract(Alembic)
Sources
Summarize(WebSumm)
Cluster/Mine(QueryFlocks)
Collaborate(KEAN, Scout,
ExpertFinder, XperNET)
Translate(CyberTrans)
Browse/Visualize(GeoNODE)
Monitor(SIAM)
Finance Energy Trans. Telecomm Z-Ave
Disseminate/Retrieve(TIDES, QANDA)
Detect, Translate, Extract, Summarize
Tamil document
•Liberation Tigers of Tamil Eelam (LTTE)•Sri Lanka•Velupillai Pirapaharan•Rebellion
Topic Detection
Source: Ron Larson (DARPA TIDES)
The objective of the Sinhala chauvinists was to utilize maximum man power and fire power to destroy the military capability of the LTTE and to bring an end to the Tamil freedom movement. Before the launching of the operation "Jayasikuru" the Sri Lankan political and military high command miscalculated the military strength and determination of the LTTE.
The objective of the Sinhala chauvinists was to utilize maximum man power and fire power to destroy the military capability of the LTTE and to bring an end to the Tamil freedom movement. Before the launching of the operation "Jayasikuru" the Sri Lankan political and military high command miscalculated the military strength and determination of the LTTE.
Summarization
Org Leader HQ LossesSinhala Kumaratunga 3000LTTE Pirapaharan Wanni 1300
Extraction
Today is a significant day in the history of our national liberation struggle, it marks the end of a year during which we have resisted and fought against the biggest ever offensive operation launched by the Sri Lankan armed forces code named "Jayasikuru”...Translation
DARPA TIDES
What is the status of thecurrent Ebola outbreak?
The epidemic is contained;as of 12/22/00, there were 421 cases with 162 deaths
Interaction
CDCWHO
Medicalliterature
Email:ProMed
~ 2500 stories/day
InternlNewsSources
Capture
Translingual
Information Detection Extraction Summarization
Unidentified hemorrhagic fUnidentif ied hemorrhagic f
Ebola hemorrhagic fever in
Re: Ebola hemorrhagi...Re: Ebola hemorrhagi...
ProMEDAnnotatorJane Analyst
10/17/00 19:3710/17/00 20:4210/18/00 7:42
HighNormalNormal
readreplied
ProMED
10/18/00 12:34 High unread
Ebola hemorrhagic fever in
SourceDate
Priority Status
10 99
0 105
1 57
0 10
2 34
0 50
1 1
0 25
5 200
0 45
0 0
0 0
0 0
0 0
0 6
0 32
0 3
0 1
HighNormalHighHigh
Ebola hemorrhagic fever - Uganda
Unfiltered
Outbreak
Cholera
Dengue Fever
Ebola
Infrastructure…
Natural Disas...
Spills
Accidents
WMD Trackin...
Suspicious Il ln...
Suspicious De...
Possible Biolo...
Pathogen threa…
----------------------------
Workspace
Ebola
Drafts
Reports
Disease
Re: Ebola hemorrhagi...
Location
UNKUNKEbolaEbolaEbolaEbolaRabiesRabies
UgandaUgandaUgandaKenyaUganda
IHTProMEDWHO
Joe Analyst
Date
10/14/00 23:0610/15/00 10:5010/16/00 21:4510/17/00 19:12
readreadreadread
unread
Date: 10/16/00
Disease: EbolaDescriptor: hemorrhagic fever
Location: Uganda
Disease Date: 10/14/00Hospital: missionary hospital in Gulu
New cases: at least 7Total cases: 51
Total dead: 31
Ebola hemorrhagic fever -
Ugandan Ministry identifies Ebola virus as the cause of the outbreak. KAMPALA:The dreaded Ebola virus that struck over 300 people in Kikwit, in the DemocraticRepublic of Congo in 1995, has ki lled 31 people in northern Uganda. A UgandanMinistry of Health statement said laboratory tests had revealed that the Ebola viruswas the cause of the epidemic hemorrhagic fever which has been raging in the Guludistrict since September. Three of the dead were student nurses , who treated the firstEbola patients admitted to a Lacor missionary hospital in Gulu town. A task forceheaded by Gulu district administrator, Walter Ochora, has been set up to co-ordinateefforts to control the epidemic. Field officials in Gulu told the Kampala-based New
Http://tides2000.mitre.org/ProMED/10162000/34n390h.html
Uganda
News Repository
CATALYSTCATALYSTEvent ExtractionTime TaggingTDTTranslationSummarization AlertingChange detectionCross-language IR
1604/18/23 20:08
MITRE
TIDES Portal
Metaqueriessupported formultiple sources
Translingual system supportsforeign-languagesources
Multiple mediaexploited
Government &private sourcesutilized
16
Translingual
Information Detection Extraction Summarization
1704/18/23 20:08
MITRE
Geospatial News on Demand (GeoNode)
InformationExtraction
Data MiningAnd Clustering
ttopic
t
Map overview
Topic Timeline
News histogram
•Navigate•Filter•Indexed access•Animate reporting trends•Create reports/ web
BNN Story skim
IndexingAnd NewsModeling
DataAcquisition/Pre-process
GeoNODE Database
News Sources:
Broadcast
Specialist Archives
World Wide Web
Intel. Msg Traffic
1804/18/23 20:08
MITRE
Person1 Person2 Location Support
Mobutu Sese Seko Laurent Kabila Kinshasa 7
0 Two-way associations: people and locations
0 Three-way associations, only one association with support of at least 5 (confidence is 50%: 1/2 of the stories mentioning any item
also mention the other two)
6612 stories, with 13,737 distinct concepts mentioned. The associations between pairs of concepts are ranked by support: the number of documents containing the
pair). All correlations have at least 50% confidence: At least 50% of the stories mentioning one item in a pair also mention the other
Type Value Type Value SupportPerson Natalie Allen Person Linden Soles 117Person Leon Harris Person Joie Chen 53Person Ron Goldman Person Nicole Brown Simpson 19Person Dole Person Bob Dole 18Person Forbes Person Dole 16Person Forbes Organization New Hampshire 15Person Bill Cosby Location Cosby 14Person Pat Buchanan Person Forbes 12Person Steve Forbes Organization New Hampshire 12Person Mobutu Sese Seko Location Kinshasa 10
Data Mining: “Find Significant Warlords in a Region”
1904/18/23 20:08
MITRE
The Web Has Gone Multilingual
•Last year, the web became more than 50% non-English
•Only 15% of Europe's half a billion population speaks English as a first language
•Only 28% speaks English at all
•Only 32% of Web surfers on the European continent consult the Web in English. [Source: Global
Reach:www.euromktg.com/eng/GR/]
•45% of Internet users from non English-speaking countries
•By 2002, analysts estimate that 66% of Internet use and 40% of e-commerce revenue will come from outside the U.S. [Source: IDC]
•300,000 Japanese patents filed annually
"If I'm selling to you, I speak your language. If I'm buying,
dann müssen Sie Deutsch sprechen”
Willy Brandt, former German chancellor
2004/18/23 20:08
MITRE
Open Source Analysis of Latin America
Event Timeline
•10Mar98 Pinochet resigns•17Mar98 Cuban defector, pitcher Orlando
Hernandez•15Apr98 Execution of Paraguayan Angel Breard,
convicted killer in US•21Apr98 Plane crash: Bogota, Columbia to
France•17Oct98 - 27Oct98 Pinochet’s arrest by Scotland
yard while getting medical treatment•26Oct98 House of Lords deny Pinochet
diplomatic immunity•28Nov98 Columbian man surrenders near
Bogota, accused of shooting DEAAgent Moreno
•04Dec98 Iranian Woman has been charged by Argentina's Supreme Court for the1992 bombing of Israeli embassy in Buenos Aires, Argentina
•07Jan99 - 18Jan99 Brazilian Financial crisis
2104/18/23 20:08
MITRE
Multiple Knowledge Sources and
Multiple Applications
GeoNODE BNN
Doc collection
orthe Web
Semi-structured database
RelationalDatabase
Knowledgesources
Userapplications
Q and AEngine
IRS hotlineoperator
MII
Question and Answering - QANDA
Concept + Information Volume + Time + Location + Source => Social Interest
Concept mapped to query sets (large numbers of queries) to address topic specificity and coverage. Query Class, Retrieval Rank and/or Relevance can be used to weight items
Search agent collects from
multiple search engines,
link traversal, and sampling strategies
used to collect relevant items,
and scale up results
to population levels.
Normalized Volume-->Importance
Windowing
collection to
specific time
periods e.g.,
4 quarters per
year
Spatial tags for tracking
by country or region
Source weighting (Optional) for weighting importance based on source type and website structure, e.g., logical location within a Gov. site
Social Interest of Topic X Country Y = F(relevance,scaled- volume,location, time, source)
Issue
Social Indicators Analysis Method (SIAM)
Gartner Group: “Year 2000 World Status,
2Q99: The Final Countdown
Other Gartner Reports (periodic
assessments, Gartner
Interactive Reports,…)
Lou Marcoccio Interview
U.S. Senate : Investigating The Year 2000
Problem: The 100 Day Report”
Department of State: Biannual Consular
Advisory
MITRE Y2K Assessment
US Agency for International Development
(USAID)
International Y2K Cooperation Center
(IY2KCC)
International Monitoring (IM) … a London-
based consultancy
Y2K Assessments are Generally Survey-
Based
Example: Gartner Group
Survey: 330 Questions
Performed Quarterly
600 People Involved
1500+ Companies
3 Calls/Company
11 Universities Provide Analysis
COMPARE Statistic Computed from Survey
Example: Department of State has an
ongoing Survey among its 260 Posts.
Reporting Bias Observed by Gartner...others.
SIAM Processing Costs
New Domain: 1 Day
Quarterly Analysis: 2 Days
Download Time: 1-2 Days for Current Data
Results Analysis and Fine tuning: 2 Days
Today’s Manual Approach
CNTRY Finance Energy Trans. Telecomm Z-Aveargentina 1.476278 2.434932 0.8366 1.9553 1.67579bermuda 0.888375 -0.06594 1.9661 0.1754 0.740966bolivia -0.65204 -0.55287 -0.6155 -0.6380 -0.61458brazil 1.869995 0.27763 0.7220 1.5664 1.109025chile 1.242407 0.065475 1.9946 0.8680 1.042629colombia -0.5338 -0.38638 -0.4981 -0.5096 -0.48196costa_rica -0.39694 -0.29373 -0.4429 -0.2979 -0.35789ecuador -0.6348 -0.54902 -0.6117 -0.6289 -0.60608el_salvador -0.6304 -0.52669 -0.5945 -0.6204 -0.59299guatemala -0.65019 -0.54851 -0.6130 -0.6366 -0.61208guyana -0.65496 -0.55555 -0.6146 -0.6410 -0.61652honduras -0.65361 -0.55464 -0.6161 -0.6409 -0.61632mexico 1.656809 2.306324 1.1496 2.0101 1.780716nicaragua -0.60775 -0.52549 -0.5961 -0.6028 -0.58303panama -0.64749 -0.54845 -0.6090 -0.6352 -0.61004paraguay -0.64199 -0.54268 -0.6052 -0.6190 -0.60222peru -0.42739 -0.31945 -0.4733 -0.3972 -0.40431puerto_rico -0.65496 -0.55555 -0.6162 -0.6401 -0.6167uruguay 0.987168 1.667379 1.2754 1.2915 1.305344venezuela -0.33471 -0.2268 -0.4382 -0.3592 -0.33974
High Risk Inferred
Moderate Risk Inferred
Low Risk Inferred
SIAM “Status Board”
VENEZUELACOLOMBIA
BRAZIL
PARAGUAY
URUGUAYARGENTINA
CHILE
BOLIVIAPERU
ECUADOR
MEXICO
NICARAGUA
HONDURAS
COSTA RICA
PANAMA
EL SALVADOR
GUATEMALABELIZE
2504/18/23 20:08
MITRE
Senate Report: “Peru is, across many sectors, either
not Y2k ready or that public information is inadequate.”
… “The Gartner Group and the World Bank offer
contradictory information, ranking Peru as one of the
better prepared in South America.”
mexico 1.656809 2.306324 1.1496 2.0101nicaragua -0.60775 -0.52549 -0.5961 -0.6028panama -0.64749 -0.54845 -0.6090 -0.6352paraguay -0.64199 -0.54268 -0.6052 -0.6190peru -0.42739 -0.31945 -0.4733 -0.3972
SIAM Indicator Board
SIAM Scores Peru as Moderate Readiness Ranking 10th out of 20 Countries
Variance of Opinion
2604/18/23 20:08
MITRE
Knowledge on Demand
•Knowledge Management Strategy
•Knowledge Extraction and Discovery- TIDES, GeoNODE, QANDA, SIAM
•Expert and Expert Community Discovery- ExpertFinder, XperNET
•Facilitating Group Knowledge Creation- KEAN, OWL, SCOUT
•Facilitating Knowledge Communication/Exchange- CVW, TrIM
•Conclusion- Knowledge Management Lessons Learned- Grand Challenges
2704/18/23 20:08
MITRE
Expert Discovery
•Find global Experts- quick- accurate- comprehensive
•Challenge: Overcome limitations of manually managed skills/expertise databases (e.g. Dataware - experts self nominate)- incomplete - expensive- out of date
2804/18/23 20:08
MITRE
Related Work
•Autonomy- document based (docs, Notes discussions, email)- dynamic expert profiling- Problem: reading/writing not always correlated
w/expertise
•Abuzz- Beehive email routes questions to experts based on
expert profile (must seed this)- Expertise validated by community (+/- satisfaction with
answers) updates profiles- Problem: Seeding/Learning curve
•MIT’s ExpertFinder (Vivacqua)- expertise models from software library use
•Tacit - email based keyword profiling
2904/18/23 20:08
MITRE
Uniqueness
•This work:- Implicitly determines expertise from multiple sources
of evidence including intellectual products (e.g., briefings, papers, web pages) and information seeking actions (e.g., web logs)
- Leverages intranet publishing (staff, corporate news letters), corporate directory services, project leadership information
- Exploits recent advances in information extraction (language processing) technology
Expertise Management Architecture
E-dB
Finder
ServiceBroker
MII
WWW
FinderAgencies
ConsultingGroups
Q&A
Services
Resources
Registration
Qualification
Selection
Expert FinderGoal: Place a user within one phone call of an expert
Enterprise EmployeeProject Database
EmployeesRanked by Mentions
Mentions of Employee inCorporate
Communications
IntegratedEmployeeDatabase
User Issues Simple Query
Relevant EmployeePublications
3204/18/23 20:08
MITRE
ExpertFinder Algorithm
Initial Form
Call Search Engines
Parse Results
Gather All URLs
Find Mentioned People
Find Published People
Combine Info
Add Phone Book Info
Weigh Evidence
Display Results
CachedResults?
yes
no
3304/18/23 20:08
MITRE
Evaluation
•Compare performance of ExpertFinder with (20) expert human resource managers
•Task: Find top 5 corporate experts in a given domain
•Measures- Agreement among humans- Agreement of machine with human(s)
•Precision
•Recall
•Chance: # experts/4500 employees = often less than .1%
3404/18/23 20:08
MITRE
The Questionnaire
1. Who are the top 5 "data mining" experts at MITRE (List them in rank order, most expert first. List as many as you can but no more than 5)?
2. the top 5 "collaboration" experts? 3. the top 5 "chemical" experts? 4. the top 5 "human computer interaction" experts? 5. the top 5 "network security" experts? 6. What is your top area of expertise (in a few words)
and who do you consider to be the top 5 people in the company in your area of expertise?
I am performing an experiment. Your participation will remain anonymous if you so desire and should only take a few short minutes. Please answer the following questions (preferably without any assistance, but if you use assistance indicate what kind you used):
3504/18/23 20:08
MITRE
Human vs ExpertFinder
•Comparison to 20 human resource managers
•Agreement = in top 5 IDed experts
•Precision = # correctly IDed experts / # IDed
•Recall = # correctly IDed experts/ # actual experts
Expert Area Human Agreement(1st, 2nd, 3rd)
ExpFinder Precision
ExpFinderRecall
Data mining 70%, 49%, 24.5% 60% 40%
Chemical 40%, 8%, 0.8% 60% 40%
HCI 90%, 36%, 11% 60% 40%
Network Security 50%, 10%, 0.4% 20% 20%
Collaboration 70%, 35%, 17.5% 5% 5%
AVERAGE 63%, 28%, 11% 41% 29%
3604/18/23 20:08
MITRE
Pattern Analysis•Work/Activity Sampling•Feature Extraction•Topic Detection•Social Network Generation•Community of Practice “Registration”…
Community of Interest Modeling
Organizational Theory
Project InformationWeb Pages
Meetings/Conferences/...Share Folders
Published documents...
Clustering techniquesSocial network analysis methods
Summarization
Emergence…Monitoring
CommunicationSharing
Expert FindingOther Applications
3704/18/23 20:08
MITRE
Expert Communities: XperNet Network
Core Group Expanded Group
Automatic Network Expansion
Network Membership Ratings
0.00010.00020.00030.00040.00050.00060.00070.00080.00090.000
100.000
1 3 5 7 9
11
13
15
17
19
21
23
25
27
29
31
Member Rank
Me
mb
ers
hip
Sco
re
3904/18/23 20:08
MITRE
Knowledge on Demand
•Knowledge Management Strategy
•Knowledge Extraction and Discovery- TIDES, GeoNODE, QANDA, SIAM
•Expert and Expert Community Discovery- ExpertFinder, XperNET
•Facilitating Group Knowledge Creation- KEAN, OWL, SCOUT
•Facilitating Knowledge Communication/Exchange- CVW, TrIM
•Conclusion- Knowledge Management Lessons Learned- Grand Challenges
4004/18/23 20:08
MITRE
Words of “Wisdom”
•Bentov’s Law: One’s level of ignorance increases exponentially with accumulated knowledge. When one acquires a bit of new information, there are many new questions that are generated by it, and each new piece of information breeds five-ten new questions. These questions pile up at a much faster rate than does accumulated knowledge. Therefore, the more one knows, the greater his level of ignorance.
•Allen’s Tenet - The strength’s of one’s opinion on any matter or controversy is inversely proportional to the amount of knowledge that person has on that subject.
•BB’s Dictum - In a group, the unknowing will try to teach the lesser-skilled or knowing
4104/18/23 20:08
MITRE
Collaboration Taxonomy
Multiple levels of virtual teamingMultiple levels of virtual teaming
Awareness
Information Sharing
Joint Efforts
Alignment
LeadershipIntent
Incr
easin
g
Inte
ract
ion
Increasing
OverheadCoordination
4204/18/23 20:08
MITRE
Knowledge Exchange and Annotation eNgine (KEAN): Search
SEARCH by
- Subject- Keyword- Employee- Rating Level- Time
boykin
4404/18/23 20:08
MITRE
Questions answered (with justification) by KEAN (e.g., data mining)
•What information does Chris (expert in data mining) think is useful for data mining?
•What information do people in the data mining community of practice find useful on data mining?
•What information does everyone think is useful on data mining in the past few weeks?
•What information on data mining have I found to be useful in the past?
4504/18/23 20:08
MITRE
KEAN Evaluation
•26 individuals on 295 URLs - length of time on page (“reading” time)- explicit utility rating
•Focused task - directory services questions- Which standards organization defines the X.500
specification?”- “How does LDAP differ from X.500?”- “Name some of the data types that can be stored in an
LDAP attribute.”
•After the experiment, rate utility 1-10 (10 highest)
•Regression test yielded positive correlation - explicit utility = .0113*time read- 66% of all URLs “read” for greater than 78 seconds
were classified as high utility (6-10)
•Time --> utility
4604/18/23 20:08
MITRE
Organizational Wide Learning (OWL):Word Command Usage by Type
Command Usage by Type
0% 10% 20% 30% 40% 50% 60%
Edit
File
Format
View
Window
Insert
Tools
Table
Help
4704/18/23 20:08
MITRE
OWL Data: Word’s Top 10 Commands
Sequence Command PercentCumulativePercent
1 Edit Delete 34.2% 34.2%2 File Save 10.5% 44.8%3 File Open 8.7% 53.5%4 Edit Paste 7.9% 61.4%5 File DocClose 5.1% 66.5%6 Edit Copy 4.2% 70.7%7 Format Bold 3.7% 74.4%8 File Print 2.8% 77.2%9 Edit Cut 2.4% 79.7%
10 File SaveAs 1.7% 81.3%
4804/18/23 20:08
MITRE
Organizational Knowledge & Ignorance Some individuals never use a number of the more frequently-used commands
Count of Users & Usage
0
5
10
15
20
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49
Command Sequence
Use
rs
1
10
100
1000
10000
100000
Usa
ge
Users Usage
4904/18/23 20:08
MITRE
OWL recommends what to learn next(unique to each individual at each point in time)
USER #314 ExpectedObservedInstructionEdit Paste 170 274 OKEdit Delete 129 0 NewEdit Copy 107 97 OKEdit Cut 48 100 OKEdit Undo 16 14 OKEdit Find 12 1 MoreEdit SelectAll 9 12 OKEdit DeleteWord 4 0 NewEdit Replace 3 0 NewEdit PasteSpecial 2 0 New
5004/18/23 20:08
MITRE
Do OWL Users Use Help?
Use of MS Word Help & Office Assistant
0
2
4
6
8
10
12
14
16
Daily Weekly Monthly Annually Never
Frequency of Use
Num
ber
of U
sers
Office Assistant Help
• More than half (12/20) use help at least monthly.
• Only a few (6/20) use Office Assistant
5104/18/23 20:08
MITRE
Cooperative Searching (Monitoring Group Information Seeking Activities)
-- a multi-user collaborative retrieval tool. The “next generation” in IR systems addresses multi-user, coordinated searching, shared analysis, and has a built-in recommender system. Tracks topics, users, and provides a persistent knowledge store.
5204/18/23 20:08
MITRE
Cooperative Searching
Hypothesis
Group (coordinated) searching can be more effective than multiple (independent) searchers
working autonomously
5304/18/23 20:08
MITRE
Cooperative Searching-- Initial Prototype
Local Shared Workspace
Collect
Users Generate Ad Hoc Queries
Off-line Queries or Web-page Monitoring SupportedUsers Generate Ad Hoc Queries
Off-line Queries or Web-page Monitoring Supported
Users Generate Task FoldersUsers Generate Task Folders
5404/18/23 20:08
MITRE
Local Shared Workspace
Collect
Retrieved Information is Organized by Domain
Search Engine Statistics Provided
Offline Cluster Analysis and Categorization Provided
Retrieved Information is Organized by Domain
Search Engine Statistics Provided
Offline Cluster Analysis and Categorization Provided
Cooperative Searching
5504/18/23 20:08
MITRE
Cooperative Searching
Local Shared Workspace
Collect
User Actions Infer Relevance
Ratings/Annotations/Actions Are Stored User Actions Infer Relevance
Ratings/Annotations/Actions Are Stored
Access Count and Evaluation Status Provided
5604/18/23 20:08
MITRE
Local Shared Workspace
Collect
Rated Items with Annotations are
integrated into a Shared Context Rated Items with Annotations are
integrated into a Shared Context
Cooperative Searching
5704/18/23 20:08
MITRE
Knowledge on Demand
•Knowledge Management Strategy
•Knowledge Extraction and Discovery- TIDES, GeoNODE, QANDA, SIAM
•Expert and Expert Community Discovery- ExpertFinder, XperNET
•Facilitating Group Knowledge Creation- KEAN, OWL, SCOUT
•Facilitating Knowledge Communication/Exchange- CVW, TrIM
•Conclusion- Knowledge Management Lessons Learned- Grand Challenges
5904/18/23 20:08
MITRE
Translingual Instant Messaging (TrIM)
•Integration of - Simple Instant Messaging Protocol (simp.mitre.org) - CyberTrans machine translation framework
•Supports multilingual chat
•Research Issues:- Quality of conversational translation using document
translation engines- Presentation (monolingual, multilingual)- Data collection to learn language models
6004/18/23 20:08
MITRE
Knowledge on Demand
•Knowledge Management Strategy
•Knowledge Extraction and Discovery- TIDES, GeoNODE, QANDA, SIAM
•Expert and Expert Community Discovery- ExpertFinder, XperNET
•Facilitating Group Knowledge Creation- KEAN, OWL, SCOUT
•Facilitating Knowledge Communication/Exchange- CVW, TrIM
•Conclusion- Knowledge Management Lessons Learned- Grand Challenges
Knowledge Management Capability Maturity Model (KM CMM)
Level 4: Managed• Integrated knowledge processes• Quantitative process management
Level 4: Managed• Integrated knowledge processes• Quantitative process management
Level 5: Optimizing• Business process alignment• Process change management
Level 5: Optimizing• Business process alignment• Process change management
Wherewe are
‘00
Where we aregoing
‘01
Where we want to
be
Level 1: Initial• Adhoc processes• Partial technical infrastructure
Level 2: Repeatable• Program planning • Content QA process• Requirements process • KFP identification
Level 3: Defined• Organizational processes • Knowledge mapping• Intergroup coordination • Training program
Level 0:Not Practiced• Failure to perform KM• Culture counter to learning, sharing
6204/18/23 20:08
MITRE
Strategy
Peter Senge
“Learning Organizations”
Process
Takeuchi and Nonaka
“Organizational Knowledge Creation”
Benchmarking
Norton and Kaplan
“Balanced Scorecard”
a
C I I SCENTER FOR INTEGRATEDINTELLIGENCE SYSTEMS
MITRE
6304/18/23 20:08
MITRE
Lessons Learned
People, and the cultures that influence their behaviors, are the single most critical resource for successful knowledge creation, dissemination, and application. Understand and influence them.
Cognitive, social, and organizational learning processes are essential to the success of a knowledge management strategy. Focus your strategy on enhancing these processes.
Measurement, benchmarking, and incentives are essential to accelerate the learning process and to drive cultural change. Create a tailored balanced scorecard to target what you want to improve.
Knowledge management programs can yield impressive benefits to individuals and organizations if they are purposeful, concrete, and action-oriented. Make yours so.
In times of profound change, learners inherit the Earth,
while the learned find themselves beautifully
equipped to deal with a world that no longer exists.
- Al Rogers -
6504/18/23 20:08
MITRE
Some Grand Challenges
•User, Group and Organization Modeling, including knowledge, beliefs, goals and plans
•(U, G, O) Tailored presentation of knowledge
•Ontological integration of distributed DB & KB
•Universal knowledge access independent of user physical, perceptual, cognitive, cultural characteristics
•Organizational strategies for knowledge sharing
•Knowledge strategies in global, multicultural enterprises
•Privacy and Security
6604/18/23 20:08
MITRE
Events of Interest
•8th Internat. Conference on User ModelingSonthofen, Germany July 13-17, 2001www.dfki.uni-sb.de/cgi-bin/um2001
•Workshop on Human Language Technology and Knowledge ManagementToulouse, FranceJuly 6-7, 2001www.elsnet.org/acl2001-hlt+km.html
•Our work: www.mitre.org/resources/centers/it
Joint EACL - ACL Meeting
6704/18/23 20:08
MITRE
Acknowledgements
•Knowledge Management - Cynthia Small, Jean Tatlias
•TIDES - Lynette Hirschman, Jay Ponte et al.
•GeoNODE - Rod Holland, John Griffith et al.
•QANDA - Marc Light
•OWL - Frank Linton
•CVW - Jay Carlson, Deb Ercolini et al.
•TrIM - Rod Holland, John Ramsdell, Flo Reeder, Jay Carlson, Justin Richer, Galen Williamson, Michael Krutsch, Keith Crouch, Keith Miller
•SIAM, XperNET, SCOUT - Ray D’Amore, Manu Konchady
•KEAN - Daryl Morey, Tim Frangioso
•ExpertFinder - Dave Mattox, Inderjeet Mani, David House