Upload
dorthy-mcbride
View
217
Download
3
Embed Size (px)
Citation preview
Levels of Searching
ADEPT
Glasgow
2nd October 2003
Judgingthe
Levelof
Search
Two epidemiologists are leaving a party late one night and one drops their car keys.
The other following behind offers to help scrabble around on the ground to look for them.
"Where did you drop them?" asks the second.
"Over there" says the first, pointing to a spot some feet away.
"Then why are we looking over here?"
"Because there's a street light on over here" comes the reply.
Practical Examples from...
• Rapid Evidence Retrieval to Systematic Searching
• NES/SCPMDE and BEME Collaboration (Alex Haig)
• Beatson (Annette Thain)
• Others...
• Balancing sensitivity and specificity
Sensitivity & Specificity
• Sensitivity (recall) - percentage of “gold standard”
• Specificity (precision) - positive predictive value
Searching across thehealth disciplines
• Inter-professional direction
• Search experience is extendible– same core skills for systematic and rapid
searches– core search skills are transferable
Medical Education Searching
• No filters
• Problematic study designs
• Grey literature in large quantities
• Dispersed sources– databases– grey literature– web– hand searches– expert opinion
• the dissemination of information which allows medical teachers, institutions and all concerned with medical education to make decisions on the basis of the best evidence available
• the production of appropriate systematic reviews of medical education which reflect the best evidence available and meet the needs of the user, and
• the creation of a culture of best evidence medical education amongst individual teachers, institutions and national bodies.
Topic Review Groups
• High Fidelity Simulation in Medical Education
• The impact of feedback in formative and summative assessment and its effectiveness on the improvement of knowledge, skills and attitudes of health science students during their learning process and on teaching skills of tutors.
• What are the features of faculty development that make it effective?
Topic Review Groups
• What conditions are necessary or required for assessment and feedback to enhance physician improvement?
• A systematic review of educational interventions on the communication skills of medical clinicians
• Performance-based instruments that measure medical communication competence - an update and expansion of previous reviews.
Topic Review Groups
• Predictive values of assessment measurements obtained in medical schools and future performance in medical practice
• What does early clinical experience contribute to the basic education of health professionals?
• The impact of inter-professional education on health and social care practitioners, professional practice, patient/client related health and well being, and public health and social care outcomes.
• International
• Educational
• Catholic inclusion of evidence
• Psychologists estimate that we each make up to 1,200 decisions every day.
• Increasingly, professionals are being expected to base their decisions on evidence.
“Systematic Searching in Medical Education”
September 2001, Berlin
Methods
• Topic - Feedback in assessment (BEME pilot and current topic review group, Barcelona/Tel Aviv)
• Software - Ovid [CGI version 7.8]
• Databases - Medline, Embase, ERIC
Methods - journal selected
• limited time and resources
• required a title that was most comprehensively indexed
• Academic Medicine 1996-present
Methods - strategies
• Three levels of strategy: standard (most users; limited search syntax) enhanced (some use of search syntax) expert (full use of search syntax)
• syntax includes: free-text, controlled vocabulary, term explosions, phrase lists, subheadings, sub strings, filters, proximity operators, etc...
Methods - handsearching
• “...refers to the planned searching of a journal page by page (i.e. by hand), including editorials, letters, etc., to identify all relevant items.”
• time consuming and meticulous
• produces the “gold standard” by which search efficiency can be measured
Sensitivity
• Sensitivity (recall) - percentage of “gold standard”
• Sensitivity = total retrieved by search
total of the hand-search
Gold Gold Standard = Standard = 4646
Embase Sensitivity (n)Basic 4.3% 2Enhanced 10.7% 5Expert 15.2% 7
ERIC Sensitivity (n)Basic 0% 0Enhanced 4.3% 2Expert 6.5% 3
Medline Sensitivity (n)Basic 0% 0Enhanced 10.7% 5Expert 19.6% 9
GS=46GS=46
Specificity Specificity (precision) - positive predictive value
Specificity = relevant records identified
total retrieved by search
Embase Specificity (n)Basic 40% 5Enhanced 33% 15Expert 30.4% 23
ERIC Specificity (n)Basic 0% 0Enhanced 40% 5Expert 37.5% 8
Medline Specificity (n)Basic 0% 0Enhanced 31.3% 16Expert 32.1% 28
A Note of Caution
• Academic MedicineAcademic Medicine is a journal that specialises in medical education :– more likely to be indexed for context– journal presents information for better retrieval
• Other journals will fare worse
• Other specificity scores for BEME pilots (not limited to one journal) ranged from 6 to 34%, with feedback in assessment at 17.8%
Reasons for shockingly poor performance
Incomplete coverage of journals No indexed database for medical education Existing controlled vocabularies are
inadequate for medical education
MeSH - Feedback
(Mapped Term)
1. Feedback, Biochemical
• Feedback, Biochemical: A mechanism of communication among life processes to co-ordinate development, reproduction, and homeostasis. In humans, feedback loops are especially important for communication between organs that are spatially separated. Virtually all hormones from the nervous and endocrine systems are under feedback control: by peripheral hormones, cations, metabolites, osmolarity or extracellular fluid volume.
2. Feedback
• Feedback: A mechanism of communication within a system in that the input signal generates an output response which returns to influence the continued activity or productivity of that system.
Feedback, Psychological
• Feedback, Psychological: A mechanism of information stimulus and response that may control subsequent behaviour, cognition, perception, or performance. (From APA Thesaurus of Psychological Index Terms, 8th Ed.)
Knowledge of results (Psychology)
• A principle that learning is facilitated when the learner receives immediate evaluation of learning performance. The concept also hypothesises that learning is facilitated when the learner is promptly informed whether a response is correct, and, if incorrect, of the direction of error.
AND exp education, medical/
• Feedback, biochemical - 0
• Feedback - 385
• Feedback, psychological - 10
• Knowledge of results (psychology) - 9
Results are transferable
• … to medical education searches and searches in general– lexical ambiguity of terms– inappropriate and/or inconsistent indexing– free text sacrifices specificity
Two unsatisfactory options:
• ignore the evidence • base your decisions on biased, irrelevant, or incorrect information
• hand search journals for every question
• 6 journals• 10 years• 30 minutes/issue
= 27 working days
Two (slightly less) unsatisfactory options:
• design extremely broad search strategies
• time intensive sorting through results
• rely on partial evidence from searches
• perhaps the best bet
One barrier to systematic searching:
Mortality
What is to be done?
Essentials for anyLevel of Searching:
1. Query formulation
2. Sources
3. Resources
4. Filters
5. Syntax
Define the search query
Identify and expand
concepts
Set the scope of the query
An Iterative Process:
1. Query formulation
McDonald S, Taylor L, Adams C. Searching the right database: a comparison of four databases for psychiatry journals. Health Libraries Review; 16: 151-6, 1999.
2. Sources
3. Resources
• Dedicated time
• Administrative support
• Professional support
4. Filters
• Know them
• Use them
5. Syntax
• Know it
• Use it
OSCE
• Objective structure clinical examination
• Objective structure clinical exam$
• Objective structure clinical test$
• Multiple station examination…
• Simulated patient examination...
• Standardi#ed patient examination...
• GOSCE, OSLER, etc.
Searching at the Correct Level of Detail
What has been done?• Raised Awareness
– Amongst teachers, researchers, students, deans, other practitioners in med ed
– Amongst information professionals and education technologists
• Published study guide for medical education• Creation of search filters• Completed systematic reviews• METRO
• Medical Education Taxonomy Research Organisation
• May 2002
• UK wide initiative
• LTSN-01 funding
• METRO-1 complete; METRO-2 beckons
• Annette Thain - Clinical Librarian Services
• Clinical teams had different needs for clinical librarian services
• Offered weekly search service after meetings
• SDIs
• Current Awareness Bulletins
• Often very specialised questions to help with unusual /rare presentations or complications
• Want max of 10 papers
• Prefer easy accessible as need to read paper that week before next clinic
• Would check – guidelines– Cochrane– Databases– Core/ respected journals– known authors– large trials– reviews– UK papers
Types of questions:• Often wanted info for the patient, or to answer a question the
patient had posed and the consultant needed the evidence.• Treatment options often effect of a combination• Clinical trials - often heard of trial at a meeting and wanting
any papers published and results to know if it might affect their patient.
• Want patient information on a rare topic usually tried websites of associations if usual sources failed
“Rapid” Systematic Reviews
• Policy Evaluation Unit
• to provide the evidence base for Ministers to make decisions
• team of researchers and information professionals• 85% of evidence retrieved and synthesised in 6-8
weeks• the remaining 15% is added when available
fin