Optimising the Utility of the NeLH VBL for Musculoskeletal Diseases—Technical Considerations

MN Kamel BoulosConcepts from HealthCyberMap—<http://healthcybermap.semanticweb.org>

Optimising the Utility of the NeLH VBL for Musculoskeletal Diseases—Technical

Considerations

Dr. Maged N. Kamel BoulosCentre for Measurement and Information in Medicine,

City University, LondonE-mail: [email protected]

27 November 2002


Introduction

• A major goal for any online clinical information service should be to maximise return on investment (human and other resources, time and money used to develop and maintain the service).

• Return on investment depends on (target) users’ perceived usefulness (or utility) of the service. A highly useful service is a successful service.


Suggested Recipe for Success

• Maximising utility by embedding contextual knowledge and clinical practice guidelines directly into users’ workflow, e.g., patient records (direct problem-to-knowledge linking).

• Maximising utility by maintaining high quality/ currency of VBL content.

• Maximising utility (usefulness) through usability (user-service interface issues).

• Maximising utility by ensuring high relevance of VBL content returned to users in response to their queries (i.e., minimal noise and silence).


Usability - 1Goal: A Powerful Easy-to-use Service

• Usability: The ease with which a user can learn to operate, prepare inputs for, and interpret outputs of a service or component of it (IEEE).

• Usability: The effectiveness, efficiency, and satisfaction with which specified users achieve specified goals in particular environments (ISO).

• Low service usability might mask useful service features resulting in decreased users’ perceived usefulness (or utility) of the service.


Usability - 2Some Design Guidelines

• Clean, elegant and consistent screens that avoid detail overload.

• Easy-to-learn/ easy-to-use functions and user-service interfaces that apply the famous principle of “recognition not recall” (Nielsen, 1994).

• Prevent user errors and provide error correction/ tolerance (e.g., spelling mistakes/ variants in user-typed query strings).


Error Correction/ Tolerance, e.g., ostoarthritisosteoarthritis

Usability - 3


Clean, Elegant Consistent Screens that are Easy to Understand and Easy to Use

Usability - 4



Usability - 5



Usability - 6



Usability - 7


Recognition Not Recall: A Comparison of Two Graphical (Visual) InterfacesScreenshot of parts of HealthCyberMap and Visual Net navigational maps for “heart diseases”. Notice the difference in map iconicity between HealthCyberMap and Visual Net approaches, and the map clutter resulting from Visual Net’s way of representing each resource directly on the map using a distinct point symbol.

Usability - 8


Relevance - 1Locating the Needle in the Haystack

• The concept of “relevance” is a fundamental concept of information retrieval.

• The design and evaluation of search engines/ information retrieval services should be based on relevance metrics:– recall (silence); and– precision (noise).


Relevance - 2• The measurement of precision requires a qualified

individual, or group of individuals, to inspect the output from a search and to sort the output into two groups of resources—relevant and not relevant (precision = number of relevant resources in query result/total number of resources, relevant + irrelevant, in query result).

• The measurement of recall requires that the individual or group of individuals also have access to the complete set of resources that was searched (recall = number of relevant resources in query result/number of relevant resources in the queried resource pool).


Relevance - 3Problems with Free-text, Word-/ Phrase-based Search Engines • The sought page might be using a different term

(synonym) that points to the same concept. “Myocardial infarction” and “coronary thrombosis” cannot be matched, although they are the same.

• Spelling mistakes and variants are considered as different terms. For example, “psoriasis” (correct spelling) and “psoriaisis” (typographical error) cannot be matched. Similarly, “anaemia” (correct UK spelling) and “anemia” (correct US spelling) cannot be matched.


Relevance - 4Problems with Free-text, Word-/ Phrase-based Search Engines • Search engines cannot process clinical documents intelligently and

are unaware of the actual context and content meaning of different Web resources. For example, searching for resources on “psoriasis” will retrieve all the documents containing this word, but many of these resources might not be relevant (“psoriasis” was just mentioned by the way in these documents, e.g., under a “See also” heading, and is not their actual topic).

• If you are searching for a particular finding using plain free text search, you might also pick documents where the clinician/ author explicitly negates the presence of the finding (e.g., “no dysuria” will be picked if you are looking for “dysuria”).

• Non-textual resources (images, audio, video) cannot be properly indexed and retrieved.


Relevance - 5A Proposed Solution

Metadata PLUS “intelligent” inference/ reasoning with both metadata and user queries.


Relevance - 6Metadata—Always Read the Label!

• Metadata are information about information (e.g., author, subject, type, format, etc. of NeLH resources/ documents). It can be compared to the labels you put on your baggage (for unambiguous identification).

• Besides collecting metadata describing information resources, two other types of metadata should be gathered:– user profiles (information about user, including user’s location profile which

directly affects user’s health/ health information needs—e.g., location-specific disease rates, guidelines and healthcare services); and

– device descriptions (information about the connection/ device/ browser that the user is using to access the service).

• An ideal service should be able to reason with all three types of metadata to personalise and optimise a Web user’s experience (serve suitable content in a suitable form and format).


Relevance - 7Resource Metadata—Problems and Limitations

• Metadata can greatly enhance information retrieval, but this depends on the quality of the metadata we are using and on our resource indexing granularity [do we treat big sites, collections and lengthy documents (e.g., a clinical guideline) as a single resource or index individual pages, smaller collections and subsections from these sites and documents that cover individual (specific) topics as distinct resources].

• Using keywords in resource metadata to describe the content of a resource is not the optimal solution. The user might not know which keywords or terms were used to index the resource and can thus miss a relevant resource when performing a search.



• A further improvement would be to use a thesaurus to care for more synonyms or force the user to select keywords from a predefined collection of terms we are using in our metadata. But, thesauri also have their own shortcomings. They do not allow users to ask questions like “Get pages describing the complications of diabetes mellitus” and retrieve relevant pages, say on “peripheral neuropathy”. This is due to the fact that a thesaurus cannot know the semantic relationship between “diabetes mellitus” and “peripheral neuropathy” (“peripheral neuropathy” is not a synonym or variant of “diabetes mellitus”).



• A coding scheme (controlled vocabulary or concept- and knowledge-based terminology) would take thesauri one step further by offering a collection of terms along with a hierarchy that tells us which terms are kinds of one another (and other relationships). We can use the hierarchy to ask more general questions. Queries for “endocrine disorders” can now pick up “diabetes mellitus” and “Grave’s disease”. Unfortunately, (older/ first-generation) coding schemes still have their drawbacks.



• A term has to be introduced for anything that we want to represent (enumerative approach). This can lead to an unmanageable number of terms (and many concepts will still be missing at the end, irrespective of how hard we try).

• Adding compositional (concepts can be constructed from primitive building blocks, governed by validation rules) and multiple parentage capabilities to coding schemes can solve this. For example, “surgical operation” acts as parent for “transplant operation” and “kidney operation”, which are both in turn parents of “kidney transplant”. A search for either “transplant operation” or “kidney operation” would find “kidney transplant”, thanks to multiple parentage.

• This is the approach adopted in SNOMED-CT (Systematised Nomenclature of Medicine–Clinical Terms, released in February 2002—<http://www.snomed.org>) for example.


Relevance - 11Resource Metadata—”Intelligent” Inference/ Reasoning

• It’s not practical or computationally efficient to encode (using concepts from a suitable terminology) all semantic relationships and other possibilities of related topics in an information resource or a metadata record of it (especially given the fact that resource indexing is still a largely manual process).

• The ideal system should be able, given just the main (most specific) concept code(s) of a resource topic, to automatically infer all allowed textual synonyms/ descriptions (even in multiple languages) for this topic, as well as the codes of any other relevant topics related to this resource via semantic relationships.


Relevance - 12Resource Metadata—”Intelligent” Inference/ Reasoning

• A terminology server (a terminology inference engine shared by many applications like NeLH, EPR, etc.) allows, given a terminology concept, the retrieval of synonyms and related broader/ narrower concepts (parent, cousin, uncle, sibling and child concepts) from the underlying clinical terminology.

• Ideally, a terminology server should support concept mapping, which involves processing free text queries to identify corresponding terms from a controlled vocabulary. This relieves users from any restrictions while ensuring accurate results and can also support spelling variants and, if necessary, multiple languages (SNOMED-CT concepts are language neutral and so can serve different natural languages).


HealthCyberMap (HCM) Proposed Architecture: Explicit concepts in resource metadata map onto a domain ontology (a clinical terminology) allowing a semantic search engine to infer implicit meanings (synonyms and semantic relationships) not directly mentioned in either the resource or its metadata. Similarly, user queries would map to the same ontology allowing the search engine to infer the implicit semantics of user queries and use them to optimise retrieval.

Relevance -13


Problem-to-Knowledge Linking - 1The library goes to clinicians instead of clinicians going to the library

• Problem-to-knowledge linking aims at providing contextually appropriate medical knowledge in the right place and at the right time by embedding contextual knowledge and clinical practice guidelines directly into users’ workflow, e.g., at the point of care in electronic patient records (EPR).


Problem-to-Knowledge Linking - 2The library goes to clinicians instead of clinicians going to the library

• Clinical codes provide a common backbone language (ontology) for proper communication between the EPR and online health/ clinical information services like NeLH.

• The actual success of problem-to-knowledge linking will depend on the quality and granularity of the metadata it uses, the topical coverage and quality of resources it points to, and the use of a suitable concept qualifier mechanism to maximise contextual relevance (by better defining a topic, narrowing retrieval, or expressing a certain aspect of a main concept).

NeLH

EPR

SSNNOOMMEEDD


Problem-to-Knowledge Linking - 3Beyond static knowledge: Self-executing guidelines that work for you!

• Computer-interpretable guidelines (CIG) aims at delivering patient-specific recommendations that are integrated with electronic patient records at point of care, i.e., integrated into workflow.

• CIG are used to generate automated reminders/ alerts; in decision support and task management; to perform retrospective analysis to test if patients were treated appropriately; to check order entry appropriateness, referral criteria; for background monitoring, execution of care plans and quality review.

• Individual coded patient data from the electronic patient record are matched to coded guideline terms and flowchart; the recommendations in guidelines are matched to actions in an order entry system or for prescription printing.

• The ultimate goal is to fully apply guidelines to clinical practice, and continually evaluate their application and modify/ refine guidelines accordingly (protocol-guided care).

Knowledge Guidelines Workflow (Practice)


Quality and Currency - 1• Health and medical Web resources are not all written by qualified,

unbiased professionals, hence the need for rigorous quality benchmarking.

• Currency (up-to-dateness) is one aspect of information quality.

• A quality metadata element should be introduced to store a resource’s level of evidence (whether it is an official guideline, systematic review, randomised controlled trial—RCT, other peer-reviewed study, official critically appraised topic—CAT, or expert opinion), and any other relevant information regarding its compliance to a recognised code of ethics (e.g., Health On the Net—<http://www.hon.ch/>) or quality seal, and whether it has been published by a trusted publisher or listed in trusted directory.


Quality and Currency - 2• We need to define a consistent way for storing this quality

information and all its facets to ensure reliable processing of it later on, especially if NeLH metadata is to be made available for use by other external services as well.

• Emerging quality “standards” like HIDDEL (Health Information Disclosure, Description and Evaluation Language—<http://www.medcertain.org/english/about_us/overview.htm>) and European Commission Guidelines (eEurope 2002/ eHealth Quality Criteria for Health Related Websites <http://europa.eu.int/information_society/eeurope/ehealth/quality/draft_guidelines/index_en.htm>) must be also taken into consideration.


Quality and Currency - 3A Plethora of Kitemarks

• New: MedCIRCLE—Collaboration for Internet Rating, Certification, Labeling and Evaluation of Health Information (<http://www.medcircle.org/>). It is the follow-up project to

MedCERTAIN and implements the HIDDEL vocabulary.

• Recommended review of the different quality benchmarking tools and checklists for medical/ health Web resources: Kamel Boulos MN, Roudsari AV, Gordon C, Muir Gray JA. The Use of Quality Benchmarking in Assessing Web Resources for the Dermatology Virtual Branch Library of the National electronic Library for Health (NeLH). J Med Internet Res. 2001;3(1):e5. Available from: <http://www.jmir.org/2001/1/e5/>.


Quality and Currency - 4Managing Web Resources for Persistent Access

• The success of a distributed information system such as the Web for research depends on the long-term consistency of the inter-links between online resources and the persistence of the links that are provided in the catalogues, indexes and listings of resource discovery services.

• The National Library of Australia has recently published a document titled “Managing Web Resources for Persistent Access” on how to overcome the deadly broken link message “HTTP 404 Not Found”.

National Library of Australia. Managing Web Resources for Persistent Access. Available from: <http://www.nla.gov.au/guidelines/2000/persistence.html> and <http://www.nla.gov.au/guidelines/2000/webresources.html>.


Quality and Currency - 5Managing Web Resources for Persistent Access

• What NeLH can do is to regularly run a link checker on its database of resource addresses to identify links that are no more functional (and possibly exclude/ delete them). Replacing these links with more current, functional versions still requires manual intervention.

• NeLH should also perform regular quality checks on listed resources (in case they have changed or the information they contain is no more valid). This is another manual task for humans to complete, as it depends on their discernment capabilities.

• Some of the National Library of Australia guidelines also apply to NeLH as a service Web site, so that links to it from other sites around the Web and within it between its different components remain valid.


Conclusion and Recommendations• Learn who are your target users and what are their needs. Evaluate

your service continuously.(Extremely useful: NIH Web Site Evaluation and Performance Measures Toolkit—<http://irm.cit.nih.gov/itmra/weptest/acknow.htm>).

• Collaborate with relevant stakeholders, e.g., other NeLH VBLs, EPR and Standards Bodies.

• A highly useful service is a successful service. The following four ingredients are key to high service usefulness:– usability;

– relevance;

– integrability into work practices; and

– quality and maintainability.

Documents

Optimising the Utility of the NeLH VBL for Musculoskeletal Diseases—Technical Considerations