Informatics in Radiology - Stanford University · *Radiology services, as in the rest of healthcare, entail a long chain of steps for each patient (events) ultimately leading to a

Note: This copy is for your personal non-commercial use only. To order presentation-ready copies for distribution to your colleagues or clients, contact us at www.rsna.org/rsnarights.

1511INFORMATICS

Daniel L. Rubin, MD, MS

Quality is becoming a critical issue for radiology. Measuring and im-proving quality is essential not only to ensure optimum effectiveness of care and comply with increasing regulatory requirements, but also to combat current trends leading to commoditization of radiology services. A key challenge to implementing quality improvement pro-grams is to develop methods to collect knowledge related to quality care and to deliver that knowledge to practitioners at the point of care. There are many dimensions to quality in radiology that need to be measured, monitored, and improved, including examination appropri-ateness, procedure protocol, accuracy of interpretation, communica-tion of imaging results, and measuring and monitoring performance improvement in quality, safety, and efficiency. Informatics provides the key technologies that can enable radiologists to measure and improve quality. However, few institutions recognize the opportunities that informatics methods provide to improve safety and quality. The infor-mation technology infrastructure in most hospitals is limited, and they have suboptimal adoption of informatics techniques. Institutions can tackle the challenges of assessing and improving quality in radiology by means of informatics. ©RSNA, 2011 • radiographics.rsna.org

Informatics in RadiologyMeasuring and Improving Quality in Radiology: Meeting the Challenge with Informatics1

Abbreviations: CAD = computer-aided detection, PACS = picture archiving and communication system

RadioGraphics 2011; 31:1511–1527 • Published online 10.1148/rg.316105207 • Content Codes: 1From the Department of Radiology, Stanford University, Richard M. Lucas Center, 1201 Welch Rd, Office P285, Stanford, CA 94305-5488. Pre-sented as an education exhibit at the 2007 RSNA Annual Meeting. Received September 22, 2010; revision requested January 5, 2011, and received June 28; accepted June 30. The author receives research support from GE Healthcare. Address correspondence to the author (e-mail: [email protected]).

©RSNA, 2011

1512 October Special Issue 2011 radiographics.rsna.org

IntroductionRadiology is coming under increasing scrutiny by payers and regulators (1–4), with all parties ques-tioning the value and effectiveness of practitio-ners (5,6). Several important trends are making quality the center of attention for both radiolo-gists and the parties judging them: (a) radiology is becoming more visible and central in health-care delivery, (b) there is an exponential growth in medical imaging, and (c) imaging is increas-ingly performed by nonradiologists or by radiolo-gists at remote locations who may not have access to the same information as local practitioners.

The need for quality assessment and quality assurance in radiology has consequently moved to the forefront, primarily in the critical areas of utilization, medical errors, and patient safety. Hospitals are responding by looking for ways to track quality indicators and deliver vital knowl-edge to physicians to prevent errors and improve measurement and monitoring of practice effi-ciency and patient safety. The processes currently being adopted are labor-intensive and are often implemented by personnel in the role of quality manager who undertake chart reviews, dissemi-nate practice guidelines, and alert practitioners to potential quality issues.

Current approaches to quality assessment and improvement are costly, time-consuming, and incomplete. The tasks required are voluminous and data-intensive, challenges for people but not for machines. Thus, informatics methods could potentially streamline and improve quality initiatives, similar to the impact these methods are having in other electronic patient systems in hospitals. I believe that informatics methods are critical to enable the myriad of quality-related initiatives that hospitals need to undertake in radiology. However, few hospitals and radiol-ogy practices have implemented these methods. While cost may be a factor hindering adoption of informatics technologies, the lack of education is also important: Few radiologists and administra-tors are aware of the potential of informatics to provide the functionality they need.

In this article, I review some of the key aspects of quality in radiology and describe the ways in which informatics methods can enable quality measurement and improvement initiatives. The

learning objectives of this article are (a) to make radiologists aware of the increasing public scrutiny on medical quality and why it is crucial to radiolo-gy; (b) to review informatics methods for measur-ing, monitoring, and helping improve the quality of care; and (c) to show how adopting informatics methods can enable and accelerate quality assess-ment programs. First, I present the motivation for focusing on quality by discussing the pressures and incentives in radiology. I conclude with an overview of the methods and challenges of mea-suring and improving quality in radiology.

Motivation: Pressures and Incentives to Focus on Quality

Why focus on quality now? There are several major incentives for radiologists: (a) increasing public scrutiny on medical errors and demands to improve healthcare quality, (b) impending pay-for-performance and similar initiatives becom-ing part of everyday practice, (c) the threat of radiology becoming a commodity in the era of the Internet and international teleradiology, and (d) the responsibility for quality that is part of the professionalism of being a physician.

Quality has become a hot topic in recent years, sparked by the Institute of Medicine study (7) and flared by recent public outcries in the lay press (8). The entire medical enterprise is being pressured to address quality not only by the pub-lic, but also by payers of healthcare services and by regulatory agencies that insist hospitals and physicians measure and improve healthcare qual-ity. Many aspects of healthcare quality have been identified that are lacking and that need regular measurement and improvement. While much of the discussion about quality focuses on medicine, an increasing literature is also scrutinizing the quality of radiology services (9–11).

Regulatory agencies and payers of healthcare services are taking note of these issues. Medicare recently announced that it will not reimburse for care due to medical errors (12). It is expected that private insurers will soon follow with simi-lar reimbursement restrictions. Many insurers currently track a variety of clinical indicators and adjust payment to physicians accordingly per pay-for-performance and similar measures (13,14). The Deficit Reduction Act of 2005 kept most Medicare reimbursements stable until 2008, but only in exchange for voluntarily

RG • Volume 31 Number 6 Rubin 1513

complying with pay-for-performance reporting measures. Quality measures are becoming part of the regulatory, compliance, and reimbursement framework. Finally, hospital credentialing offices have begun incorporating quality indicators into their physician evaluation programs. In response to these changes in the healthcare environment, radiologists and hospital administrators are being spurred to plan and implement quality measure-ment and improvement procedures.

Beyond the direct pressures on the medical system to improve healthcare quality, there is also a business case for quality in radiology. With the advent of picture archiving and communica-tion systems (PACS), radiology is under threat of becoming a commodity (15,16). There is a growing perception that all radiologists provide an equivalent service globally and that cost is the only factor that need be considered in the mar-ketplace. However, radiologists can differentiate themselves from competitors if they can demon-strate better quality.

Stephen Swensen, past chairman of radiology at Mayo Clinic, makes a compelling argument (17): “Radiology as a commodity will crash and burn in this flat world…. For cents on a dollar, you can have images interpreted in other parts of the planet using teleradiology. Unless we can differentiate our product by quality—meaning quality as a combination of outcomes, safety, and service—then why wouldn’t someone send their images to Bangalore, India, for that dramatic sav-ings in a commodity market? We have to be able to not just say that we’re better; we have to be able to prove it.”

Finally, quality is ultimately the core aspect of the professionalism of medicine. Responsibil-ity for quality is fundamental to the practice of radiology. Ultimately, radiologists are the best equipped to discover the problems limiting the effectiveness of their practice and to guarantee the quality of their services.

Dimensions of Quality in Radiology

What is quality? Hillman et al (18) provide this definition: “Quality is the extent to which the right procedure is done in the right way at the right time, and the correct interpretation is ac-curately and quickly communicated to the patient and referring physician.” This single statement

mentions all the key components of quality: (a) appropriateness of the examination, (b) the procedure protocol, (c) accuracy of interpreta-tion, (d) communication of results, and (e) mea-suring and monitoring performance improvement in quality, safety, and efficiency.

In the definition of quality of Hillman et al (18), appropriateness of the examination is rep-resented by the term the right procedure. There are two aspects: appropriateness of the examination requested by the referring physician and appro-priateness of the examination performed (the im-aging protocol). The latter is discussed in the next paragraph as the procedure protocol; I use the term appropriateness to refer to appropriateness of the imaging procedure requested. Radiologists and referring physicians must be knowledgeable about which imaging procedure is appropriate for each clinical indication.

In the definition of quality, the procedure protocol is represented by the term the right way. Once the correct procedure is requested, the cor-rect protocol for the procedure must be selected and communicated to the technologist who will perform the study.

Accuracy of interpretation is represented by the term the correct interpretation. Once the imag-ing procedure has been performed, the images are reviewed by the radiologist. The radiologist’s task is to accurately perceive and interpret the imaging observations (radiologic diagnosis).

Communication of results is represented by the phrase accurately and quickly communicated. Once the radiologist provides an interpretation and recommendation, those results must be com-municated to the referring physician and the pa-tient in a timely manner, depending on the type of result (ie, critical results vs noncritical results).

Finally, measuring and monitoring perfor-mance improvement in quality, safety, and ef-ficiency is represented by the phrase patient and referring physician. Ultimately, the effectiveness of radiology is judged by the accuracy of radiolo-gist performance, efficient service, and avoidance of unintended patient complications. Radiolo-gists and institutions must measure and monitor indicators of quality, safety, and efficiency in their services to prove that imaging and their interven-tions are of high quality.


Sequence of Events in the Radiology Care Process*

Event Metric

Referring physician orders examination Appropriateness guidelines, intended examination (ordering error)Appointment scheduled Access timesInitial radiology encounter Patient wait time, patient education (preparation)Protocol selection Standardized protocol: best practicePatient examination Environment of care, safety, and comfort; procedural complicationsInterpretation (peer-reviewed credentials) Correct subspecialty interpretation, accuracy, structured report,

report answers clinical questionFinalization errors Timelines, succinctnessCommunication (emergent or important) Referring physician satisfaction, query answered or addressedMeasuring and monitoring performance

improvement in quality, safety, and ef-ficiency

Turnaround time in examinations and reporting, rate of overread-ing agreement, patient complications, patient satisfaction

*Radiology services, as in the rest of healthcare, entail a long chain of steps for each patient (events) ultimately leading to a diagnosis and treatment. Multistep processes can fail at any step, and issues related to quality need to be considered at every step in the radiology care process.

Figure 1. Radiology appropriateness guidelines. A portion of a guideline from the American College of Radiology’s Appropriateness Criteria (19) is shown. Radiology appro-priateness guidelines specify a clinical context (eg, palpable breast mass in a woman young-er than 30 years of age), a list of potential imaging procedures (eg, breast ultrasonography [US], mammography), and the corresponding appropriateness ratings for each modality (on a nine-point rating scale, where 1–3 = usually not appropriate, 4–6 = may be appropriate, and 7–9 = usually appropriate). While guidelines such as this provide a list of references, it is challenging for a referring physician to connect particular recommendations in such guidelines to the evidence supporting them. For example, if the physician noted a palpable breast mass in a 40-year-old patient with a suspicious mammographic result, the physician would not be able to easily see the evidence supporting the recommendation against order-ing magnetic resonance (MR) imaging (appropriateness rating = 3) by looking at this guide-line. NS = not significant, RRL = relative radiation level.


These aspects of quality arise directly out of the sequence of events that make up the health-care process in radiology (Table). Swensen and Johnson (11) describe this sequence of events as the “value map” for radiology because it speci-fies each of the steps that can go wrong during the care of the patient. Consequently, each of these steps should be part of a quality assurance program.

Many of these aspects of quality are cur-rently being addressed in hospitals in a variety of focused quality improvement programs, in which common problems in work flow or care processes are systematically identified and changed to improve practice. However, some components of quality are difficult to address by purely opera-tional or administrative measures because they are information-intensive and challenge human capability. All aspects of quality can be enabled by informatics, as described in the following section.

Enabling Quality through Informatics

Appropriateness of the ExaminationAppropriateness of radiologic imaging examina-tions encompasses (a) measuring the impact of radiologic procedures on patient care in tems of defining guidelines for imaging given particular clinical indications and (b) evidence-based deci-sion support to alter clinical practice.

Defining Guidelines for Imaging.—Guidelines on the appropriateness of imaging (also called practice guidelines) are being created to prompt providers to make the best choices for imaging their patients. The guidelines vary in terms of the organization and presentation of their synthesis of recommendations and the evidence support-ing them. Some guidelines contain specifications of the different clinical contexts for imaging, the possible imaging modalities appropriate to those contexts, and the appropriateness rating for each modality in that context (Fig 1) (19,20).

Guidelines are being developed by a variety of organizations, hospitals, and commercial entities.

A large set of practice guidelines is collected by the National Guideline Clearinghouse (http://www.guideline.gov). Guidelines are being devel-oped from several different resources (alone or in combination): peer-reviewed articles in the literature, professional organizations, expert pan-els and consensuses of expert opinions, and local institutional experience based on mining clinical records.

While guidelines compiled in these ways can be helpful, specializations in practice in individual institutions could require customized guidelines appropriate to particular institutions that are specific to their patient populations. Informat-ics methods can help institutions determine customizations needed for generic examination appropriateness guidelines by means of data min-ing (computerized query and analysis of data in databases) in their electronic information sys-tems to (a) measure utilization; (b) measure and monitor performance improvement in quality, safety, and efficiency; and (c) enable definition of evidence-based radiology in their particular clini-cal settings.

The term data mining refers to analytical queries in electronic databases. The wealth of electronically accessible data collected by hos-pitals in the course of routine clinical care can be mined to discover imaging procedure utiliza-tion and positivity rates to guide development of institution-specific guidelines for imaging examination appropriateness (21). Physician-specific practice patterns can also be disclosed by means of data mining (Fig 2) and the re-sults used to educate physicians about imaging appropriateness (21,22). In addition, if strong patterns in physician ordering are apparent, that could be an indicator of the need for a hospital to consider adapting a generic national guideline to be more representative of its particular clinical population. Data mining can also support track-ing of the communication of critical test results (Fig 2b), another component of quality that is described later.


Evidence-based Decision Support.—Evidence-based decision support is the process by which referring physicians are guided into making better decisions about the appropriate imaging proce-dures for their patients. Evidence-based decision support is currently implemented by disseminat-ing practice guidelines to physicians; however, this has unfortunately been having little impact in practice (23). Incorporating guidelines into daily practice is challenging for two reasons. First, it is difficult to make all referring physicians aware of these guidelines (ie, dissemination of guidelines).

Guidelines are published and distributed as paper or Portable Document Format (PDF) documents and are not readily accessible at the point of care. Even if referring physicians are aware of them, there are many different guidelines according to imaging modality and clinical indication, making it cumbersome for physicians to access them and apply them to individual patients.

The second challenge in adopting guidelines to practice is that referring physicians often ques-tion their applicability in particular patients. They must see the evidence that supports these guide-lines to be convinced that an alteration in the initial course of care makes sense (ie, evidence

Figure 2. Data mining in radiology. Informatics methods can be used to mine electronic patient data in hospital databases to measure indicators of quality and discover patterns useful for defining institution-specific appropriateness. (a) Graph obtained by using the Brigham and Women’s Hospital’s quality improvement system to mine the physician orders during 1 year shows that some physicians are heavy users of MR imaging (eg, the physician represented by the pink bars); patterns in physi-cian ordering are apparent. Similar data mining can enable evidence-based radiology by defining ap-propriateness. (b) Communication of critical test results can also be tracked. Red indicates critical results, blue indicates noncritical results C-Spine = cervical spine, L-Spine = lumbar spine. (Courtesy of Ramin Khorasani, MD, Brigham and Women’s Hospital, Boston, Mass.)


supporting guidelines). Linking examination ap-propriateness to the evidence on which it is based is thus central to evidence-based radiology.

Informatics can help to overcome these challenges by means of computerized evidence-based decision support, methods that provide knowledge to physicians at the point of ordering imaging studies. These systems address the two challenges for evidence-based decision support:

As far as dissemination of guidelines, the guidelines can be encoded in computerized form with the key aspects (clinical indication, imag-ing procedure, and appropriateness) specified (24,25). Coded controlled terms in a representa-tion called an ontology can be particularly useful to provide a human-readable and machine-inter-pretable representation of guideline knowledge (Fig 3). Thus, the guidelines can be incorporated into physician order entry systems and deliver knowledge to referring physicians at the point of

order, eliminating the need for them to be aware of, or refer to, the paper-based criteria (26).

As for evidence supporting guidelines, infor-matics systems can present the evidence sup-porting each of the recommendations. This is critical functionality for guidelines to convince practitioners about the appropriate course of care. Computerized order entry systems can incorporate electronic representations of appro-priateness guidelines, examine the clinical history and indications for an ordered procedure, and alert physicians if that procedure is not appropri-ate (Fig 4) (27,28). These systems thus deliver evidence-based patient-specific appropriateness guidance just-in-time and at the point of order, a paradigm that can improve the adoption and impact of appropriateness guidelines (29).

Figure 3. Computer representation of appropriateness guidelines. A guideline for radiology appropriateness specifies the appropriateness ratings of different potential imaging procedures in different clinical contexts. They are generally created and disseminated in paper form (Fig 1), which is human readable but not computer accessible. The key knowledge in radiology guidelines can be put into computer-accessible format by extracting the key content and representing it as a graph, as shown below. The nodes in the graph correspond to the guideline content: (a) a radiologic subspecialty applicable to the procedure and broad classes of clinical conditions representing the indications for the procedure, (b) a set of clinical variants on each indication, and (c) ap-propriateness ratings. All possible combinations of clinical indications, imaging procedures, and their appropriateness ratings are built by specifying the possible paths in the graph. For example, a kidneys, ure-ters, and bladder study (part of gastrointestinal [GI] imaging) ordered for unexplained dysphagia has an appropriateness rating of 2. Abd = abdominal, CT = computed tomography, CV = cardiovascular, IR = interventional radiology, MSK = musculoskeletal.


Figure 4. Computerized evidence-based radiology. Electronic versions of practice guidelines (Fig 3) can be incorporated into clinical order entry systems, ensuring that referring physicians are aware of the most appropriate radiology procedures to be ordered in the specific con-texts of their patients. (a) Order entry screen from a system that incorporates guidelines for radiology appropriateness. Referring physicians record patient information and indications for requested procedures; the system provides immediate feedback if the radiologic examination is not appropriate given the clinical context. ID = identification, MRN = medical record number, N/A = not applicable. (b) The system can show physicians the evidence supporting the appropri-ateness recommendations, enhancing their acceptance and serving to educate them about ex-amination appropriateness. The system also provides suggestions for alternate imaging exami-nations that are appropriate in the clinical context, which reduces the number of calls to radiol-ogists by referring physicians to determine the type of examination that should be ordered. (Courtesy of Ramin Khorasani, MD, Brigham and Women’s Hospital, Boston, Mass.)

Procedure ProtocolRadiologic imaging modalities are complex, re-quiring that many different parameters be speci-fied to customize the imaging procedure to the patient and clinical indication. These parameters include use of contrast material, section thickness,

acquisition protocols, and a variety of postpro-cessing steps. CT may be the most appropriate examination for imaging a patient (eg, one with a suspected pancreas mass), but if the procedure is not performed with contrast material and thin sections, it provide less information than an exam-ination performed with this protocol. The param-eters specified for performing imaging are referred


to as the imaging protocol. Most hospitals have collections of numerous protocols for each of the major indications for imaging; these protocols can be difficult to manage and communicate effective-ly to the technologists performing the imaging.

Informatics methods can help ensure that patients receive the best protocol for their clinical conditions by means of standardization of pro-tocols and electronic communication of them to the imaging unit. Work is under way to standard-ize imaging protocols. The Uniform Protocols for Imaging in Clinical Trials initiative (30) is a broad coalition of clinical imaging stakeholders (eg, radiologists, oncologists, device manufactur-ers, the pharmaceutical industry) convened by the American College of Radiology to develop universally acceptable image-acquisition proto-cols to improve consistency and enable estab-lishment of a community consensus and greater uniformity in imaging. Controlled terminology such as RadLex (see the section on controlled terminology and structured reporting) can greatly facilitate the clarity of representation and inter-pretation of such protocols by providing standard names for procedures and imaging parameters.

Accuracy of InterpretationRadiologists vary in the accuracy of image inter-pretation (31–34). Variation in interpretation is perhaps the weakest aspect of clinical imaging (10). In fact, it has been asserted that “the num-ber one killer disease in the U.S. is not cancer or heart disease but variability in care” (35). A variety of resources have been developed to provide information to radiologists (36–38), but such general reference sources are not sufficient; tools are needed to help radiologists make the best diagnostic and treatment decisions for their individual patients.

Radiology interpretation comprises three steps: (a) perception of image findings, (b) interpreta-tion of those findings to render a diagnosis, and (c) decisions and recommendations about case management (next tests or treatments). Each of these steps poses pitfalls to accurate image inter-pretation. Informatics methods can support radi-ologists and help them reduce errors during each of these steps: These methods include just-in-time methods to deliver knowledge at the point of care, computer-aided detection (CAD) to assist with perception, and decision support applications to reduce variation in interpretation

Just-in-Time Information Systems.—Just-in-time informatics techniques deliver knowledge needed by radiologists during the routine work flow. The

commonest approach to doing this is to make electronic books available online (39) as well as through the PACS. These resources are generally searchable, permitting radiologists to find infor-mation about particular diseases or radiologic findings.

Radiology-specific resources on the Web in-clude GoldMiner (36) and Yottalook (37). These tools, like electronic texts, provide canonical knowledge—facts that are generally true about all patients in regard to the disease in question. However, such resources do not actually pro-vide radiologists with patient-specific diagnostic or management advice. For such “personalized medicine” tasks, CAD and decision support have a role.

Computer-aided Detection.—CAD is an infor-matics method for improving quality by helping radiologists perceive abnormal imaging obser-vations. In CAD systems, a computer program “reads” the images, detecting particular types of imaging findings that it has been trained to recognize. The central task of these systems is detection of particular imaging findings, such as calcifications, masses, or nodules.

A related task is diagnosis (ie, interpretation of imaging findings) (see the section on decision support). Because CAD systems seek specific types of image findings, the radiologist should not consider these systems a substitute for evaluating the entire image, as there are many other types of image findings that could be present beyond those the CAD system is trained to detect. Fur-thermore, CAD systems may not detect lesions that they are built to recognize.

CAD systems generally display regions of sus-pected abnormality as annotations on the image that the radiologist reviews. The CAD programs are usually trained to be very sensitive (so as not to miss any true-positive lesions on the images). Consequently, there will often be one or more false-positive findings—CAD annotations on the image that the radiologist believes do not repre-sent abnormalities and can be ignored.

Thus, the CAD reading is often regarded as a second opinion. The diagnosis is ultimately made by the radiologist, who takes into account the CAD output. Many studies have shown that such second opinions, whether rendered by a radiolo-gist or a computer, increase the overall accuracy of the radiologist (40,41).


Decision Support.—Radiologists make deci-sions as part of the interpretation process, such as deciding on the most likely disease, additional recommended imaging tests or follow-up, and potential interventions such as biopsy. Informat-ics can play a major role in helping the radiologist make the best decisions by means of decision support, applications that use a model of dis-ease (such as a Bayesian network) (Fig 5) (42) to inform the radiologist about the most likely diseases for their particular patient.

The input is usually a controlled terminology-encoded structured radiology report, which con-veys the key imaging observations in the image as well as clinical information that informs the decision process. The decision support applica-tion uses its model of disease to deduce the best course of action for the practitioner given the in-

put information. For example, a decision support system for mammography was recently created to inform radiologists as to the likely diagnosis and help them decide whether to biopsy suspicious breast lesions (42).

The radiologist records the findings using a structured reporting form, and information in that form is processed by the system to produce its output (in this case, a differential diagnosis ranked by probability of disease) (Fig 5). The system suggests whether the practitioner should biopsy a lesion depending on whether malignant diagnoses exceed a predetermined probability threshold. If the practitioner does biopsy a lesion, another decision support system can help make the decision about whether a benign biopsy result might be due to sampling error (42).

Communication of ResultsOptimal communication of clinical information between the radiologist and referring physician

Figure 5. Example of a decision support application for mammography. Decision support is a type of computer application that helps radiologists make decisions, such as the diagnosis for a particular patient and what other courses of action should be taken (decision making for management). The example application includes a structured reporting form (see also Fig 7) and a Bayesian network that processes the radiologist’s observations and provides a differential diagnosis for the likely diseases in the patient, ranked by the probability of disease. Such information can be used to guide the radiologist in managing the individual case (ie, personalized care). CA = cancer; ca = calcifica-tions; Cy = cyst; DCIS = ductal carcinoma in situ; DCNOS = ductal carcinoma not otherwise specified; FA = fibro-adenoma; FC = fibrocystic change; FF = focal fibrosis; FH = family history; HTTP = Hypertext Transfer Protocol; LN = lymph node; P/A/O = present, absent, or obscured; Pap = papillary cancer; RS = radial scar.


is critical to patient care. This communication is bidirectional. First, the referring clinician must communicate the clinical context: the reason for the examination, the patient history, and the clinical suspicions. Second, the radiologist must communicate the imaging results: what was found and the likely diagnoses. These com-munications are critical to quality to prevent delayed diagnosis, delayed treatment, and harm to patients. Most institutions have policies in place about communicating critical test results by phone, but such manual methods are prone to failure and malpractice cases have arisen from failing to get the message to referring physicians.

There are two informatics methods that can be used to improve communication between refer-ring physicians and radiologists: (a) controlled terminology and structured reporting for record-ing the radiology results and (b) electronic notifi-cation and reminder systems for communicating the radiology results to referring physicians.

Controlled Terminology and Structured Report-ing.—Controlled terminologies are enumerated sets of terms and of attributes specifying the characteristics of each term and are intended to describe a domain explicitly and unambiguously. These terminologies are controlled because a single term represents all the ways of referring to that term. For example, if a terminology contains the term Salmonella meningitis, this term would

always be used to refer to that condition, regardless of other ways in which it could be expressed. Con-trolled terminologies are also referred to as termi-nologies, vocabularies, and sometimes thesauri.

RadLex® (ie, the radiology lexicon) (http:// radlex.org) is a controlled terminology that pro-vides a list of standard terms for radiology report-ing, teaching, and research (43). It is a project of the Radiological Society of North America and a joint effort with professional organizations and standards bodies. By providing standard terms, communication is improved because each term has a definition as well as links to synonyms. Rad-Lex will also be incorporating exemplary images to illustrate its terms to reduce ambiguity and improve the consistency of language in radiology.

Controlled terminologies such as RadLex can improve communication in radiology in two ways. First, they provide a single way of describing radiologic imaging results (by resolving synonyms to a single preferred term) and supply a standard-ized list of abbreviations, acronyms, symbols, diagnoses, and imaging observations (Fig 6). Ac-cordingly, radiologists and physicians can speak the same language. Second, a controlled termi-nology such as RadLex provides a set of standard names for radiologic procedures (which can have many names, causing potential ambiguity for providers), thus enabling accurate ordering.

Figure 6. RadLex, a controlled terminology for radiology, as visualized in a graphical ter-minology browser (BioPortal; http://bioportal.bioontology.org). Controlled terminologies like RadLex contain a list of terms (shown as a hierarchy of terms on the left side of the image). Each term contains attributes such as a unique identifier, a preferred name, comments, and illustrative images. The terms can be visualized as a hierarchical list (left) or a graph (right). The terms and their attributes are usually not accessed directly by the radiologist; instead, they are usually embedded in applications such as structured reporting, voice recognition, and natural language processing.


Structured reporting can also enable better communication of radiologic results (44,45). Structured reporting is an informatics technol-ogy that enables radiologists to completely report the necessary aspects of an imaging study using a template and controlled terminology (Fig 7). The structured report not only improves the clarity of communication but can also feed decision sup-port applications, as described earlier.

Electronic Notification and Reminder Systems.—Timely report turnaround is important to quality in radiology. However, timely report turnaround does not ensure that the results of imaging procedures have been communicated and acted on. Electronic notification and reminder systems track patient events and clinical data and alert physicians if prespecified criteria are met or if particular actions should be taken. In the imag-ing domain, these systems can enable radiologists to categorize their interpretations (eg, routine, important, urgent, stat) and automatically invoke the appropriate notification procedures (eg, dial phone, page, e-mail) for communicating results to referring physicians and patients. These sys-tems can even track the receipt of the message

and send reminders or invoke escalation process-es to ensure that the information is acted on.

Electronic alerts and reminders have been shown to be effective outside of radiology. The Regenstrief Institute (Indianapolis, Ind) created a system that provides automated reminders to physicians for preventive therapies. The system was shown to significantly increase the rate of delivery of such therapies (46). Other informatics systems have been created to identify patients at risk for deep-vein thrombosis (47); these systems markedly reduced the rates of deep-vein throm-bosis and pulmonary embolism. Other systems have detected and corrected errors potentially leading to adverse drug reactions (48).

Electronic alerts and reminders also have potential in radiology, for issues such as contra-indications to contrast agents, allergies, and need for follow-up examinations. At this point, how-ever, few such systems for radiology are available commercially, and institutions interested in such functionality would need to create such systems in-house.

Measuring and Monitoring Performance Improvement in Quality, Safety, and EfficiencyUltimately, the goal of quality improvement is to provide high-quality radiology service effi-

Figure 7. Example of structured reporting, an informatics technology that enables radiologists to use controlled terminology to create radiology reports using intuitive paradigms such as fill-in forms. In this example, the radiolo-gist reports a mammography case by selecting Breast Imaging Reporting and Data System (BI-RADS) descriptors for each lesion reported. CA = cancer.


Figure 8. Example of a quality dashboard display. The display is produced by an online analytical processing tool, which queries a database containing real-time clinical data to produce the desired chart. A variety of indicators are summarized. Deviations from predetermined benchmarks are displayed, making it easy for hospital staff to detect quality issues for which prompt attention is needed. ED = emergency department, NICU = neonatal intensive care unit. (Courtesy of Ramin Khorasani, MD, Brigham and Women’s Hospital, Boston, Mass.)

ciently and avoid unintended harm to patients. Measuring and monitoring the quality, safety, and efficiency of radiology services is a labori-ous procedure; at present, it is mostly performed by hospital staff. Informatics technologies can automate this work by means of automated cap-ture and analysis of quality indicators. In radiol-ogy, quality indicators include measures such as examination appropriateness, availability or waiting time for imaging, examination through-put or performance, communication of results, and safety (ie, wrong patient, procedure, or site) (49,50).

Once an organization decides on the quality indicators important for improvement in quality, safety, and efficiency, informatics methods can help track all these indicators automatically on an ongoing basis and send alerts when devia-tions in the indicators are detected. A common approach to monitoring quality indicators is to create computerized quality dashboards, which are graphical displays of the indicators being measured at multiple points in time (23,51). Such dashboards often include visual alerts to deviations from benchmarks and provide hospital staff with a quick overview of the status of quality in the institution (Fig 8) (49).

A common method of implementing such dashboards is to create a database linked to the hospital patient record system that is populated by the necessary data feeds. Online analytical pro-cessing tools (programs that query and visualize the contents of databases) can be useful for creat-ing charts that summarize particular indicators on an ad hoc basis. Data visualization tools can be applied to produce graphical displays of the quality indicators of the dashboard on an ongo-ing basis (Fig 8), including alerts when there are deviations from predetermined benchmark values.

Business intelligence systems (also called business analytics systems) represent a related informatics technology, in which rules are cre-ated to trigger actions such as automated alerts and reminders based on detection of outliers in quality indicators (52,53). Many hospitals are currently implementing such systems. Business analytics systems can be useful for demonstrating meaningful use of order entry decision support. Such applications permit users to mine trans-actional data dynamically to find information of interest relevant to performance monitoring, tracking, and improvement initiatives (Fig 9).


Overview of Methods and Challenges

Quality in medicine is a hot topic and the focus of attention in many medical specialties outside of radiology. The importance of quality in radiol-ogy has received less attention. However, it is vital for radiologists to understand how quality is-sues affect them and their institutions, since pub-lic and regulatory pressures will continue making quality assessment and improvement critical to measure and improve. Quality should be re-garded not as a burden but as an opportunity for radiology. Quality is an issue that organizations can leverage to remain competitive and valued by distinguishing themselves through measurable benchmarks of quality.

This article provides an overview of informat-ics methods to help radiologists improve their ap-proaches to quality. Some of the technologies are currently available in commercial systems, while others are still being developed with limited de-ployment in academic institutions. With respect to guidelines for imaging, several computer order entry decision support systems are currently available commercially. In addition, many elec-tronic medical record vendors provide a means of implementing rules and scripting languages

that permit implementation of practice guide-lines for imaging, potentially avoiding the need to purchase a dedicated commercial system. Those hospitals with in-house informatics groups have developed their own custom radiology informat-ics solutions (54).

A number of systems to help radiologists with interpretation are available, particularly CAD systems for detecting calcifications and masses in mammography, lung nodules, and colonic polyps. Several products are on the market to deliver knowledge at the point of care, some of which are even integrated into the PACS. Finally, many electronic medical record products incor-porate electronic alerts and reminders, but these focus on medical domains outside of radiology; it would be useful if the vendors extended their current systems to radiology.

Commercial adoption has been slower for decision support systems and for reporting sys-tems that incorporate controlled terminology or use it to improve report quality. In addition, few products are available that target the measure-ment and monitoring of quality, possibly because of the lack of standardized quality metrics in radiology and the great diversity of approaches being adopted at different hospitals. Advances will likely accrue faster as the radiology commu-nity develops standard measures of quality, such as radiation dose.

Figure 9. Example of a display from a business analytics application. The display is produced by a query to the hospital database containing real-time clinical data to display a variety of metrics of perfor-mance. Such displays facilitate rapid review of a range of measures needed to monitor and track quality improvement initiatives. L Spine = lumbar spine. (Courtesy of Ramin Khorasani, MD, Brigham and Women’s Hospital, Boston, Mass.)


Several of the informatics methods discussed in this article are relevant to meaningful use criteria for incentive payments under the recent healthcare reform legislation. For example, order entry decision support and communication of results are specific meaningful use criteria. As meaningful use criteria continue to be developed with greater attention to radiology, additional informatics methods highlighted in this article will likely become relevant to meaningful use compliance.

There are two key challenges hindering the widespread adoption of quality improvement pro-grams in radiology. The first challenge is defin-ing exactly what constitutes quality in radiology. General definitions of quality have been provided (18), but such broad definitions do not provide specific guidance to healthcare organizations on how to measure and improve quality. Radiol-ogy departments need to define for themselves a set of indicators and benchmarks that need to be established as part of a quality improvement program. It would be helpful if standard qual-ity indicators and metrics of performance were defined by professional radiology organizations.

The second challenge hindering adoption of quality improvement in radiology is that imple-menting quality initiatives can be costly and difficult. Assessing quality depends on the collec-tion of massive amounts of information to assess quality indicators (or quality dashboards). It is this information-intensive challenge for which informatics offers direct assistance, since com-puters are much better suited to processing large amounts of information quickly and continuously than are people.

The informatics methods described in this article can meet these challenges. Radiology departments should consider adopting quality improvement measures addressing the elements of quality described earlier: (a) appropriateness of radiologic examinations, (b) correct proce-dure protocols, (c) accuracy of interpretation, (d) communication of results, and (e) measur-ing and monitoring performance improvement in quality, safety, and efficiency. As described earlier, informatics provides key tools that can ensure imaging is appropriate and effective.

Informatics applications span these elements of quality: (a) data mining to measure utilization and assess the utility of imaging to establish crite-ria for appropriateness in radiology; (b) comput-erized evidence-based decision support to enable dissemination and enforcement of appropriate-ness guidelines for radiology; (c) CAD systems to help radiologists avoid missing important imag-ing findings; (d) just-in-time methods to deliver

knowledge to radiologists during image interpre-tation and decision support applications to help them make better diagnoses and management decisions; (e) controlled terminology, standard-ized protocols, and structured reporting to make radiologic imaging protocols and results machine accessible; (f) electronic notification and remind-er systems for communicating radiologic proto-cols and results; and (g) dashboards of quality indicators and business intelligence systems to analyze clinical information, enabling institutions to measure and monitor quality in real time.

Although the informatics methods discussed are promising for quality programs, there could be challenges in adopting these methods. The first challenge is that commercial products vary in terms of their features and ability to deliver the desired functionality. In addition, the perfor-mance of these systems may vary across different institutions or different patient populations. A second challenge is that most informatics ap-proaches require hospitals to integrate patient data that exist in many different proprietary information systems. Although standards such as Health Level Seven (HL7) and Digital Imaging and Communications in Medicine (DICOM) have improved interoperability in terms of mes-saging and storage of data, these standards do not address directly linking data from the same patient across disparate systems—a task required for many of the informatics approaches discussed in this review. As standard terminologies and ontologies emerge, such data integration interop-erability challenges may become more tractable in the future.

A final challenge to adopting informatics meth-ods to enable quality is cost. Informatics systems, programming staff for in-house solutions, and consultation and support require investment, both financial and human resources. However, in the increasingly competitive environment of radiology, such costs could clearly prove to be a valuable investment. Moreover, the alternative to not adopting informatics approaches could prove more costly; hiring quality managers to perform data-intensive tasks that could be automated by machines will likely prove more costly and less ef-ficient than investing in informatics solutions.

ConclusionsQuality in radiology encompasses many dimen-sions, including approropriateness of the proce-dure, the protocol used to perform it, accuracy of the interpretation, communication of the results, and measuring and monitoring performance


improvement in quality, safety, and efficiency. Computer applications to measure and improve quality can be successfully deployed. Informat-ics methods should not be regarded as futuristic developments on the horizon; such applications are already in routine use at many institutions and will likely become more prevalent in the future. Ultimately, as radiologists, quality is not just our goal, it is our responsibility, and deploy-ing informatics methods will help us achieve our objectives.

Acknowledgment.—I wish to acknowledge and thank Ramin Khorasani, MD, for reviewing the manuscript, providing very helpful comments, and contributing figures.

References 1. Steele JR, Schomer DF. Continuous quality im-

provement programs provide new opportunities to drive value innovation initiatives in hospital-based radiology practices. J Am Coll Radiol 2009;6(7): 491–499.

2. Blachar A, Tal S, Mandel A, et al. Preauthoriza-tion of CT and MRI examinations: assessment of a managed care preauthorization program based on the ACR Appropriateness Criteria and the Royal College of Radiology guidelines. J Am Coll Radiol 2006;3(11):851–859.

3. Smulowitz PB, Ngo L, Epstein SK. The effect of a CT and MR preauthorization program on ED uti-lization. Am J Emerg Med 2009;27(3):328–332.

4. Brenner DJ, Hricak H. Radiation exposure from medical imaging: time to regulate? JAMA 2010; 304(2):208–209.

5. Shaw LJ, Iskandrian AE, Hachamovitch R, et al. Evidence-based risk assessment in noninvasive im-aging. J Nucl Med 2001;42(9):1424–1436.

6. Steele JR, Hovsepian DM, Schomer DF. The Joint Commission practice performance evaluation: a primer for radiologists. J Am Coll Radiol 2010; 7(6):425–430.

7. Stefl ME. To err is human: building a safer health system in 1999. Front Health Serv Manage 2001; 18(1):1–2.

8. Pollack A. Who’s reading your x-ray? The New York Times, November 16, 2003.

9. Johnson CD, Swensen SJ, Glenn LW, Hovsepian DM. Quality improvement in radiology: white pa-per report of the 2006 Sun Valley Group meeting. J Am Coll Radiol 2007;4(3):145–147.

10. Robinson PJ. Radiology’s Achilles’ heel: error and variation in the interpretation of the Röntgen im-age. Br J Radiol 1997;70(839):1085–1098.

11. Swensen SJ, Johnson CD. Radiologic quality and safety: mapping value into radiology. J Am Coll Ra-diol 2005;2(12):992–1000.

12. Sack K. Medicare won’t pay for medical errors. The New York Times, September 30, 2008.

13. McVey LR. Pay-for-performance radiology: a new concept. Radiol Manage 1999;21(3):18–21.

14. Moser JW, Wilcox PA, Bjork SS, et al. Pay for per-formance in radiology: ACR white paper. J Am Coll Radiol 2006;3(9):650–664.

15. Hwang NH. You’ve got mail: the concerns of elec-tronically outsourcing radiological services over-seas. J Leg Med 2004;25(4):469–484.

16. Wong WS, Roubal I, Jackson DB, Paik WN, Wong VK. Outsourced teleradiology imaging services: an analysis of discordant interpretation in 124,870 cases. J Am Coll Radiol 2005;2(6):478–484.

17. Valenza T. Making quality the differentiator in a flat world. Imaging Economics, February 2007. http://www.imagingeconomics.com/issues/ar-ticles/2007-02_01.asp. Accessed March 15, 2010.

18. Hillman BJ, Amis ES Jr, Neiman HL; FORUM Participants. The future quality and safety of medi-cal imaging: proceedings of the third annual ACR FORUM. J Am Coll Radiol 2004;1(1):33–39.

19. American College of Radiology. ACR Appropriate-ness Criteria. http://www.acr.org/secondarymain-menucategories/quality_safety/app_criteria.aspx. Accessed March 25, 2010.

20. Chou R, Qaseem A, Snow V, et al. Diagnosis and treatment of low back pain: a joint clinical practice guideline from the American College of Physicians and the American Pain Society. Ann Intern Med 2007;147(7):478–491.

21. Reiner B. Uncovering and improving upon the in-herent deficiencies of radiology reporting through data mining. J Digit Imaging 2010;23(2):109–118.

22. Prevedello LM, Andriole KP, Hanson R, Kelly P, Khorasani R. Business intelligence tools for radiol-ogy: creating a prototype model using open-source tools. J Digit Imaging 2010;23(2):133–141.

23. Bautista AB, Burgos A, Nickel BJ, et al. Do clini-cians use the American College of Radiology ap-propriateness criteria in the management of their patients? AJR Am J Roentgenol 2009;192(6):1581–1585.

24. Tjahjono D, Kahn CE Jr. Promoting the online use of radiology appropriateness criteria. RadioGraph-ics 1999;19(6):1673–1681.

25. Sistrom CL; American College of Radiology. In support of the ACR Appropriateness Criteria. J Am Coll Radiol 2008;5(5):630–635; discussion 636– 637.

26. Morin RL, Rosenthal DI, Stout MB. Radiology or-der entry: features and performance requirements. J Am Coll Radiol 2006;3(7):554–557.

27. Khorasani R. Clinical decision support in radiol-ogy: what is it, why do we need it, and what key features make it effective? J Am Coll Radiol 2006; 3(2):142–143.


28. Rosenthal DI, Weilburg JB, Schultz T, et al. Radiol-ogy order entry with decision support: initial clini-cal experience. J Am Coll Radiol 2006;3(10):799–806.

29. Sistrom CL, Dang PA, Weilburg JB, Dreyer KJ, Rosenthal DI, Thrall JH. Effect of computer-ized order entry with integrated decision support on the growth of outpatient procedure volumes: seven-year time series analysis. Radiology 2009; 251(1):147–155.

30. CTSA Clinical Trials Service Awards. Imaging in Clinical Trials/UPICT Group. http://ctsa-imaging.org/upict.cfm. Accessed June 25, 2011.

31. Armato SG 3rd, McNitt-Gray MF, Reeves AP, et al. The Lung Image Database Consortium (LIDC): an evaluation of radiologist variability in the identi-fication of lung nodules on CT scans. Acad Radiol 2007;14(11):1409–1421.

32. Barlow WE, Chi C, Carney PA, et al. Accuracy of screening mammography interpretation by char-acteristics of radiologists. J Natl Cancer Inst 2004; 96(24):1840–1850.

33. Hillman BJ, Hessel SJ, Swensson RG, Herman PG. Improving diagnostic accuracy: a comparison of interactive and Delphi consultations. Invest Radiol 1977;12(2):112–115.

34. Halladay JR, Yankaskas BC, Bowling JM, Alexan-der C. Positive predictive value of mammography: comparison of interpretations of screening and di-agnostic images by the same radiologist and by dif-ferent radiologists. AJR Am J Roentgenol 2010;195 (3):782–785.

35. Ward P. Fix your systems to cut error rates. Di-agnosticImaging.com. http://www.dimag.com/ecr2007/showArticle.jhtml;jsessionid=VIANMV0XXWCLIQSNDLQSKHSCJUNN2JVN?articleID=198000214. Published March 12, 2007.

36. Kahn CE Jr, Thao C. GoldMiner: a radiology im-age search engine. AJR Am J Roentgenol 2007;188 (6):1475–1478.

37. Sharpe RE Jr, Sharpe M, Siegel E, Siddiqui K. Uti-lization of a radiology-centric search engine. J Digit Imaging 2010;23(2):211–216.

38. Kitchin DR, Applegate KE. Learning radiology: a survey investigating radiology resident use of textbooks, journals, and the Internet. Acad Radiol 2007;14(9):1113–1120.

39. Chew FS, Llewellyn K, Olsen KM. Electronic pub-lishing in radiology: beginnings, current status, and expanding horizons. J Am Coll Radiol 2004;1(10): 741–748.

40. Doi K. Computer-aided diagnosis in medical imag-ing: historical review, current status and future po-tential. Comput Med Imaging Graph 2007;31(4-5):198–211.

41. Yoshida H, Näppi J. CAD in CT colonography without and with oral contrast agents: progress and challenges. Comput Med Imaging Graph 2007;31 (4-5):267–284.

42. Burnside ES, Rubin DL, Shachter RD, Sohlich RE, Sickles EA. A probabilistic expert system that provides automated mammographic-histologic cor-relation: initial experience. AJR Am J Roentgenol 2004;182(2):481–488.

43. Langlotz CP. RadLex: a new method for indexing online educational materials. RadioGraphics 2006; 26(6):1595–1597.

44. Kahn CE Jr, Langlotz CP, Burnside ES, et al. To-ward best practices in radiology reporting. Radiol-ogy 2009;252(3):852–856.

45. Reiner BI. The challenges, opportunities, and im-perative of structured reporting in medical imag-ing. J Digit Imaging 2009;22(6):562–568.

46. Dexter PR, Perkins S, Overhage JM, Maharry K, Kohler RB, McDonald CJ. A computerized re-minder system to increase the use of preventive care for hospitalized patients. N Engl J Med 2001; 345(13):965–970.

47. Kucher N, Koo S, Quiroz R, et al. Electronic alerts to prevent venous thromboembolism among hospi-talized patients. N Engl J Med 2005;352(10):969–977.

48. Raschke RA, Gollihare B, Wunderlich TA, et al. A computer alert system to prevent injury from ad-verse drug events: development and evaluation in a community teaching hospital. JAMA 1998;280 (15):1317–1320.

49. Khorasani R. Setting up a dashboard for your prac-tice. J Am Coll Radiol 2008;5(4):600.

50. Olsha-Yehiav M, Einbinder JS, Jung E, et al. Qual-ity dashboards: technical and architectural consid-erations of an actionable reporting tool for popula-tion management. AMIA Annu Symp Proc 2006: 1052.

51. Morgan MB, Branstetter BF 4th, Lionetti DM, Richardson JS, Chang PJ. The radiology digital dashboard: effects on report turnaround time. J Digit Imaging 2008;21(1):50–58.

52. Bellazzi R, Arcelloni M, Ferrari P, et al. Manage-ment of patients with diabetes through information technology: tools for monitoring and control of the patients’ metabolic behavior. Diabetes Technol Ther 2004;6(5):567–578.

53. Noirot LA, Blickensderfer A, Christensen E, et al. Implementation of an automated guideline moni-tor for secondary prevention of acute myocardial infarction. Proc AMIA Symp 2002:562–566.

54. Bui AAT, Taira RK. Medical imaging informatics. New York, NY: Springer, 2010.

Teaching Points October Special Issue 2011

Informatics in Radiology Measuring and Improving Quality in Radiology: Meeting the Challenge with InformaticsDaniel L. Rubin, MD, MS

RadioGraphics 2011; 31:1511–1527 • Published online 10.1148/rg.316105207 • Content Codes:

Page 1513“Quality is the extent to which the right procedure is done in the right way at the right time, and the cor-rect interpretation is accurately and quickly communicated to the patient and referring physician.”

Page 1517Informatics can help to overcome these challenges by means of computerized evidence-based decision support, methods that provide knowledge to physicians at the point of ordering imaging studies.

Page 1519Radiology interpretation comprises three steps: (a) perception of image findings, (b) interpretation of those findings to render a diagnosis, and (c) decisions and recommendations about case management (next tests or treatments). Each of these steps poses pitfalls to accurate image interpretation. Informatics methods can support radiologists and help them reduce errors during each of these steps:

Page 1521There are two informatics methods that can be used to improve communication between referring physi-cians and radiologists: (a) controlled terminology and structured reporting for recording the radiology results and (b) electronic notification and reminder systems for communicating the radiology results to referring physicians.

Page 1523A common method of implementing such dashboards is to create a database linked to the hospital patient record system that is populated by the necessary data feeds. Online analytical processing tools (programs that query and visualize the contents of databases) can be useful for creating charts that summarize par-ticular indicators on an ad hoc basis.

Documents

Informatics in Radiology - Stanford University · *Radiology services, as in the rest of healthcare, entail a long chain of steps for each patient (events) ultimately leading to a