Predicting Cancer Prognosis Using Interactive Online Tools ......Review Predicting Cancer Prognosis Using Interactive Online Tools: A Systematic Review and Implications for Cancer

2013;22:1645-1656. Published OnlineFirst August 16, 2013.Cancer Epidemiol Biomarkers Prev Borsika A. Rabin, Bridget Gaglio, Tristan Sanders, et al. Systematic Review and Implications for Cancer Care ProvidersPredicting Cancer Prognosis Using Interactive Online Tools: A

Updated version

10.1158/1055-9965.EPI-13-0513doi:

Access the most recent version of this article at:

Material

Supplementary

http://cebp.aacrjournals.org/content/suppl/2013/11/08/1055-9965.EPI-13-0513.DC1.html

Access the most recent supplemental material at:

Cited Articles

http://cebp.aacrjournals.org/content/22/10/1645.full.html#ref-list-1

This article cites by 29 articles, 9 of which you can access for free at:

E-mail alerts related to this article or journal.Sign up to receive free email-alerts

Subscriptions

Reprints and

[email protected]

To order reprints of this article or to subscribe to the journal, contact the AACR Publications Department

Permissions

[email protected]

To request permission to re-use all or part of this article, contact the AACR Publications Department at

on January 21, 2014. © 2013 American Association for Cancer Research. cebp.aacrjournals.org Downloaded from

Published OnlineFirst August 16, 2013; DOI: 10.1158/1055-9965.EPI-13-0513



http://cebp.aacrjournals.org/lookup/doi/10.1158/1055-9965.EPI-13-0513

http://cebp.aacrjournals.org/lookup/doi/10.1158/1055-9965.EPI-13-0513





http://cebp.aacrjournals.org/cgi/alerts

http://cebp.aacrjournals.org/cgi/alerts

mailto:[email protected]




http://cebp.aacrjournals.org/




Review

Predicting Cancer Prognosis Using Interactive Online Tools:A Systematic Review and Implications for Cancer CareProviders

Borsika A. Rabin1, Bridget Gaglio4, Tristan Sanders2, Larissa Nekhlyudov6, James W. Dearing1,Sheana Bull3, Russell E. Glasgow5, and Alfred Marcus3

AbstractCancer prognosis is of keen interest for patientswith cancer, their caregivers, andproviders. Prognostic tools

have beendeveloped to guide patient–physician communication anddecision-making. Given the proliferation

of prognostic tools, it is timely to review existing online cancer prognostic tools and discuss implications for

their use in clinical settings. Using a systematic approach, we searched the Internet, Medline, and consulted

with experts to identify existing online prognostic tools. Each was reviewed for content and format. Twenty-

two prognostic tools addressing 89 different cancers were identified. Tools primarily focused on prostate

(n¼ 11), colorectal (n¼ 10), breast (n ¼ 8), and melanoma (n¼ 6), although at least one tool was identified for

most malignancies. The input variables for the tools included cancer characteristics (n ¼ 22), patient

characteristics (n ¼ 18), and comorbidities (n ¼ 9). Effect of therapy on prognosis was included in 15 tools.

Themost common predicted outcomewas cancer-specific survival/mortality (n¼ 17). Only a few tools (n¼ 4)

suggested patients as potential target users. A comprehensive repository of online prognostic toolswas created

to understand the state-of-the-art in prognostic tool availability and characteristics. Use of these tools may

support communication and understanding about cancer prognosis. Dissemination, testing, refinement of

existing, anddevelopmentof new tools underdifferent conditions areneeded.CancerEpidemiol Biomarkers Prev;

22(10); 1645–56. �2013 AACR.

IntroductionPatient–provider communication in oncology often

involves conveying a large amount of highly complexinformation (1). Cancer prognosis is one of the leadingtopics of interest for patients with cancer, their caregiversand providers, and is relevant at all stages of the cancercontinuum (2–5). Prognostic information can supportdecision-making and communication around therapeuticand palliative treatment decisions, management ofcomorbid conditions, palliative care, and decisionsregarding prioritization of management of other chronicconditions andother life events (2, 6, 7). Information about

prognosis can also address psychosocial needs of patientswith cancer and their caregivers, including uncertaintyand empowering the patient and family to participate inthe decision-making process and help patients under-stand how behavior changes may impact prognosis(2, 6, 8, 9).

Many compelling challenges arise when discussingprognosis with patients with cancer. Among these areuncertainties regarding the integrity of the data uponwhich the prognosis is based, individual variation inprognosis and response, which resources or tools to useto estimate prognosis, the timing of when to discussprognosis given a patient’s informational, cognitive, andsocial support needs, how best to frame and presentprognostic probabilities so that the patient interpretsprobabilities correctly, given health literacy and numer-acy challenges, and the time required to convey prognos-tic information to patients. Clear and understandablecommunication by a provider also requires communica-tion skills and the willingness of the health care providerto engage in this complex information exchange withpatients, many of whom are facing the most profoundcrisis of their lives (2, 10–12). For providers, the commu-nication of prognosis about cancer is complex and emo-tional and can involve the use of qualitative statements(e.g., good chances or not so great chances of survival),comparison with peers (more or less likely to die from

Authors' Affiliations: 1Cancer Research Network Cancer CommunicationResearch Center, Institute for Health Research; 2Kaiser Permanente Color-ado; and 3University of Colorado, Denver, Colorado; 4Mid-Atlantic Perma-nente Research Institute, Kaiser Permanente; 5National Cancer Institute,Rockville, Maryland; and 6Department of Population Medicine, HarvardMedical School and Department of Medicine, Harvard Vanguard MedicalAssociates, Boston, Massachusetts

Note: Supplementary data for this article are available at Cancer Epide-miology, Biomarkers & Prevention Online (http://cebp.aacrjournals.org/).

Corresponding Author: Borsika A. Rabin, Kaiser Permanente Colorado,Legacy Highlands Building, 10065 E. Harvard Ave., Suite 300, Denver, CO80231. Phone: 303-614-1295; Fax: 303-614-1285; E-mail:[email protected]

doi: 10.1158/1055-9965.EPI-13-0513

�2013 American Association for Cancer Research.

CancerEpidemiology,

Biomarkers& Prevention

www.aacrjournals.org 1645





cancer than peers), and actual numbers (e.g., percentages,fractions, or relative or absolute risk probabilities over agiven period of time).

Providers can use a number of resources tomake cancerprognostic estimations, including their own experienceand intuition, clinical data from scientific publications(especially clinical trials), guidelines such as those pub-lishedby theNationalComprehensiveCancerNetwork aswell as textbooks, look-up tables, andnomograms (13, 14).Literature suggests that nomograms that use prognosticalgorithms integrating several predictors improve prog-nostic accuracy (15, 16). More recently, there has beenincreasing interest in the availability anduseofweb-basedtools by both health care providers and patients forobtaining information about health promotion and dis-easemanagement (17–20). This is especially evident in theemergence of web-based prognostic tools for cancer. Can-cer prognostic tools are designed to support decision-making including treatment decisions through providingcancer outcome predications that take diverse patientcharacteristics into account. These web-based tools, themajority of which are also available to patients, have notbeen systematically described or reviewed, nor their usein real world settings often described. Furthermore, clearcriteria for the development of an evaluation of decisionaids only exist for patient-facing tools. No guidelines havebeen developed to assess the quality of provider-facingdecision aids (21). The goals of this study were to reviewcancer prognostic tools available online, describe theircontent and format, and consider the implications for useof these tools for different purposes by cancer careproviders.

Materials and MethodsInteractive cancer prognostic tools were identified

using three approaches. First, the Internet search engineGoogle was used with a combination of search termsdescribing cancer (i.e., cancer, leukemia, lymphoma, car-cinoma, malignancy, hematologic malignancy, and mel-anoma), prognosis (i.e., prognosis, survival, predictive,and prediction), and tool (i.e., tool, calculator). For eachsearch, we reviewed the first 10 pages of results forrelevant tools (100 per search term for a total of 5,600search results). Second, we searched Medline for peer-reviewed publications from 1996 through July 2011 usingthe same search terms as in the web search. Titles andabstracts and when necessary, full text articles werereviewed to identify additional prognostic tools. Third,we sought input from cancer specialists regarding anyexisting or emerging prognostic tools that they might beaware of or use in their own practice.

Tools identified through one of the three searchapproaches were then assessed for eligibility. To be eli-gible, tools had to have an English version, a focus oncancer, have an interactive component (i.e., data or infor-mation entered by the clinician or patient is manipulatedvia an algorithm that draws on a dataset to produceprognostic estimates), and provide output measures. The

following output measures were part of the inclusioncriteria: (i) cancer- or noncancer-specific mortality/over-all survival; (ii) disease-free survival (DFS) or progres-sion-free survival (PFS); (iii) clinical response to treat-ment; or (iv) cancer therapy-induced side effects. Toolsfocusing solely on prevention and the risk of developingcancer were not included in this review.

Some of the tools were not publicly available. To accessthese tools we contacted the developers of each tool viaemail and/or telephone. We made a minimum of threeattempts to gain access via thedevelopers. Tools forwhichwe were unable to obtain access were excluded from thisanalysis.

We developed and refined an abstraction protocolbased on existing literature on patient-centered com-munication, decision aids, usability testing and inputfrom providers, and cancer communication and eHealthresearchers (6, 22). The abstraction protocol was thenpilot tested by three abstractors (B.A. Rabin, T. Sanders,and B. Gaglio) and further refined. Domains for theabstraction included (i) general characteristics and pur-pose of the calculator i.e., goal, cancer sites (e.g., breast,prostate, etc.), cancer stage, and patient population bestserved by the tool, intended users (including patientuser), and disclaimer for users; (ii) data (i.e., algorithm,validation, data source, and publications); (iii) inputfactors (i.e., patient demographic and genetic character-istics, health status/comorbidities, modifiable risks,impact of therapy, and adjuvant treatment); (iv) outputmeasures (i.e., prognostic measures, side effects, qualityof life, recurrence, and spread); and (v) website speci-fications (i.e., access, funding source, and developer).Additional domains of presentation of results, usability,and patient centeredness were also abstracted and willbe summarized in a separate report. The completeabstraction protocol is available upon request from thefirst author.

All three abstractors coded the same six calculators(emerging from two tools) separately. After review of theinitial attempt, the same calculators were reviewed againby the abstractors. These two rounds resulted in accept-able consistency. Then, each abstractor reviewed andabstracted information on a set of tools individually.Issues that arose during the abstraction were resolved bythe three abstractors using a consensus approach.

ResultsOverview

Thirty-six interactive cancer prognostic toolswere iden-tified through the three search approaches. We wereunable to obtain access to six tools. Eight tools did notmeet our inclusion criteria largely due to lack of interac-tive component (n¼ 5) or lack of focus ondesiredoutcomemeasures (n ¼ 3). Thus, we were able to review andabstract information on 22 tools (See Fig. 1). There wasa high degree of variability with regard to the number ofembedded calculators in each tool (a total of 107 calcula-tors;minimum:1,maximum: 22) and the tools addresseda

Rabin et al.

Cancer Epidemiol Biomarkers Prev; 22(10) October 2013 Cancer Epidemiology, Biomarkers & Prevention1646





total of 89 different cancers (minimum: 1, maximum: 84).We classified unique cancer sites under 13 main catego-ries. A list of abstracted tools, their brief name developedfor the purpose of this article (and used throughout thearticle to refer to the tools), the number of includedcalculators and main cancer categories included are sum-marized in Table 1. A full list of calculators, the name of

the developer/developing institution and their webaddress is provided in Table 2.

Some of the identified tools were cancer-specific tools(prostate, n ¼ 6 and breast, n ¼ 2), although about half ofthe tools (n¼ 12%or 55%) addressedmultiple cancer sites.Overall, tools focused on prostate (n¼ 11), colorectal (n¼10), breast (n¼ 8), andmelanoma (n¼ 6), although at least

Figure 1. Distribution of prognostictools throughout the identificationprocess.

Review of Online Cancer Prognostic Tools

www.aacrjournals.org Cancer Epidemiol Biomarkers Prev; 22(10) October 2013 1647





Tab

le1.

List

ofprogn

ostic

tools

Can

cersites

Tooln

ame

Brief

name

No.o

fca

lculators

Prostate

Colorectal

dBreas

tOther

gen

itourinarye

Melan

oma

Other

gas

trointestinal

fTho

racicg

Hea

dan

dne

ckh

Gyn

ecologic

iSoft

tiss

uej

End

ocrinek

Hem

atologic

lNervo

ussy

stem

m

Adjuva

nton

linea

Adjuva

nt4

þþ

þAJC

C—individua

lized

melan

omapatient

outcom

epredictio

ntools

AJC

C1

þ

Artificial

neural

netw

orks

in

prostateca

ncer

ANN

3þ

Bioch

emical

recu

rren

ce-

free

survival

prediction

mod

el

BioChe

mical

1þ

Can

cerM

ath

Can

cerM

ath

10þ

þþ

þþ

UCSF—

capra

score

Cap

ra1

þCan

cersu

rvival

que

ry

system

b

CSQS

2þ

þ

DFS

calculator

forEBRT,

brach

ythe

rapy

,and

combinations

ofthe2

EBRT

1þ

FinP

rogon

line

FinP

rog

1þ

Nom

ogramsforp

redictiong

survival

ofGBM

patients

GBM

1þ

TheHan

tables

Han

1þ

IBTR

—breas

tca

ncer

mod

uleve

rsion2.0

IBTR

1þ

Knigh

tCan

cerInstitu

te—

survival

predictio

ntools

KCI

7þ

þþ

þ

Lerner

Res

earchInstitu

te—

riskca

lculators

Lerner

22þ

þþ

þþ

þþ

þþ

MAASTR

Oprediction

web

site

MAASTR

O6

þþ

þ

MDAnd

erso

nclinical

calculators

MDAnd

erso

n9

þþ

þ

Mem

orialS

loan

-

Ketterin

g—predictio

n

toolsc

MSK

18þ

þþ

þþ

þþ

þ

Unive

rsity

ofMon

trea

l—

nomog

rams

Nom

ograms

6þ

þþ

May

oclinic

adjuva

nttool

(num

erac

y)

Num

erac

y2

þþ

Progn

ostig

ram

Progn

ostig

ram

1þ

þþ

þþ

þþ

þþ

þþ

þþ

QxM

D—ca

lculatea

QxM

D8

þþ

þ

(Con

tinue

don

thefollo

wingpag

e)

Rabin et al.






one tool was identified for most malignancies. Othergenitourinary cancers were also commonly evaluated(n ¼ 6). Characteristics of each tool are summarizedin Table 3 and are discussed in the next section. Cancersite–specific descriptions of the tools are provided inSupplementary Tables S1.1–S1.13. Some of the toolsincluded multiple calculators for the same cancer sitefocusing on different outcomes. To facilitate the reviewof the tools, we combined information from these calcu-lators under the tool’s cancer-specific summary in theSupplementary Tables.

Input variables included in prognostic estimatesTherewaswide variation in factors thatwere accounted

for in the prognosis estimates. We classified these factorsas cancer characteristics, demographic characteristics,comorbidities, therapy impact, genetic characteristics,and modifiable risk factors (see Table 3). The cancercharacteristics category referred to any clinical and path-ologic results that are associated with the patient’s cancerdiagnosis (e.g., tumor size, nodal status, grade, histology,etc.). All abstracted tools included information about atleast one cancer characteristic. Demographic characteris-tics (e.g., age, gender, marital status, etc.) were accountedfor in most (n ¼ 18 or 82%) tools. In contrast, only ninetools included noncancer comorbidities or health status intheir estimates. There was variability in how these factorswere captured; for example, using the health status of thepatient (good, bad, etc.; Adjuvant) versus a complexcomorbidity calculator that takes into account 16 itemizedconditions (CSQS). The differential impact of therapy onprognosis was accounted for in at least one calculator of15 tools. However, there was also variation in how ther-apy was captured. Some tools’ inputs included detailedinformation about specific chemo- or radiotherapy agentsand dose, whereas, others weremore simple (e.g., chemo-therapy: yes–no).

Only five tools took genetic/genomic characteristicsinto account. This included information about breastcancer receptor status [estrogen receptor (ER) in adjuvant,ER/progesterone receptor/HER-2 in Finn Prog] and anti-gen Ki67 for mantel cell lymphoma (QxMD). Further-more, one tool (MSK) assessed for an ovarian cancer-specific indicator (hereditary breast and ovarian cancer,HBOC syndrome). Modifiable risk factors were also verysparsely factored into the calculations of these tools (onlyn¼ 2% or 9%) and solely focused on smoking status (lungcancer calculator for MAASTRO and bladder cancer cal-culator for Nomograms).

Predicted outcomesThe most common predicted outcome was cancer-spe-

cific survival/mortality (n ¼ 17) followed by DFS/PFS(n ¼ 16). A few tools also included noncancer-specificsurvival (n¼ 4), clinical response to treatment (n¼ 2), andtherapy-induced side effects (n ¼ 2) as outcomes. No toolused quality of life or quality adjusted life years as one ofits outcomes.

Tab

le1.

List

ofprogn

ostic

tools

(Con

t'd)

Can

cersites

Tooln

ame

Brief

name

No.o

fca

lculators

Prostate

Colorectal

dBreas

tOther

gen

itourinarye

Melan

oma

Other

gas

trointestinal

fTho

racicg

Hea

dan

dne

ckh

Gyn

ecologic

iSoft

tiss

uej

End

ocrinek

Hem

atologic

lNervo

ussy

stem

m

Calcu

latorfores

timating

overalllife

expec

tanc

y

andlifetim

eris

kfor

prostateca

ncer

dea

thin

newly

diagn

osed

men

man

aged

with

out

defi

nitiv

eloca

lthe

rapy

Ros

wellP

ark

1þ

aNee

dto

crea

teon

-site

loginto

acce

ss.

bNee

dto

contac

tdev

elop

erforac

cess

.cNee

dto

contac

tdev

elop

erforac

cess

topan

crea

ticca

lculator.

dColon

,rec

tum,co

lorectal,c

ecum

,he

patic

flex

ure,

splenicflex

ure,

sigm

oidco

lon,

largeintestine,

rectos

igmoid,a

nus,

andan

alca

vity.

eBladder,k

idne

y/rena

lcell,pen

ile,tes

tis,o

ther

malege

nitalo

rgan

s,ureter,a

ndothe

rurinaryorga

ns.

f Gas

tric,e

sopha

geal,p

ancrea

s,ga

llbladde

r,stom

ach,

smallintes

tine,

liver,b

ileduc

t,othe

rbiliary,

retrop

erito

neum

,perito

neum

,omen

tum,a

ndap

pen

dix.

gLu

ng,m

esothe

lioma,

bronc

hus,

pleura,

trac

hea,

andmed

iastinum

.hOral,larynx

,lip,ton

gue,

salivaryglan

d,fl

oorof

mou

th,g

um,n

asop

harynx

,ton

sil,orop

harynx

,hyp

opha

rynx

,other

buc

cala

ndpha

rynx

,nas

alca

vity,m

idea

r,sinu

s,an

dey

ean

dorbit.

i Ova

rian,

endom

etria

l,ce

rvix,u

terus,

ovary,

vagina

,vulva

,and

othe

rfemalege

nitalo

rgan

s.j Bon

esan

djoints,K

apos

isarco

ma,

softtis

suesa

rcom

a,an

duterineleiomyo

sarcom

a.k A

drena

l,thyroid,a

ndothe

ren

doc

rine.

l Lym

pho

ma,mye

loma,leuk

emia,H

odgk

inslympho

madisea

se/nod

al,non

-Hod

gkinslympho

ma,multip

lemye

loma,ac

utelymphc

yricleuk

emia,chron

iclymph

ocyricleuk

emia,acu

tegran

uloc

yticleuk

emia,chron

icgran

uloc

yticleuk

emia,acu

temon

ocyric

leuk

emia,c

hron

icmon

ocyric

leuk

emia,o

ther

acuteleuk

emias,

andothe

rch

ronicleuk

emias.

mBrain,o

ther

nervou

ssy

stem

.







Table 2. List of prognostic tools with their brief name, developer, and web address

Tool name Brief name Developer Web address

Adjuvant Online Adjuvant Adjuvant Inc. http://www.adjuvantonline.com/AJCC—individualizedmelanoma patientoutcome prediction tools

AJCC American JointCommittee onCancer

http://www.melanomaprognosis.org/

Artificial neural networks inprostate cancer

ANN Institute for DynamicEducationalAdvancement

http://www.prostatecalculator.org/

Biochemical recurrence-free survival predictionmodel

BioChemical Duke ProstateCenter

http://eurology.surgery.duke.edu/Aspx/PredictionModel/NomogramsModel.aspx

CancerMath CancerMath CancerMath.net http://www.lifemath.net/cancer/UCSF—capra Score Capra University of

California SanFranciscoMedicalCenter

http://urology.ucsf.edu/patientGuides/uroOncPt_Assess.html#capra

Cancer survival querysystem

CSQS National CancerInstitute

http://www.csqs.cancer.gov/

DFS calculator for EBRT,brachytherapy andcombinations of the two

EBRT Professor LesBradbury

http://www.prostate-cancer-radiotherapy.org.uk/calculator.htm

FinProg online FinProg The FinProgResearch Group

http://www.finprog.org/CM/CM2.asp?pi ¼ 1

Nomograms for predictiongsurvival of GBM patients

GBM EuropeanOrganization forResearch andTreatment ofCancer

http://www.eortc.be/tools/gbmcalculator/model1.aspx

The Han tables Han James BuchananBrady UrologicalInstitute–JohnsHopkins Medical

http://urology.jhu.edu/prostate/hanTables.php

IBTR—breast cancermodule version 2.0

IBTR Unknown http://160.109.101.132/ibtr/

Knight Cancer Institute—survival prediction tools

KCI Knight CancerInstitute atOregonHealth andScienceUniversity

http://skynet.ohsu.edu/nomograms/

Lerner Research Institute—risk calculators

Lerner Cleveland Clinic -Lerner ResearchInstitute

http://www.lerner.ccf.org/qhs/risk_calculator/

MAASTRO predictionwebsite

MAASTRO MAASTRO Clinic http://www.predictcancer.org/

MD Anderson clinicalcalculators

MD Anderson MD AndersonCancer Center

http://www.mdanderson.org/education-and-research/resources-for-professionals/clinical-tools-and-resources/clinical-calculators/index.html

Memorial Sloan-Kettering—predictiontools

MSK Memorial SloanKettering CancerCenter

http://www.mskcc.org/cancer-care/prediction-tools

(Continued on the following page)

Rabin et al.






More than half of the tools (n¼ 12) reported onmultipleoutcomes. Outcomes were reported for one or multipletimeframes, most commonly 1 and 5 years. Format forpresenting prognosis varied, with half of the tools pre-senting only numerical presentation (i.e., ANN, Biochem-ical, Capra, EBRT, GBM, Han, Lerner, Nomograms,Numeracy, QxMD, and Roswell). The remainder used acombination of numerical and graphical display.

Purpose and intended usersAmong the stated purposes abstracted from the respec-

tive websites, the tools were primarily designed to sup-port decision-making about treatment.However, the toolswere also intended to be used the following treatmentcompletion.Most of the abstracted cancer prognostic toolswere designed with cancer specialists or physicians asexclusive users. Only four tools (i.e., BioChemical, GBM,MSK, and Nomograms) mentioned patients as potentialusers, but recommended patients to discuss results withtheir cancer specialists. None of the tools stated intentwasto support care transitions or were specifically design forthe primary care context.

Development/validation and fundingTools were classified on the basis of their primary

sources of data as population-based (i.e., underlying datais population-based dataset, often a national registry) orclinic-based (i.e., underlying data is based on a patientpopulation from one ormultiple health care settings and/or clinical trials). A number of tools (n¼ 4% or 18%) usedmixed data sources, most commonly a population-baseddataset such as the Surveillance, Epidemiology, and EndResults (SEER) database complemented by data frominstitutional patient databases or clinical trials.All except for one tool (QxMD) have reported on val-

idation of at least one of their calculators and these

validation efforts were published either as peer-reviewedmanuscripts or as technical reports. Only one tool (Adju-vant) reported any information on actual use of the tool inreal world settings. Funding was received from diversesources including federal, institutional, and industry.

Modality and accessAll except for oneprognostic tool (QxMD)wasavailable

in a web-based format. QxMD works as an applicationdesigned for handheld devices such as the iPhone, Black-berry, orAndroid.Adjuvant is also available for handhelddevice and on CD-ROM and EBRT had a downloadableexcel version. All abstracted tools were free of charge forusers and most were open access (n ¼ 20% or 91%). Twotools (adjuvant and QxMD) required login informationbefore accessing the tool; (CSQS) and (MSK pancreaticcancer calculator) required access to be granted by thedeveloper.

DiscussionIn this systematic review we identified and summa-

rized information on 22 interactive cancer prognostictools. Our review showed that advanced and multipleprognostic tools for cancer prognosis are currently freelyavailable online for a large number of cancer sites, diversetypes of patients with cancer, and clinical scenarios. Wefound great variability in terms of number of tools fordifferent cancer sites ranging from 11 tools for prostatecancer to two tools for several cancer sites. Overall, wenoted a major variation across existing tools in terms offormat and content. This lack of standardization is notsurprising given that this review focuses on the firstgeneration of cancer prognostic tools and to date most ofthese tools were developed in isolation by separateresearch and practice groups in the general absence ofguidelines or recommended practices. Use of these tools

Table 2. List of prognostic tools with their brief name, developer, and web address (Cont'd )

Tool name Brief name Developer Web address

University of Montreal—nomograms

Nomograms University ofMontreal

http://nomogram.org/

Mayo clinic adjuvant tool(numeracy)

Numeracy Mayo Clinic http://www.mayoclinic.com/calcs/

Prognostigram Prognostigram WashingtonUniversity in St.Louis

http://otooutcomes.wustl.edu/research/topics/cancer/Pages/Prognostigram.aspx

QxMD—calculate QxMD QxMD http://www.qxmd.com/apps/calculate-by-qxmdCalculator for estimatingoverall life expectancyand lifetime risk forprostate cancer death innewly diagnosed menmanaged withoutdefinitive local therapy

Roswell Park Roswell ParkCancerInstitute

http://www.roswellpark.org/apps/prostate_cancer_estimator/







Table 3. Characteristics of cancer prognostic tools

Features Adjuvanta AJCC ANNa BioChemical CancerMatha Capraa

OutcomesOverall survival/mortalityCancer-specific þb þ � � þ þb

Noncancer-specific þ � � � þ �DFS/PFS þ � þb þ þ� þClinical response � � � � � �Therapy-induced side effects � � � � � �

Cancer characteristics þ þ þ þ þ þDemographic characteristics þ þ � þ þ þComorbidities þ � � � � �Therapy impact þ � � þ þb �Genetic factors þ � � � � �Modifiable factors � � � � � �Patient user recommended � ND ND þ � �Data source Mixed Clinic Clinic Clinic Mixed ClinicValidation þ þ þ þ þ þFunder Astra Zeneca The

Greenberg BreastCancer Foundation

ND IDEA, ANNs, NCI ND ND UCSF

ModalityWeb-based þ þ þ þ þ þOther modality þb � � � � �

Updated June 2006 ND October 2007 2011 April 2009 2010

Features CSQS EBRT FinProg GBM Han

OutcomesOverall survival/mortalityCancer-specific þ � þ þ �Noncancer-specific þ � � � �

DFS/PFS � þ þ � þClinical response � � � � �Therapy-induced side effects � � � � �

Cancer characteristics þ þ þ � þDemographic characteristics þ � þ þ �Comorbidities þ � � þ �Therapy impact � þ þ þ þb

Genetic factors � � þ � �Modifiable factors � � � � �Patient user recommended � � � þ �Data source Population Clinic Population Clinic ClinicValidation þ þ þ þ þFunder NCI ND Helsinki University

Research Funds,the Academy ofFinland, the CancerSociety of Finland

ND ND

ModalityWeb-based þ þc þ þ þOther modality � � � � �

Updated 2010 ND ND ND 2011

Features IBTR KCIa Lernera MAASTROa MD Andersona MSKa

OutcomesOverall survival/mortalityCancer-specific � þ þb þb þb þb

Noncancer-specific � � � � � �

(Continued on the following page)

Rabin et al.






in clinical practice may support communication andunderstanding about cancer prognosis, but further testingand dissemination of these tools in clinical settings isneeded. We found limited reports on actual use andapplication in real world clinical settings for only onetool. Validation of the underlying algorithm is only thefirst of many issues to be considered in the use of prog-nostic tools. If the purpose of clinical shared decision-

making is a goal of an instrument, the scientific validityalone does little good if patients, family members, ordifferent groups of providers cannot adequately under-stand and use the information presented (23). This isespecially critical given the emerging information on lowhealth literacy and numeracy (12, 24).

Our review found that cancer prognostic tools differedmarkedly in the type and number of input variables they

Table 3. Characteristics of cancer prognostic tools (Cont'd )Features Adjuvanta AJCC ANNa BioChemical CancerMatha Capraa

DFS/PFS þ � þb þb þb þb

Clinical response � � � � þb þb

Therapy-induced side effects � � � þb � �Cancer characteristics þ þ þ þ þb þDemographic characteristics þ þ þb þb þb þb

Comorbidities � � þb þb � þb

Therapy impact þ � þb þb þb þb

Genetic factors � � � � � þModifiable factors � � � � � �Patient user recommended � � � � � þData source Clinic Populationb

MixedbClinic Clinic Populationb

ClinicbClinic

Validation þ þb þ þ þb þFunder ND Knight Cancer

InstituteND ND ND ND

ModalityWeb-based þ þ þ þ þ þb

Other modality � � � � � þb

Updated ND ND ND July 2011 ND ND

Features Nomogramsa Numeracy Prognostigram QxMD Roswell Park

OutcomesOverall survival/mortalityCancer-specific þb þb þ þb þNoncancer-specific þb � � � �

DFS/PFS þb þ � þb �Clinical response � � � � �Therapy-induced side effects � � � � �

Cancer characteristics þb þ þ þ þDemographic characteristics þb þb þ þb �Comorbidities þ � þ þb �Therapy impact þb þb þ � �Genetic factors � � � þ �Modifiable factors þ � � � �Patient user recommended þ � � � NDData source Populationb

ClinicbClinic Mixed Clinicb Mixed

Validation þb þ þ ND þFunder University of Montreal ND ND ND NDModalityWeb-based þ þ þ � þOther modality � � � þ �

Updated ND August 2011 ND December 2011 ND

Abbreviations: Mixed, used a combination of clinic and population-based sample; ND, not defined.aIndicates multiple calculators for cancer sites in the tool.bIndicates that at least one but not all calculators has the feature.cExcel version downloadable.

Table 3. Characteristics of cancer prognostic tools (Cont'd)

Features IBTR KCIa Lernera MAASTROa MD Andersona MSKa







took into account. The differences in input variablesacross cancer site–specific calculators are partially due tothe variability in the available scientific information oncertain input variables across cancer sites. Most toolsincluded basic tumor and patient characteristics, butlacked information about the effect of genomics as wellas patient comorbidities. As genomic medicine continuesto evolve, incorporation of these factors into prognostictools as well as environmental exposures will be critical.Furthermore, understanding how cancer relates to overallprognosis in the context of other conditions (e.g., cardio-vascular disease) is important as the population of cancerpatients’ ages. Very few tools accounted for modifiablerisk factors including smoking, overweight/obesity, alco-hol consumption, physical activity, or other lifestyle intomeasures of prognosis. As data continue to add supportfor the potential effect of these factors on prognosis,refinement of tools will be needed. Until prognostic sys-tems include all of these risk factors, the promise ofprecision medicine will only be partially fulfilled.

While most tools focused on similar outcomes, therewas some variation with respect to the time framesreported as well as presentation of estimates. Further-more, the development of the tools was based on variabledatasets, ranging from single site clinic population topopulation-based datasets such as SEER. The lack ofstandardization makes comparison of results from differ-ent tools difficult (25). This might require that usersexperiment with different tools to determine whichtool(s) fit bestwith their clinical practice or specific patientwith cancer.

Only two tools included potential adverse impacts oftherapy as an outcome variable and there was no tool thatused quality of life as one of its outcomes. As suggested byBrowman and colleagues: "experienced oncologistsunderstand that for significant portion of patients withcancer, maintaining and improving health-related qualityof life is an important objective when considering treat-ment options" (26). As quality of life and side effect-related outcomes are most meaningful and important topatients with cancer when making treatment decisions,prognostic tools should strive to include them as one keyoutcome variable. For this to happen, data on both thesevariables and the earlier mentioned modifiable risk fac-tors should be consistently collected and made available.

The purpose of the cancer prognostic tools was notalways clearly defined and ranged from providing base-line prognostic estimates to supporting treatment deci-sions after diagnosis or adjuvant treatment decisions afterprimary (e.g., surgery) treatment. It is important to clearlydifferentiate tools that provide information on prognosiswithout manipulation of treatment and those that canaccount for prognosis differences based on therapy ofchoice. Similarly, only two tools accounted for potentialimpact of behavior change. Such component could be animportant motivator as well as a key aspect for informeddecision-making.Discussion of cancer prognosismaybe aclassic "teachable moment or setting" (27, 28). Further-

more, it is important that tools directly specify theiroptimal users and timing of use. Specifically, the predic-tion tools developed by the Memorial Sloan-KetteringCancer Center (MSK tool; New York, NY) included anintroductory page for each cancer site that includes a clearlisting of the mission of each calculator, who should usethem, thedataset it is building on, the typeof outcomes thecalculator will provide, the data elements needed to usethe calculator (e.g., age, cancer stage, etc.) and any limita-tions and special notes to the user. A similar functionalitywas developed for the revised version of the CSQS tool(developed by the National Cancer Institute), but notfound on most other tools.

Moreover, it is important that the tools specify whetherdevelopers intend tools for use by patients. As indicatedby our review, almost all of the cancer prognostic toolswefoundwere accessible to the general public although onlyfour tools mentioned patients as potential users. Cancerprognosis is among the leading topics of information thatis sought by patients with cancer (5, 29). Anecdotal evi-dence and simple review of online patient discussiongroups, indicates that patients with cancer already accessthese tools and discuss and compare prognostic resultsfrom the tools with each other (30). While studies haveshown that patients may benefit from cancer prognostictools (31) studies have not assessed the impact of cancerprognostic tools when directly accessed by patients orhow theymayoptimally beusedwith bothpatient/familyand provider focused interfaces to facilitate betterinformed decisions.

Finally, we found that only two of the reviewedcancer prognostic tools had applications for a handhelddevice. Recent research suggests that there is an increas-ing number of health care providers (especially physi-cians) who own smart phones and access medical infor-mation via their handheld devices (32, 33). There is alsoincreasing desire to access medical applications such asdecision aids remotely and at the place of service (34).Making existing prognostic tools accessible throughmultiple devices would increase their access and use.Furthermore, methods to integrate the use of these toolsinto patient care, including cancer care plans are stillunclear (35).

Limitations of this report include potential underesti-mation of the number of prognostic tools eligible. Fromour initial review, we found six tools, which were exclud-ed because of inability to gain access to the site. On thebasis of our search criteria, these sites were eligible forinclusion in this review. We included tools identifiedthrough May 2011. There could have been more toolspublished since then, although in scanning the literaturein March 2013, no additional tools were identified. Fur-thermore, our review did not compare the algorithmsdriving each tool as this was beyond the scope of ourarticle. However, despite the potential limitations, ourreview used comprehensive search criteria, had clearenumeration of criteria and recommendations for report-ing, and compared a large number of tools. To our

Rabin et al.






knowledge, this is the first such review and compilationundertaken to date.In summary, our study systematically identified and

reviewed 22 cancer prognostic tools on several importantcriteria. Cancer prognostic tools hold great promise infacilitating patient-centered communication and deci-sion-making and helping patients prepare for life posttreatment. Providers can also anticipate that increasingnumberofpatientswill use, ormayalreadybeusing, thesetools especially in early stages of the cancer journey. Theprocess of identifying available toolswas time consumingas there was no one location where all existing cancerprognostic tools were easily accessible and compared.Thus, finding appropriate tools and evaluating them foruse may be challenging if not impossible for a busypracticing clinician. The intent of this review is to assistproviders and health care teams/systems in becomingmore aware of available interactive cancer prognostictools so that informed decisions about their use in clinicalpractice can be made. This review makes a significantcontribution to the literature and has considerable poten-tial to enhance clinical practice. Technology has affordedunprecedented opportunities to assist cancer care specia-lists to improve care delivery and to support treatmentdecisions of the patients. To maximize these opportu-nities, we advocate clear delineation between tools andpages on websites that are intended for providers andthose that are intended for patients, with careful attentionto state-of-the-art and science of health communication todesign appropriate content and format for each audience.

Implications for research and practiceFuture dissemination, testing, and refinement of exist-

ing tools anddevelopment of new tools based on scientificand risk communication evidence, as well as evolvingtechnologies will be needed. More specifically the follow-ing next steps and recommendations emerge from ourwork:

1. Clear criteria should be developed to guide thedevelopment of new and evaluation of existingprovider-facing tools in terms of content and format.This could include best practices guidelines on thestandardization of prognostic tools and adequateinformation about intended use and limitations;eventually lead to accreditation of existing tools.

2. Several prominent cancer sites have only smallnumber of tools available indicating need for

additional developmental work focusing on cancersthat may not be well represented to date.

3. There should be a special focus on adaptation ofexisting and new tools to newly emerging platformssuch as iPads, smart phones, and mobile devices.

4. Furthermore, integration of tools into electronicmedical recordswould further increase their use andsmooth integration into practice.

5. Tools should be compared in terms of theconsistency and reliability of algorithms and theirapplicability and usefulness in real world settings.

6. Tools should include factors and outcomes that haveincreasing relevance to providers and patientsincluding impact of modifiable risk factors andgenomic, epigenetic, environmental and patientbehavior, and preference factors and measures ofquality of life.

Disclosure of Potential Conflicts of InterestNo potential conflicts of interest were disclosed.

DisclaimerThe opinions or assertions contained herein are the private ones of the

authors andarenot considered as official or reflecting the viewsof theNIH.

Authors' ContributionsConception and design: B.A. Rabin, T. Sanders, L. Nekhlyudov, S. Bull,R.E. GlasgowDevelopment of methodology: B.A. Rabin, B. Gaglio, T. Sanders, L.Nekhlyudov, R.E. GlasgowAcquisition of data (provided animals, acquired and managed patients,provided facilities, etc.): B.A. Rabin, B. Gaglio, T. SandersAnalysis and interpretation of data (e.g., statistical analysis, biostatis-tics, computational analysis): B.A. Rabin, B. Gaglio, T. Sanders, L. Nekh-lyudov, S. BullWriting, review, and/or revision of themanuscript:B.A. Rabin, B. Gaglio,T. Sanders, L. Nekhlyudov, J.W. Dearing, S. Bull, R.E. Glasgow, A.MarcusAdministrative, technical, or material support (i.e., reporting or orga-nizing data, constructing databases): B.A. Rabin, T. SandersStudy supervision: B.A. Rabin

AcknowledgmentsThe authors thank Ms. Sara Hoerlein and Michelle Henton for their

support with the preparation of the manuscript.

Grant SupportThis work was supported by the National Cancer Institute (P20

CA137219). The authors of this manuscript (all except R.E. Glasgow) havereceived financial support from the National Cancer Institute to evaluatethe feasibility and small-scale implementation of one of the cancer prog-nostic tools (Cancer Survival Query System) included in this review.

ReceivedMay 15, 2013; revised August 5, 2013; acceptedAugust 5, 2013;published OnlineFirst August 16, 2013.

References1. Epstein RM, Street RLJ. Patient-centered communication in cancer

care: promoting healing and reducing suffering. In: NIHPublicationNo.07-6225. Bethesda, MD: National Cancer Institute; 2007.

2. Hagerty RG, Butow PN, Ellis PM, Dimitry S, Tattersall MH. Commu-nicating prognosis in cancer care: a systematic review of the literature.Ann Oncol 2005;16:1005–53.

3. de Bock GH, Bonnema J, Zwaan RE, van de Velde CJ, Kievit J,Stiggelbout AM. Patient's needs and preferences in routine

follow-up after treatment for breast cancer. Br J Cancer 2004;90:1144–50.

4. Smith TJ, Dow LA, Virago E, Khatcheressian J, Lyckholm LJ, Mat-suyamaR.Giving honest information to patientswith advancedcancermaintains hope. Oncology 2010;24:521–5.

5. Rutten LJ, Arora NK, Bakos AD, Aziz N, Rowland J. Information needsand sources of information among cancer patients: a systematicreview of research (1980–2003). Patient Educ Couns 2005;57:250–61.







6. Shariat SF, Karakiewicz PI, Roehrborn CG, Kattan MW. An updatedcatalog of prostate cancer predictive tools. Cancer 2008;113:3075–99.

7. Weeks JC, Cook F, O'Day S, Peterson LM, Wenger N, Reding D, et al.Relationship between patients' predictions of prognosis and theirtreatment preferences. JAMA 1998;279:1709–14.

8. Peele PB, Siminoff LA, Xu Y, Ravdin PM. Decreased use of adjuvantbreast cancer therapy in a randomized controlled trial of a decisionaid with individualized risk information. Med Decis Making 2005;25:301–7.

9. Jefford M, Tattersall MH. Informing and involving cancer patients intheir own care. Lancet Oncol 2002;3:629–37.

10. Mills ME, Sullivan K. The importance of information giving for patientsnewly diagnosed with cancer: a review of the literature. J Clin Nurs1999;8:631–42.

11. Davis TC,WilliamsMV,Marin E, ParkerRM,Glass J.Health literacy andcancer communication. CA Cancer J Clin 2002;52:134–49.

12. BerkmanND, Sheridan SL, Donahue KE, Halpren DJ, Viera A, Crotty K,et al. Health literacy interventions and outcomes: an updated system-atic review. Rockville, MD: Agency for Healthcare Research andQuality (US); 2011. Evidence Reports/Technology Assessments,No. 199.

13. Shariat SF, Karakiewicz PI, Suardi N, Kattan MW. Comparison ofnomograms with other methods for predicting outcomes in prostatecancer: a critical analysis of the literature. Clin Cancer Res 2008;14:4400–7.

14. Lamont EB, Christakis NA. Complexities in prognostication inadvanced cancer: "to help them live their lives the way they want to".JAMA 2003;290:98–104.

15. Ross PL, Gerigk C, Gonen M, Yossepowitch O, Cagiannos I, SoganiPC, et al. Comparisons of nomograms and urologists' predictions inprostate cancer. Semin Urol Oncol 2002;20:82–8.

16. Mackillop WJ, Quirt CF. Measuring the accuracy of prognostic judg-ments in oncology. J Clin Epidemiol 1997;50:21–9.

17. Higgins O, Sixsmith J, Barry MM, Domegan C. A literature review onhealth information-seeking behaviour on the web: a health consumerand health professional perspective. Stockholm: Technical Report,European Centre for Disease Prevention and Control; 2011.

18. Hesse BW,Moser RP, Rutten LJ. Surveys of physicians and electronichealth information. N Engl J Med 2010;362:859–60.

19. Kreps GL, Neuhauser L. New directions in eHealth communica-tion: opportunities and challenges. Patient Educ Couns 2010;78:329–36.

20. O'Connor AM, Wennberg JE, Legare F, Llewellyn-Thomas HA, Moul-ton BW, Sepucha KR, et al. Toward the "tipping point": decision aidsand informed patient choice. Health Aff 2007;26:716–25.

21. Elwyn G, O'Connor A, Stacey D, Volk R, Edwards A, Coulter A,et al. Developing a quality criteria framework for patient decision

aids: online international Delphi consensus process. BMJ 2006;333:417.

22. Waters E, Sullivan H, Nelson W, Hesse BW. What is my cancer risk?How internet-based cancer risk assessment tools communicate indi-vidualized risk estimates to the public: content analysis. JMed InternetRes 2009;11:e33.

23. KohHK,BrachC,Harris LM, ParchmanML.Aproposed "health literatecare model" would constitute a systems approach to improvingpatients' engagement in care. Health Aff 2013;2:357–67.

24. Woloshin S, Schwartz LM, Moncur M, Gabriel S, Tosteson ANA.Assessing values for health: numeracy matters. Med Decis Making2001:380–8.

25. Bardia A, Loprinzi C, Grothey A, Nelson G, Alberts S, Menon S, et al.Adjuvant chemotherapy for resected stage II and III colon cancer:comparison of two widely used prognostic calculators. Semin Oncol2010;37:39–46.

26. BrowmanGP, Berrang T, Smith S. Prognostic tools for cancer survival:a secondary role for quality-of-life measurement. J Clin Oncol2009;27:2902–4.

27. Hochbaum GM. Public participation in medical screening programs:a sociopsychological study. U.S. Department of Health, Education,and Welfare, Public Health Service Publ. No. 572. Washington, D.C.:U.S. Government Printing Office; 1958.

28. Demark-Wahnefried W, Aziz NM, Rowland JH, Pinto BM. Riding thecrest of the teachable moment: promoting long-term health after thediagnosis of cancer. J Clin Oncol 2005;23:5814–30.

29. Mistry A, Wilson S, Priestman T, Damery S, Haque MS. How do theinformation needs of cancer patients differ at different stages ofthe cancer journey? A cross-sectional survey. J R Soc Med2010;1:30.

30. Breastcancer.org: Discussion Boards [Internet]. What prognosis cal-culators are out there? [cited 2013 Aug 1]. Available from: http://community.breastcancer.org/forum/67/topic/797024.

31. Belkora JK, Rugo HS, Moore DH, Hutton DW, Chen DF, Esserman LJ.Oncologist use of the Adjuvant! model for risk communication: a pilotstudy examining patient knowledge of 10-year prognosis. BMC Can-cer 2009;9:127.

32. Peskin SR. Is "mobile health" revolution made for managed care?Managed Care 2010;19:20–3.

33. PandeyA,HasanS,DubeyD, Sarangi S. Smartphone apps as a sourceof cancer information: changing trends in health information-seekingbehavior. J Cancer Educ 2013;28:138–42.

34. Prgomet M, Georgiou A, Westbrook JI. The impact of mobilehandheld technology on hospital physicians' work practices andpatient care: a systematic review. J Am Med Inform Assoc 2009;16:792–801.

35. Whelan TJ, Loprinzi C. Physician/patient decision aids for adjuvanttherapy. J Clin Oncol 2005;23:1627–30.

Rabin et al.






Documents

Predicting Cancer Prognosis Using Interactive Online Tools ......Review Predicting Cancer Prognosis Using Interactive Online Tools: A Systematic Review and Implications for Cancer