Artificial Intelligence and the Practice of Pathology

Toby C. Cornish, MD, PhDAssociate Professor of PathologyMedical Director of Informatics

Department of PathologyUniversity of Colorado SOM

DISCLOSURE

In the past 12 months, I have had a significant financial interest or other relationship with the manufacturer(s) of the following product(s) or provider(s) of the following service(s) that will be discussed in my presentation.

• Leica Biosystems, Inc. Pathology Imaging Advisory Board: Consulting Fees

Who am I?

• Anatomic Pathologist (GI Pathology)• Clinical Informaticist• Medical Director of Informatics

– University of Colorado SOM Dept of Pathology• Medical Director of the LIS

– UCHealth

• Flagship: University of Colorado Hospital (Aurora, CO)• 9 additional hospitals in Denver/Fort Collins/Colorado

Springs/Steamboat Springs area• 3 affiliated hospitals in Wyoming and Nebraska• 1+ hospitals under construction (Highlands Ranch)

UCHealth University of Colorado Hospital

UCHealth Broomfield Hospital

UCHealth Memorial Hospital CentralUCHealth Memorial Hospital NorthUCHealth Pikes Peak Hospital

UCHealth Grandview Hospital

UCHealth Longs Peak Hospital

WYOMINGCOLORADO

NEBRASKA

UCHealth Yampa Valley HospitalUCHealth Poudre Valley Hospital

UCHealth Medical Center of the Rockies

Ivinson Memorial Hospital

Cheyenne Regional Medical Center Sidney Regional Medical Center

June 7, 2017

UCHealth Grandview Hospital

AI and Computational Pathology

Artificial intelligence

• Production of intelligent-seeming behavior by machines

• Concept dates back to the early 1950s

• Comprises a broad range of approaches

Hierarchy of artificial intelligence

ArtificialIntelligence

MachineLearning

DeepLearning

Machine learning

• Machine learning allows computers to predict outcomes from data without being explicitly programmed

• Refines a model that predicts outputs using sample inputs (features) and a feedback loop

• Classical machine learning relies heavily on extracting or selecting salient features, which is a combination of art and science (“feature engineering”)

ArtificialIntelligence

MachineLearning

DeepLearning

Deep learning

• The concept dates back to the 1940s• Previously:

– “cybernetics” (1940s-60s)– “connectionism” / “artificial neural networks” (1980s-90s)

• “Deep learning” (c. 2006)• Multiple hidden layers in an artificial neural network• Computationally intensive• Until recently not feasible with large datasets

ArtificialIntelligence

MachineLearning

DeepLearning

Deep learning

• Avoids the need to define specific features in the data as inputs

• Discovers the features from the raw data provided during training

• Hidden layers in the artificial neural network represent increasingly more complex features in the data

• Convolutional Neural Networks (CNNs), a type of deep learning, are commonly used for image analysis

ArtificialIntelligence

MachineLearning

DeepLearning

Machine learning v. deep learning

http://adilmoujahid.com/posts/2016/06/introduction-deep-learning-python-caffe/

A deep learning model example

Goodfellow et al., Deep Learning. 2016

Output (object identity)

3rd hidden layer (object parts)

2nd hidden layer (corners and contours)

1st hidden layer (edges)

Visible layer (input pixels)

Strong v. Weak AI

• Strong AI– AI capable of solving difficult, generalized

problems including the ability to deal with unexpected circumstances

– e.g. perform “universal” histopathologic diagnosis

• Weak AI (Narrow AI)– AI capable of solving specific tasks– e.g. identify breast cancer metastasis in a lymph

node

Ken Jennings, after losing to IBM’s Watson on Jeopardy! in 2011

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Artificial intelligence Machine learning Deep learning

Watson winsJeopardy

Google Search Trends

Gartner Hype Cycle

Expe

ctat

ions

TimeTechnology Trigger

Peak of Inflated Expectations

Trough of Disillusionment

Slope of Enlightenment

Plateau of Productivity

Hype Cycle for Emerging Technologies (2018)

http

s://w

ww.

gartn

er.c

om/s

mar

terw

ithga

rtner

/top-

trend

s-in

-the-

gartn

er-h

ype-

cycl

e-fo

r-em

ergi

ng-te

chno

logi

es-2

018/

Expe

ctat

ions

Time

Deep Learning

1st “AI Winter” 2nd “AI Winter” 3rd “AI Winter”?

Expert Systems

1987 19931974 1980 ? ?

Early approaches

?

What is an “AI Winter”?

• Extended period of time in which enthusiasm for and investment in AI “dried up”

• Generally attributed to:– Over-promising– Under-delivering– Creation of an economic bubble– Infighting– Inadequate computational hardware

• Two major “winters” are recognized with many smaller events

Beyond Jeopardy: The IBM Watson Legacy

Journal of Clinical Oncology 31, no. 15_suppl (May 2013) 6508-6508.

Background: Electronic decision support is increasingly prevalent in clinical practice. Traditional tools map guidelines into an interactive platform. An alternative method builds on experience-based learning.

Methods: Memorial Sloan-Kettering (MSK), IBM and WellPoint teamed to develop IBM Watson – a cognitive computing system leveraging natural language processing (NLP), machine learning (ML) and massive parallel processing – to help inform clinical decision making. We made a prototype for lung cancers using manufactured and anonymized patient cases. We configured this tool to read medical language and extract specific attributes from each case to identify appropriate treatment options benchmarked against MSK expertise, anonymized patient cases and published evidence. Treatment options reflect consensus guidelines and MSK best practices where guidelines are not granular enough to match treatments to unique patients. Analysis and building accuracy is ongoing and iterative.

Aug. 11, 2018

https://www.statnews.com/wp-content/uploads/2018/09/IBMs-Watson-recommended-unsafe-and-incorrect-cancer-treatments-STAT.pdf

https://www.statnews.com/wp-content/uploads/2018/09/IBMs-Watson-recommended-unsafe-and-incorrect-cancer-treatments-STAT.pdf

https://hackernoon.com/%EF%B8%8F-big-challenge-in-deep-learning-training-data-31a88b97b282

Training data in machine learning

• Large amounts of data and expert annotation are the keys to successful training of machine learning algorithms

• The amount of data needed is typically proportional to the difficulty of the task

Training data for self driving cars

• Professor Amnon Shashua, co-founder and CTO of Mobileye:– “Mobileye has 600 people on staff annotating images and

expects to have 1,000 or more shortly.”

https://www.teslarati.com/mobileye-cto-building-autonomous-driving-systems/

https://aws.amazon.com/blogs/machine-learning/aws-partners-with-mapillary-to-support-the-large-scale-scene-understanding-challenge-at-cvpr-2017/

AI and Computational Pathology

Definition: Computational Pathology• An approach to diagnosis that incorporates multiple sources of

raw data (eg, clinical electronic medical records, laboratory data including “-omics,” and imaging [both radiology and pathology imaging]); extracts biologically and clinically relevant information from these data; uses mathematic models at the molecular, individual, and population levels to generate diagnostic inferences and predictions; and presents this clinically actionable knowledge to customers through dynamic and integrated reports and interfaces, enabling physicians, patients, laboratory personnel, and other health care system stakeholders to make the best possible medical decisions.

Computational Pathology: A Path Ahead. Louis DN, et al. Arch Pathol Lab Med. 2016 Jan;140(1):41-50.

3 5 6 8 11 514

7 12 18 199

269

179

1726

52

88

124141

155 155

119128

137

201

250

309294

406

384

11 1 1 1 2 4 10 9 11

0

50

100

150

200

250

300

350

400

450

1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019

Pubm

edAr

ticle

s

Pathology and ("Artifical Intelligence" or "Machine Learning" or "Deep Learning") "Computational Pathology"

Bartels PH et al. Anal Quant Cytol Histol. 1988 Aug;10(4):299-306

So which is it: AI or Computational pathology?

• AI applied to pathology is a form of Computational Pathology

• It doesn’t matter what you call it

• I will likely use these terms interchangeably in this talk

How will AI affect the Practice of Pathology?

The future of AI in pathology

• Variety of opinions– Practicing pathologists– AI / Digital Pathology enthusiasts– Data scientists– Informaticists– Vendors

The future of AI in pathology

• Depends on the time scale under consideration– 1 year?– 5 years?– 10 years?– 20 years?– 30 years?– 100 years?

The future of AI in pathology

• Depends on what the underlying technology is– Deep learning?– The next generation of AI?

Impact of AI

Potentialof AI

Weak(Narrow)

Strong(General)

Negative Positive

Skeptics

Dystopians Utopians

Pragmatists

• "artificial intelligence is the future and it always will be"

The pragmatist’s view of AI in Pathology

A pragmatist’s view of AI in Pathology

• Barring disruptive changes in the AI paradigm, such as– A non linear increase in computing power– New methods of building AI

• A General AI that could replace a pathologist is far off, but not impossible

• Narrow AI tools will reach the early market within 3 - 5 years and the mainstream market within 5 – 8 years

A pragmatist’s view of AI in Pathology

• AI will become just another tool we adopt and use alongside our traditional tools

• Consider previous scenarios:– External examination v. dissection– Macroscopic examination v. microscopic examination– Electron microscopy v. light microscopy– IHC v. H&E– Molecular pathology v. histopathology

The fact that immunocytochemistry of human tissues is now a rich and complex field (in terms both of the different techniques available and the large number of different molecular entities detectable in human tissues) has tended to cause a polarisation among pathologists. There are the "monoclonal antibody revolutionaries" and an opposing group of "contras". This polarisationis not, as far as we know, described in the literature but is nevertheless to be found in pathology departments throughout the world.

Mason DY & Gatter KC. J Clin Pathol. 1987, 40(9):1042-54

Mason DY & Gatter KC. J Clin Pathol. 1987, 40(9):1042-54

Skeptic?

Mason DY & Gatter KC. J Clin Pathol. 1987, 40(9):1042-54

Dystopian?

Utopian?

0

5000

10000

15000

20000

25000

30000

35000

EM IHC / ICC Mol Dx AI DP

Pubm

edar

ticle

s

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

EM IHC / ICC Mol Dx AI DP

Perc

enta

ge o

f tot

al a

rticl

es

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

EM IHC / ICC Mol Dx AI DP

Cum

ulat

ive

perc

enta

ge o

f tot

al a

rticl

es

Pathologist benefits

• Use CP as an adjunct to other traditional tests and methods• Derive diagnostic, prognostic and predictive information not

possible before• Increase visibility and importance of pathologists in the healthcare

enterprise• Create new billable tests (where applicable)• “Automate” the boring stuff• “Automate” the time-consuming stuff• “Automate” the hard stuff

Challenges for CP in Digital Pathology

• Accumulation of large well-annotated image datasets and related data

• Creation of robust, generalizable models• Demonstration of value to pathologists, clinicians, administrators• Overcome stakeholder skepticism• Insufficient substrate for implementation (i.e. not enough clinical

deployment of WSI)• Lack of CP expertise amongst pathologists

Creating a “Digital Substrate” for pathology

• The “Digital Substrate” represents the digitization of all pathology slides to permit higher order processes that add value to the practice of pathology

• Value of using digital pathology increases with the percentage of digital pathology adopters (width of the digital substrate)

• Both requires and enables integration of digital pathology functions at a variety of levels

Is it this?

DICOM

• Digital Imaging and Communications in Medicine• DICOM supplement 145 for WSIs (2011)• Still not widely implemented or supported

http://dicom.nema.org/dicom/dicomwsi/Connectathon/index.html

Systems integration (single vendor)

DSR

LIS

ImageAnalysis

Digitalslides

WSIViewer

Image dataAnnotation

IA orderOrders?Results?

Systems integration: 3rd party?

DSR

LIS ImageAnalysis

Digitalslides

No interfaces to 3rd party image Analysis system

WSIViewer Image data

IA order

Clinical slide archiving

Accessioning

Specimen processing

Slide creation

Case distribution

Pathologist review & sign out

Archives

Slide scanning

Contributing Pathologists

Central Lab

Consulting Pathologists

ConsultationNetwork

Secondary diagnosis telepathology

Systems integration: Clinical lab

LISEHR/CPOE MiddlewareProvider

Test QueryOrders

Results

Orders

Results

Labels

Orders

Results

Instrument

Samples

1 4

Test Orders

Results

2

3

6

5

7

891011

LIS

PACS

MiddlewarePathologist

Results

Orders

Results

Orders

Results

1 4

Results

9

10

11

WSIWSI

WSI

AI

Figure 2: Architecture of a computational pathology app and platform. All work is done by the platform, but the app contains all the ‘intelligence’; it defines the algorithm, on which slides to run it, and how to present the results to the pathologist. This upfront programming consists of two steps: 1) select applicable slide types (stain, organ, etc.), and 2) program the app’s tasks on the analytics engine. These tasks are performed automatically in step 3) when a relevant slide is scanned. Finally when a pathologist reviews a relevant case, step 4) displays and allows interaction with the pre-calculated results.

Systems integration

DSR

LIS ImageAnalysis

Digitalslides

IA orders/results

Image, Annotation,

resultsAPIs

WSIViewer

Image dataAnnotation

IA order

No DP

No AI=

Systems integration

PACS(DICOM

)

LIS ImageAnalysis

Digitalslides

IA orders/results

Image, Annotation,

resultsAPIs

WSIViewer

IA order

Image dataAnnotation

Summary: systems integration

• Current interfacing is very lightweight• No standard model exists for robustly interfacing image analysis,

LIS and DSR components– Orders in– Results out– Use of a single image repository (i.e. no duplication of image data)

• Most productive labs rely heavily on FTEs as “human interfaces” or have spent considerably on custom programming

• Current solutions not scalable (think CP)

Summary: systems integration

• Lab information systems will be the biggest impediment to systems integration for DP & CP

• Many are aging poorly and lack a modern UX• Existing data models and APIs are inadequate for many

proposed CP workflows– For example, LISes do not discretely know what type of tissue is in a

block or on a slide

https://ai.googleblog.com/2018/04/an-augmented-reality-microscope.html

Left: Schematic overview of the ARM. A digital camera captures the same field of view (FoV) as the user and passes the image to an attached compute unit capable of running real-time inference of a machine learning model. The results are fed back into a custom AR display which is inline with the ocular lens and projects the model output on the same plane as the slide. Right: A picture of our prototype which has been retrofitted into a typical clinical-grade light microscope.

Example view through the lens of the ARM. These images show examples of the lymph node metastasis model with 4x, 10x, 20x, and 40x microscope objectives.

https://webcast.aacr.org/console/player/38653?sec=1&mediaType=slideVideo&&crd_fl=0&ssmsrq=1524692831210Hipp & Stumpe. Advancing Cancer Diagnostics with Artificial Intelligence. AACR 2018 Annual Meeting. Chicago, IL

Addressing challenges:“technopanic”

“AI Technopanic”

Daniel Castro and Alan McQuinn. 9/10/2015. The Information Technology & Innovation Foundation

Thoughts from another domain…

Kasparov v. Deep Blue

• May 11, 1997• Match result: Deep Blue (W) - Kasparov:

3½ - 2½

http

s://w

ww.

flick

r.com

/pho

tos/

amitr

ajit/

http

s://c

omm

ons.

wik

imed

ia.o

rg/w

iki/F

ile:K

aspa

rov-

29.jp

g

What have they been up to since?

Deep Blue

Computer History Museum (CHM), Mountain View, California, US

https://www.flickr.com/photos/amitrajit/

Garry Kasparov

• In June 1998, Kasparov played the first public game of human-computer collaborative chessagainst Veselin Topalov

• Each used a regular computer with off-the-shelf chess software (Kasparov: Fritz 5, Topalov: ChessBase 7.0)

• The match ended in a 3 - 3 tie• Kasparov called this “advanced chess,” and it has later been

called “centaur chess”• Retired in 2005

“Freestyle” chess

• 2005: the first “freestyle” chess tournament• Teams could consist of any number of humans or computers• Some teams consisted of chess grand masters• The most powerful chess computer at the time was also

entered• The winning team consisted of young, amateur players,

Steven Cramton and Zackary Stephen and their computers

- Garry Kasparov"The Chess Master and the Computer“ The New York Review of Books, 2/11/2010

“weak human + machine + better process was superior to a strong computer alone and, more remarkable, superior to a strong human + machine + inferior process.”

http://www.nybooks.com/articles/2010/02/11/the-chess-master-and-the-computer/

Summary

• Computational pathology is coming with near term applications of narrow (weak) AI to clinical problems

• Abundant, expertly labeled (annotated) training data is the key factor to successful CP solutions

• With some exception, the “gold standard” for labeling pathology images will continue to be the pathologist

• Diligence is required to ensure that our efforts represent ground truth

• Labeling of training data may shift to directly measures of clinical outcome such as response to treatment

Questions? Toby C. Cornish, M.D., Ph.DAssociate Professor of Pathology

University of Colorado School of Medicinetoby.cornish@ucdenver.edu