Translational Gastroenterology Unit John Radcliffe ... · Translational Gastroenterology Unit John Radcliffe Hospital and Linacre College, ... Comparing disease activity indices for

UCEIS in Practice

Date of preparation September 2013 AXHUG130882an

Simon Travis Translational Gastroenterology Unit John Radcliffe Hospital and Linacre College, University of Oxford, UK Chris A Bernhardt Bernhardt Regulatory Consulting, Ohio, USA

Discussion agenda

Background

What is the UCEIS?

Why was it developed?

How was it developed?

What does it offer?

Putting it into practice

Review of process

Making the UCEIS available

Feedback & comments re: plans/approach


Interobserver variation in histopathology

• 25 histopathologists

• Examined biopsies from 60 cases, two rounds, rectal and full colonic

series

What is the UCEIS?

UCEIS Ulcerative Colitis Endoscopic Index of Severity

Properties

Reliable (intra & inter-investigator performance)

Accounts for 88% of variance in severity

Simple (sum of 3 descriptors)


Why was it developed?

Endoscopic severity assessment important

Diagnostic tool

Clinical trial decision making – Qualification for enrollment

– Outcome assessment

Clinical treatment decision

Variability of classic methods (Baron)

Imprecise


Your evaluation of the endoscopic severity – just as you would do in your daily practice


1. Travis SPL, et al. Gut 2012;61:535–42

1. Remission

2. Mild

3. Moderate

4. Severe


1. Travis SPL, et al. Gut 2012;61:535–42

Question: Would you score this

Travis SPL, et al. Gut 2012;61:535-42. Copyright © BMJ Publishing Group Ltd & British Society of Gastroenterology. All rights reserved.

Distribution of Baron score among specialists in the phase 1 panel as a function of the level assigned by a central reader


0

20

40

60

80

100

Ass

ign

ed

rat

ing

by

ph

ase

1 p

ane

l (%

)

0 1 2 3 Rating of central reader

(40/6) (30/5) (57/8) (29/5) n/s =

Baron 3

Baron 2

Baron 1

Baron 0

27%

53%

20%

7%

70%

16%

7%

12%

23%

37%

28%

7%

17%

76%

How do indices compare?


Endoscopic activity indices for IBD

Ulcerative colitis

1. Matts

2. Baron score

3. Modified Baron endoscopy score

4. Mayo Clinic endoscopy subscore

5. Modified Mayo Clinic endoscopy subscore

6. Sutherland UCDAI mucosal appearance

7. Powell Tuck Index

8. Blackstone Endoscopic interpretation

9. Rachmilewitz CAI index

10. Endoscopy Activity Index

11. Ulcerative Colitis Endoscopic Index of

Severity (UCEIS)

12. Ulcerative Colitis Colonoscopic Index of

Severity (UCCIS)

Crohn’s disease

1. CDEIS

2. Rutgeerts’ post-operative index

3. SES-CD

4. Absence of ulcers

So CD should be simple!

D’Haens & Sandborn et al Gastroenterology 2007;132:763-86

Palmer et al Assessing activity in UC in Lichtenstein et al Springer 2013 (in press)

Descriptions matter

Baron score Baron et al. BMJ 1964;1:89–92

Remission 0 Mild 1 Moderate 2 Severe 3 Normal: matt mucosa,

ramifying vascular

pattern clearly visible

throughout, no

spontaneous bleeding, no

bleeding to light touch

Abnormal but not

haemorrhagic:

appearances

between (0) and

(2)

Moderately

haemorrhagic: bleeding

to light touch, but no

spontaneous bleeding

seen ahead of instrument

on initial inspection

Severely

haemorrhagic:


seen ahead of

instrument at initial

inspection and bleeds

to light touch

Mayo Clinic Endoscopy subscore Kamm et al. Gut 2008;57:893-902

0 1 2 3

Normal Erythema, decreased

vascular pattern and

minimal granularity

Marked erythema,

friability, granularity,

absent vascular pattern,

bleeding with minimal

trauma and no ulcerations

Ulceration and


Baron does not describe ulceration; Mayo combines terms; neither define terms

How do indices compare?

100 patients with UC prospectively evaluated in the Oxford IBD clinic, each by four specialists, followed by videosigmoidoscopy, later scored by each specialist.

Results – Inter-observer agreement for SCCAI (k= 0.75, 95% CI 0.70-0.81)

– Mayo Clinic index (k = 0.72, 95% CI 0.67-0.78)

– Seo index (k = 0.89, 95% CI 0.83-0.95)

– Endoscopy in the Mayo Clinic index had the greatest variation (k = 0.38)

Inter-observer variation alone would have excluded up to 1 in 5 patients from recruitment or remission criteria in representative trials.

Comparing disease activity indices for ulcerative colitis. Walsh AJ et al J Crohns Colitis (2013 in press)


Development of the UCEIS

Wide inter-observer variation in endoscopic assessment – 76% agreement for ‘severe’ and 27% agreement for ‘normal’

endoscopic mucosal appearances between 10 experienced investigators and a central reader

– 30 investigators then rated 25/60 different videos for 10 descriptors and assessed overall severity on a 0-100 visual analogue scale

– Kappa statistics tested inter- and intra-observer variability for each descriptor (0.34-0.65 and 0.30-0.45 within and between observers for the 10 descriptors) Different models to predict the overall assessment of severity (visual analogue scale) used general linear mixed regression

– The final model incorporated 3 descriptors, each with precise definitions


Descriptive terms for UC

Descriptor

Vascular pattern

Bleeding

Erosions and ulcers

Extent of erosions & ulcers

Incidental friability

Mucopus

Mucosal surface

Mucosal surface

Mucosal erythema

Contact Friability Test

Transition to normal mucosa

0.00 means no agreement beyond chance, 1.00=perfect; Weighted kappa puts value on

scores from different readers being close (eg when reader 1 says 4, reader 2 says 3)

Between endoscopists:

Weighted kappa

0.42

0.37

0.45

0.42

0.40

0.40

0.34

0.34

0.35

-

-

Descriptive terms for UC

Descriptor

Vascular pattern

Bleeding

Erosions and ulcers

Extent of erosions & ulcers

Incidental friability

Mucopus

Mucosal surface

Mucosal surface

Mucosal erythema

Contact Friability Test

Transition to normal mucosa

Descriptor

(score most severe

lesions)

Likert scale

anchor points

Vascular pattern

Normal (0)

Patchy loss (1)

Obliterated (2)

Bleeding

None (0)

Mucosal (1)

Luminal mild (2)

Luminal severe (3)

Erosions and

ulcers

None (0)

Erosions (1)

Superficial ulcer (2)

Deep ulcer (3)

The three terms in combination account for 88%

of variance between observers and 96% of the

scale of the full range of severity on VAS

Travis et al Gut 2012;61:935-42

Travis et al Gastroenterology 2013 (in press)

Gut 2012; 61:535-42

Development of the UCEIS

Followed published scientific steps for scale development

Defined descriptors of severity (Phase 1)

Constructed scale (Phase 2) – Minimum number of descriptors to account for 90% of variability

Validated scale (Phase 3)

Assess symptom knowledge on UCEIS (Phase 4)


Phase 3 Correlation coefficients Partial Correlation Coefficients (adjusted for investigators) n=548

Variable Vascular

Pattern

Bleeding Erosions &

Ulcers

UCEIS

(unW)

UCEIS

Weighted

VAS

Vascular

Pattern

0.684

(<0.0001)

0.728

(<0.0001)

0.848

(<0.0001)

0.872

(<0.0001)

0.808

(<0.0001)

Bleeding 0.712

(<0.0001)

0.891

(<0.0001)

0.881

(<0.0001)

0.818

(<0.0001)

Erosions &

Ulcers

0.912

(<0.0001)

0.903

(<0.0001)

0.889

(<0.0001)

UCEIS

(unW)

0.999

(<0.0001)

0.937

(<0.0001)

UCEIS

Weighted

0.936

(<0.0001)

VAS

R2 = (0.936 squared): accounts for 88% of the variance in overall assessment of severity

between observers

Validation framework

Face validity Your expertise, Literature (1,2)

– Items ask about ‘right sort of things’

Content validity Other Scales, FA, VAS (2,3) – Items cover (all) the important aspects

Construct validity Mayo Stratum, VAS (2,3) – Sensible relationships with other variables

Internal consistency Cronbachs Alpha (3) – Items sufficiently related to each other

Reproducible Kappas, Inter-Investigator ICC (3)

– Different observers agree

Repeatable Kappas, Intra-Investigator ICC (3) – Same observer agrees on different occasions


What does it offer?

Reliable across range of severity

Known inter- and intra-investigator statistics – ICC & RR Statistics

– Kappa

Simplicity with documented process


ICC, Intraclass Correlation Coefficient; RR, relative risk

Mean assessment of overall severity as a function of its rank among all mean evaluations of severity, based on 750 evaluations performed by 30 investigators on 25 out of 60 videos


Travis SPL, et al. Gut 2012;61:535-42. Copyright © BMJ Publishing Group Ltd & British Society of Gastroenterology. All rights reserved.

0

10

20

30

40

50

60

70

80

90

100

Rank-order based on mean severity evaluation

Me

an s

eve

rity

eva

luat

ion

0 10 20 30 40 50 60

Normal Mayo 0 Mayo 1–2 Mayo 3–5 Mayo 6–7 Mayo 8–9 Mayo 10–11 Severe

Mayo stratum

Impact of knowledge of clinical details on the mean UCEIS score

25 investigators evaluated 28 videos selected from a library of 40. There was 1 video (in the Mayo 0 stratum) for which the UCEIS score was significantly different when clinical details were supplied (p=0.021)


Travis SPL, et al. Gastroenterology 2013 (epub 24 July).

0

Rank-order based on mean UCEIS evaluation

Me

an U

CEI

S ev

alu

atio

n

1

1

2

3

4

5

6

7

8

3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39

Blinded With Sx

Reading status

Inter- and intra-investigator agreement on UCEIS and VAS (ICC, RR)

Inter-investigator

VAS ICC = 0.78

UCEIS ICC = 0.88

UCEISw ICC = 0.88

Intra-investigator

VAS ICC = 0.87

UCEIS ICC = 0.96

UCEISw ICC = 0.96


Travis SPL, et al. Gastroenterology 2013 (epub 24 July).

Inter- and intra-investigator agreement on descriptor responses

Inter-investigator ** Non-repeat evaluations

VP κ=0.54 (0.50, 0.57) M

B κ=0.48 (0.46, 0.50) M

E&U κ=0.53 (0.51, 0.57) M

Intra-investigator ** Repeat evaluations

VP κ=0.87 (0.74, 1.00) V

B κ=0.47 (0.27, 0.67) M

E&U κ=0.81 (0.67, 0.94) V


VP: Vascular Pattern; B: Mucosal bleeding; E&U: Erosions & Ulcerations Travis SPL, et al. Gastroenterology 2013 (epub 24 July).

References

Development of an ulcerative colitis endoscopic index of severity (UCEIS). Travis SPL, Schnell D, Krzeski P, Abreu MT, Altman DG, Colombel JF, Feagan BG, Hanauer SB, Lémann M, Lichtenstein GR, Marteau PR, Reinisch W, Sands BE, Yacyshyn BR, Bernhardt CA, Mary JY, Sandborn WJ.

Gut 2012;61:535-42.

Reliability and Initial Validation of the Ulcerative Colitis Endoscopic Index of Severity (UCEIS). Travis SPL, Schnell D, Krzeski P, Abreu MT, Altman DG, Colombel JF, Feagan BG, Hanauer SB, Lichtenstein GR, Marteau PR, Reinisch W, Sands BE, Yacyshyn BR, Schnell P, Bernhardt CA, Mary JY, Sandborn WJ.

Gastroenterology 2013 (epub 24 July)


UCEIS

Simple three descriptive terms

Vascular pattern (3-point Scale)

Bleeding (4-point Scale)

Erosions & ulcerations (4-point Scale)

Scores the most severely affected area at flexible sigmoidoscopy

Uses uniform training (exposure) to terms

Step 1: Introduces terms with examples

Step 2: Confirms understanding – Blinded evaluation of sample videos


GRADE DEFINITION PHOTO

Normal 0 Normal vascular pattern with arborisation of capillaries clearly defined

Patchy loss 1 Patchy loss or blurring of vascular pattern

Obliterated 2 Complete loss of vascular pattern

Vascular pattern reference


Images courtesy of C Bernhardt Adapted from Travis SPL, et al. Gut 2012;61:535-42


None 0 No visible blood See normal under vascular pattern

Mucosal 1 Some spots or streaks of coagulated blood on the surface of the mucosa ahead of the scope, which can be washed away

Luminal mild 2 Some free liquid blood in the lumen

Luminal severe 3 Frank blood in lumen ahead of endoscope or visible oozing from mucosa

Bleeding reference


Images courtesy of C Bernhardt Adapted from Travis SPL, et al. Gut 2012;61:535-42


None 0 Normal mucosa, no visible erosions, ulcers

See normal under vascular pattern

Erosions 1 Tiny (5 mm) defects in the mucosa, of a white or yellow colour with a flat edge

Superficial ulcer 2 Larger (>5 mm) defects in the mucosa, which are discrete fibrin-covered ulcers when compared to erosions, but remain superficial

Deep ulcer 3 Deeper excavated defects in the mucosa, with a slightly raised edge

Erosions and ulcerations reference


Images courtesy of C Bernhardt

Sco

re m

ost

se

vere

lesi

on

s

Adapted from Travis SPL, et al. Gut 2012;61:535-42

UCEIS© 2013

Descriptor (score most severe lesions)

Likert scale anchor points

Definition

Vascular pattern

Normal (0) Normal vascular pattern with arborisation of capillaries clearly defined, or with blurring or patchy loss of capillary margins

Patchy loss (1) Patchy obliteration of vascular pattern

Obliterated (2) Complete obliteration of vascular pattern

Bleeding

None (0) No visible blood

Mucosal (1) Some spots or streaks of coagulated blood on the surface of the mucosa ahead of the scope, which can be washed away

Luminal mild (2) Some free liquid blood in the lumen

Luminal severe (3) Frank blood in the lumen ahead of endoscope or visible oozing from mucosa after washing intra-luminal blood, or visible oozing from a haemorrhagic mucosa

Erosions and ulcers

None (0) Normal mucosa, no visible erosions or ulcers

Erosions (1) Tiny (<5mm) defects in the mucosa, of a white or yellow colour with a flat edge

Superficial ulcer (2) Larger (>5mm) defects in the mucosa, which are discrete fibrin-covered ulcers when compared to erosions, but remain superficial

Deep ulcer (3) Deeper excavated defects in the mucosa, with a slightly raised edge

Score the worst affected area at sigmoidoscopy, especially on insertion Date of preparation September 2013 AXHUG130882an Travis SPL, et al. Gastroenterology 2013 (in press).

Putting the UCEIS into practice

2 steps needed

Training

Review of definitions and examples (9 video clips)

Verification of understanding (qualification)

Rate 4 standardized videos (Stage 2 agreement)

Qualification criteria – VP and bleeding – within 1 unit of Stage 2 mean – E&U – match Stage 2 mean

Re-qualify if needed – Evaluate 3 videos of ‘missed term’

Score the worst affected area

Evaluate on insertion (esp. bleeding)


Training example: UCEIS = 0 (normal)


VP training example


Images courtesy of S Travis

E&U training example



Importance of 2 steps

Design confirmed understanding of terms

Stage 3 and 4

UCEIS characteristics reflect design

About 50% qualified on 1st attempt

About 45% qualified on 2nd attempt

About 5% did not participate


Qualification example



Question: Score this

Total score (most

severe area)

0

1

2

3

4

5

6

7

8


Descriptor (score most severe lesions)

Likert scale anchor points

Definition

Vascular pattern

Normal (0) Normal vascular pattern with arborisation of capillaries clearly defined, or with blurring or patchy loss of capillary margins

Patchy loss (1) Patchy obliteration of vascular pattern

Obliterated (2) Complete obliteration of vascular pattern

Bleeding

None (0) No visible blood

Mucosal (1) Some spots or streaks of coagulated blood on the surface of the mucosa ahead of the scope, which can be washed away

Luminal mild (2) Some free liquid blood in the lumen

Luminal severe (3) Frank blood in the lumen ahead of endoscope or visible oozing from mucosa after washing intra-luminal blood, or visible oozing from a haemorrhagic mucosa

Erosions and ulcers

None (0) Normal mucosa, no visible erosions or ulcers

Erosions (1) Tiny (<5mm) defects in the mucosa, of a white or yellow colour with a flat edge

Superficial ulcer (2) Larger (>5mm) defects in the mucosa, which are discrete fibrin-covered ulcers when compared to erosions, but remain superficial

Deep ulcer (3) Deeper excavated defects in the mucosa, with a slightly raised edge

Making UCEIS available

Target user groups

Clinical trial users

– Central readers

– Investigators

Clinical practice

– Group practice

– Individual

Endoscopy trainees

Validate in Asia

‘Copyright’

Refers only to the wording of the descriptive terms


Vision

Make UCEIS readily available for broad use

Internet access for professionals – Europe

– Asia

–North America

– South America

No cost to user

Publication acknowledges copyright of UCEIS – “© 2013 Warner Chilcott. All Rights Reserved. The UCEIS Index

was developed by the UCEIS Study Group”


Concluding remarks

Steering Committee

Co-Chairs: Drs. Travis and Sandborn

Members: Drs. Dan Schnell, Piotr Krzeski, Maria Abreu, Doug Altman, Jean-Frédéric Colombel, Brian G Feagan, Stephen B Hanauer, Gary R Lichtenstein, Philippe Marteau, Walter Reinisch, Bruce E Sands, Bruce R Yacyshyn, Jean-Yves Mary, and Chris Bernhardt


Documents

Translational Gastroenterology Unit John Radcliffe ... · Translational Gastroenterology Unit John Radcliffe Hospital and Linacre College, ... Comparing disease activity indices for