1

Statistics and Image Evaluation

Oleh Tretiak

Medical Imaging Systems

Fall, 2002

2

Which Image Is Better?

3

Which Image is Better?

4

Overview

• Why measure image quality?• How to measure image quality• Statistical variation and probability theory• Some results in probability theory• Some results in statistics• Experimental design• Subjective quality measurement• ROC theory and estimation

5

Why Measure Image Quality

• Market studies (sell films)

• Market studies (sell equimpent)

• Test if equipment is working up to specification

• Measure effect of equipment on radiologists performance

• Measure the ability to perform diagnosis

6

Types of ‘Quality’

• Viewer preference– Relevant for entertainment, home viewing

• Technical quality– Process control (equipment maintenance)

• Utility– Ability to perform diagnosis, drive a remote

vehicle, locate enemy weapons

7

How to Measure Image Quality

• Viewer preference– Viewer trials

• Technical quality– Phantoms, resolution targets, expert viewers

• Utility– Viewer trials with expert viewers

8

Variation and Probability Theory

• Sources of variation– Input: Different patients are different– Equipment: Different equipment is different,

same equipment is different at different times– Subjective: Different viewers report different

opinions, same viewers report different findings at different times

9

Probability Theory

• Probability theory used almost universally

• Models– Independent trials– Dependency effects

• Same films viewed by different viewers

• Same patients imaged by different modalities

• Same patients viewed by several radiologists, multiple reading

10

Example

• Goal: evaluate quality of television set

• Method: Ask viewers to report quality of image, standard viewing conditions

• Viewers report a quality number 1-5– 1-Terrible, 2-Bad, 3-So-so, 4-Good, 5-

Excellent

• Evaluation: Compute average response

11

Probabilistic Model

• Reported values are iid random variables.– Set {1, 2, 3, 4, 5}– Probabilities p(1), p(2), p(3), p(4), p(5)– Probabilities are unknown!– Example: 100 observation, computed average

value is ?

12

Breakout

• Excel visq.xls

• Blackboard

13

Conclusions

• Results depend on who you ask

• Average result measured from a sample varies from sample to sample

• Prob. Theory tells us that with large samples, the average is equal to expected value

• Why does this matter?

14

Some Results from Prob. Theory

• Mean (Expected value)

• Variance, Standard Deviation

• Sample mean, sample variance

• Law of large numbers

• Central limit theorem

15

Some Results in Statistics

• Statistical problems– Estimation– Hypothesis testing– Confidence level

16

Estimate of the Mean

• For a Gaussian (Normal) random variable with known standard deviation,

• ~ is the confidence level €

Pr[X −1.96σ X ≤ μ ≤ X +1.96σ X ] = 0.95

€

μ =X ± zα /2

σn

17

Practical Issues

• Can use for non-normal distributions (CLT)

• If n is large and st. dev. Is not known, use sample st. dev.

• Small sample, unknown st. dev. — use the Student t statistic.

18

Experiment

Quality 5 Quality 1

19

Case A

20

Case B

21

More Statistics

• Estimate of variance– chi-squared (ki-squared)– Depends on number of observations (degrees of

freedom)

• Estimate ratio of independent normal variances – Fisher f distribution– Most important