An Introduction to Things to Come!

Administrivia

• The details on the class can be found here: www.stanford.edu/class/hrp259

• Class participation plus lots of little assignments and 2 exams determines your final grade.

• If you send me a computer virus or other malicious code (even unintentionally) you will fail the course.– Visit ess.stanford.edu to get a virus scanner and keep

it up to date by clicking LiveUpdate….

• Dozens of new viruses are unleashed on the world every day. Please update your virus definitions and scan everything before you send it to me.

How to Predict The Future

• Either the world is driven completely by random chance events (and your best bet for predicting the future is using Tarot cards or a Magic 8 Ball™) or there are detectable patterns in the world.

• If you talk to a preschool teacher or a PhD in math, they will tell you that math is all about pattern detection.

Deterministic Modeling• To predict the future using a deterministic model,

you would say that if event A happens then B will happen. For example, in a deterministic world, if you know that a baby gestates for exactly 280 days (40 weeks), its weight will be exactly 7 ½ lbs (3.2 kg). We know that in reality it is typically in the range of 5.5–10 pounds (2.7–4.6 kg).

• For better or for worse, we do not live in a deterministic (Orwellian 1984ish) world and we can not usually make EXACT predictions in medicine.

What We Want….

• You want to be able to fill in a table like this:

…and express it with a simple formula like this:

lbs = weeks * something

Weeks of Gestation (in weeks)

Weight at birth (in lbs)

29 weeks

38 weeks39 weeks40 weeks

…

The Process

• The process of going from a single predictors or a set of predictors to a predicted outcome is called statistical modeling.

• People get far too excited about figuring out which statistic (with accompanying p-values anxiety) to use for the factors that are used in models.

• Today I want to talk about the process of building and using the most common models that you will see in medicine.

What is a Model and Why Care?

• The predictors and the outcomes can be on a continuous scale (time in days) or categorical factors (mom smoked, yes or no).

• Generally we try to use all the information available when we make a prediction about the future.– The amount of blood ejected each time the heart

beats (continuous scale) as opposed to whether or not the heart is beating

– The number of cancer cells seen on a slide (or the presence or absence of malignant cells)

• The models we build are remarkably similar regardless of whether we have categorical or continuous outcomes.

The Structure of a Model

• All the models I learned in school were formulated at their core like this:

Outcome = baseline + predictor + predictor

• The math can get ugly very quickly depending on the properties of the outcome (continuous, count, categories) but the core idea is that these models are all using additive contributions from some predictors!

Baby’sWeight

Impact of time

Impact of being a smoker

Weeks * a number a number

some number

How many predictors?

• You will be faced with the HARD question of how many predictors you have.– Stop and think about the LEVELS of your predictors. If you have

100 births in a dataset and 10 variables, many of which are categorical, you can quickly find yourself making predictions based on one or two births.

• Male babies (50%) from smoking mothers (20%) where this is the mom’s 2nd or later birth (35%) = SMALL numbers.

• From100 in the sample, you end up with 5.25 births in this level.• Do you want to make generalizations to the WORLD based on 5

children?

• Are the factors correlated?• Are some cheap to measure?• Can you use some as proxies for others?

Models

• One of my favorite statistics books is Michael J. Crawley’s Statistical Computing. He says:

1. All models are wrong.

2. Some models are better than others.

3. The correct model can never be known with certainty.

4. The simpler the model, the better it is.

What Makes a Bad Model

• Predicts some outcomes poorly

• Is strongly influenced by a small number of data points

• Shows systematic patterns in how it fails to predict

Types of Statistical Modeling

• Think about the extremes of what you could use when making a model:

• Null model – use the mean of everyone• A theoretically minimally adequate model• Current model you have specified• A theoretical maximal model with every predictor• Saturated model

– Every predictor and every interaction– Have everyone predict themselves

What is an interaction?

• Sometimes factors in a model behave weird together. A mad scientist invented a love potion and she looked at happiness in the two genders and people on and off her drug. There was a huge increase in happiness if you were male and on drug but minimal impact of gender or drug by themselves.

Interaction Gender by Drugdrugged

F

0

5

10

15

20

25

30

35

Perc

ent

Mean 99.88

Standard Dev. 13.78

normal

Mean 103.7

Standard Dev. 16.70

M

50 60 70 80 90 100 110 120 130 140

0

5

10

15

20

25

30

35

Perc

ent

Mean 120.1

Standard Dev. 12.73

50 60 70 80 90 100 110 120 130 140

Mean 101.8

Standard Dev. 11.18

happiness

Interactions Again

• Imagine that are two factors that impact a man’s risk of prostate cancer, being Japanese and living in Japan when you are a pre-teen. If you are Japanese, your risk of prostate cancer is lower than for men of any other race. If you were raised in Japan, your risk is lower than for men raised in the USA.

• The interaction term would measure if there is extra protection for being both Japanese and having been raised in Japan.

Categorical Interaction

• IsJapanese = 0 or 1 (1= yes and 0 = no) • IsRaisedJapan = 0 or 1• IsJapanese * isRaisedJapan = ??

Probablity of Prostate Cancer

0

0.01

0.02

0.03

0.04

0.05

0.06

0.07

0.08

0.09

0.1

Born USA Born Japan

Pro

babl

ity a

nnua

lly

Not Japanese

Japanese

You can ALWAYS do better.

• You can keep adding more and more predictors to a model but the price is the loss of generalizability. Will you get another child who is exactly that weight, who’s mother smoked that much, of that ethnicity, that gender, whose father weighed that much, mother weighed that much, etc….?

• Modeling methods compare the models and frequently use criteria that penalize for extra factors (AIC criteria).

Goals

• I see modeling as having two goals.• Estimate parameters.

– How much weight gain occurs each week as a baby is developing?

• Estimate how well it describes your data.– How far off will my guess be when I predict the next

child?– Are there regions where my guesses are far off, like

premature or late deliveries?– Is there a lot of variability at one point and not at

others?– Can I see any problems when I fit the model to THIS

data?

Looking for Errors

• Statisticians use the word “error” differently than everyone else.– You know that you will not have perfect

prediction. Instead, you will be off. That is error. It does not mean somebody made a mistake! It just means you can’t make a perfect prediction.

– Specifying how far you will be off is the fun and interesting part of statistics. The rest is just math.

Looking at Errors

Outcome = baseline + predictor + predictor + error

Baby Weight Impact of time

Impact being a smoker

Weeks * a number a number

some number

a numberdrawnfrom a

bell shaped distribution

Looking for Errors

• Hopefully you will see that, given any specific predictor value, your guessed values for the outcome will be close to the values you actually observe in the outcome. Also, any observed outcome values that stray too far from your guess are unlikely.

• That pattern of how far off your guesses are from your observed data can frequently be described by a bell-shaped (“normal”) histogram. So, if you measure errors between your prediction and the observed outcomes, the distribution should be “normal.”

Guesses and Errors

I guessed way too high rarely

I guessed way too low rarely

My model guesses7.5 lbs

9.5 lbs5.5 lbs

Histogram of actual weights at 40 week births

Histogram of errors at 40 weeks

Most errors are off by just a bit

0 error if child was 7.5 lbs

Looking at Errors

• There are some kinds of errors that you will be unwilling to accept.

• If I want to predict the number of times an evil lackey proposes marriage to a mad scientist, I will not accept a negative number!

• If I am predicting the chance of someone developing cancer, I will not accept a number less than 0% or greater than 100%.

• Specifying the type of errors is a critical part of building a model.

More on Errors

• In addition to specifying the range of legal values, another critical component is specifying the variability in the errors.

• If your data is constrained to lie between 0 and 1, what is the variability like?– If the average is about .5, then you can have scores

that are both above and below that and the variability drawn as a histogram may be well described by a bell shaped curve. What is the variability like if the average is about .95? Now the whole right side of your curve is hanging off the side of your page!

• If you have count data (e.g., number of cancer cells), your variability increases with the average count.

0 1.5

0 1.73

0 1.9

0 5000 10000 15000 20000

05

00

01

00

00

15

00

02

00

00

mean

vari

an

ce

If you pretend your variability is normally distributed but your outcome has a limited range, you clearly have problems.

In theory, the variance of count data increases with the mean.

Ordinary Least Squares

• Perhaps the easiest models to draw and understand are ones where you have a continuous outcome like weight and a continuous predictor like time.

• The model is just a line….

• Y = mX + bWeight = estimated weight gain each week after conception * number

of weeks + weight at 0 weeks

30 35 40

10

00

20

00

30

00

40

00

50

00

GWKS_DEL

FE

TA

L_

WG

T_

Maximum Likelihood Visual

Bad Models

• All models are wrong.

• Your data is sacred (after you remove the pregnant men) and you fit models to the data. You do not fit data to a model. That difference is not a semantic minor detail.

Poor Predictions

• Sometimes you have data points that are not well fit by the model. Go to extreme measures to document those points. If the data is not a true error, then run the analysis with it and without it. Include the point(s) in all your plots with a special symbol and if one person changes your inferences, consider excluding them. – You may have different subgroups that you

have not identified yet.

A True Outlier

30 35 40

10

00

20

00

30

00

40

00

50

00

GWKS_DEL

FE

TA

L_

WG

T_

Induced because of HUGE size

Looking at Residuals

• A critical step in examining the quality of a model is graphically looking at the residuals.

• Residuals are the differences between the estimated values and the observed values for each person/critter/observation.

• Look for curves, changing variability across the range of values or changes over time.

Patterns in Residuals

From Crawley: Statistical Computing

Partial Residuals

• When you have multiple predictors in a model, you can ask for residuals where you have “controlled for” or “removed the effect of” the other factors.

• Evaluate software packages on their ability to produce these graphics. They are completely missing from Excel, they are not built into SAS (but can be done with minor work), but they are trivially easily with R or S-plus.

Curve Fitting

• Linear models can model curves– The math is not too bad….

• You can use explicit mathematical formulas. If you see curves in your residuals, you can use things like:– Polynomials or inverse polynomials– Exponentials– Power functions

Keywords in Parametric Modeling and What They Look Like

S-shaped curve

From Crawley Statistical Computing

Nonlinear Regression

• Often the formulas to describe your data are extraordinarily complicated and you want to use non-linear or non-parametric modeling instead.

• Key words you will see include:– Non-parametric smoothing

• Lowess regression• Spine regression

– GAM– Tree models

A Bad Fit

• What happens when you fit a straight linear model to curvilinear data?

0 10 20 30 40 50

02

04

06

08

01

00

12

01

40

X = Age

Y=

Siz

e

0 10 20 30 40 50

02

04

06

08

01

00

12

01

40

X = Age

Y=

Siz

e

Is this better than a flat line at the mean?

residual

Is it good?

• A tiny p-value does not mean a good model!

• Where on the output does it tell that this is a good or a poor model?

Residuals?

0 10 20 30 40 50

02

04

06

08

01

00

12

01

40

X = Age

Y=

Siz

e

Flatten the line, then look up and down to see if you are systematically off.

Curve Fitting!

• You can build a model that has a curve using a polynomial… the degree of the polynomial determines how many “bends” appear in a curve. So a 2nd degree polynomial would use x and x2 while a 3rd degree polynomial would use x and x2 and x3. These squared or cubed values don’t do anything especially complicated. They are just like adding new variables.

0 10 20 30 40 50

02

04

06

08

01

00

12

01

40

X = Age

Y=

Siz

e

Polynomials

size = intercept + X * something + X2*something else

0 10 20 30 40 50

02

04

06

08

01

00

12

01

40

X = Age

Y=

Siz

e

size = intercept + X * something + X2* something else +X3 * another thing

poly2 = lm(y~poly(x,2)) poly3 = lm(y~poly(x,3))

What is a good fit?

• Choosing where to stop adding terms to a model is as much an art as a science. You can do comparisons between the models and ask to see if it is a statistically significant difference.

• There are systems for penalizing your model as you add more and more factors to a model like AIC.

Generalized Linear Models

• You will eventually move out of the realm of predicting continuous outcomes with normal error. When you do, you will move into the realm of Generalized Linear Models (GLM).

• You want to have a linear model predicting an outcome where you restrict the possible outcome values (e.g., only allow values between 0 and 1) and deal with errors not being consistently normal across the entire range.

• You can change (transform) your outcome and model this with just another linear model similar to what I have shown.

GLM in English

• If you are predicting the number of bacteria you see in a Petri dish, you can not possibly see a negative number of bacteria. A GLM model can be written so that your predicted values can not be negative.

• Contrast this with the baby weight example where with a bit of bad data for your predictor value, you could have the formula spit out a negative weight or a baby weighing a ton.

GLM

• Instead of modeling like this:Outcome =

baseline + predictor + predictor + error

• You can model with GLM like this:Tweaked outcome =

baseline + predictor + predictor + not normal error

normal/bell-shaped

Ordinary Regression

• So, the ordinary least squares regression models that I have shown are really just a case of GLM. In these cases I specify that the tweak to the outcome is to just make the outcome identical to what it was originally and the error is normal.

• The tweak to the outcome is called the link and this case the link is called identity.

Link Functions

• The tweaks to the outcome are called links:

• Identity link = predicting a continuous outcome (baby weight)

• Log link = if you can’t have negative values

• Logit link = if you have to restrict the range to between 0 and 1

• There are other links.

Error Structure

• Why bother to specify an error structure other than normal?– Strong skew, kurtosis errors, bounded errors, negative counts

• The shape of the error distribution is not a bell-shaped curve. Rather than worrying about the math to describe those curves, you simply need to know that different types of data have different error structures.– Normal errors – continuous outcomes– Poisson errors - counts– Binomial errors - proportions– Gamma errors - variation

Binary Response

• If you are not dealing with a continuous outcome, or count data, you will likely have a binary (yes/no scored as 1 or 0) outcome.

• Clearly you need to do some major tweaking to the outcome because linear models, as we have seen, can predict very large and small numbers.

• Also, the variability of a binary outcome is very different from a continuous variable.

Logistic Regression

• The solution is to specify a link that limits values to be between 0 and 1 (think of the changed outcome as being the probability of being scored 1) and use an error term that behaves well with binary outcomes.

• This is a GLM with a logit link and binomal errors.

• This kind of analysis is so popular most people don’t know it is a GLM. Rather, they know it only as logistic regression.

Before Next Time

• The first assignment is on the class website and is due before the start of class Wednesday.

• Print the slides as close as possible to class time.