Do NFL Coaches Coach to Win?

An examination of the relation between in-game decision making in the NFL and a

coach’s job securityBy Mitch Levy

NFL Rules Background

• What are two point conversion attempts?

• What are fourth down attempts?

Theoretical vs. Actual Strategy

• Modeling suggests coaches should attempt these plays more often

• ~2.5 times as often for fourth down attempts*

• ~5 times as often for two point conversions*

*From model by footballcommentary.com

Theories about the Discrepancy

• Modeling error

• Loss aversion/delaying outcome

• Lack of knowledge

• Dislike of criticism

Hypotheses

• Coaches are aware, that their decisions are suboptimally conventional (but not necessarily the degree)

• Non-expected utility theory• Dislike of criticism plays a meaningful role• Stronger incentives to win produce better

decisions

Idea of Study

• Evaluate coaches with different degrees of job security and see how their decisions change

• Hypothesis: coaches with their jobs on their lines will endure more discomfort to win more

Details of Study

• Coaches of teams that made the playoffs the previous year will be deemed safe – all other coaches are unsafe

• Evaluate conformity to suggestions of Markov model for “safe” vs. “unsafe” coaches

Some Possible Objections

• Playoff teams may have better conversion odds

• Playoff teams may have better coaches

• Playoff teams may have older, more conventional coaches

• Non-playoff teams may be worse, and pursue a high-variance strategy

Data Sources

• NFL Plays, 2002-2012 (Courtesy of Brian Burke, advancednflstats.com)

• Suggestions by model: nflcommentary.com, run by (NAME HERE)

• Model is a Markov Chain analysis

Two Point Attempts: Model Info

• Input variables: time remaining, score discrepancy

• Output variable: probability of success that justifies attempting the two point conversion, rather than kicking an extra point

All Two Point Attempts: Summary

• Total Two Point Attempts: 622

• Total Two Point Conversions: 319

• Overall Success Rate: 51.28%

• This rate used to determine which conversions should be attempted

2 Pt. Play, Safe vs. Unsafe Coaches

• Safe Coaches:– 159 Attempts, 720 Non-Attempts– Correct attempt rate: 18.1%

• Unsafe Coaches:– 331 Attempts, 1212 Non-Attempts– Correct attempt rate: 21.5%

“The Belichick Effect”

• Bill Belichick is an outlier, both in appearing in the playoffs 10/11 times in the sample (no other coach appears more than 4 times) and in his aggressiveness.

• The numbers, controlling for Belichick, are even more compelling

2 Pt. Play, Safe vs. Unsafe Coaches, Controlling for the Belichick effect

• Safe Coaches:– 141 Attempts, 688 Non-Attempts– Correct attempt rate: 17.0%

• Unsafe Coaches:– 328 Attempts, 1205 Non-Attempts– Correct attempt rate: 21.4%

Statistical Significance

• Without Belichick Effect?

• With Belichick Effect?

Fourth Down Attempts: Model Info

• Input variables: time remaining, score discrepancy, position on the field

• Output variable: probability of success that justifies attempting the fourth down play, rather than punting or kicking a field goal

Fourth Down Attempts• Conversion success on 4th and ...• 1 : 0.664885088188, 2 : 0.584415584416, 3 :

0.555831265509• 4 : 0.544668587896, 5 : 0.493710691824, 6 :

0.527659574468• 7 : 0.492462311558, 8 : 0.436090225564, 9 :

0.485148514851• 10 : 0.474137931034, 11 : 0.511904761905, 12 :

0.426666666667• 13 : 0.338461538462, 14 : 0.388888888889, 15 :

0.446153846154

Determining whether a coach “should” attempt a 4th down play

• If 4th and 1, use determined value of 0.665

• Otherwise, use 0.61 – 0.015*yds_to_go

4th Down, Safe vs. Unsafe Coaches

• Safe Coaches:– 910 Attempts, 1437 Non-Attempts– Correct attempt rate: 38.8%

• Unsafe Coaches:– 1768 Attempts, 2708 Non-Attempts– Correct attempt rate: 39.5%

4th Down, Safe vs. Unsafe Coaches (Controlling for the Belichick effect)

• Safe Coaches:– 859 Attempts, 1377 Non-Attempts– Correct attempt rate: 38.4%

• Unsafe Coaches:– 1755 Attempts, 2692 Non-Attempts– Correct attempt rate: 39.5%

Statistical Significance

• Without Belichick Effect?

• With Belichick Effect?

Implications

Controlling for Offensive Strength

• Calculate separate conversion probabilities for playoff, non-playoff teams

• Same methodology: separate 4th and 1 probability, linear regression on 4th and 2 through 4th and 15

Controlling for Offensive Strengthon 4th Down Attempts

• Safe Coaches:– 882 Attempts, 1429 Non-Attempts– Correct attempt rate: 38.1%

• Unsafe Coaches:– 1755 Attempts, 2692 Non-Attempts– Correct attempt rate: 39.5%

Model