Upload
abdel-salam-sayyad
View
910
Download
1
Tags:
Embed Size (px)
Citation preview
CS 573 – ADVANCED DATA MINING
SUBSTITUTE LECTURE 1/30/2013
ABDEL SALAM SAYYAD, Graduate Research Assistant
SPOT QUIZ
Before this class you should have watched:
1) Reddit.com interview with Peter Norvig.
2) Nate Silver at the Chicago Humanities Festival.
Please turn in:
1) Answers to spot quiz.
2) Answers to homework 2.
SPOT QUIZ
1. Where does Peter Norvig work?
2. What’s Peter Norvig’s favorite programming language?
3. Where does Nate Silver work?
4. What’s the title of Nate Silver’s book?
5. In predicting the outcome of the presidential elections in 2012,
how many states (out of 50) did Nate Silver predict correctly?
OUTLINE
• Peter Norvig lecture
• 5-minute break
• Hans Rosling on the beauty of data
• Review of Nate Silver’s book
PETER NORVIG - THE UNREASONABLE
EFFECTIVENESS OF DATA
Duration: 1 hour 3 minutes.
http://www.youtube.com/watch?v=yvDCzhbjYWs
HANS ROSLING: USING STATS TO DEBUNK MYTHS
ABOUT THE THIRD WORLD
Only the first 6 minutes.
http://www.youtube.com/watch?v=9GKy8nCFsJU
If you like to see more data come to life, read Nate Silver’s book!
NATE SILVER
• FiveThirtyEight.com
• On Time magazine’s 100
most influential list of 2009.
• Recent prediction:
The 49ers win the Super Bowl.
• The one before it:
This would be a Seahawks-Patriots Super Bowl…
• For fun, check out #DrunkNateSilver
THE SIGNAL AND THE NOISE: WHY SO MANY
PREDICTIONS FAIL – BUT SOME DON’T
• SPOILER ALERT:
– If you’ve watched the video, much has already been spoiled!
– I’ve got some more spoilers, but I also have teasers.
– Two thumbs up! Go read the book.
A VARIETY OF TOPICS
1. Financial Crisis of 2008
2. Election outcomes
3. Baseball
4. Weather forecasting
5. Earthquakes
6. GDP Growth
7. Spread of epidemics
8. Sports betting
9. Kasparov vs. Deep Blue
10. Poker
11. Stock Markets
12. Climate Change
13. Pearl Harbor and 9/11
ECONOMICS, POLITICS, SCIENCE, & SPORTS!
1. Financial Crisis of 2008
2. Election outcomes
3. Baseball
4. Weather forecasting
5. Earthquakes
6. GDP Growth
7. Spread of epidemics
8. Sports betting
9. Kasparov vs. Deep Blue
10. Poker
11. Stock Markets
12. Climate Change
13. Pearl Harbor and 9/11
SUCCESSES, FAILURES, & POLITICIZED!
1. Financial Crisis of 2008
2. Election outcomes
3. Baseball
4. Weather forecasting
5. Earthquakes
6. GDP Growth
7. Spread of epidemics
8. Sports betting
9. Kasparov vs. Deep Blue
10. Poker
11. Stock Markets
12. Climate Change
13. Pearl Harbor and 9/11
CH. 1 PREDICTING THE COLLAPSE OF THE HOUSING
MARKET IN 2008
• SPOILER: The financial companies claimed that they didn’t see it coming.
• TEASER: Were there economists who correctly predicted the collapse of the housing market? Was there a basis in the data for such prediction?
• LESSON: The problem with risk calculations for combined mortgages: the independence assumption.
• LESSON: Never predict for a situation you’ve never had based on many situation you’ve had before, i.e. when you’re out of sample.
CH. 2 POLITICAL FORECASTING
• SPOILER: Predictions made by the McLaughlin Group are not so good after all.
• TEASER: Two different approaches to prediction: the fox approach and the hedgehog approach.
• LESSON: Three useful principles for political prediction:– Principle I: Think Probabilistically
– Principle II: Today’s forecast is the First Forecast of the Rest of Your Life.
• “When the facts change, I change my mind. What do you do, sir?” –Keynes.
– Principle III: Look for consensus.
CH. 3 BASEBALL – SCOUTS VS. STATHEADS
• TEASER : Silver talks about his own
experience in baseball forecasting.
• TEASER : The movie “Moneyball” is
mentioned a lot.
CH. 4 WEATHER FORECASTING: A SUCCESS STORY
• SPOILER: Predicting the track of hurricane Katrina (2005) was a success story! (same for Sandy in 2012).
• TEASER: Why was there so much blame to spread around after Katrina?
• TEASER : Why is weather forecasting very successful?
• TEASER : The effect of marketing/customer satisfaction in commercial weather forecasting.
CH. 5 PREDICTING EARTHQUAKES
• SPOILER: A failure story with catastrophic results.
• TEASER: Some people were actually put on trial for failing to warn the public about a pending earthquake!
• LESSON: Overfitting is your enemy.*** This is the second most important lesson in the book***
– Recall: minute 16 of the CHF video.
CH. 6 HOW TO DROWN IN 3 FEET OF WATER
• SPOILER: Failure to communicate uncertainty, e.g. Grand Forks, ND in 1997.
– Recall: minute 13 of the CHF video.
• TEASER: How economists at Google can predict unemployment!
• LESSON: Beware of correlation without causation.
CH. 7 PREDICTING EPIDEMICS
• TEASER: Why did the flu predictions fail in 1976, and then again in 2009?
• LESSON: The danger of extrapolation.
• LESSON: Self-fulfilling predictions.
– e.g. The Iowa presidential primaries.
• LESSON: Self-canceling predictions.
– e.g. Predicting epidemics.
CH. 8 SPORTS BETTING & BAYES' THEOREM
(LESS AND LESS AND LESS WRONG)
• TEASER: A success story about sports better
Bob Voulgaris.
• LESSON: Bayes’ Theorem.
*** This is the most important lesson in the book***
• TEASER: The first example Silver provides is
about “the probability of spousal infidelity”.
CH. 8 BAYES' THEOREM
(LESS AND LESS AND LESS WRONG)
SPOILER:
CH. 8 BAYES' THEOREM
(LESS AND LESS AND LESS WRONG)
continued:
CH. 9, 10, 11, 12, 13
The remaining topics:
• Kasparov vs. Deep Blue
• Poker
• Stock Markets
• Climate Change
• Pearl Harbor and 9/11
– Teaser: Donald Rumsfeld is interviewed for this chapter