The Improbability Principle Returns Luck, Lotteries, and Laura · The Improbability Principle Returns ‐ Luck, Lotteries, and Laura ... Alex and Donna Voutsinas, looking at old family

RSS Statistical fallacies meeting 1

The Improbability Principle Returns ‐

Luck, Lotteries, and Laura

David J. Hand Imperial College, London

and Winton Capital Management

May 2015


Émile Borel: Borel measure

Borel‐Cantelli lemma

Heine‐Borel theorem

. . . . .


Borel’s law: sufficiently improbable events are impossible

Nonsense?


Borel’s law: sufficiently improbable events are impossible

Nonsense? What he meant was, we should behave as if such events are impossible:

If an event is so improbable that you’d expect it to occur just once in the history of the universe, then it would be irrational to behave as if it would happen

But nonetheless people do see extraordinarily improbable events happen


‐ John Ironmonger: The Coincidence Authority

‐ about a London‐based professor who studies coincidences ‐ whose birth day is 30th June, my birthday ‐ and the female protagonist works at UCL, where my wife works

‐ King James Bible was published in the year that Shakespeare turned 46. The 46th word from the start of Psalm 46 is "shake" and the 46th word from the end is "spear"

‐ Shakespeare and Cervantes both died on 23 April 1616 ….?


The RSS and ASA 104th presidents each held office in the same year RSS Presidents (normally) serve 2 years ASA Presidents (normally) serve 1 year The RSS started in 1834 The ASA started in 1839


Edward Jarvis served 30 years as ASA president: 1852‐1882 Other early presidents served more than 1 year as ASA pres.


Seeking to understand these apparently impossible events, people have come up with all sorts of bizarre explanations:

‐ new physical forces (synchronicity)

‐ clairvoyance

‐ intuition

‐ supernatural beings

‐ morphic resonance

‐ etc


But none of these things are necessary The highly improbable events can be explained in perfectly sound mathematical terms In fact, the improbability principle says

extremely improbable events are commonplace and why


The improbability principle An intertwining of five mathematical laws and the human mind


1. The law of inevitability One of the complete set of all possible outcomes must occur How to guarantee a lottery win More subtle variants: the pigeonhole principle: if n items are put into m containers, with n > m, then at least one container must contain more than one item


2. The law of truly large numbers

If a highly improbable event has enough opportunities to happen, it becomes almost certain

The Silverton family: 4 generations, 16 male offspring, no female

216 = 65,536 7 bn people: track forward for 16 births for each: about 100,000 such families

Alex and Donna Voutsinas, looking at old family photos a week before getting married, come across a picture of 5 year old Donna at DisneyWorld In the background is Alex, in a stroller


3. The law of selection

Changing what you choose from (the reference class) can dramatically alter probabilities

The map at a railway station saying “You are here”

Penis enlargement emails

Stephen Jay Gould

More subtle variants: regression to the mean:


4. The law of the probability lever Changing the circumstances a little can have a huge impact on probabilities The Sally Clark case and the independence assumption Gilbert and Sullivan opera The Yeomen of the Guard, Act 2: Fairfax: ‘And didst thou see all this?’ Point: ‘Aye, with both eyes ‐ this and that. The testimony of one eye is naught ‐ he may lie. But when it is corroborated by the other, it is good evidence that none may gainsay’


5. The law of near enough

Relax your criteria for a match and you increase the chance of a match

What’s a lottery “win”? “A man is celebrating winning the lottery twice in four months by matching the same five numbers and the bonus ball. Mike McDermott's initial win of £194,501 on the Lotto was followed by another £121,157 last weekend.” Telegraph 12 Oct, 2002

“we take wins at face value but come up with excuses to think of almost wins as actual wins or should‐have‐been wins” Matthew Hutson


Combinations of the laws

The classic case of Laura Buxton Laura Buxton, a ten year old girl from Stoke‐on‐Trent, released a balloon with a label attached, saying ‘ please return to Laura Buxton”, and gave her address on the other side It was blown 140 miles south west, landing in Milton Lilborne, where it was found by a ten year old girl named Laura Buxton

The two Laura Buxtons: ‐ are both fair haired ‐ are both in year 5 at school, ‐ both have three 3 year old female black labradors ‐ both have a rabbit ‐ and a guinea pig


Incredible coincidence ! To give some perspective, there are just 28 voters called Laura Buxton in the UK


But

The law of truly large numbers applies ‐ how many people release balloons each year?

What’s the chance that someone releases a balloon which is found by someone with the same name?

‐ 42 million people on the electoral register ‐ 12 million different names in the UK

And the law of near enough also applies:

‐ actually it was a neighbour of Laura 2 who found it. Was about to bin it when he saw the label, so delivered the balloon to Laura 2

‐ Laura 1 was only nearly 10


People notice and remember the matching characteristics Why would you notice and remember that the Lauras ‐ had different named parents ‐ had different age guinea pigs ‐ families had different types of car ‐ took holidays in different places ‐ fathers had different jobs ‐ had differently named friends ‐ . . . . . . ‐ and so on endlessly


And on top of this there is the human mind A collection of heuristics created by aeons of evolution To make quick rough judgements which are often correct But which also take consequences into account And so are vulnerable to false positives etc


Hindsight bias Self‐serving bias Overconfidence bias Base rate fallacy Confirmation bias Conjunction fallacy Gamblers’ fallacy The need for narrative – everything has a cause . . . . . .


Hindsight bias

A manifestation of the law of selection

“It is all too common for caterpillars to become butterflies and then to maintain that in their youth they had been little butterflies” George Valliant

“Life is lived forwards, but understood backwards” Kierkegaard

“After the event, of course, a signal is always crystal clear ... but before the event it is obscure and pregnant with conflicting meanings.” Leonard Mlodinow


“History is written by the victors”

“Hindsight is 20/20”

“Should have known”


Why? ‘Your act was unwise,’ I exclaimed, ‘as you see by the outcome.’ He solemnly eyed me. ‘When choosing the course of my action,’ said he, ‘I had not the outcome to guide me.’

Ambrose Bierce

Looking back, one can always find a sequence leading to the known now

‐ Recognise this in statistical modelling ?


World Trade Centre:

‐ 10 July 2001, CIA received intelligence that Al‐Qaeda might be planning an attack on the US

‐ numerous publications describing deliberately crashing planes into buildings

‐ unusually large numbers of shares in insurance companies and airlines sold before the attack


World Trade Centre:

‐ 10 July 2001, CIA received intelligence that Al‐Qaeda might be planning an attack on the US

‐ numerous publications describing deliberately crashing planes into buildings

‐ unusually large numbers of shares in insurance companies and airlines sold before the attack

But how many other strands of intelligence were there?


Pearl Harbour:

‐ Autumn 1941: US intercepted message asking a Japanese spy to divide Pearl Harbour into five areas, and summarise the ships in those areas;

‐ weeks later US monitors lost track of radio messages from Japanese fleets;

‐ December, Japanese change call signs for second time in a month (instead of every 6 months);

‐ two days later, messages to Japanese diplomats instructing them to destroy codes and secret documents;

‐ etc


Same issue as overfitting in statistics


Same issue as overfitting in statistics One can always fit a conspiracy theory to the data


Also

People have an inbuilt inclination to consistency: when asked about their earlier state of mind, they tend to reconstruct their current beliefs

After an event occurs, people give a higher probability to their earlier belief that the event would occur

One consequence is that people underestimate how much they are surprised by events

Self fulfilling prophecies


Also We choose which events to remember:

The German “Karicartoon” calendar, published by a small Berlin‐based company, gives a witty cartoon for each day of the year.

On Sunday 10 February 2011, it showed a drawing of the Pope sitting in front of the TV while the lottery numbers are announced.

On finding that he has all seven numbers correct, the caption shows him exclaiming: “Holy s**t! I’ll resign tomorrow!”


Also We choose which events to remember:

The German “Karicartoon” calendar, published by a small Berlin‐based company, gives a witty cartoon for each day of the year.

On Sunday 10 February 2011, it showed a drawing of the Pope sitting in front of the TV while the lottery numbers are announced.

On finding that he has all seven numbers correct, the caption shows him exclaiming: “Holy s**t! I’ll resign tomorrow!”

11 Feb 2013, Vatican announced that the Pope would resign


And which to emphasise: In The Old Curiosity Shop, Charles Dickens describes the conversation between the mothers of two characters, Tim and Barbara, who are meeting for the first time:


And which to emphasise: In The Old Curiosity Shop, Charles Dickens describes the conversation between the mothers of two characters, Tim and Barbara, who are meeting for the first time:

“And we are both widows too!” said Barbara’s mother. ‘We must have been made to know each other’....tracing things back from effects to causes, they naturally reverted to their deceased husbands, respecting whose lives, deaths, and burials, they compared notes, and discovered sundry circumstances that tallied with wonderful exactness; such as Barbara’s father having been exactly four years and ten months older than Kit’s father, and one of them having died on a Wednesday and the other on a Thursday, and both of them having been of a very fine make and remarkably good‐looking, with other extraordinary coincidences.”


Let’s take the famous Lincoln/Kennedy coincidence:

Both were assassinated in office ‐ Lincoln's name has 7 letters ‐ Kennedy's name has 7 letters ‐ In Lincoln's and Kennedy's names the vowels and consonants fall in exactly the same place; in the order c, v, c, c, v, c, c ‐ Lincoln was elected to Congress in 1846 ‐ Kennedy was elected to Congress in 1946 ‐ Lincoln was elected president in 1860 ‐ Kennedy was elected president in 1960


‐ Kennedy had a secretary named Lincoln ‐ War was thrust upon Lincoln almost immediately after inauguration ‐ War was thrust upon Kennedy almost immediately after inauguration ‐ Lincoln ordered the Treasury to print its own money ‐ Kennedy ordered the Treasury to print its own money ‐ Lincoln gave black Americans freedom and legalized equality ‐ Kennedy enforced equality for black Americans ‐ Lincoln delivered the Gettysburg Address on November 19, 1863 ‐ Kennedy was assassinated on November 22, 1963


‐ Lincoln was loved by the common people and hated by the establishment ‐ Kennedy was loved by the common people and hated by the establishment ‐ Lincoln was succeeded, after assassination, by vice‐president Johnson ‐ Kennedy was succeeded, after assassination, by vice‐president Johnson ‐ Andrew Johnson was born in 1808 ‐ Lyndon Johnson was born in 1908 ‐ Andrew Johnson's name has 13 letters ‐ Lyndon Johnson's name has 13 letters ‐ Andrew Johnson had a pug nose and slicked‐back hair ‐ Lyndon Johnson had a pug nose and slicked‐back hair


‐ Lincoln was sitting beside his wife when he was shot ‐ Kennedy was sitting beside his wife when he was shot ‐ Rathbone, who was with Lincoln when he was shot, was injured (by being stabbed) ‐ Connally, who was with Kennedy when he was shot, was injured (by being shot) ‐ Rathbone's name has 8 letters ‐ Connally's name has 8 letters ‐ Lincoln's wife held his head in her lap after he was shot ‐ Kennedy's wife held his head in her lap after he was shot ‐ Lincoln was shot on a Friday ‐ Kennedy was shot on a Friday


‐ Lincoln was shot in a theater named Ford ‐ Kennedy was shot in a car named Ford ‐ Kennedy was shot in a car also named Lincoln ‐ Lincoln's bodyguard was away from his post at the door of the President's box at the theater ‐ Kennedy's bodyguards were away from their posts on the running‐boards of the President's car ‐ Lincoln was shot in a theater and his assassin ran to a warehouse ‐ JFK was shot from a warehouse and his assassin ran to a theater ‐ Lincoln's assassin had a three‐word name, John Wilkes Booth ‐ Kennedy's assassin had a three‐word name, Lee Harvey Oswald


‐ John Wilkes Booth has 15 letters ‐ Lee Harvey Oswald has 15 letters ‐ John Wilkes Booth was born in 1839 (s/b 1838) ‐ Lee Harvey Oswald was born in 1939 ‐ Lincoln didn't die immediately after being shot ‐ Kennedy didn't die immediately after being shot ‐ Lincoln and Kennedy died in places beginning with the initials P and H ‐ Lincoln died in Petersen's house ‐ Kennedy died in Parkland Hospital ‐ Booth was shot and killed in police custody before going to trial ‐ Oswald was shot and killed in police custody before going to trial


‐ Andrew Johnson was a heavy drinker with crude behavior ‐ Lyndon Johnson was a heavy drinker with crude behavior ‐ There were conspiracy theories that Johnson was knowledgeable about Lincoln's assassination ‐ There were conspiracy theories that Johnson was knowledgeable about Kennedy's assassination ‐ Days before it happened Lincoln told his wife and friends about a dream he'd had of being shot by an assassin ‐ Hours before it happened Kennedy told his wife and friends it would be easy for an assassin to shoot him from a crowd ‐ Shortly after Lincoln was shot the telegraph system went down ‐ Shortly after Kennedy was shot the telephone system went down


‐ Kennedy's father had been the Ambassador to England at the Court of St James ‐ Lincoln's son became the Ambassador to England at the Court of St James ‐ Lincoln and Kennedy were 2 of the greatest presidents of the nation ‐ Lincoln's wife tastefully and expensively re‐decorated the White House ‐ Kennedy's wife tastefully and expensively re‐decorated the White House ‐ Lincoln loved great literature and could recite poetry by heart ‐ Kennedy loved great literature and could recite poetry by heart ‐ Lincoln had young children while living at the White House ‐ Kennedy had young children while living at the White House ‐ Lincoln's sons had ponies they rode on the White House grounds ‐ Kennedy's daughter had a pony she rode on the White House grounds


‐ Lincoln lost a child (12 year old son) to death while President ‐ Kennedy lost a child (newly born son) to death while President ‐ Lincoln had 2 sons named Robert and Edward. Edward died young and Robert lived on.

‐ Kennedy had 2 brothers named Robert and Edward. Robert died young and Edward lived on

‐ Lincoln let his children run and play in his office ‐ Kennedy let his children run and play in his office ‐ After Lincoln's assassination the nation experienced an emotional convulsion ‐ After Kennedy's assassination the nation experienced an emotional convulsion


‐ Lincoln's funeral train traveled from Washington DC to New York ‐ Kennedy's brother's funeral train traveled from New York to Washington DC ‐ Lincoln Assassination conspiracy theories are believed all these years later ‐ Kennedy Assassination conspiracy theories are believed all these years later ‐ Abraham was the first name of the man who filmed Kennedy's murder in the Lincoln ‐ The man running alongside Kennedy's car snapping pictures with his 35mm camera was a salesman of Lincoln cars ‐ Jefferson Davis was the name of the President of the Confederate States while Lincoln was president of the United States ‐ Jefferson Davis Tippit was the name of the police officer killed by Kennedy's assassin


‐ Lincoln was famous for his wit and for telling hilarious stories and anecdotes ‐ Kennedy was famous for his wit and for telling hilarious stories and anecdotes ‐ Lincoln was sitting in a rocking chair at Ford's Theater when he was shot ‐ Kennedy had a special rocking chair he sat in at the White House ‐ Kennedy's seat in the Lincoln he was sitting in when he was shot is in Ford's museum ‐ Lincoln's seat in the Ford he was sitting in when he was shot is in Ford's museum ‐ John Kennedy is the name of a character in a 1951 movie about a detective traveling by train to thwart the assassination of President Lincoln ‐ John Kennedy is the name of the real‐life detective who traveled in the train with President Lincoln in 1860 to thwart his assassination


You will note, however, that the names Abraham Lincoln and John F. Kennedy have 12 and 14 letters respectively.


You will note, however, that the names Abraham Lincoln and John F. Kennedy have 12 and 14 letters respectively. However, as one of my correspondents helpfully points out Abraham is an elongated form of Abram, so that Abram Lincoln has 12 letters. My correspondent added “this is what’s known as a ‘clue’ ”


Hindsight bias is not merely a failure of the naive: The eminent historian E.H.Carr said “When I studied ancient history in this university many years ago, I had as a special subject ‘Greece in the period of the Persian Wars’. I collected fifteen or twenty volumes on my shelves and took it for granted that there, recorded in these volumes, I had all the facts relating to my subject. Let us assume ‐‐ it was very nearly true ‐‐ that those volumes contained all the facts about it that were then known, or could be known. It never occurred to me to inquire by what accident or process of attrition that minute selection of facts, out of all the myriad facts that must once have been known to somebody, had survived to become the facts of history.”


Hindsight bias as a form of selection bias Survivorship bias Length bias Dropout bias Publication bias . . . . . .


The tendency to regard escaping from a disaster as good luck

Many lucky events are associated with narrowly avoiding bad outcomes.


The tendency to regard escaping from a disaster as good luck

Many lucky events are associated with narrowly avoiding bad outcomes.

Divide a roulette wheel into (i) three wedges of equal size, red yellow blue (ii) 18 sections of equal size, 6 of each colour

85% of subjects felt it was luckier to land on the red in case (ii) than on case (i).

[Because in case (ii) landing on red is always a close call ?]


Subtleties ! ‐ the Monty Hall problem ‐ the exchange paradox ‐ the sibling gender problem ‐ . . . . . . . . . . . . . . . . . . . . . . H T H H T . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . H H H H H. . . . . . . . . . . . . . . . . . . . . .


So: Statistical fallacies are only statistical in subject matter They are not failures of statistics Any more than “probability paradoxes” represent contradictions in mathematics It’s a question of improved understanding, education, and intuition

Documents

The Improbability Principle Returns Luck, Lotteries, and Laura · The Improbability Principle Returns ‐ Luck, Lotteries, and Laura ... Alex and Donna Voutsinas, looking at old family