Upload
others
View
1
Download
0
Embed Size (px)
Citation preview
http://statwww.epfl.ch
Week 1: Introduction
Contents
Organization of course.
Motivation for probability and statistics.
Basic notions of sets and of combinatorics.
References: Ross (Chapter 1); Ben Arous notes (Chapter 1).
Exercises: 1–16 of Recueil d’exercices.
Probability and Statistics I — week 1 1
http://statwww.epfl.ch
Organization
Lecturer: Professor A. C. Davison
Assistants: D. Baraka, S. Brahim Belhaouar, S. Salom
Lectures: Monday 14.15–16.00, CO1
Exercises: Monday 16.15–18.00, CO1, CO5. Students should come
in alternate weeks. Those whose surnames begin with letters A–L
should come in odd weeks (starting today), and those whose
surnames begin with letters M–Z should come in even weeks (starting
next week).
Tests: December 15 2003, March 29 2004, June 14 2004.
Probability and Statistics I — week 1 2
http://statwww.epfl.ch
Course Material
Books: Roughly the first two-thirds of the course is on probability,
and a good account is:
Ross, S. M. (1999) Initiation aux Probabilites. PPUR: Lausanne.
For notes in French by Professor Gerard Ben Arous, see
http://dmawww.epfl.ch/benarous/Pmmi/prost1/prost1_fr00.htm
There are many other excellent introductory books: look in the
library.
References on statistics will be given later.
Exercises: We use the Recueil d’Exercices, which is also available
electronically:
http://ima.epfl.ch/cours/cours.htm
Probability and Statistics I — week 1 3
http://statwww.epfl.ch
Intellectual Motivation
Probability and statistics provide mathematical tools and models for
studying random events:
• lotteries, weather forecasting, finance (Nobel Prize, 2003), . . .;
• numbers of junk emails I receive today;
• burstiness of internet traffic;
• noise affecting transmission of a signal or an image;
• errors in coding of signals.
They provide optimal methods for forecasting, for filtering out the
noise, for suggesting how traffic should be handled, and for
reconstruction of the true signal or image.
Probability and Statistics I — week 1 4
http://statwww.epfl.ch
The scum of the universe
Log gamma ray counts indexed by galactic latitude and longitude.
0
1
2
3
4
5
6EGRET measurements
50100150200250300350
20
40
60
80
100
120
140
160
180
Probability and Statistics I — week 1 5
http://statwww.epfl.ch
Markov random field
c
c
c
c
c
c
c
c
c
c
c
c
c
c
c
c
c c
c
c
Probability and Statistics I — week 1 6
http://statwww.epfl.ch
One-dimensional slice
50 100 1500
5
10
15
20
25
EGRET measurements
50 100 1500
5
10
15
20
25
Wavelet−Anscombe estimate
50 100 1500
5
10
15
20
25
Wavelet estimate
50 100 1500
5
10
15
20
25
MRF−l1 estimate
Probability and Statistics I — week 1 7
http://statwww.epfl.ch
A better view?
EGRET measurements
100200300
50
100
150
Wavelet−Anscombe estimate
100200300
50
100
150
Wavelet estimate
100200300
50
100
150
MRF−l1 estimate
100200300
50
100
150
Probability and Statistics I — week 1 8
http://statwww.epfl.ch
Practical Motivation
Many subsequent SSC courses use probability ideas:
Learning and neural networks (Hasler/Thiran);
Performance evaluation (Le Boudec);
Statistical signal processing and applications (Vetterli);
Automatic speech processing (Bourlard);
Biomedical signal processing (Vesin);
Stochastic models for communications (Thiran);
Signal processing for communications (Prandoni);
Information theory and coding (Teletar);
. . .
Probability and Statistics I — week 1 9
http://statwww.epfl.ch
Notes and Plan
The transparencies will be available about a week before each lecture,
and should be downloaded, printed two-to-a-side, and brought to the
lecture. See
http://statwww.epfl.ch/davison/teaching/ProbStatSC/20032004/
I will bring NO copies of the transparencies to the lectures.
Week 1: preliminaries on sets and counting
Week 2: basic notions of probability spaces
Probability and Statistics I — week 1 10
http://statwww.epfl.ch
Preliminaries on Sets
Definition: A set A is a collection of objects, x1, x2, . . . , xn, . . .:
A = x1, x2, . . . , xn, . . . .
We write x ∈ A to mean ‘x is an element of A’, or ‘x belongs to A’.
The collection of all possible objects in a given context is called the
universal set Ω.
Examples of sets are
CH = Geneve, Vaud, . . . , Grisons set of Swiss cantons
0, 1 = finite set consisting of elements 0 and 1
N = natural numbers, countable set
R = real numbers, uncountable set
∅ = empty set, has no elements
Probability and Statistics I — week 1 11
http://statwww.epfl.ch
Subsets
Definition: A set A is a subset of a set B if x ∈ A implies that
x ∈ B: we write A ⊂ B.
If A ⊂ B and B ⊂ A, then every element of A is contained in B and
vice versa, so A = B: both sets contain precisely the same elements.
Notice that ∅ ⊂ A for any set A.
Thus for example:
∅ ⊂ 1, 2, 3 ⊂ N ⊂ Z ⊂ Q ⊂ R ⊂ C, I ⊂ C
Venn diagrams are useful for understanding elementary set
relations, but beware: they can be misleading (not every relation can
be so represented).
Probability and Statistics I — week 1 12
http://statwww.epfl.ch
Cardinal of a Set
Definition: A finite set A has a finite number of elements, and this
is called its cardinal(ity):
card A, #A, |A|.
Obviously |∅| = 0 and |0, 1, | = 2, but | R | does not exist (at least
for this course!)
Exercise: Show that if A and B are finite and A ⊂ B, then
|A| ≤ |B|.
Probability and Statistics I — week 1 13
http://statwww.epfl.ch
Boolean Operations
Definition: Let A, B ⊂ Ω. Then we define three operations:
the union of A and B is A ∪ B = x ∈ Ω : x ∈ A or x ∈ B;
the intersection of A and B is A ∩ B = x ∈ Ω : x ∈ A and x ∈ B;
and the complement of A in Ω is Ac = x ∈ Ω : x 6∈ A.
Obviously A ∩ B ⊂ A ∪ B, and if the sets are finite, then
|A| + |B| = |A ∩ B| + |A ∪ B|, |A| + |Ac| = |Ω|.
We can also define the difference of A and B to be
A \ B = A ∩ Bc = x ∈ Ω : x ∈ A and x 6∈ B,
(note that A \ B 6= B \ A), and the symmetric difference
A 4 B = (A \ B) ∪ (B \ A).
Probability and Statistics I — week 1 14
http://statwww.epfl.ch
Boolean Operations
If Aj∞
j=1is an infinite set of subsets of Ω, then
∞⋃
j=1
Aj = A1 ∪ A2 ∪ A3 ∪ · · · : all x ∈ Ω in at least one Aj
∞⋂
j=1
Aj = A1 ∩ A2 ∩ A3 ∩ · · · : all x ∈ Ω in every Aj
The following are easy to show (mostly using Venn diagrams):
• (Ac)c = A, (A ∪ B)c = Ac ∩ Bc, (A ∩ B)c = Ac ∪ Bc
• A∩ (B∪C) = (A∩B)∪ (A∩C), A∪ (B∩C) = (A∪B)∩ (A∪C)
• (⋃
∞
j=1Aj)
c =⋂
∞
j=1Ac
j , (⋂
∞
j=1Aj)
c =⋃
∞
j=1Ac
j
Probability and Statistics I — week 1 15
http://statwww.epfl.ch
Partition
Definition: A partition of Ω is a collection of non-empty subsets
A1, . . . , An of Ω such that
1. the Aj are exhaustive, that is, A1 ∪ A2 ∪ · · · ∪ An = Ω, and
2. the Aj are disjoint, that is, Ai ∩ Aj = ∅, whenever i 6= j.
A partition can also have an infinite number of sets Aj∞
j=1.
Example 1.1: Let Aj = [j, j + 1), for j = . . . ,−1, 0, 1, . . .. Do the
Aj partition Ω = R?
Example 1.2: Let Aj be the set of all natural numbers divisible by
j, for j = 1, 2, . . .. Do the Aj partition Ω = N?
Probability and Statistics I — week 1 16
http://statwww.epfl.ch
Cartesian Product
Definition: The Cartesian product of two sets A, B is the set of
ordered pairs
A × B = (a, b) : a ∈ A, b ∈ B.
Likewise
A1 × · · · × An = (a1, . . . , an) : a1 ∈ A1, . . . , an ∈ An.
If A1 = · · · = An = A, then we write A1 × · · · × An = An.
As the pairs are ordered, A × B 6= B × A unless A = B.
If A1, . . . , An are all finite, then
|A1 × · · · × An| = |A1| × · · · × |An|.
Example 1.3: Let A = a, b, B = 1, 2, 3. Write down A × B.
Probability and Statistics I — week 1 17
http://statwww.epfl.ch
Preliminaries on Combinatorics
Combinatorics is the mathematics of counting. Two basic principles:
• addition: if I have m red hats and n blue hats, then I have
m + n hats in total;
• multiplication: if I have m hats and n scarves there are mn
different ways I can combine them.
In mathematical terms, let A1, . . . , Ak be sets. Then
|A1 × · · · × Ak| = |A1| × · · · × |Ak|,
and if the Aj are disjoint, then
|A1 ∪ · · · ∪ Ak| = |A1| + · · · + |Ak|.
Probability and Statistics I — week 1 18
http://statwww.epfl.ch
Examples
Example 1.4: Six dice are rolled.
(a) How many outcomes are there?
(b) For how many of these do the dice show six different faces?
Example 1.5: I have 5 hats and 5 friends, and I want to give one
hat to each friend. How many ways can I do this?
Example 1.6: I have 5 hats and 3 friends, and I want to give one
hat to each friend. How many ways can I do this?
Example 1.7: How many different ways can I arrange my 4
probability books on a shelf?
Probability and Statistics I — week 1 19
http://statwww.epfl.ch
Permutations: Ordered Selection
Definition: A permutation of n distinct symbols is an ordering of
them.
Theorem: Given n distinct symbols, the number of distinct
permutations (without repetition) of length r ≤ n is
n (n − 1) (n − 2) · · · (n − r − 1) =n!
(n − r)!.
Thus there are n! permutations of length n.
Theorem: Given n =∑r
i=1ni symbols of r distinct types, where ni
are of type i and are otherwise indistinguishable, the number of
permutations (without repetition) of all n symbols is
n!
n1! n2! · · · nr!.
Probability and Statistics I — week 1 20
http://statwww.epfl.ch
Example
Example 1.8: You are playing bridge, and when you pick up your
cards you notice that the suits are already grouped: the clubs are all
adjacent to each other, the hearts likewise, and so on. Your hand
contains 4 spades, 4 hearts, 3 diamonds and 2 clubs.
(a) How many permutations of your hand are there?
(b) How many of these permutations have the cards grouped together
as described?
Probability and Statistics I — week 1 21
http://statwww.epfl.ch
Example
Example 1.9: A class of 20 students elects a committee of size 4 to
organise a voyage detudes. How many ways are there of choosing the
committee if:
(a) their roles are indistinguishable?
(b) there are 4 distinct roles (president, secretary, treasurer,
travel-agent)?
(c) there is a president, a treasurer, and two travel-agents?
(d) there are two treasurers and two travel-agents?
Probability and Statistics I — week 1 22
http://statwww.epfl.ch
Multinomial and Binomial Coefficients
Definition: Let n1, . . . , nr lie in the range 0, 1, . . . , n, with total
n1 + · · · + nr = n. Then(
n
n1, n2, . . . , nr
)
=n!
n1! n2! · · · nr!,
is called a multinomial coefficient. The case r = 2 is most
common:(
n
k
)
=n!
k!(n − k)!
(
= Ckn in some older books
)
is called a binomial coefficient.
Example 1.10: Compute these coefficients for n = 1, 2, 3, 4.
Probability and Statistics I — week 1 23
http://statwww.epfl.ch
Combinations: Unordered Selection
Theorem: The number of ways of choosing a set of r symbols from a
set of n distinct symbols without repetition is
n!
r!(n − r)!=
(
n
r
)
.
Theorem: The number of ways of dividing n distinct objects into r
distinct groups of sizes n1, . . . , nr, where n1 + · · · + nr = n is
n!
n1! n2! · · · nr!.
Probability and Statistics I — week 1 24
http://statwww.epfl.ch
Properties of Binomial Coefficients
Theorem: If n, m are non-negative integers and r ∈ 0, . . . , n, then:(
n
r
)
=
(
n
n − r
)
;
(
n + 1
r
)
=
(
n
r − 1
)
+
(
n
r
)
, (Pascal’s triangle);
r∑
j=0
(
m
j
)(
n
r − j
)
=
(
m + n
r
)
, (Vandermonde’s formula);
(a + b)n =n
∑
r=0
(
n
r
)
arbn−r, (Newton’s binomial formula);
(1 − x)−n =
∞∑
j=0
(
n + j − 1
j
)
xj , |x| < 1 (negative binomial series).
Probability and Statistics I — week 1 25
http://statwww.epfl.ch
Partitions of Integers
Example 1.11: How many ways are there of putting 6 identical
balls into 3 boxes, so that each box contains at least one ball?
Example 1.12: How many ways are there of putting 6 identical
balls into 3 boxes?
Theorem: (a) The number of distinct vectors (n1, . . . , nr) of positive
integers satisfying n1 + · · · + nr = n is(
n − 1
r − 1
)
.
(b) The number of distinct vectors (n1, . . . , nr) of non-negative
integers satisfying n1 + · · · + nr = n is(
n + r − 1
n
)
.
Probability and Statistics I — week 1 26