Estimators and estimates: An estimator is a mathematical formula

Estimators and estimates:

An estimator is a mathematical formula.

An estimate is a number obtained by applyingthis formula to a set of sample data.

1

ESTIMATORS

It is important to distinguish between estimators and estimates. Definitions are given above.

EMU, ECON 503, M. Balcılar

Population characteristic Estimator

Mean: X

4

ESTIMATORS

n

iixn

X1

1

A common example of an estimator is the sample mean, which is the usual estimator of the population mean.



Mean: X

4

ESTIMATORS

n

iixn

X1

1

Here it is defined for a random variable X and a sample of n observations.



Mean: X

Population variance:

4

ESTIMATORS

Another common estimator is s2, defined above. It is used to estimate the population variance, X

2.

2X

n

iixn

X1

1

n

ii Xx

ns

1

22 )(1

1


Estimators are random variables

8

ESTIMATORS

)...(11

11

n

n

ii xxn

xn

X

An estimator is a special kind of random variable. We will demonstrate this in the case of the sample mean.



8

ESTIMATORS

)...(11

11

n

n

ii xxn

xn

X

iXi ux

We saw in the previous sequence that each observation on X can be decomposed into a fixed component and a random component.



8

ESTIMATORS

)...(11

11

n

n

ii xxn

xn

X

iXi ux

uunn

uunn

X

XX

nXX

)(1

)...(1

)...(1

1

So the sample mean is the average of n fixed components and n random components.



8

ESTIMATORS

It thus has a fixed component X and a random component u, the average of the random components in the observations in the sample.

)...(11

11

n

n

ii xxn

xn

X

iXi ux

uunn

uunn

X

XX

nXX

)(1

)...(1

)...(1

1


10

ESTIMATORS

probability density

function of X

X XXX

probability density

function of X

The graph compares the probability density functions of X and X. As we have seen, they have the same fixed component. However the distribution of the sample mean is more concentrated.


10

ESTIMATORS

Its random component tends to be smaller than that of X because it is the average of the random components in all the observations, and these tend to cancel each other out.

probability density

function of X

X XXX

probability density

function of X


Unbiasedness of X:

1

UNBIASEDNESS AND EFFICIENCY

XXn

nn

nn

xExEn

xxEn

xxn

EXE

1)(...)(

1

)...(1

)...(1

)(

1

11

Suppose that you wish to estimate the population mean X of a random variable X given a sample of observations. We will demonstrate that the sample mean is an unbiased estimator, but not the only one.


Unbiasedness of X:

2


XXn

nn

nn

xExEn

xxEn

xxn

EXE

1)(...)(

1

)...(1

)...(1

)(

1

11

We use the second expected value rule to take the (1/n) factor out of the expectation expression.


Unbiasedness of X:

3


XXn

nn

nn

xExEn

xxEn

xxn

EXE

1)(...)(

1

)...(1

)...(1

)(

1

11

Next we use the first expected value rule to break up the expression into the sum of the expectations of the observations.


Unbiasedness of X:

4


XXn

nn

nn

xExEn

xxEn

xxn

EXE

1)(...)(

1

)...(1

)...(1

)(

1

11

Each expectation is equal to X, and hence the expected value of the sample mean is X.


probabilitydensityfunction

X

estimator B

How do we choose among them? The answer is to use the most efficient estimator, the one with the smallest population variance, because it will tend to be the most accurate.


estimator A

12EMU, ECON 503, M. Balcılar


estimator B

In the diagram, A and B are both unbiased estimators but B is superior because it is more efficient.


estimator A

13

X


1

CONFLICTS BETWEEN UNBIASEDNESS AND MINIMUM VARIANCE

Suppose that you have alternative estimators of a population characteristic , one unbiased, the other biased but with a smaller population variance. How do you choose between them?


estimator B

estimator A



One way is to define a loss function which reflects the cost to you of making errors, positive or negative, of different sizes.

2

error (positive)error (negative)

loss


3


A widely-used loss function is the mean square error of the estimator, defined as the expected value of the square of the deviation of the estimator about the true value of the population characteristic.


222 )()()(MSE ZZZEZ

estimator B


4


The mean square error involves a trade-off between the population variance of the estimator and its bias. Suppose you have a biased estimator like estimator B above, with expected value Z.


Z

bias

222 )()()(MSE ZZZEZ

estimator B


5


The mean square error can be shown to be equal to the sum of the population variance of the estimator and the square of the bias.


Z

bias

222 )()()(MSE ZZZEZ

estimator B


6


To demonstrate this, we start by subtracting and adding Z .

22

22

22

22

22

2

2

)(

))((2)(

)()(2)(

))((2)()(

))((2)()(

)(

)()(MSE

ZZ

ZZZZZ

ZZZZ

ZZZZ

ZZZZ

ZZ

ZE

ZEEZE

ZZE

ZE

ZEZ


7


We expand the quadratic using the rule (a + b)2 = a2 + b2 + 2ab, where a = Z - Z and b = Z - .

22

22

22

22

22

2

2

)(

))((2)(

)()(2)(

))((2)()(

))((2)()(

)(

)()(MSE

ZZ

ZZZZZ

ZZZZ

ZZZZ

ZZZZ

ZZ

ZE

ZEEZE

ZZE

ZE

ZEZ


8


We use the first expected value rule to break up the expectation into its three components.

22

22

22

22

22

2

2

)(

))((2)(

)()(2)(

))((2)()(

))((2)()(

)(

)()(MSE

ZZ

ZZZZZ

ZZZZ

ZZZZ

ZZZZ

ZZ

ZE

ZEEZE

ZZE

ZE

ZEZ


9


The first term in the expression is by definition the population variance of Z.

22

22

22

22

22

2

2

)(

))((2)(

)()(2)(

))((2)()(

))((2)()(

)(

)()(MSE

ZZ

ZZZZZ

ZZZZ

ZZZZ

ZZZZ

ZZ

ZE

ZEEZE

ZZE

ZE

ZEZ


10


(Z - ) is a constant, so the second term is a constant.

22

22

22

22

22

2

2

)(

))((2)(

)()(2)(

))((2)()(

))((2)()(

)(

)()(MSE

ZZ

ZZZZZ

ZZZZ

ZZZZ

ZZZZ

ZZ

ZE

ZEEZE

ZZE

ZE

ZEZ


11


In the third term, (Z - ) may be brought out of the expectation, again because it is a constant, using the second expected value rule.

22

22

22

22

22

2

2

)(

))((2)(

)()(2)(

))((2)()(

))((2)()(

)(

)()(MSE

ZZ

ZZZZZ

ZZZZ

ZZZZ

ZZZZ

ZZ

ZE

ZEEZE

ZZE

ZE

ZEZ


12


Now E(Z) is Z, and E(-Z) is -Z.

22

22

22

22

22

2

2

)(

))((2)(

)()(2)(

))((2)()(

))((2)()(

)(

)()(MSE

ZZ

ZZZZZ

ZZZZ

ZZZZ

ZZZZ

ZZ

ZE

ZEEZE

ZZE

ZE

ZEZ


13


Hence the third term is zero and the mean square error of Z is shown be the sum of the population variance of Z and the bias squared.

22

22

22

22

22

2

2

)(

))((2)(

)()(2)(

))((2)()(

))((2)()(

)(

)()(MSE

ZZ

ZZZZZ

ZZZZ

ZZZZ

ZZZZ

ZZ

ZE

ZEEZE

ZZE

ZE

ZEZ


14


In the case of the estimators shown, estimator B is probably a little better than estimator A according to the MSE criterion.


estimator B

estimator A


n

1 50

1

The sample mean is the usual estimator of a population mean, for reasons discussed in the previous sequence. In this sequence we will see how its properties are affected by the sample size.

probability density function of X

50 100 150 200

n = 1

EFFECT OF INCREASING THE SAMPLE SIZE ON THE DISTRIBUTION OF x

0.08

0.04

0.02

0.06

X


n

1 50

2

Suppose that a random variable X has population mean 100 and standard deviation 50, as in the diagram. Suppose that we do not know the population mean and we are using the sample mean to estimate it.

50 100 150 200

n = 1


0.08

0.04

0.02

0.06


X


n

1 50

3

The sample mean will have the same population mean as X, but its standard deviation will be 50/ , where n is the number of observations in the sample.

50 100 150 200

n = 1


n

0.08

0.04

0.02

0.06


X


n

1 50

4

The larger is the sample, the smaller will be the standard deviation of the sample mean.

50 100 150 200

n = 1


0.08

0.04

0.02

0.06


X


n

1 50

5

If n is equal to 1, the sample consists of a single observation. X is the same as X and its standard deviation is 50.

50 100 150 200

n = 1


0.08

0.04

0.02

0.06


X


n

1 504 25

6

We will see how the shape of the distribution changes as the sample size is increased.

50 100 150 200

n = 4


0.08

0.04

0.02

0.06


X


n

1 504 25

25 10

7

The distribution becomes more concentrated about the population mean.

50 100 150 200

n = 25


0.08

0.04

0.02

0.06


X


n

1 504 25

25 10100 5

8

To see what happens for n greater than 100, we will have to change the vertical scale.

50 100 150 200

0.08

0.04

n = 100


0.02

0.06


X


n

1 504 25

25 10100 5

9

We have increased the vertical scale by a factor of 10.

50 100 150 200

n = 100


0.8

0.4

0.2

0.6


X


n

1 504 25

25 10100 5

1000 1.6

10

The distribution continues to contract about the population mean.

50 100 150 200

n = 1000


0.8

0.4

0.2

0.6


X


n

1 504 25

25 10100 5

1000 1.65000 0.7

11

In the limit, the variance of the distribution tends to zero. The distribution collapses to a spike at the true value. The sample mean is therefore a consistent estimator of the population mean.

50 100 150 200

n = 5000


0.8

0.4

0.2

0.6


X


Documents

Estimators and estimates: An estimator is a mathematical formula