In this segment, we derive and discuss the weak
law of large numbers.
It is a rather simple result, but plays a central role
within probability theory.
The setting is as follows.
We start with some probability distribution that has a
certain mean and variance, which we assume to be finite.
We then draw independent random variables out of this
distribution so that these Xi's are independent and
identically distributed, i.i.d.
for short.
What's going on here is that we're carrying out a long
experiment during which all of these random
variables are drawn.
Once we have drawn all of these random variables, we can
calculate the average of the values that have been
obtained, and this gives us the so-called sample mean.
Notice that the sample mean is a random variable because it
is a function of random variables.
It should be distinguished from the true mean, mu, which
is the expected value of the Xi's, which is a number.
It is not random.
And mu is some kind of average over all the possible outcomes
of the random variable Xi.
The sample mean is the simplest and most natural way
for trying to estimate the true mean, and the weak law of
large numbers will provide some support to this notion.
Let us now look at the properties of the sample mean.
Let us calculate its expectation.
By the way, this object here involves two different kinds
of averaging.
The sample mean averages over the values observed during one
long experiment, whereas the expectation averages over all
possible outcomes of this experiment.
The expectation is some kind of theoretical average because
we do not get to observe all the possible outcomes of this
experiment, but the sample mean is something that we
actually calculate on the basis of our observations.
In any case, the expected value of the sample mean, by
linearity, it is the expected value of the numerator divided
by the denominator.
Using linearity once more, the expected value of the sum is
the sum of the expected values, and since each one of
those expected values is equal to mu, we obtain n times mu
divided by n, which leaves us with mu.
So the theoretical average, the expected value of the
sample mean, is equal to the true mean.
Let us now calculate the variance of the sample mean.
The variance of a random variable divided by a number
is the variance of that random variable divided by the square
of that number.
Now, since the Xi's are independent, the variance is
the sum of the variances.
And therefore, we obtain n times the variance
of each one of them.
And after we simplify, this leaves us with sigma
squared over n.
We're now in a position to apply the Chebyshev
inequality.
The Chebyshev inequality tells us that the distance of a
random variable from its mean, being larger than a certain
number, has a probability that's bounded above by the
variance of the random variable of interest divided
by the square of the number that we have here.
We have already calculated the variance, and so this quantity
is sigma squared over n times epsilon squared.
And now, if we consider epsilon as a fixed number and
let n go to infinity, then what we obtain is a limiting
value of 0.
So the probability of falling far from the mean diminishes
to zero as we draw more and more samples.
That's exactly what the weak law of large numbers tells us.
If we fix any particular epsilon, which is a positive
constant, the probability that the sample mean falls away
from the true mean by more than epsilon, that probability
becomes smaller and smaller and converges to 0 as n goes
to infinity.
Let us now interpret the weak law of large numbers.
As I already hinted, we have to think in terms of one long
experiment, and during that experiment, we draw several
independent random variables drawn from the same
distribution.
One way of thinking about those random variables is that
each one of them is equal to the mean, the true mean, plus
some measurement noise, which is a term that has zero
expected value.
And all of these noises are independent.
So we have a collection of noisy measurements, and then
we take those measurements and form the average of them.
What the weak law of large numbers tells us is that the
sample mean is unlikely to be far off from the true mean.
And by far off, we mean at least epsilon distance away.
So the sample mean is, in some ways, a good way of estimating
the true mean.
If n is large enough, then we have high confidence that the
sample mean gives us a value that's close to the true mean.
As a special case let us consider a probabilistic model
in which we repeat independently many times the
same experiment.
There's a certain event A associated with that
experiment that has a certain probability, and each time
that we carry out the experiment, we use an
indicator variable to indicate whether the outcome was inside
the event or outside the event.
So Xi is 1 if A occurs, and it is 0 otherwise.
The expected value of the Xi's, the true mean in this
case, is equal to the number p.
In this particular example, the sample mean just counts
how many times the event A occurred out of the n
experiments that we carried out, so it's the frequency
with which the event A has occurred.
And we call it the empirical frequency of event A. What the
weak law of large numbers tells us is that the empirical
frequency will be close to the probability of that event.
In this sense, it reinforces or justifies the
interpretation of probabilities as frequencies.