The purpose of this segment is to give you a little bit of
the bigger picture.
We did discuss some inequalities, we did discuss
convergence of the sample mean--
that's the weak law of large numbers-- and we did discuss a
particular notion of convergence of random
variables, convergence in probability.
How far can we take those topics?
Let's start with the issue of inequalities.
Here, one would like to obtain bounds and approximations on
tail probabilities that are better than the Markov and
Cherbyshev inequalities that we have seen.
This is indeed possible.
For example, there is a so-called Chernoff bound that
takes the following form.
The Chernoff bound tells us that the probability that the
sample mean is away from the true mean by at least a, where
a is a positive number, this probability is bounded above
by a function that falls exponentially with n and where
the exponent depends on the particular number, a, that we
are considering.
But in any case, this term in the exponent
is a positive quantity.
Notice that this is much better, much stronger than
what we obtained from the Cherbyshev inequality because
in the Cherbyshev inequality, we only obtain an inequality
for this probability that falls off as the
rate of 1 over n.
So this falls much faster, and so it tells us that this
probability is indeed much smaller than what the
Cherbyshev inequality might predict.
However, this inequality requires some additional
assumptions on the random variables involved.
Another type of approximation on this tail probability can
be obtained through the central limit theorem, which
will actually be the next topic
that we will be studying.
Very loosely speaking, the central limit theorem tells us
that the random variable M sub n, which is the sample mean,
behaves as if it were a normal random variable with the mean
and the variance that it should have.
We know that this is the mean and the variance of the sample
mean, but the central limit theorem tells us that in
addition to that, we can also pretend that the sample mean
is normal and carry out approximations as if this were
a normal random variable.
Now, this statement that I'm making here is
only a loose statement.
It is not mathematically completely accurate.
We will see later a more accurate statement of the
central limit theorem.
In a different direction, we can talk about different types
of convergence.
We did define convergence in probability, but that's not
the only notion of convergence that's
relevant to random variables.
There's an alternative notion, which is convergence with
probability one.
Here is what it means.
We have a single probabilistic experiment.
And within that the experiment, we have a sequence
of random variables and another random variable, and
we want to talk about this random variable converging to
that random variable.
What do we mean by that?
We consider a typical outcome of the experiment, that is,
some omega.
Look at the values of the random variable Yn under that
particular omega, and look at that sequence of values, the
values of the different random variables under that
particular outcome.
Under that particular outcome, Y also has a certain numerical
value, and we're interested in whether this convergence takes
place as n goes to infinity.
Now for some outcomes, omega, this will happen.
For some, it will not happen.
We will say that we have convergence with probability
one if this event has probability equal to 1.
That is, there is probability one, that is, essential
certainty, that when an outcome of the experiment is
obtained, the resulting sequence of values of the
random variables Yn will converge to the value of the
random variable Y.
Now, this definition is easy to write down, but to actually
understand what it really means and the ways it is
different from convergence in probability is not so easy.
It does take some conceptual effort, and we will not
discuss it any further at this point.
Let me just say that this is a stronger notion of
convergence.
If you have convergence with probability one, you also gets
convergence in probability.
And it turns out that the law of large numbers also holds
under this stronger notion of convergence.
That is, we have that the sample mean converges to the
true mean with probability one.
This is the so-called strong law of large numbers, and
because this is a stronger notion of convergence, a more
demanding one, that's why this is called the strong law.
Incidentally, at this point, you might be quite uncertain
and confused as to what is really the difference between
these two notions of convergence.
The definitions do look different, but what is the
real difference?
This is quite subtle, and it does take
quite a bit of thinking.
It's not supposed to be something that is obvious.
So the purpose of this discussion is only to point
out these further directions but without, at this point,
going into it in any depth.
Finally, there is another notion of convergence in which
we're looking at the distributions of the random
variables involved.
So we may have a sequence of random variables.
Each one of them has a certain distribution described by a
CDF, and we can ask the question, does this sequence
of CDFs converge to a limiting CDF?
If that happens, then we say that we have convergence in
distribution, and this is more or less the type of
convergence that shows up when we deal with the central limit
theorem because this is really a statement about
distributions, that the distribution of the sample
mean in some sense starts to approach the distribution of a
normal random variable.