Mathematically speaking, the Chebyshev inequality is just a
simple application of the Markov inequality.
However, it contains a somewhat different message.
Consider a random variable that has a
certain mean and variance.
What the Chebyshev inequality says is that if the variance
is small, then the random variable is unlikely to fall
too far off from the mean.
If the variance is small, we have little randomness.
And so X cannot be too far from the mean.
In more precise terms, we have the following inequality.
The probability that the distance from the mean is
larger than or equal to a certain number is, at most,
the variance divided by the square of that number.
So if the variance is small, the probability of falling far
from the mean is also going to be small.
And if the number c is large, so that we're talking about a
large distance from the mean, then the probability of this
event happening falls off at a rate at
least 1 over c squared.
By the way, I should add here that c is assumed to be a
positive number.
If c was negative, then the probability that we're looking
at would be equal to 1 anyway.
And there isn't any point in trying obtain a bound for it.
To prove the Chebyshev inequality, we will apply the
Markov equality as follows.
The probability of interest is the same as the probability
that the square of this quantity is larger than or
equal to the square of c.
But now, here we have a non-negative random variable.
And we can apply the Markov inequality with X replaced by
this random variable and with a replaced by c squared.
So this gives us the expected value of the random variable
of interest divided by c squared.
But we recognize that the
numerator is just the variance.
And this is the Chebyshev inequality that we claimed.
As an application of the Chebyshev inequality, let us
look at the probability of this event that the distance
from the mean is at least k standard deviations, where k
is some positive number.
Using the Chebyshev inequality with c replaced by k times
sigma, we obtain sigma squared over c squared, which in our
case is k squared times sigma squared,
which is 1 over k squared.
So what this is saying is that if you take, for example, k
equal to 3, the probability that you fall three standard
deviations away from the mean or more, that probability is
going to be less than or equal to 1 over 9.
And this is true no matter what kind of
distribution you have.
Let us now revisit our earlier example, where X is an
exponential random variable.
And we're interested in the probability that the random
variable takes a value larger than or equal to a.
The Markov inequality gave us a bound of 1 over a.
And as we recall, the exact answer to this probability was
e to the minus a.
Let us see what we can get using the Chebyshev
inequality.
Now, our random variable has a mean of 1.
Let us assume that a is bigger than 1, so that we're
considering an event that we fall far away from the mean by
a distance of at least a minus 1.
That is we write the probability that X is larger
than or equal to a as the probability that the distance
of X from the mean is larger than or equal to a minus 1.
And now, this event is smaller than the event that the
absolute value of X minus 1 is larger than a minus 1.
This is because if this event is true, then that event will
also be true.
And now, we can apply the Chebyshev inequality.
Here we have the distance of X from the mean.
So the Chebyshev inequality applied to the random variable
X will have up here the variance of X,
which is equal to 1.
And in the denominator, we will have a minus 1 squared.
Notice that if a is a large number, this quantity here
behaves like 1 over a squared, which falls off much faster
than 1 over a.
So at least for large a's, the Chebyshev bound is going to
give us a smaller bound and, therefore, more informative
than what we obtained from the Markov inequality.
In most cases, the Chebyshev inequality is, indeed,
stronger and more informative than the Markov inequality.
And one of the reasons is that it exploits more information
about the distribution of the random variable X. That is it
uses knowledge, not just about the mean
of the random variable.
But it also uses some information about the variance
of the random variable.