We will now go through another example to consolidate our
intuition about the content of the law of iterated
expectations and the law of the total variance.
The example is as follows.
We have a class, and that class consists of 30 students
in total who are divided into sections--
the first and the second section.
Let xi be the score of students i, let's say the
final grade in the class.
We consider the following probabilistic experiment.
We pick a student at random, uniformly, so that each
student is equally likely to be picked.
And we define two random variables--
X is a numerical random variable that gives us the
score of the selected student.
So if student i is selected, the value of the random
variable capital X is xi.
And capital Y is defined as the random variable, which is
the section of the selected student, so that y takes
values 1 or 2.
We're given some information.
For the first section, the average of the
student scores is 90.
For the second section, the average of the
student scores is 60.
Given that information, what is the expected value of the
student score?
Well, each student is equally likely to be picked, so has
probability 1 over 30 to be picked.
And this multiplies the score of the student, so this is the
expected value of the random variable of interest.
What is this number?
Well, we need to calculate the sum of the xi's.
The sum of the first 10 xi's is equal to 90 times 10, and
the sum of the xi's in the other section is
equal to 60 times 20.
And we carry out the calculation, and we find that
the answer is 70.
Now let us look at conditional expectations.
If Y is equal to 1, this means that a student from section
one was picked.
And within that section, each student is equally likely to
be picked, so the outcome of this random variable is
equally likely to be any one of these xi's.
Each xi gets picked with probability of 1 over 10.
And so, the expected value of this random variable is 90.
Similarly for the second section, the expected value of
the score of a randomly selected student, given that
the student belongs in that section, is equal to 60.
With this information available, now we can describe
the abstract conditional expectation,
which is a random variable.
This random variable takes the value of 90 if a student from
the first section was picked, and the value of 60 if a
student from the second section was picked.
What is the probability of this event that the student
from the first section was picked?
Given that the first section has 10 out of a total of 30
students, this probability is 1/3, and therefore, this
probability is 2/3.
Now that we have the distribution of this random
variable, we can calculate the expected value of this random
variable, which is 1/3 times 90 plus 2/3 times 60.
And this number evaluates to 70, which of course, it's no
coincidence, it's the same as the average
over the entire class.
By the law of iterated expectations, we know that
this quantity should be the same as this quantity.
So the law of iterated expectations allows us to
calculate the overall average in the entire class by taking
the section averages, and weigh them according to the
sizes of the different sections.
It's a divide and conquer method, and it is similar to
what we have been doing when we use the total expectation
theorem to divide and conquer.
We continue with our example, and here is a summary of what
we found so far.
The conditional expectation is a random variable that takes
these two values with certain probabilities.
And the mean of this random variable is equal to 70.
Let us now calculate the variance
of this random variable.
This random variable, with probability 1/3, takes a value
90, which is this much away from the mean of this random
variable, which we square.
And with probability 2/3, it takes a value of 60, which is
this much away from the mean of the random variable.
We square this, as well.
And when we carry out the calculation, we find that this
number is equal to 200.
Let us now continue.
And suppose that somebody gave us this piece of information.
For the first section, this is the deviation of the i-th
student from the mean of that section.
So this is the sum of the squares of the deviations and
then we average over all the students.
We will use this data to calculate certain quantities--
for example, the variance of the scores
in the first section.
Now in the first section, with probability 1/10, we pick the
ith student that has this score.
And this is the deviation of that student from the mean of
that section.
So this is the same as the mean squared deviation from
the mean of the section.
And this is exactly the variance within that section.
It is the variance of the random variable, which is the
score of a random student, given that we are selecting a
student from the first section.
For the second section, the story similar.
We're given this information, and this tells us the variance
of the student scores within the second section.
So now we can describe the abstract conditional variance.
It is a random variable that takes this value with
probability equal to the probability of selecting
someone from this section, which is 1/3.
Or it takes a value of 20, which is the variance in the
second section.
And the second section is selected with probability 2/3.
With this information at hand, now we can calculate the
expected value of this random variable, which is 1/3 times
10 plus 2/3 times 20, which is 50/3.
At this point, we have the two quantities that are necessary
to apply the law of total variance.
According to the law of total variance, the variance of the
student scores throughout the entire class is equal to this
number, which is 50/3, plus this number, which is 200.
And this is the overall variance.
Now let us interpret the law of total
variance in this context.
The interpretation is as follows.
The variance of the student scores in the entire class
consists of two pieces.
The first piece looks at the variance inside each section,
which is 10 or 20, depending on which section
we're looking at.
And we take the average over the different sections.
So we look at the variability of the scores within a typical
section, and then we average over all the sections.
The other term looks at the means in the different
sections, and figures out how different are these means.
How much do they vary from the overall class average?
It measures the variability between different sections.
So the overall randomness in the test scores can be broken
down into two pieces of randomness.
One source of randomness is that the different sections
have different means.
The other source of randomness is that inside each section,
the students are different from the
means of their section.
And these two pieces of randomness together add up to
the total randomness of the student scores as measured by
the variance of the entire class.