Let's now go through another example,
which will be a little more challenging.
We're going to revisit an old problem.
We have a coin that has an unknown bias, Theta.
And we have a prior distribution on this Theta.
We fix some positive integer, n, we flip a coin
n times, that has this unknown bias.
And we record the number of heads.
On the basis of the number of heads that have been observed,
we wish to estimate the bias, Theta, of the coin.
To make things more concrete, we're
going to assume a prior distribution on Theta that
is uniform on the unit interval.
Now, this is a problem we have considered before.
We have calculated the expected value of Theta given X.
And we did find that the expected value takes
this particular form.
Now, notice that this is a linear function of X.
And if it turns out the least mean squares estimator
is a linear function of X, then we're
guaranteed, since this is the best,
that this is also the best within the class
of linear estimators.
So we immediately have the conclusion
that the linear least mean squares
estimator is this particular function of X.
So there's not much left to do.
On the other hand, just for practice,
let us derive this answer directly from the formulas
that we have for the linear least mean squares estimator,
and see whether we're going to get the same answer.
So we want to use this formula.
And in order to apply this formula, all that we have to do
is to calculate these expected values, this variance,
and this covariance.
So now let's move on to this particular calculational
exercise.
Let's start by writing down what we
know about the random variables involved in this problem.
About Theta, we know that it is uniform.
And so it has a mean of 1/2 and a variance of 1/12.
About X, what we know is the following.
If you fix the bias of the coin, then the number
of heads you're going to obtain in n flips
has a binomial distribution, with parameters n and Theta.
But of course, Theta itself is a random variable.
So for this reason, this is a conditional distribution.
But within the conditional universe,
we know the mean and the variance of a binomial,
and they are as follows.
The mean of a binomial is n times the bias of the coin.
But because we're talking about the conditional universe,
this is a conditional expectation.
And it's a random variable, because it's
affected by the value of the random variable Theta.
And similarly, for the variance, it's
the usual formula for the variance of a binomial,
except that now the bias itself is a random variable.
So now let's continue with the calculation of the quantities
that we need for the formula for our estimator.
Let's start with the expected value of X.
Since we know the conditional expectation of X,
we can use the law of iterated expectations.
The unconditional expectation is the expected value
of the conditional expectation, which is n times Theta.
And since the mean of Theta is 1/2, we obtain n/2.
Let us now continue with the calculation of the variance.
There are different ways that we can calculate it.
One could be the law of total variance.
But we will take the alternative approach,
which is to use the general formula for the variance,
that the variance is equal to the expected
value of the square of a random variable,
minus the square of the expected value.
We know the expected value of X.
So all that's left is to calculate
the expected value of X squared.
How are we going calculate it?
Well, we know the conditional distribution of X.
So it should be easy to calculate
the conditional expectation of X squared
in the conditional universe, and then
use the law of iterated expectations
to obtain the unconditional expectation.
So now, we need to calculate this conditional expectation
here.
How do we do it?
The expected value of a square of a random variable
is always equal to the variance of that random variable
plus the square of the expected value.
We're going to use this property,
but we're going to use it in the conditional universe.
So in the conditional universe, this
is going to be equal to the variance,
in the conditional universe, which is n times Theta times 1
minus Theta, plus the square of the expected value of X,
but the expected value in the conditional universe, which
is this quantity-- n times Theta.
So we obtain another term-- n squared, Theta squared.
So now we can go back to our previous calculation,
and right here, the expected value
of this expression, which is n times Theta.
And then we have some Theta squared terms.
One is n squared.
The other is a minus n.
So we obtain plus n squared minus n Theta squared.
The expected value of n times Theta
is n times the expected value of Theta, which is 1/2.
So we obtain a factor of n/2.
But then we have this additional term here.
We need the expected value of Theta squared.
What is it?
Well, since we know the mean and the variance of Theta,
we can calculate the expected value of Theta squared.
It is equal to the variance plus the square of the mean.
And this evaluates to 1/3.
So from here, we're going to obtain
plus n squared minus n divided by 3.
And you can rewrite this term in a different way
by collecting first the n terms, n/2 minus n/3.
This gives us n/6.
And then there's the n squared term,
which is n squared over 3.
Now that we found the expected value of X squared,
we can go back to this calculation.
We have n/6 plus n squared over 3
minus the square of the expected value of X,
which is this expression here.
So we obtain minus n squared over 4.
1/3 minus 1/4-- that makes 1/12.
So we obtain n/6 plus n squared over 12.
Or another way of writing this is n times n plus 2 over 12.
And this completes the calculation
of the variance of X.
The last quantity that's left for us to calculate
is the covariance of Theta with X.
We're going to calculate it using the alternative formula
for the covariance, which is the expectation of the product
minus the product of the expectations.
We have the expectations, but we do not
have the expectation of the product.
So we need to calculate it.
Once more, it's going to be the same trick.
We're going to condition on Theta,
and then use the law of iterated expectations.
So the law of iterated expectations,
when you condition on Theta, takes this form.
And to continue here, we need to find
this conditional expectation in the inside.
This conditional expectation-- what is it?
If I give you Theta, then you know Theta.
It is becoming now a constant.
There's nothing random to it, so it
can be pulled outside the expectation.
And we obtain Theta times the conditional expectation of X.
We know what the conditional expectation of X is.
It's n times Theta.
So from here, we obtain overall, a term n times Theta squared.
So now we can go back here.
We have the expected value.
And the term in the inside-- we just found it.
It's n times Theta squared.
And since the expected value of Theta squared
is 1/3, from here, we obtain n/3.
And now we can go back, finally, to the calculation
of the covariance.
It's going to be n/3, minus the expected value of Theta,
which is 1/2, times the expected value of X, which is n/2.
So it's minus n over four.
And this evaluates to n/12.
So we have succeeded in calculating all the quantities
that are needed in the formula for the linear least
mean squares estimator.
We can now take those values that we have just found
and substitute them into this formula.
And after a little bit of algebra and moving terms
around, everything simplifies to this expression.
Just to verify that this makes sense,
what is the coefficient next to X?
It's the covariance divided by the variance.
n/12 divided by this expression--
this n cancels that n.
This 12 cancels that 12.
We're left with an n plus 2 in the denominator.
And indeed, the coefficient that multiplies X
is the term n plus 2 in the denominator.
And you can similarly verify that the constant term as well
is the correct one.
So of course, this answer is what we had found in the past
to be the optimal, the least mean squares
estimator of X. As we discussed earlier, when this is linear
in X, it has to be the same as the linear least mean squares
estimator.
So this answer is not a surprise,
but it was an interesting and perhaps useful
exercise to go through the details of this calculation
to see what it takes to figure out
the different terms in this formula.