After our warm-up, we can now come to the real problem.
We have, again, a random variable Theta
with a known prior distribution.
And we're interested in a point estimate.
What will be different this time,
however, is that we now have an observation.
And we also have a model of that observation
as a conditional distribution given
the value of the true parameter.
We observe a value of that random variable.
That value is little x.
And on the basis of that value, we
would like to now come up with a point
estimate of the unknown random variable Theta.
How do we proceed?
We can, of course, use the Bayes rule.
And the Bayes rule is going to give us
a distribution for the unknown random variable given
the observation that we have obtained.
And that distribution could be discrete or continuous.
Let me just plot something as if it's continuous.
And now that we have the posterior distribution
of Theta, we would like to come up with a point estimate.
How do we do it?
Remember our earlier conclusion.
If we do not have any observations,
if we live in a universe where we have a distribution of Theta
and we want a point estimate, the optimal,
the one that minimizes the mean squared error,
is the expected value of our random variable.
But now we live in a different universe,
in a universe where we have a conditional distribution
of Theta.
We want to minimize the conditional mean squared error,
because this is the mean squared error that
applies to this conditional universe in which we have
obtained a particular observation.
What is going to be the result of this minimization?
Well, this is a problem that's identical to the problem
of minimizing this quantity, except that now this problem is
posed in a conditional universe.
So we just follow the same steps.
And obtain the same solution, the solution
is going to be the expected value
of the unknown random variable, except that now we
live in a conditional universe.
And therefore, we should take the relevant expected value
which is the conditional expectation given
the information that we have available in our hands.
So to summarize, what we obtain is that the optimal estimate
is the conditional expectation.
And this is a relation between numbers.
But if we want to think about it more abstractly, we have
designed an estimator which is based on a random variable,
capital X, and calculate the expected value
of our random variable that we're trying to estimate,
namely Theta, on the basis of X.
Let us now continue with some observations.
Remember that the expected value of Theta
minimizes this quantity.
And we can write this more explicitly
in terms of the following inequality--
that if we use the expected value as an estimate,
the resulting mean squared error is less than
or equal to the mean squared error
that we would have obtained if we
had used any other estimate, c.
So this is a relation that's true for all c.
Now, let us take this inequality and translate it
into our more interesting context where
we have an observation available.
Once more, the conditional expectation
minimizes the mean squared error.
Let us write out explicitly what this
means in a form analogous to what we wrote down earlier.
What it means is that the expected value
of Theta minus the estimate, namely
the conditional expectation, squared.
In this conditional universe in which we live,
this is less than or equal to the mean squared error
that we would have obtained if we had used any other estimate
in the place of the conditional expectation.
So for any value g of x that we might have used,
the error would have been at least as large.
Why am I using this notation g here?
Let us go back to the bigger picture.
What we have is that we are obtaining a numerical value x.
We do some processing to it which
corresponds to some function g.
And we come up with an estimate which
is a function of the little x that we have observed.
So no matter what estimate we use,
the mean squared error is going to be at least as large
as the mean squared error that we obtain
if we use the conditional expectation.
Now, let us take this inequality here and write it
in a more abstract form.
Suppose that we have settled on some particular estimator
and we want to compare this estimator with the expected
value estimator.
Then we're going to get the following inequality.
If we use the conditional expectation
as an estimator in a conditional universe
where we know the value of the random variable X,
the corresponding mean squared error
is going to be less than or equal to the mean squared error
obtained by the alternative estimator g.
What does this inequality say?
This inequality is simply an abstract version
of the previous inequality.
The previous inequality is true for all little x.
Here we have an inequality between random variables.
This random variable here is a random variable
that takes this specific numerical value, whenever
capital X takes the value little x.
When X takes the value little x, we're
conditioning on this event.
And when X is equal to little x, this quantity
takes on this particular numerical value.
And similarly, on the other side,
this is a random variable that takes
this particular numerical value whenever
capital X is equal to little x.
Now that we have an inequality between random variables,
actually between conditional expectations,
we can take expectations of both sides.
And we use the law of iterated expectations.
The expectation of a conditional expectation
is an unconditional expectation.
So we obtain this as the expected
value of this quantity.
And it's less than or equal to the expected value
of this quantity using, again, the law
of iterated expectations.
We obtain this relation here.
And what this inequality is saying
is that the overall mean squared error,
if we use the conditional expectation, now
as an estimator, is less than or equal to the mean squared error
that we would obtain if we had used any other estimator.
So this inequality refers to the following picture.
We obtain an observation which is a random variable.
We process that random variable to come up
with an estimator which is a function of the random variable
that we observe and so is itself a random variable.
So when we use this random variable to estimate Theta,
we obtain a certain mean squared error.
This is going to be at least as large as the mean squared error
that we obtain if we use the conditional expectation
as our estimator.
So to summarize, the conditional expectation
of Theta, viewed as a random variable,
as an estimator, what we call the LMS estimator of Theta,
has the property that it minimizes
the mean squared error over all possible alternative
estimators.
So if you want to design this box using some other function
g, you're going to obtain a mean squared error that's
going to be no better than what you obtain if you were
to use the conditional expectation.