We have presented the complete solution
to the liner least mean squares estimation problem, when
we want to estimate a certain unknown random variable
on the basis of a different random variable X
that we get to observe.
But what if we have multiple observations?
What would be the analogous formulation of the problem?
Here's the idea.
Once more, we restrict ourselves to estimators
that are linear functions of the data, linear functions
of the observations that we have.
And then we pose the problem of finding the best
choices of these coefficients a1 up to a n and b.
What does it mean to find the best choices?
It means that if we fix certain choices,
we obtain an estimator, we look at the difference
between the estimator and the quantity
we're trying to estimate, take the square,
and then take the expectation.
So once more, we're looking at the mean squared error
of our estimator and we try to make it as small as possible.
So this is a well-defined optimization problem.
We have a quantity, which is a function of certain parameters.
And we wish to find the choices for those parameters,
or those coefficients, that will make
this quantity as small as possible.
One first comment is similar to the case
where we had a single measurement [and]
is the following.
If it turns out that the conditional expectation
of Theta given all of the data that we
have is linear in X, if it is of this form, then what happens?
We know that this is the best possible estimator.
If it is also linear, then it is the best estimator
within the class of linear estimators as well
and, therefore, the linear least mean squares estimator
is the same as the general least mean squares estimator.
So if for some problems it turns out that this is linear,
then we automatically also have the optimal linear estimator.
And this is going to be the case, once more,
for certain normal problems with a linear structure of the type
that we have studied earlier.
Now, let us look into what it takes
to carry out this optimization.
If we had a single observation, then we
have seen a closed form formula, a fairly simple formula,
that tells us what the coefficients should be.
For the more general case, formulas
would not be as simple, but we can
make the following observations.
If you take this expression and expand it,
it's going to have a bunch of terms.
For example, it's going to have a term of the form a1 squared
times the expected value of X1 squared.
It's going to have a term such as twice a1, a2 times
the expected value of X1, X2.
And then there's going to be many more terms to some of them
will also involve products of Theta with this.
So we might see that we have a term of the form a1 expected
value of X1 Theta.
And then, there's going to be many, many more terms.
What's the important thing to notice?
That this expression as a function of the coefficient
involves terms either of this kind
or of this kind, or of that kind,
first-order or second-order terms.
To minimize this expression, we're
going to take the derivative of this and set it equal to 0.
When you take the derivative of a function that
involves only quadratic and linear terms,
you get something that's linear in the coefficients.
The conclusion out of all this discussion
is that when you actually go and carry out this minimization
by setting derivatives to zero, what you
will end up doing is solving a system of linear equations
in the coefficients that you're trying to determine.
And why is this interesting?
Well, it is because if you actually
want to carry out this minimization,
all you need to do is to solve a linear system, which is easily
done on a computer.
The next observation is that this expression only
involves expectations of various terms
that are second order in the random variables involved.
So it involves the expected value of X1 squared,
it involves this term, which has something
to do with the covariance of X1 and X2.
This term that has something to do with the covariance of X1
with Theta.
But these are the only terms out of the distribution of the X's
and of Theta that will matter.
So similar to the case where we had a single observation,
in order to solve this problem, we
do not need to know the complete distribution of the X's
and of Theta.
It is enough to know all of the means,
variances, and covariances of the random variables
that are involved.
And once more, this makes this approach
to estimation a practical one, because we do not
need to model in complete detail the distribution
of the different random variables.
Finally, if we do not have just one unknown random variable,
but we have multiple random variables that we
want to estimate, what should we do?
Well, this is pretty simple.
You just apply this estimation methodology
to each one of the unknown random variables separately.
To conclude, this linear estimation methodology
applies also to the case where you have multiple observations.
You need to solve a certain computational problem in order
to find the structure of the best linear estimator,
but it is not a very difficult computational problem,
because all that it involves is to minimize
a quadratic function of the coefficients
that you are trying to determine.
And this leads us to having to solve
a system of linear equations.
For all these reasons, linear estimation,
or estimation using linear estimators, is quite practical.