Let us now discuss in some more detail
what it takes to carry out Bayesian inference,
when both random variables are discrete.
The unknown parameter, Theta, is a random variable
that takes values in the discrete set.
And we can think of these values as alternative hypotheses.
In this case, we know how to do inference.
We have in our hands the Bayes rule
and we have seen plenty of examples.
So instead of going through one more example in detail,
let us assume that we have a model, that we have observed
the value of X, and that we have already determined
the conditional PMF of the random variable Theta.
As a concrete example, suppose that Theta
can take values 1, 2, or 3.
We have obtained our observation,
and the conditional PMF takes this form.
We could stop at this point or we
could continue by asking for a specific estimate of Theta--
our best guess as to what Theta is.
One way of coming up with an estimate
is to use the maximum a posteriori of probability
rule, which looks for that value of theta that
has the largest posterior, or conditional, probability.
In this example, it is this value,
so our estimate is going to be equal to 2.
An alternative way of coming up with an estimate
could be the LMS rule, which calculates
an estimate equal to the conditional expectation
of the unknown parameter, given the observation that we
have made.
This is just the mean of this conditional distribution.
In this example, it would fall somewhere around here,
and the numerical value, as you can check, is equal to 2.2.
Next, we may be interested in how good a certain estimate is.
And for the case where we interpret the values of Theta
as hypotheses, a relevant criterion
is the probability of error.
In this case, because we already have
some data available in our hands and we're
called to make an estimate, what we care about
is the conditional probability, given the information
that we have, that we're making an error.
Making an error means the following.
We have the observation, the value of the estimate
has been determined, it is now a number,
and that's why we write it with a lowercase theta hat.
But the parameter is still unknown.
We don't know what it is.
It is described by this distribution.
And there's a probability that it's
going to be different from our estimate.
What is this probability?
It depends on how we construct the estimates.
If in this example, we use the MAP rule
and we make an estimate of 2, there
is probability 0.6 that the true value of Theta
is also equal to 2, and we are fine.
But there's a remaining probability of 0.4
that the true value of Theta is different than our estimate.
So there's probability 0.4 of having made a mistake.
If, instead of an estimate equal to 2,
we had chosen an estimate equal to 3,
then the true parameter would be equal to our estimate
with probability 0.3, but we would have made an error
with probability 0.7, which would
be a bigger probability of error.
More generally, the probability of error
of a particular estimate is the sum
of the probabilities of the other values of Theta.
And if we want to keep the probability of error small,
we want to keep the sum of the probabilities
of the other values small, which means
we want to pick an estimate for which its own probability is
large.
And so by that argument, we see that the way
to achieve the smallest possible probability of error
is to employ the MAP rule.
This is a very important property of the MAP rule.
Now, this is the conditional probability
of error, given that we already have data in our hands.
But more generally, we may want to compare estimators or talk
about their performance in terms of their overall probability
of error.
We're designing a decision-making system
that's going to process data and making decisions.
In order to say how good our system is,
we want to say that overall, whenever you use the system,
there's going to be some random parameter,
there's going to be some value of the estimate.
And we want to know what's the probability that these two will
be different.
We can calculate this overall probability of error
by using the total probability theorem.
And the conditional probabilities of error as
follows.
We condition on the value of X. For any possible value of X,
we have a conditional probability of error.
And then we take a weighted average
of these conditional probabilities of error.
There's also an alternative way of using the total probability
theorem, which would be to first condition on Theta
and calculate the conditional probability of error
for a given choice of this unknown parameter.
And both of these formulas can be used.
Which one of the two is more convenient
really depends on the specifics of the problem.
Finally, I would like to make an important observation.
We argued that for any particular choice
of an observation, the MAP rule achieves the smallest
possible probability of error.
So under the MAP rule, this term is as small
as possible for any given value of the random variable,
capital X.
Since each term of this sum is as small as possible
under the MAP rule, it means that the overall sum will also
be as small as possible.
And this means is that the overall probability of error
is also smallest under the MAP rule.
In this sense, the MAP rule is the optimum way
of coming up with estimates in the hypothesis-testing context,
where we want to minimize the probability of error.