We will now go through an example that
involves a continuous unknown parameter,
the unknown bias of a coin and discrete observations,
namely, the number of heads that are
observed in a sequence of coin flips.
This is an example that we will start in some detail now,
and we will also revisit later on.
And in the process, we will also have the opportunity
to introduce a new class of probability distributions.
This example is an extension of an example
that we have already seen, when we first
introduced the relevant version of the Bayes rule.
We have a coin.
It has a certain bias between 0 and 1, but the bias is unknown.
And consistent with the Bayesian philosophy,
we treat this unknown bias as a random variable,
and we assign a prior probability distribution to it.
We flip this coin n times independently,
where n is some positive integer,
and we record the number of heads that are obtained.
On the basis of the value of this random variable,
we would like to make inferences about Theta.
Now to make some more concrete progress,
let us make a specific assumption.
Let us assume that the prior on Theta
is uniform on the unit interval, in some sense reflecting
complete ignorance about the true value of Theta.
We observe the value of this random variable, some little k,
we fix that value, and we're interested in the functional
dependence on theta of this particular quantity,
when k is given to us.
How do we do this?
We use the appropriate form of the Bayes rule, which
in this setting is as follows.
it is the usual form, but we have
f's indicating densities whenever we're
talking about the distribution of Theta,
because Theta is continuous.
And whenever we talk about the distribution of K, which
is discrete, we use the symbol p,
because we're dealing with probability mass functions.
As always, the denominator term is such
that the integral of the whole expression over theta
is equal to 1.
This is the necessary normalization property,
and because of this, this denominator term
has to be equal to the integral of the numerator
over all theta, which is what we have here.
So now let us move, and let us apply this formula.
We first have the prior, which is equal to 1.
Then we have the probability that K is equal to little k.
This is the probability of obtaining exactly k heads,
if I tell you the bias or the coin.
But if I tell you the bias of the coin,
we're dealing with the usual model of independent coin
flips, and the probability of k heads
is given by the binomial probabilities, which
takes this form.
And finally, we have the denominator term,
which we do not need to evaluate at this point.
Now, I said earlier that we're interested in the dependence
on theta, which comes through these terms.
On the other hand, the remaining terms
do not involve any thetas, and so they
can be lumped together in just a constant.
And so we can write the answer that we
have found in this more suggestive form.
We have some normalizing constant,
and here we keep separately the dependence on theta.
Of course, this answer that we derived
is valid for little theta belonging to the unit interval.
Outside the unit interval, either the prior density
or the posterior density of Theta would be equal to 0.
This particular form of the posterior distribution
for Theta is a certain type of density,
and it shows up in various contexts.
And for this reason, it has a name.
It is called a Beta distribution with certain parameters,
and the parameters reflect the exponents
that we have up here in the two terms.
Note that these parameters are the exponents augmented by 1.
This is for historical reasons that do not concern us here.
It is just a convention.
The important thing is to be able to recognize what it takes
for a distribution to be a Beta distribution.
That this that the dependence on theta is of the form theta
to some power times 1 minus theta to some other power.
Any distribution of this form is called a Beta distribution.
So now, let's continue this example
by considering a different prior.
Suppose that the prior is itself a Beta distribution
of this form where alpha and beta are
some non-negative numbers.
What is the posterior in this case?
We just go through the same calculation as before,
but instead of using one in the place of the prior,
we now use the prior that's given to us.
The probability of k heads in the n tosses,
when we know the bias, is exactly as before.
It is given by the binomial probabilities.
And finally, we need to divide by the denominator term, which
is the normalizing constant.
What do we observe here?
The dependence on theta comes through these terms.
The remaining terms do not involve theta,
and they can all be absorbed in a constant.
Let's call that constant d, and collect the remaining terms.
We have theta to the power of alpha plus k,
and then, 1 minus theta to the power of beta plus n minus k.
And once more, this is the form of the posterior
for thetas belonging to this range.
The posterior is 0 outside this range.
So what do we see?
We started with a prior that came
from the Beta family of this form,
and we came up with a posterior that is still
a function of theta of this form,
but with different values of the parameters alpha and beta.
Namely, alpha gets replaced by alpha plus k,
beta gets replaced by beta plus n minus k.
So we see that if we start with a prior
from the family of Beta distributions,
the posterior will also be in that same family.
This is a beautiful property of Beta distributions
that can be exploited in various ways.
One of which is that it actually allows
for recursive ways of updating the posterior of Theta
as we get more and more observations.