So now we can come to the central topic
of our lecture, which describes the conditions under which
a Markov chain reaches steady state.
The question that we are asking, and which
we motivated in the previous lecture
by looking at an example with a simple, two-state Markov chain
is the following-- we are asking whether the probability
of being in state j at time n, given that you started at time
0 in state i, converges to some constant, pi of j.
In fact, that question consists of two parts.
Do we have convergence?
And is it independent of i?
We have seen an example where it is not always the case.
For example, in this Markov chain,
you have two recurrent classes.
This is one current class here.
And then there's a second recurrent class here.
And we know that if we are interested
in the long-time probability of being in that state, assuming
that you started in one of these states here,
the probability will be 0 to be here.
But if you started in one of these two states,
the probability would be positive.
So clearly here, the initial conditions
will matter whenever you have two or more recurrent classes.
So what would happen if you have only one recurrent class?
So let's remove this one and consider this situation here,
where you have only one recurrent class.
In that case, what we have seen--
this is still not sufficient.
Indeed, if you look at this recurrent class
and you are interested in 9 and assume
that you started at 9, then at time 1,
you will be either here or here.
And at time 2, you will be back at 9 for sure.
And in general, for time n even, the probability will be one,
and for time n odd, it will be zero.
So that specific n-step transition probability
in that situation here will never converge.
It will keep oscillating between 0 and 1.
So the issue here is that we had a periodic recurrent class,
and the period in that case was 2.
So let us consider now the final case where
you have only one recurrent class.
And that recurrent class is not periodic.
And how do we realize that this is not periodic here?
Well, we have a self transition here.
So now that we have one recurrent class,
and this recurrent class is aperiodic, the question is--
do you have this kind of convergence here?
And it turns out-- and this is the big theory of Markov chains
under the name of the steady-state convergence
theorem-- that indeed, yes.
The rij's do converge to a steady-state limit, which
we call a steady-state probability as
long as these two conditions are satisfied.
So in summary, not only these two conditions
are necessary, like we had seen with our counter example,
but they are sufficient.
We're not going to prove this theorem here.
It's a little bit complicated.
But what is the intuitive idea behind this theorem?
Well, let us think intuitively as
to why the initial state does not matter,
when the chain has a single recurrent
class and no periodic states.
The technique is pretty classical.
The idea is the following-- think
about two independent copies of that Markov chain,
starting at two different initial conditions.
So for example, think about a red copy.
And the red copy would initially start at state 2.
And then at each unit of time will jump to the next state,
according to the transition probabilities of this Markov
chain.
So for example, so this is at time 0, which was here.
Time 1 might come here.
Time 2, 3, 4, 5 and so forth.
So this is one copy of the Markov chain.
So think about another copy, the blue copy.
And assume that the blue copy started here at time 0.
And again, independently of the other,
but during the same unit of time,
it will jump from state to state according
to the transition probabilities again.
So think that maybe this one will go here.
Then will go here.
Times 2, 3 will be here, 4 here, maybe 5 here.
And so forth.
Now look at these two independent copies.
There will be a time and in that case, for our little example
here, is the time 4, where for the first time
they collide, in the sense that they
jump to the same state at the same time.
So at time 4, both of them are here.
Now, think a little bit about the future evolution
of these two independent copies, given that they are in state 4
now.
And here we are using the Markov property
to say that the future evolution of the blue copy
is independent of the previous path.
Given that you are in state 4, the fact that you started in 1
does not matter for the future evolution of that blue copy.
And for the red copy, given that you are in 4,
the fact that you started in 2 does not
matter to characterize the future evolutions
of that red copy.
So in some sense, probabilistically speaking,
these two copies cannot be distinguished for their future
evolutions, given that they both are at state 4.
So this means that the initial conditions for these two
copies, given that these two copies met at a given state,
at a given time-- probabilistically speaking,
nothing can differentiate them in the future because
of the Markov property.
That's essentially the high-level idea of this proof.
Now, the key thing here mathematically
is to prove that whenever you have a Markov chain that
has a single recurrent class and this single recurrent class is
not periodic, and you start from any initial conditions,
the two copies will eventually meet in a given
state at a given time with probability 1.
OK.
So now let's assume that the theorem holds.
That means that yes, indeed, we have
proved the existence of these steady-state probabilities.
The question is now how to calculate them.
Well, the way to do it is to start from our key recursion
that we had for the m-state transition probabilities.
So where we assume here that we have m
states, and we are going to take the limits
on both sides of this equality.
So when n goes to infinity, we know that rij of n
will go to pi of j.
And here, when n goes to infinity,
in some sense n minus 1 also goes to infinity.
And so rik of n minus 1 should go to pi of k.
And so we are using that property.
And again, we take the limit as n goes to infinity.
And we say that rij of n converges to here.
Now, the limit on this side-- you
have a limit of a finite sum.
You can exchange the summation and the limit.
And so you take the limit inside.
The limit of rik of n minus 1, when n goes to infinity,
goes to pi of k.
And then you have the resulting term.
And so from that, taking the limit
again as n goes to infinity on both sides,
you end up with this equation here for j.
Now, you can do that for any of the j of your Markov chain.
So you have m states, so you end up having m equations.
And you have m unknowns, the m pi j's.
So this is a system of m equations with m unknowns.
Unfortunately, this system is singular
and it has multiple solutions.
And one way to see that is the solution pi j
equals 0 for all j is a valid solution to the system.
Zero equals zero.
So clearly this is not very informative.
So maybe we need one more condition
to get a uniquely solvable system of linear equations.
It turns out that the system of equations
has a unique solution if you impose an additional condition,
which is pretty natural, which means that the pi
j's are actually probabilities.
They should all sum to 1.
In other words, in the future, if you
ask yourself what is the probability of being in state
j, and you get pi of j, the summation
of all of the possible states have to be 1.
If you consider that additional equation, plus that system
here, so if you consider this extended system,
then you can show that this has a unique solution.
And this unique solution cannot be this one.
And so in conclusion, we can find
the steady-state probabilities of the Markov chain
by just solving these linear equations, which
should be numerically a straightforward procedure.