So let us first review the steady state
behavior of Markov chains.
Consider again the following example.
This chain has some recurrent states, some transient states,
and a single recurrent class.
So for example, state 9 is recurrent.
State 3 is recurrent.
State 5 is recurrent.
And why are they recurrent?
Because whenever you are in 9, no matter where you go,
there's always a way to come back.
You can go to 3 and come back, go 5 and come back.
And actually, this is a recurrent class.
And this is a recurrent class, because all
these recurrent states communicate between each other.
What about the other states-- well they are not recurrent.
So for example, state 1-- and once you are here,
there is a possibility that you go there,
and you will never come back.
So it can not be recurrent.
So it's transient.
What about 4?
4 also has the possibility at one point to go there.
And then from there, it will never come back.
So 4 is also transient.
As for 2, no matter where it goes, well,
it's going to reach or touch a transient state.
So by definition, it will be also transient.
So they have three transient states
and three recurrent states.
Also, this recurrent class is not periodic.
So it's aperiodic.
And why is it not periodic?
Well, here, there is a simple way to tell.
We have the existence of a self-transition probability.
And that's enough to show that this recurrent class is not
periodic.
So this is one of the nicest possible Markov
chains in the sense that they have the following property--
the probability that you find yourself
at some particular state j.
At time n, when n is very large, it
converges to a steady-state value
that we denote by pi of j.
There are two aspects to this property.
First, the limit exists, so the probability of state j
does not fluctuate.
It settles to something in the long run.
And furthermore, that probability
is not affected by i.
Now, if we don't know where the chain started, and we
want to know the unconditional probability of being in state j
in the long run, when n is large,
then either we are given an initial distribution
over the states, or we can choose
any initial distribution.
For example, we can assume that all initial states are equally
likely-- or any other type of distributions.
And then you can condition over all the initial states,
use the total probability theorem,
and you're going to get the same answer, pi of j, in the limit.
Let's see how to do that.
So this is the summation of all i.
So you condition on that state i.
So it's Rij of n times the initial probability
distribution of your choosing.
So this is the total probability theorem.
Now, in the limits, when n goes to infinity,
this goes to pi of j, independent of i.
So you can take this expression, the limit,
and take it out of the summation.
And then you have the summation of-- probability of x0
equals 1.
These are probabilities, so they sum to 1.
So in the end, you have that converges to pi of j.
So the conditional probability, given the initial state,
is in the limit, the same as the unconditional probability
when n is large.
And in that sense, it tells us that x of n and x of 0
are approximately independent.
They become independent in the limit as n goes to infinity.
So if the Markov chain has run for a long time,
and you are asked the question, "Where is the chain now,"
then your answer should be, I don't know.
It's random.
But it's going to be in a particular j
with that particular probability, pi of j.
So in that sense, the steady-state probabilities
are valuable information.
So how do we compute them?
Well, for transient states, like these, they are 0.
So pi of 1 is 0.
Pi of 2 is 0.
And pi of 4 is 0.
And why is that?
Well, if your initial state were one of the states here,
the probability of being in here is 0, no matter what.
But even if you started here initially,
in one of these states, you might, for a while,
fluctuate and turn around like that.
But eventually, after a finite amount of time,
you will go to that class and never come back to 1.
So in the long run, the probability
of finding yourself in state 1 will be 0.
And this is the same for 2 and 4.
Now, how do we calculate these?
Well, for these states in the recurrent class,
we compute them by solving a linear system of equations,
which are called the balance equation-- these--
together with an extra condition.
The normalization equation here has
to be satisfied, because these are probabilities,
and they have to sum up to 1.
And we have seen that the system of m plus 1 equation
provides a unique solution to this kind of system
for the recurrent class.
So you would apply that to that recurrent class.
And in that example, you have three states,
so you would choose m equals 3 for that example.
And you would solve the system to get the pi j.
Now, what if we had multiple recurrent classes?
Consider this chain.
It is an expanded version of the previous one
with additional states.
Some of these are recurrent, and one is transient.
But now we have two recurrent classes.
And that was our 1 class, so class 1.
And now we have a second recurrent class, class 2.
So what happens in the long run, when
you have situations like that?
Well, in the long run, if you start from here,
you're going stay here.
And in some sense, the study of that recurrent class
is the same as the study of that recurrent class.
And in order to find the steady-state probabilities
of these states, assuming that you started in one of these,
will be exactly the same as before.
So you will use the same system, with m equals 3 here.
Now, if you had started here instead,
again, this is a recurrent class,
and you have m equals 2 states here.
And in order to find what is the steady-state probabilities
of these two states, you could use the same kind of result
here, but you apply it with m equals 2 in isolation.
So you just concentrate on that.
If, on the other hand, your Markov chain started from here,
for example, for that specific example,
you're guaranteed that the next transition you'll end up here.
And then you can do the same thing as before.
We still know that the steady-state probability of 8
will be 0 and 0 and 0 and 0.
Now, what would happen if you started from here,
from one of these states?
Well again, for a while, you might
travel throughout this system here.
But eventually, you're going to move away from that.
And you will either go through a transition going
into that recurrent class via this transition
or via this transition.
And once you're in there, essentially, the chain
will remain there.
And so you do the same calculation as before.
And if, on the other hand, you transition away from that class
and arrived in this recurrent class,
then you would apply the result that you had here.
So in some sense, conditional on the fact that you left
the states and you arrived there--
in that conditional world, you can isolate yourself and really
solve the problem for that class-- and the same
from that class.
Now, of course, this raises the question, if I start from here,
how do I know whether I'm going to get here or here?
Well, we don't know.
It's random.
So we will be interested in calculating the probabilities
that eventually you end up here versus here.
And this is something that we are
going to do towards the end of today's lecture.