In this video, we continue our exploration
of quantities of interest associated
with the short-term behavior of Markov chain.
This time, we suppose that we have a Markov chain composed
of a single recurrent class, such as this one.
Here, we have our recurrent class.
And these are transient states.
We also assume that we're interested
in a specific recurrent state-- let's say 9.
So this is my state s.
And we will assume that the Markov chain starts
from a given initial state, state i,
and assume that i is 1.
Now the question we ask is, given that you started in 1,
how long is it going to take to reach 9 for the first time?
We know this is a recurrent class,
so we know that the Markov chain eventually
will come to that class and circulate here.
So you know that the Markov chain will get to that state 9.
You are interested in knowing when
it does so for the first time.
Of course, we don't know for sure.
This is random.
It is a random variable.
And let's try to calculate the expected value
of this random variable.
More precisely, this is what we would like to do.
We would like to find the mean first passage time from i
to s, from any starting state i.
Here, we're going to illustrate that with 1.
And mathematically, this is what we have here.
So here again, what you have is that the Markov chain
started in state i, at time 0.
And now you are looking at this set here.
And the set records all the time n such
that your Markov chain, Xn, is in the state s of interest.
So out of this set, you're looking at the minimum, n,
that verifies that.
So this is going to be a random variable.
And you take the expected value of that.
And this is what we call the mean first passage time from i
to s.
Again, this is the expected number of steps
in order to get from i to s for the first time.
So how do we go about calculating such a quantity?
Well, let us think a little bit.
We are looking at the event that you
will visit 9 from 1 for the first time.
What happened after visiting 9 is
of no relevance in our calculation.
In other words, our calculation will never involve this arc
and will not involve this arc either.
Because in order for the Markov chain
to traverse this arc or this one,
it would have to visit 9 first.
So what it means is that we can forget about this arc,
and we can forget about this arc-- in a sense,
that they don't matter in the calculation
of the mean first passage time to s.
Now, removing these arcs entirely
means that you would have to increase
the probability of that self transition from 0.2 to 1.
So what do you get?
Well, you get this following graph.
We have removed these arcs, and we
have increased the probability 0.2 here to 1.
And again, we have argued that calculating
the mean first passage time from i to s in this graph
is the same as doing the same thing for this graph.
But here, we have a very special structure.
We have only one recurrent state left, which is state 9.
And that state is an absorbing state.
All the other state-- this one was
transient, transient, transient.
This one becomes a transient state.
And this one is also a transient state in this new transition
probability diagram.
So we get a situation where we have one single absorbing
state, and we are interested in calculating
the probability starting from i to reach that absorbing state
and also the number of steps it takes
to do that, which is what we did in the previous video.
So let us repeat what we had seen before.
So this is going to be the unique solution to this system.
t of s equals 0, of course.
Since you start from s, you are in s.
And so the amount of time it takes to get to s is 0.
And otherwise, for all the other states that are not s,
this is the resulting system of equation that we have.
And the unique solution of this system
gives you the result of finding the expected amount of time
to go to the absorbing state s.
So using this simple trick, we've
been able to use the previous calculation
to calculate something else.
Let us consider a related question, which
we called the mean recurrence time of s.
Here, let's go back to the original graph.
And this is our state s.
And now the question is the following--
given that you are in s, what is the amount of time
it will take for the Markov chain
to return to s for the first time?
So for this question, the Markov chain is currently in 9.
And you ask yourself, how long is it
going to take, once you leave 9, to return to 9?
Here again, we don't know for sure.
It's a random variable.
And we are interested in the expectation
of that random variable-- in other words,
in the expected number of steps it takes in order
to get back to 9 once you are in 9.
And this is what we mean by the mean recurrence time of s.
And mathematically, this is what it is.
It is almost the same formula as the one that you had here,
except that here you have n greater than/equal
to 1 as opposed to n greater than/equal to 0.
It simply means that you're not interested in the first time
that you're in s, because you start from s.
So you want to have n greater than or equal to 1. n
equals 0 would not work. ts star would have been 0.
So how would you solve that problem?
Well, here again, you use the same trick that we used before.
And think in terms of tree.
Let's look at the Markov chain in state 9.
What can happen next?
From 9, it can go to 3 or it can go to 5.
Or it can jump on itself.
So this is with a probability 0.2.
This is with a probability 0.2.
And this is with a probability of 0.6.
And now, after you make one transmission--
so let's-- you are in 3 now-- what is the time to get to 9?
Well, it's exactly t3-- where the t
is, the solution here of that previous system.
What about from 5?
t5.
And what about from 9?
Well, t9.
And t9 was 0.
So from that tree, what you have is
t9 star will be 0.2 times t3 plus 0.6 times t5 plus 0.2
times t9 plus, of course, 1, because you
have done one transition.
Where, again, this value of t3, t5, and t9 are the ones
corresponding to this solution here.
So of course, t9 is 0.
So what we have shown here, that in general,
this is actually what you would have.
And this is exactly what we have written here.
ts star is 1 plus the summation of psj of t of j, where t of j
is the solution to this system.
So again, you started from 9.
You have to do a transition first.
You can do a transition unto yourself.
You can go to 3 or you can go to 6.
After that transition, which stands for the number
1 here, what happens is you're trying
to find the solution to this system, which
is the mean first passage time, from the current state where
you are, to 9.
And then when you put these two things together,
you get the mean recurrence time of 9.