In this lecture, we will concentrate
on the study of Markov chains in the long run,
and study under what conditions a Markov chain
exhibits steady-state behavior, and under what conditions
such steady-state behavior is independent
of the initial starting state.
More precisely, we will look at long-term state occupancy
behavior-- that is, in the n-step transition probabilities
when n is large.
So assume that we have a Markov chain which is initially
in a given state i, and consider the probability
that the chain is in a specific state j after n transitions.
Question-- does that probability converge to some constant
when n goes to infinity?
And if this is the case-- second question-- can this constant be
independent of the initial state i?
We will see that for nice Markov chains,
the answers to both questions will be yes.
How to characterize nice Markov chains?
We will use several new concepts, one dealing
with a Markov chain being aperiodic or not,
and the other with the notion of recurrent classes.
Without going into details now, let
us simply mention that we will show
that the existence of convergence
will be tied to having an aperiodic Markov chain.
And in case we have convergence, the independence
from the initial state will be tied
to having a single recurrent class.
We will end this lecture by looking in detail
at the special and important class of Markov chains usually
known as birth-death processes.