In this lecture, we introduce Markov chains, a general class
of random processes with many applications
dealing with the evolution of dynamical systems.
As opposed to the Bernoulli and Poisson
processes, which are memoryless in a sense
that the future does not depend on the past,
Markov chains are more elaborate, as they
allow some dependencies between different times.
However, these dependencies are of simple and restricted
nature, captured by the so-called Markov property.
Conditional on the current state of the Markov chain, its future
and past evolutions are independent.
As mentioned in the unit overview,
we will only consider discrete time
Markov chains that evolve within finite state spaces.
This allows us to concentrate on the main concepts
without having to deal with some required technical details
needed to study general Markov processes under continuous time
and general, possibly uncountable, state spaces.
We will first introduced the basic concepts,
using the simple example of a checkout counter
at a supermarket, an example of a simple queuing system.
We will then abstract from the example
and give some general definitions, including
the central notions of states, transition probabilities,
Markov property, and transition probability graphs.
Afterwards, we will look at various questions,
such as predicting what will happen in n-steps
in the future, given the current state of our system.
We will define n-step transition probabilities exactly
and show how to calculate them efficiency.
We will also discuss what could happen
when we let the Markov chain run for a very long time.
We will end this lecture by introducing
the notions of recurrent and transient states
and their importance in studying Markov chains in the long run.