In this lecture, we introduce Markov chains, a general class
of random processes with many applications
dealing with the evolution of dynamical systems.
They have been used in physics, chemistry, information
sciences, queuing theory, internet applications,
statistics, finance, games, music, genetics, baseball,
history, you name it.
So what make these processes so powerful and practical?
Well, as opposed to the Bernoulli and Poisson
processes, which are memoryless in the sense
that the future does not depend on the past,
Markov chains are more elaborate as they
allow the representation of situations where
the future depends on the past and, to some extent,
could be predicted from the past.
More precisely, we are going to consider models
where the influence of the past on the future
is summarized by the notion of a state, which evolves
over time according to some probability distribution.
That's the link between the past and the future.
We will restrict ourselves to discrete time Markov
chains in which the state changes
at certain discrete time steps.
The state at time t plus 1, which is here,
is a function of the state at time t,
and there is some noise, or randomness.
As another view, this is what we will cover in this lecture.
We will first introduce the basic concepts
using the example of a checkout counter at the supermarket.
We will then abstract from the example
and give some general definitions.
Afterwards, we will look at various questions,
such as predicting what could happen in the future given
the current state of our systems.
We will end this lecture by giving some key
structural properties of Markov processes.
So let us start.