Before we get going with our discussion
of inference methods, it is worth
looking at the big picture for some perspective.
So far, we have concentrated on ways
to analyze probability models.
This part of the picture.
If our model has been selected in a careful way,
then it should also be relevant to the real world
and help us make predictions or decisions.
But how do we know that this is the case?
This is why we need to look at data that are generated
by the real world, and then use these to come up with a model.
This is what the field of inference and statistics
is all about.
This field has undergone a radical transformation
in recent years.
I will exaggerate a little, but in the past,
a statistician might be called to look
at problems such as this one.
You're given data on a few patients,
and you need to figure out whether a certain treatment is
effective or not.
But today, a statistician lives in a dreamland.
There are tons of data that are generated everywhere.
These allow us to build quite detailed large models involving
thousands of parameters.
And we do have the computational power to do all that.
In this landscape, the opportunities
for a statistician are endless.
So let me give you a small representative sample.
In a somewhat traditional setting,
one designs a data collection method,
and then uses these data to make a simple prediction.
This is the case, for example, in polling.
Where the purpose is to predict the outcome of an election.
Another field is marketing and advertising,
where the situation is somewhat similar.
Except now, we want to make predictions
not for a population as a whole, but
for each individual consumer.
And a particular application has to do
with so-called recommendation systems.
You collect ratings that people give to movies, as
in a famous competition that was announced by Netflix.
So you have data for every movie and the people
who have watched them.
You make a note of what rating that person
gave to a particular movie.
And now after you collect huge amounts of data of this kind,
you try to use this information to guess
whether, for example, this person is
interested in this particular movie or not.
This is a quite difficult problem.
A quite complicated one.
And it gave the community an opportunity
to develop fancier and fancier combinations of methods
in order to come up with good predictions
of unknown entries in this table.
Another field is, of course, finance.
The markets are truly uncertain.
And there are quite complete historical data.
Lots of them.
How do we use these data to make predictions?
Coming now to the natural sciences,
a revolution has been taking place in the life sciences.
There are tons of genomic data to be processed to find out
what combination of genes causes what disease.
Or we may want to find out the details
of the chemical reactions inside a living cell.
And there is an upcoming new frontier, neuroscience,
where there will be vast amounts of data that will be generated.
These will consist of brain measurements.
Of measurements of what each neuron is doing.
And hopefully, these will lead us one day
to finding out what the brain really does and how it works.
In the sciences, the list is endless.
It goes on and on.
In modeling climate and the environment,
scientists are using a huge models these days.
Which they try to calibrate using lots of available data.
And in physics as well, scientists
to use fancy inference methods trying to find
needles in a haystack.
Like rare particles or remote planets.
Finally, engineering is a fight against noise.
Engineers try to make devices that
will work in uncertain environments.
The field of signal processing is a prime example
where the generic question is to recover
the content of a signal.
For example, the content of a radio transmission
when a signal is received after it gets corrupted by noise.
I could go on and on for hours generating lists of this kind,
but we have to stop somewhere.
The bottom line is that the opportunities and the needs
are vast.
For this reason, we will look into the core methodologies
that come into play.
Fortunately for us, the fundamental concepts
and approaches turn out to be the same independent
of the particular application.