The following content is provided under a Creative
Commons license.
Your support will help MIT OpenCourseWare
continue to offer high-quality educational resources for free.
To make a donation or to view additional materials
from hundreds of MIT courses, visit MIT OpenCourseWare
at ocw.mit.edu.
JEREMY KEPNER: All right, welcome.
Thank you so much for coming.
I'm Jeremy Kepner.
I'm a fellow at Lincoln Laboratory.
I lead the Supercomputing Center there,
which means I have the privilege of working
every day with pretty much everyone at MIT.
I think I have the best job at MIT
because I get to help you all pursue your research dreams.
And as a result of that, I get an opportunity
to see what a really wide range of folks are doing
and observe patterns between what different folks are doing.
So with that, I'll get started.
This is meant to be some initial motivational material,
why you should be interested in learning
about this mathematics, this mathematics of big data
and how it relates to machine learning and other really
exciting topics.
It is a math course.
We will be going over a fair amount of math.
But we really work hard to make it very accessible to people.
So we start out with a really elementary mathematical concept
here, probably one that hopefully most of you
are familiar with.
It's the basic concept of a circle, right?
And I bring that up because many of us
know many ways to state this mathematically, right?
It's all the points that are equal distance
from a particular point.
There's other ways to describe it.
But this is a basic mathematical concept of a circle
that many of us have grown up with.
But, of course, the other thing we know
is that, right, this is the big idea.
Although I can write down an equation for circle, which
is the equation for a perfect, ideal circle,
we know that such things don't actually exist in nature.
There is no true perfect circle in nature.
Even this circle that we've drawn here, it has pixels.
If I zoomed in on it, if I zoomed in on it enough,
it wouldn't look like a circle at all.
It would look like a series of blocks.
And so that approximation process, right,
where we have a mathematical concept of an ideal circle,
right, but we know that there are not really--
they don't really exist nature, but we
understand that it is worthwhile to think
about these mathematical ideals, manipulate them and then
take the results of the manipulation
back into the real world.
That's a really productive way to think about things
and, really, the basis for a lot of what we do here at MIT.
This concept is essentially the basis of modern
or ancient Western thought on mathematics.
If you remember your history courses,
this concept of ideal shapes and ideal circles
is the foundation of platonic mathematics
some 2,500 years ago.
And at the time, though, that they were developing
that concept, this idea that there
are ideal shapes out there and that thinking about them
and manipulating them was a more effective way
to reason about the real world, there was a lot of skepticism.
You could imagine 2,500 years ago
someone is walking around and saying,
I believe there are these things called
ideal circles and ideal squares and ideal shapes.
But they don't actually exist in nature.
That would probably not be well-received.
In fact, it was not well-received.
Many of those philosophers who were thinking about this
were very negatively received.
And, in fact, if you want to learn
about how negative the response was to this,
I encourage you to go and read the Allegory of the Cave, which
is essentially the story of these philosophers talking
about how they're trying to bring
the light of this knowledge to the broader world
and how they essentially get killed because of it,
because people don't want to see it.
So that struggle they experienced 2,500 years ago,
it exists today.
You as people at MIT will try and bring mathematical concepts
into environments where people are like,
I don't see why that's relevant.
And you will experience negative inputs.
But you should rest assured that this is a good bet.
It's worked well for thousands of years.
You know, it's what I base my career on.
People ask me, well, what's the basis of it?
Well, I'm just betting on math here.
It's been a good tool.
So this is why we're beginning to think this way when
we talk about big data and machine learning.
So really looking at the fundamentals, what
are the ideals that we need in order to effectively reason
about the problems that we're facing today
in the virtual world, right, and the fact
that this mathematical concept described the natural world so
well and also described in the virtual world
is sometimes called the unreasonable effectiveness
of mathematics.
You can look that up.
But people talk about math.
Why does it do such a good job of describing so many things?
And people say, well, they don't really know.
But it seems to be a good bit of luck that it happens that way.
So circles, that gets us a certain way.
But in most of the fields that we work with,
and I would say that, in almost any introductory course
that you take in college, whatever the discipline is,
whether it be chemistry or mechanical engineering
or electrical engineering or physics or biology,
the basic fundamental theoretical ideas
that they will introduce to you will be
the concept of a linear model.
So there we have a linear model, right?
And why do we like linear models?
And again, it can be physics.
It can be as simple as F = MA Or, in chemistry, it
can be some kind of chemical rate equation.
Or in mechanical engineering it can
be basic concepts of friction.
The reason we like these basic linear models
is because we can project, right?
I know that if that solid line represents
what I believe to-- you know, if I
have evidence to support that that is correct,
then I feel pretty good about projecting maybe where I don't
have data or into a new domain.
So linear models allow us to do this reasoning.
And that's why in the first few weeks of almost
any introductory course they begin with these linear models,
because they have proven to be so effective.
Now, there are many non-linear phenomena that
are tremendously important, OK?
And as a person who deals with large-scale computation,
those are a staple of what people do.
But in order to do non-linear calculations or reason
about things non-linearly, it usually
requires a much more complicated analysis and much more
computation, much more data.
And so our ability to extrapolate
is very limited, OK?
It's very limited.
So here I am talking about the benefits
of thinking mathematically, talking about linearity.
What does this have to do with big data and machine learning?
So we would like to be able to do the same things that we've
been able to do in other fields in this new emerging
field of big data.
And this often deals with data that
doesn't look like the traditional measurements we
see in science.
This can be data that has to do with words or images, pictures
of people, other types of things that don't feel
like the kinds of data that we traditionally
deal with in science and engineering.
But we know we want to use linear models.
So how are we going to do that?
How can we take this concept of linearity,
which has been so powerful across so many disciplines,
and bring them to this field that
just feels completely different than the kinds data
that we have?
So to begin with, I need to refresh for you what it really
means to be linear.
Before, I showed you a line and, hence, the line, linear.
But mathematically, linearity means something much deeper.
And so here's an equation that you may have first seen
in elementary school.
We basically have to two times three
plus four is equal to two times three plus two times four.
That is called the distributive property.
It basically says multiplication distributes over addition.
And this is the fundamental reason
why I would say mathematics works in our world, right?
If this wasn't true very early on in the earliest
days of inventing mathematics, it would not
have been very useful, right?
To say that I have two of three plus four of something, OK,
and then I can change it and do it
in this other way, that's really what makes mathematics useful.
And from a deeper perspective, the distributive property
is basically what makes math linear.
This is the property that, if this property holds,
then we can reason about a system linearly.
Now, you're very familiar with this type of mathematics,
but there's other types of mathematics.
So if you'll allow me, hopefully you
will let me just replace those multiplication symbols
and addition symbols with this funny circle times and circle
plus.
And we'll get to why I'm going to do that.
Because it turns out that, while you
have done most of your careers with traditional arithmetic
multiplication and addition, the kind
you would do on your calculator or have
done in elementary school, it turns out
there's other pairs of operations
that also obey this property, this distributive property,
and, therefore, allow us to potentially build
linear models of very different types of data
using this property.
So, as I mentioned, the classic two
are circle plus is just equal to regular arithmetic addition,
as we show on the first line, and circle times is
equal to regular arithmetic multiplication.
So those are the standard ones.
And, by far, this pair, this is the most common pair
that we use across the world today.
But there are others.
So, for instance, I can replace the plus operation
with max and the multiplication operation with addition, OK?
And the above distributive equation
will still hold, right?
That's a little confusing.
I often get confused that multiplications is now
addition.
But this pair sometimes referred to as max plus-- you'll
sometimes hear about it as max plus algebra-- is actually
very important in machine learning and neural networks.
This is actually the back end of the rectified linear unit,
is essentially this operation.
If you didn't understand what that meant, that's OK.
We'll get to that later.
It's very important in finance.
There are certain finance operations
that rely on this type of mathematics.
There are other pairs, also.
So here's one.
I can replace addition with union and multiplication
with intersection, right?
Now, that also obeys that linear property.
This is essentially the pair of operations
that, anytime you make a transaction and work
with what's called a relational database, that's
the mathematical operation pair that's sitting inside it.
It's why those databases work.
It allows us to reason about queries, which are just
a series of intersections and unions,
and then reorder them in such a way.
In databases, this is called query planning.
And if that property wasn't true,
we wouldn't be able to do that.
So this is a deep property of that.
So we can put all different types of pairs in here
and reason about them linearly.
And this is why that many, many of the systems
we use today work.
And so this class is about really exposing
that, that, really, the mathematics that
allows us to think linearly about data that we haven't
really thought of as maybe obeying
some kind of linear model.
This is essentially the critical point of this class.
So it goes beyond that, though.
So hopefully you'll allow me to replace those numbers
with letters, right?
So that's basic algebra there.
Just for a refresher, the previous equation,
we had A = 2, B = 3, C = 4.
But we're not limited to these variables, or these letters,
to being just simple scalar numbers,
in this case, real numbers or integers or something
like that.
They can be other things, too.
So, for instance, A, B, and C could be spreadsheets.
And that's something we'll go over with extensively
in a class, so that I can basically
have A, B, and C be whole spreadsheets of data
and the linear equation will still hold.
And, in fact, that's probably the key concept in big data,
is the necessity to reason about data
as whole collections and transforming whole collections.
Going and looking at things one element at a time
is essentially the thing that is extremely difficult to do when
you have large amounts of data.
A, B, and C can be database tables, right?
Those don't differ too much from spreadsheets.
And as I talked to you in the previous slide,
that union/intersection pair naturally lines up
and we can reason about whole tables
in a database using linear properties.
They can be matrices.
I think, for those of you who have had a linear algebra
and matrix mathematics, that would have been
the first example, right, when I substituted the A, B, and C
and had these linear equations.
Often, in many of the sciences, we
think about matrix operations and linearity
as being coupled together.
And through the duality between matrices and graphs
and networks, we can represent graphs and networks
through matrices.
Any time you work with a neural network,
you're representing that network as a matrix.
And, of course, all these equations apply there as well
and you can reason about those systems linearly.
So that provides a little motivation there.
As we like to say, enough about me,
let me tell you about my book.
So this will be the text that will we use in the class.
We are not going to go through the full text,
but we have printed out copies of the first seven chapters
that we will go through.
And we will hand those out later when you do the class.
So let me now switch gears a little bit
and talk about how this relates to, I think,
one of the most wonderful breakthroughs
that we have seen, or I've seen in my career,
and many of my colleagues here at MIT have seen,
which is what's been going on in machine learning,
right, which is-- it's not hype.
There's a real real there there and it's tremendously exciting.
So let me give you a little history, basic history
of this field.
So in a certain sense, before 2010, machine learning
looked like this.
And then, after 2015, it kind of looks like this.
So when people talk about the hype in machine learning,
or AI, really deep neural networks
are the elephant inside the machine learning snake.
It has stormed onto the scene in the last five years
and basically allowed us to do things that we had almost taken
for granted were impossible.
Just the fact that you're able to talk to computers
and they can understand you, that we can have computers that
can see at least in a way that approximates the way humans do,
these are really almost technological miracles
that, for those of us who have been working
on this field for fifty years, we had almost literally given
up on.
And then all of a sudden it became possible.
So let me give you a little sense of appreciation
for this field and its roots.
So machine learning, like any field,
is defined as a set of techniques and problems.
When you ask what defines a field, you ask, well,
what are the problems that they work on that other fields don't
really work on?
And what are the techniques they employ that really are not
really being employed by them?
So the core techniques, as I mentioned earlier,
are these neural networks.
These are meant to crudely approximate maybe the way
humans think about problems.
We have these circles which are neurons.
They have connections to other neurons.
You know, those connections have different weights
associated with them.
As information comes in, they get
multiplied by those weights.
They get summed together.
And if they pass certain thresholds or criteria,
then they send a signal on to another neuron.
And this is, to a certain degree,
how we believe the human brain works and is
a natural starting point for, how could
we make computers do similar things?
The big problems that people have worked on
are these classic problems in machine learning,
are language, how do we make computers understand
human language, vision, how do we make computers
see pictures or explain pictures back to us the way
we would like, and strategy and games and other types of things
like that.
So how do we get them to solve problems?
This is not new.
These core concepts trace back to the earliest days
of the field.
In fact, these four figures here, each one
is taken from a paper that was presented
at the very first machine learning
conference in the mid-1950s.
So there was a machine learning conference in the mid-1950s.
It was in Los Angeles.
It had four papers presented.
These were the four papers.
And I will say that three of them
were done by folks at MIT Lincoln Laboratory, which
is where I work.
And so that was basically the neural networks
of language and vision.
And we didn't play games, so that was it.
And you might say, well, why is it?
Why was there so much work going on
in Lincoln Laboratory in the mid-1950s
that they would want to pioneer in these directions?
At that time, people were first building computers
and computers were very special purpose.
So different organizations around the world
were building computers to do different things.
Some were doing them to simulate complex fluid dynamics systems,
think about designing ships or other types
of things like that or airplanes.
Others were doing them to, say, like what Alan Turing was
doing, break codes.
And our task was to help people who
were watching radar scopes make decisions, right?
How could computers enable humans to watch more sensors
and see where they're going?
How could we do that?
So at Lincoln Laboratory, we were building special purpose
computers to do this.
And we built the first large computer
with reliable, fast memory.
This system had 4,096 bytes of memory, which, at the time,
people thought was too much.
What could you possibly do with 4,096 numbers?
The human brain, of course!
Right, that's enough, right?
Most of us can remember five, six, seven digits, right?
So a computer that can remember 4,096 numbers
should be able to do things like language and vision
and strategy.
So why not?
So they went out and they started
working on these problems, OK?
But Lincoln Laboratory, being an applied research laboratory,
we are required to get answers to our sponsors in a few years'
time frame.
If problems are going to take longer than that,
then they really are the purview of the basic research
community, universities.
And it became apparent pretty early
on that this problem was going to be more difficult.
It was not going to be solved right away.
So we did what we often do, is we partnered.
We found some bright young people at MIT, people
just like yourselves.
In this case, we found a young professor named Marvin Minsky.
And we said, why don't you go and get some of your friends
together and create a meeting where
you can lay out what the fundamental challenges are
of this field?
And then we will figure out how to get that funded so that you
can go and do that research.
And that was the famous Dartmouth AI conference
which kicked off the field.
And the person leading this group, Oliver Selfridge
at Lincoln Laboratory, basically arranged for that conference
to happen and then subsequently arranged for what would
became the MIT AI Lab that was founded by Professor Minsky.
And likewise, Professor Selfridge
also realized that we would need more computing power.
So he left Lincoln Laboratory and formed
what was called Project MAC, which became the Laboratory
for Computer Science.
And then those two entities later merged 30 years later
to become CSAIL.
So that was the initial thing.
Now, it was pretty clear that, when this problem was handed
off to the basic research community,
there was a feeling that these problems would
be solved in about a decade.
So we were really thinking by the mid-1960s
is when these problems would be really solved.
So it's like giving someone an assignment, right?
You all are given assignments by professors
and they give you a week to do it.
But it took a little longer.
In this case, it took five weeks or, in this case, five decades
to solve this problem.
But we have.
We have now really, using those techniques,
made tremendous progress on those problems.
But we don't know why it works.
So we made this tremendous progress
but we don't really understand why this works.
So let me show you a little bit what we have learned,
and this course will explore the deeper mathematics
to help us gain insight.
We still don't know why it works.
At least we can lay the foundations
and maybe you can figure it out.
So here I am, fifty years later, a person
from Lincoln Laboratory saying, "All right.
Question one has been answered.
Here's question two."
Ha.
Why does this work and hopefully you can begin,
be the generation figured it out.
Hopefully it'll take less than fifty years.
Historically this type once we know how it works,
it usually takes about twenty years to figure out why.
So I mean impasses but maybe maybe you
know some people are smarter and they'll figure it out faster.
So this is what a neural network looks like.
On the left you have your input, in this case, a vector, y zero.
It's just these dots that are called features.
What is a feature?
Anything can be a feature.
That is the power of neural networks,
is they don't require you to a priori state what
the inputs can be.
They can be anything.
People have said, well, you know,
neural networks, machine learning,
it's just curve fitting.
Yeah, but it's curve fitting without domain knowledge.
Because domain knowledge is so costly and expensive
to create that having a general system that can do this
is really what's so powerful.
So the inputs: we have a input feature.
It could be a vector, which we call y sub zero.
And that can just be an image, right, the canonical thing
being an image of a cat, right?
And that can just be the pixels, values just
rolled out into a vector, and they will be the inputs.
And then we have a series of layers.
These are called hidden layers.
The circles are often referred to as neurons, OK?
And each line connecting each dot
has a value associated with it, a weight.
And the strength of the connection
between any two neurons is given by that weight.
And then, ultimately, the output, in this case,
the output classification, the series of blue dots there,
are the different possible categories.
So if I put in a cat picture, one of those dots would be cat,
maybe one would be dog, maybe one would be apple or orange,
whatever I desired.
And the whole idea is that, if I put in a picture of a cat
and I set all these values correctly,
then the dot corresponding to cat
will end up with the highest score, right?
And then I mentioned earlier that each one of these neurons
collects inputs.
And if it's above a certain threshold,
it then chooses to pass on information to the next.
And that's where these b values, which are vectors,
are just the thresholds associated
with each one of those.
It's a vector, one value associated with each one
of those that does those.
This entire system can be represented relatively
simply with one equation, which is
that yi plus one, which is the next vector in the layer,
OK, can be computed by the previous vector, yi
matrix multiplied by the weight, W.
So whenever you see transformations
from one set of neurons to the next layer, you should think,
oh, I have a matrix that represents all those weights
and I'm going to multiply it by the vector to get the next one.
Then we apply these thresholds, all right?
So we add these, the bi's, and then
we have a function that we pass it through.
Typically, this h function has been given the name
rectified linear unit.
It's much simpler than that.
It's just, if the value is greater than what comes out
of this matrix multiplied, if the value is greater
than zero, don't touch it.
Just let it pass through.
If it's less than zero, make it zero, right?
You know, it's a pretty complicated name
for a very simple function.
That's actually critical.
If you didn't have that h function,
this nonlinear function there, then we
could roll up all of these together
and we would just have one big matrix equation, right?
So that's really considered a pretty important part of it.
So that's pretty much what's going on.
When you want to know what the big deal is of neural networks,
that's all that's going on.
It's just that equation.
The challenge is we don't know what the W's and the b's are.
And we don't know how many layers there should be.
And we don't know how many neurons there
should be in each layer.
And although the features can be arbitrary,
picking the right ones do matter.
And picking the right categories do matter.
So when people talk about, I do machine learning
or I'm off working on-- they're basically
playing with all of these parameters
to try and find the ones that will
work best for their problem.
And there's a lot of trial and error.
And you'll hear about there's now
systems that try and use machine learning to do
that process automatically.
You know, how do you make machines that learn
how to do machine learning?
The basic approach is a trial and error approach.
I take a whole bunch of pictures of cats
that I now have cats in them, OK, and other things, right?
And I randomly set all those weights and thresholds.
And I put in the vector and I see what the system--
I guess what I think the number of layers and neurons
and all that should be and I run it through the system
and I get an estimate or a calculation
for what I think these final values should be
and I compare it with the truth.
That is, I just basically subtract it.
And then I use those corrections to very carefully adjust
the weights.
Basically, with the last weights first, I
do what's called back propagate these little changes to try
and make a better guess on what these weights should be.
So if you hear the term back propagation,
that's that process of taking those differences
and using them to adjust these weights by about
0.01% at a time.
And then we just do this over and over
again until eventually we get a set of weights
that we think does the problem well enough for our purpose.
So that's called back propagation, all right?
Once we have the set of weights and we
have a new picture that we want to know what
it is, we drop it in there and it tells us
it's a cat or a dog or whatever.
That forward step is called inference.
These are two words you'll hear frequently
in machine learning, back propagation and inference.
And that's all there is to it.
There's really nothing else to that.
If you can understand this equation,
you'll be way ahead of most people in machine learning,
you know?
You know, there's lots of people who
understand about all the software and the packages
and the data.
All of them are just doing that.
And I'd say this is one of the most powerful ways
to be ahead in your field, is to actually understand
the mathematical principles.
Because then the software and what it's doing
is much clearer.
And other people who don't understand
these mathematical principles, they're really guessing.
They're like, oh, well, I do this
and I throw this module in.
They don't really know that all it's doing
is making adjustments to these various equations,
how many different layers there are and stuff like that.
Now, why is this important?
You're like, well, what does it matter?
As I said before, we have this system.
It works but we don't know why.
Well, why is it important to know why?
Well, there's two reasons.
One is that, if we want to be able to apply
this incredible innovation to other domains-- so many of you
probably want to do that.
Many of you want to say, how can I apply machine
learning to something else other than language or vision
or some of these other standard problems?
I kind of need some theory to know.
Like, OK, if I have a problem that's like this one over here
and I changed it in this way, there's
a good chance it'll work.
There's some basis for why I'm going to try something, right?
Right now there's a lot of trial and error.
It's like, well, it's an idea.
But if you can have some math that says,
you know, I think that will probably work,
that really is a great way to guide your reasoning
and guide your efforts.
Another reason is that-- so here's
a picture of a very cute poodle, right?
And the machine learning system correctly
identifies it as poodle.
One thing we realized is that the way you
and I see that picture is actually very, very different
than the way the neural network sees that picture, all right?
And, in fact, I can make changes to that picture that
are imperceptible to you or me but will completely
change how the neural network-- that
is, given our neural network, I can basically make it
think anything, right?
And so, for instance, this is a famous paper.
And they got the system to think that that
was an ostrich, right?
And you can basically show this for anything, right?
So what's called robust AI, or robust machine learning,
machine learning that can't be tricked,
is going to become more and more important.
And again, having a deeper understanding of the theory
is very, very critical of that.
So how are we going to do this?
What's the main concept that we are going
to go through in this class?
This has mostly been motivational.
But how are we going to understand the data at a deeper
level?
You know, what's the big idea?
And the big idea is captured now in this,
I apologize for this eye chart slide,
which is what we call declarative mathematically
rigorous data.
So we have this mathematical concept
called an associative array.
And it's corresponding algebra that basically encompasses
the data you would put in databases, the data
that you would put in graphs, the data that
would put in matrices and it makes it all a linear system.
And the key operations are outlined there at the bottom.
If you recall, we have our basic little addition
and multiplication.
And then what's going to be very important,
probably the real workhorse for this-- and i
didn't show it before-- is called essentially
array multiplication or matrix multiplication.
And that's the far one on the right
there, which we often abbreviate just with no symbol, just A B.
But if we really want to explicitly call out
that its matrix multiplication as a combination
of both multiplication and addition,
we put in what we call the punch-drunk emoji, which
is a plus dot times.
You're probably all young enough that you don't even
remember emojis when they had to type them out
with just little characters and we didn't have icons, right?
So that meant you went to the bar and lost to the fight,
right?
But, anyway, that's really going to be the workhorse of what
we're doing here.