The following content is provided under a Creative
Commons license.
Your support will help MIT OpenCourseWare
continue to offer high quality educational resources for free.
To make a donation or view additional materials
from hundreds of MIT courses, visit MIT OpenCourseWare
at ocw.mit.edu.
PROFESSOR 1: We've revised 18.05.
At MIT, 18.05 is the introductory probability
and statistics class for mostly non-math majors.
For most of them, it's the only probability and statistics
class they'll take.
A lot of them will go on to be doctors
and so our goal is to have at least statistically
literate doctors out there.
That's important.
Now let's keep it back there.
We've been supported for two years
by a Davis Foundation grant.
The PI on the grant is Haynes Miller.
And I classify him as our PI and visionary,
because the vision of bringing active learning into the math
department was his, and it takes some effort.
So thank you for that, and it's working in some way.
You'll see.
Here's the outline of the talk.
We'll talk about what we inherited,
but we came into with 18.05, what we've done,
and what we've learned.
And along the way, we'll do a little demonstration.
So what we found when we-- the state of 1805 three years ago,
was it was a traditional lecture class
taught at MIT as a lecture with a recitation component.
And the enrollment was dwindling.
Students were not taking it.
They migrated to another class.
Part of it was that the faculty who
were teaching it were not-- how do
I say it gently-- were not so interested in this section
of undergraduate education.
And in fact, the year we took it over,
I think we were faced-- Haynes can correct me
if I'm wrong-- with the choice of either us taking it
over or not teaching the class.
So the first year we did teach it as a lecture class.
John was the recitation instructor.
I was the lecturer.
And then in the next two years we
converted to a fully flipped classroom, active learning,
and we co-teach it.
So what we found is just that.
Haynes had a big interest in active learning,
he's been pushing them in recitations
for years with some success I would say now, it's built up,
and Davis and everyone is interested in the world
of online learning and what can happen there.
So we have a component of the class online.
The class itself will be put on OpenCourseWare Educator
at the end of this year.
So what did we do?
We changed everything at once.
In retrospect, I would say I don't recommend it.
But having gotten through it, you're
always glad that you did it all at once, because now we don't
have to do any of these pieces.
We can just keep refining them.
It's a lot to talk about, so we're not
going to focus much on the new curriculum.
I'll just say a few words, that we're
really proud of the curriculum that we have.
We've introduced a unit on Bayesian inference.
[INAUDIBLE] here is our Bayesian, can I say zealot?
Does it say that's your religion on your Facebook site?
PROFESSOR 2: No.
That's me.
PROFESSOR 1: Oh, that's John.
That's on John's site.
John is a convert.
A Bayesian unit.
What that has done for us is allows
us to bridge the gap between the probability and statistics
part, which is always a hard gap.
Probability I'll say is a mathematical science.
Statistic is that statistics are at least as
much art as science.
And so it allows us to bridge that.
Bayesian inference and Bayesian learning
are becoming more and more important today.
So it's good that they see it.
And finally, even if they're just
doctors who need to know what a p-value is,
we feel we can explain the p-value better
by having introduced this unit.
So we actually added three weeks worth of material to the class,
and we'll talk a little bit about that later.
Then we'll speak more about the pedagogy,
the importance of the classroom, and what technology
we have in the class.
And now we-- actually we're going to show you--
this is a video-- a little bit, where you should pay attention
while you're watching the video is
the energy in the classroom and particularly the comments
and the oohs and aahs that come at the end when you'll see John
talking some students on this.
Today we're going to do Bayesian updating, which
is to say we're going to be using Bayes' theorem
to learn from data.
Which treatment would you choose if you needed a treatment?
Treatment one cured 100% of the patients.
Treatment two cured 95% in a trial.
And treatment three cured 90%.
All right.
There's some holdouts for the 95% cure.
I'm not sure why.
What if I gave you this information.
Treatment one cured three out of three patients.
That's 100%.
Treatment two cured 19 of 20 patients.
That's 95%.
Or you have a standard treatment which
has cured 90,000 out of 100,000 patients-- 90%.
PROFESSOR 2: So suppose, in this at MIT mug,
I have dice of two types-- 4-sided and 20-sided.
I'm going to randomly pull out a die and roll it.
All right, I got a one.
Which type of die do you think I randomly chose?
OK that was actually what I had in this cup.
Does this change at all of your analysis
of which die you think I rolled and got a one with?
So this is the first board question.
All right.
I want you to practice making the table Jerry showed you.
Make one for if it were 13.
Make one if I rolled a five instead.
Or do the same question if I had rolled a nine.
MALE VOICE 1: He knew it was a 20-sided die.
PROFESSOR 2: Your beliefs have changed
based on that about the probability that you
have each of the five dice.
So if you take that and you use that as your prior
here, then the likelihood, multiply and sum,
that should give you--
FEMALE VOICE: I Like that.
PROFESSOR 1: Right?
So I've lectured for years.
I don't know that I've ever heard
of that many oohs and aahs or a statement,
I like that, at the end.
Sometimes there's a student who nods very enthusiastically.
OK.
So what I'm going to go through here is just what we do.
I'm going to resist the urge to talk
a lot about why we like doing it,
because that will come later.
And John will talk about that.
The structure is we need three times 80 minutes.
So that's 240 minutes, which matches the five
times 50 minutes that a traditional class meets.
Three lecture hours, two recitation.
So it's not more time, but at all times
we have two teachers this year and three assistant teachers
in the room.
So there's five instructors in the room at all times.
What we have them do.
Before class, they do some reading.
To get them to do the reading, there's
reading questions which they get credit for.
These are all online, it's hosted on MITx,
and they get immediate feedback.
They know whether they got the question right or wrong.
It's possible to ask lots and lots of different types
of questions, although we restrict ourselves mostly
to multiple choice and fill in the number type of question.
Jenny French, who's here, has done the same thing for 18.03,
and she's been more ambitious with the types of questions.
They come to class and we do a minimal amount of lecturing.
What gets left out of a math class, of our class,
as opposed to a traditional class, lecture class, is we
don't do examples.
We're going to have the students work examples,
and it's much more better-- much more better?--
much better for students to work one example than
for the teacher to work two.
We feel this strongly.
You saw that.
You saw a little bit of lecturing.
It was cut.
I maybe lectured a little more in the video than you saw,
but very quickly, we have them up
at the boards doing group problems
as you saw in the video, while the teachers and the assistants
are circulating and talking to people
and helping them get past stumbling points
and assessing what they're understanding and correcting
misconceptions.
Other components of the class which you didn't see as much:
we have whole-class discussions; we'll
have them discuss things that they're tables in groups;
we can operate on many levels of interaction
between students and students, and students and teachers.
We also use a little bit of technology, clicker questions,
where we'll ask a multiple choice,
usually a kind of concept question
to see if they've understood one idea,
and it also allows us to take attendance.
And the students see it that way.
Once a week, on Fridays, we actually
do a computer based studio, where
they have their computers.
It's the one time we let them take their computers out
and we do it in our-- typically we're not
introducing new material.
We're doing more advanced work or simulations
that you can do on the computer.
And finally, we do have traditional problem sets.
They have to turn in the problem set on paper once a week,
and it's graded in the usual way.
The pset checker is not traditional.
This is something we borrowed from the physics people.
Safe is here, I think.
I think you guys pioneered this.
It allows the students to check their answers
to their homework ahead of time, which allows us to tell them
we care about how you present it,
that your logic and your thinking is correct,
it allows the students-- I can't resist saying
this one thing's on, I'm sorry.
It allows the students to reflect on the piece sets
as they're doing it.
One of the things we know is when we turn back a pset a week
later, it's gone cold, they've barely looked at it
to see what they've done wrong.
Here they see immediately it's wrong, and they can reflect.
OK.
I think now we have an active part of this.
John is going to--
PROFESSOR 2: We believe strongly in active learning,
so we got to do some active learning.
But really what I want to take you through
is how we might envision a particular new concept being
introduced to the students.
What are the various components and how do they fit together?
So Jerry mentioned that we have this unit,
the middle unit of our course, on Bayesian inference.
And the first day we do a number of problems that involve dice.
In particular, at the beginning of the term,
we actually give students dice that we
expect them to keep with them throughout the term,
and we use them to generate data,
and to do probability experiments,
but also just to make something tangible.
And so, in particular on this first day,
the students will have come in having done reading, maybe we
had about 10 pages, they read, they answer some reading
questions.
And this introduces them to some new concepts,
but doesn't expect them to have mastered it.
And then, in the beginning of that class that you saw filmed,
we had some discussions surrounding these dice.
So what you see here is I have five dice.
This four-sided.
This is a six-sided regular old die.
Eight, 12, and 20.
So these are platonic dice.
And the question I have for you is as follows.
So I actually have smaller versions.
I'm going to take them and put them in this cup.
All right.
I'll shake it around.
All right.
Allison, reach in and grab whatever and don't look.
All right.
Hand it to me.
Thank you.
And then Lourdes can you come up and be our verifier here.
All right.
So four dice are still in this cup.
Lourdes is going tell--
PROFESSOR 1: John.
John.
PROFESSOR 2: No.
I'll do it in the cup.
So you're going to say what I roll.
OK.
I'm going to drop it in, shake it around.
All right.
Just say the number.
Do not say which die.
OK?
What number do you see?
AUDIENCE: Four.
PROFESSOR 2: A four.
Great?
So which die do you think I chose?
Raise your hand if you think it's the four-sided die.
Let's say which is the most likely.
Raise your hand if you think it's a six-sided.
OK.
Eight?
12?
20?
Raise your hand if you think actually they're all
equally likely at this point.
Great.
So this is an example of an excellent opportunity
to have a whole-class discussion, because clearly you
don't all agree.
So in our class, since they sit around these tables of nine
people, we might have them say, OK, well, discuss
at the level of your table-- you're all facing each other--
and see if you can come to some consensus.
So let's try to convince each other.
All right.
Well, I'm not going to tell you the answer just yet.
Let's do another roll.
Maybe that'll help.
AUDIENCE: Is that the same die?
PROFESSOR 2: It's the same die.
I didn't, You saw.
It's not magic.
All right.
What I just roll?
AUDIENCE: A five.
PROFESSOR 2: A five.
Ooh.
OK.
So now raise your hand, is it the four-sided die?
Very good.
Six-sided die?
OK.
Eight?
12?
20?
Are they all equally likely?
Raise your hand if you still think
they're all equally likely.
That's interesting, right?
So there's been some progress here.
Maybe discuss with your neighbors now.
I'll give you 10 seconds.
PROFESSOR 1: All right.
This is great.
Usually it takes us about a week and a half
before we get people to discuss this animatedly when
we do things.
They have to learn to trust us.
All right.
So you've had some discussions.
I might call on people to give their explanations,
but in the interest of time, let me just put a few more times.
Now I got a six.
Now I got a three.
Now I got a two.
Now I got a one.
Now I got a six.
OK.
Raise your hand if you think it a six-sided die?
Wonderful.
All right.
FEMALE VOICE: It is.
PROFESSOR 2: You never know in science whether you're right.
PROFESSOR 1: This is statistics.
I'm not going to tell you.
Sorry.
No, that's the difference between probabilities
and statistics.
You never know.
All right, so, so-- great.
We've gotten the students to appreciate
there's some interesting question here,
maybe some intuition.
They've discussed it.
Next, what you saw on the video was the students at the boards,
working on problems where they're actually
doing computations meant to take them
to the next level, where they're not just
beginning to appreciate that something's going on,
but to actually rigorously mathematically do
some computations.
And they've seen examples of this computation
in the reading.
And we're all going around the room,
and we're helping, interacting with the groups.
And after they do this, they come back together,
we might discuss a solution, maybe
there'll be some clicker questions involved.
And they'll still have, on Friday, the studio.
In the studio they might use R. Perhaps
we've written a script that simulates
this rolling of a die, and updates
the prior to the posterior about which die it is, and then
they'll experiment with different situations
with this simulation, or changing some component of it.
We don't expect them to be experts in R,
but we expect them to be able to understand
code we've written and modified in some way to build
their conceptual understanding.
And then they'll have a problem set.
The problem set will ask them some questions
that build further.
And there will be exams, and so on.
So that's the sort of trajectory of seeing
a topic, sort of building up your level of understanding.
So now, I'm going to say a little bit now about the why.
So Jerry told you of the what of what we do.
Here's a little bit about why we do it
and what we've learned from the experience of doing it,
and what we like.
So in terms of active learning versus traditional lecture.
The first thing to note is that just the physical act
of standing up serves many very useful purposes.
The first is that in a given 80-minute session,
students might be up on their feet
two or three times for 10 to 15 minutes.
And when they're up, they're not on the phone,
they're not on the computer, and they're not asleep.
So those are all good things.
But even more, just the act of standing,
it sort of in some ways activates your focus.
You're at the board, there aren't distractions,
you're with your group members, and you have a task at hand.
And so just the physical act of standing we find
is a way to raise the energy in the room.
Now I'll not make all you stand while we talk about this,
but I think you would agree that standing wakes you up.
The physical space.
This TEAL room is kind of amazing,
but what's most amazing about it is not
that it has all these computers at the tables, which
at this stage isn't so useful, because everyone has laptops.
It's not even really the video cameras and projectors.
It's just the physical space is such that students can rapidly
get out of the seats, find room on the board, work on a problem
comfortably, come back, sit at these tables.
Their chairs swivel without them getting up,
because they have little rollers.
It's carpeted, so it's not too noisy.
There's just lots of things about the physical space that
are really nice and make the class possible.
So if we had a choice between technology
versus just physical space, we would choose the latter,
in a heartbeat.
And that means that in classrooms where you might not
have whiteboards everywhere, you might want
to think about putting some up.
In Haiti we actually put up paper on the walls
in order to do simulations with professors there,
to get them to feel what active learning was like, at least
the way we were trying it.
Peer and teacher instruction.
So here, there's a lot of talk about peer instruction,
about how students can help teach one another,
and that's really important.
And there's also teacher instruction,
where the teacher's interacting with the student,
and helping them move along in their understanding.
And what we find, in the classroom, is really happening
is a lot of both.
They're both important.
So the peer instruction happens because they have discussions
at the table, or because they're working together
at groups in the board.
In particular, at the board, you might
have one student who's very strong,
a couple others who aren't as strong on a particular problem,
and us going around can very much
encourage the student who seems to be taking the lead
to make sure the other group members understand,
or to try to vary who's taking that lead.
But you see a lot of students explaining
to each other, the students being able to look over
at the next group if they're stuck, all of this
is a kind of peer instruction.
But there's also the teacher instruction,
which is the fact that Jerry and I, and then
a graduate student and two undergrads
are in the room, able to basically distribute ourselves
like zone defense, and monitor a certain number of groups,
and really keep tabs on the progress they're making.
Because you're going to have groups
that are faster and slower, and you want to kind of bring them
to about the same point at the same time,
so they can get the most out of the time
when you come to the explanation.
So that sort of interaction between the teachers
and the students is also really huge,
because the students actually feel like they get to know us
and we get to feel like we know them.
The student self-assessment, that's referring to the idea
that there's a really rich formative assessment happening
in the classroom, when the students are asked especially
to get up and solve problems at the board.
When you listen to teacher lecture
at you, doing an example, you may nod your head
and think you understand it.
But when you're at a board, or when you're using a clicker,
you actually have to do the problem.
You figure out really quickly if you understand it,
because if you're standing there,
and you have no idea where to get started,
you don't understand it.
And you know.
But you want to create an environment
where that's OK, because maybe your group mates know.
Or maybe we know.
And they're comfortable saying, hey John, Jerry, can
you help us get started.
Or even better, we're on top of them
even before they get to that point,
because we have a sense at this point which
groups are going to have the most trouble getting started.
And there are some groups where once you get them started,
they fly.
And it's just a matter of being there at the beginning.
There are other groups which are the reverse.
So you get to learn that through this rich interaction.
And that self-assessment the students say really
helps them know if they know.
Right?
Teacher formative assessment.
I guess I just mentioned that.
That's us being able to see where these groups are at.
So the advantage of them standing up at the board
is that from a distance you can scan the room
and see very quickly how far each student or group has
gotten on these problems, which student is holding the pen
or explaining.
By the time this period has ended
where they've been working on these problems,
you've helped several groups get past several stumbling points.
So now comes time to bring the class back together
and go through an explanation of those
parts which are most helpful, and you know,
because you actually already helped them.
It's like you had an office hour where everyone attended,
and now you get to explain the answer, right?
And so that's been really exciting for us.
So those are some of the active learning benefits we found.
And then, in terms of technology, there are, I guess,
three ways we're using technology
to enhance the class.
The reading questions, as Jerry said,
these are, by and large, a way to get
students to at least open the PDF that has the reading.
The students at MIT are going to play some game theory
and manage their time to balance a lot of obligations.
You can't expect them to say your course is
the most important course, and of course I'm
going to do this before I do anything else.
But at least if they get to the stage
where they open the document, because they
know they're going to get graded, 5% of the grade,
on answering these reading questions,
you have a good chance that they're going
to have some first exposure.
And if you create an expectation at the beginning of the term
that we're going to assume that they've done the reading,
and not go over it as if they didn't, then that also
helps to enforce and get students to actually do it.
So they are actually graded on these questions
in terms of whether they're right or wrong,
and they're not meant to be particularly hard questions.
They're more like questions that are exactly
parallel to an example, so at least they
have to look through the example, change the numbers,
plug in an answer.
Clickers and attendance.
When you do an active learning course,
it's even more important that students come to class.
You can't watch an active learning course on video
like you can watch a lecture.
And so in some sense, it's a way to bring value back to campus
in the face of MOOCs.
Well, one way to force attendance
is to actually take attendance.
But we can do that implicitly with the clickers.
So students know that if they don't show up,
they're not clicking in to certain questions,
they don't get any points for that.
That's 5% of their grade.
Now we definitely have a few students who are still
not going to show up at all, because they ace every exam,
they learn it on their own, great.
They'll get 95%, they'll get an A in the course.
Right?
And that's OK.
It's not that necessarily everyone has
to learn the same way, but it's really
important for those students who aren't
able to do everything on their own that they come to class.
And we get a pretty good attendance rate.
And we think the clickers help incentivize that.
The pset checker, Jerry mentioned,
is a tool to get students to know ahead
of time whether or not they got their answer correct, so
that they still have time to reflect and ask a friend
or think about it some more while they might still
get some credit.
So there's still incentive.
As opposed to a couple weeks later,
when the problem set comes back, they have three other problem
sets, they have an exam, it's not in RAM,
might as well not give it back to them at all.
I can tell you how big the stack of problem
sets I have that students-- I mean,
every day, every week I bring them to the class.
I'm like, please, unburden me.
Take back these problem sets.
Most students don't even want them back.
It just tells you how little they're
benefiting from the grading of the problem sets.
And I think this tool in particular transfers
very broadly very easily.
It doesn't have to be a math class or a physics course.
It just has to be a course where there's some way
to do some assessment, or some way
to evaluate if a student has made progress.
It might be there's a formula and you just
plug in some number and that's how you know.
Granted, they don't learn what the answer is.
They just get told that 3.58 is not the right answer.
It's not very efficient for them to then try 3.59, 3.57, you
know, it won't help them.
So they use it in a way that's very satisfying if you've ever
taken a MOOC, which is you get those green checks when
your answer's correct.
It feels great.
And so they know, great, I figure these out.
I got green checks.
I'm turning this in.
It's going to be correct.
But they still get graded on their logic
and on their writing.
That's the only part of the problem set
that actually counts toward the grade.
So I think those are those points.
PROFESSOR 1: Right.
I can add that since it's on MITx, we have analytics.
They use the pset checker a lot.
Most students are using it.
I'd say well over 2/3.
And they go through, you can see how many
attempts they're making for the various questions.
Next slide.
Is it me?
PROFESSOR 2: I think it's both of us.
PROFESSOR 1: Ah yeah.
So we'll give our own takes on this.
So when you talk about active learning,
there's some common questions.
So how much work was all this?
The answer was an unbelievably large amount of work for us.
But part of that was we changed everything.
So it's hard to say what it would be,
had we taught this as a lecture course for three or four years,
had we changed our curriculum and added the Bayesian
unit before converting it into an active learning class,
had we not been so ambitious in using the MITx platform,
all of those things would have lessened the work.
But I think you could say it's still a lot of work
to convert from a lecture to a active learning format.
For one thing, there's no faking what
you're going to do in an active learning format.
We have slides, we have to have questions, well thought
out questions, ahead of time.
You can't pull examples kind of out of your hat the way
I would never do, but I've heard that some people do
when they lecture.
How much are you able to cover?
This is also a typical question.
How can you cover as much?
The answers is we find we can cover at least as much.
I'll say a little more about that.
You're cutting out examples, you spend a lot of time
in a math class working through examples.
The students are doing the examples
so they're learning the material better
than you're doing the examples.
And that time from your doing the examples
is saved for them to work.
In terms of 18.05, it's a little hard to measure.
We know we've added three weeks worth of material
to the class we inherited.
It's possible that the class we inherited was underambitious.
And we could've added that in a lecture class.
We know we cover the first unit, the probability unit in--
PROFESSOR 2: I think five weeks instead of seven.
PROFESSOR 1: Five weeks instead of seven.
And they do just as well on comparable exams.
So I think I feel safe in saying that we cover
at least as much material.
You spend more time in the classroom
because, as the teachers, we're in there for all 240 minutes.
None of it is pawned off on recitation leaders.
It might be interesting to know whether that's
really necessary for all the teachers
to be there all the time.
Do you have anything to add to that, John?
PROFESSOR 2: I think for us, one thing that added a lot of work
was that when we decided to change the curriculum,
we realized there was no textbook that
taught what we wanted to teach, so we wrote the textbook.
And that takes a lot of time.
PROFESSOR 1: Yeah, I said that was in November
for a class that was starting in January that we finally
decided we had to throw away the textbook.
PROFESSOR 2: Right.
So in the end, we wrote these materials,
but I can say that the students really
appreciated that the reading they were doing
was very targeted.
It told them the objectives, exactly what
we expected of them, what kinds of problems, at what level,
and so it turned out to have other benefits we
didn't foresee going in.
And then in terms of how much we're able to cover,
like Jerry said, we gave exactly the same first midterm
three years in a row.
We're not going to give it a fourth year in a row,
so don't--
PROFESSOR 1: No.
We added one question.
PROFESSOR 2: Plus one question.
So because we gave exam at seven weeks,
the first time we taught it, that
was just the probability unit.
The second and third time we taught it-- so third time
is right now-- that was the probability
unit plus like a week or two of more stuff.
Right?
So it had one more question.
Like six instead of five questions.
And if you remove the one more question,
they actually did as well or slightly better
compared to the first unit.
If you include it, they did it pretty much the same.
But you have the same amount of time to take the exam,
so it's a pretty concrete measure that they did better.
Now we don't know that that's because in part we
actually understood a little more statistics
and could explain better the second time around.
PROFESSOR 1: No, this is just probability.
We understood probability.
PROFESSOR 2: We did understand probability.
OK.
That was good.
PROFESSOR 1: We should say, so neither of us
are statisticians going in.
We've learned a lot of statistics,
and particularly how to teach it.
I don't think, at a mathematical level,
the statistics we're doing is terribly hard,
but we all believe, I think, as teachers
that the more expertise you have.
So Sansa, who I mentioned earlier,
does know a lot of statistics and has been consulting with us
and been a tremendous asset in getting us up to speed on this.
AUDIENCE: Question.
How did you know that the students didn't
cheat, for example, and have access
to the previous years' exams?
PROFESSOR 1: You ask a really good question,
but I think you can infer that from reading their papers.
If you read the papers, students are not
getting everything right.
PROFESSOR 2: By the way, we did change the wording, the topic,
and the numbers.
I mean we didn't leave it so you could just put the same answer,
certainly.
And doing it a third year was kind
of stretching the limits of what you're exactly asking.
PROFESSOR 1: We won't do it again.
I will say this, and I often get into a discussion
about cheating, which is I could give-- I know from lecture
classes and maybe Arthur will verify this who's
sitting there, Arthur Mattuck, whose done a lot of lecturing--
I've seen when I've teach recitation that I can tell
my students, the teacher has given you practice questions.
Three of them will be on the exam.
Two of the questions on the exam will be just small changes
of this, and I still know that many students won't find time
to do the practice exam.
I think it's kind of a myth.
If they found my exam in the copier-- well actually,
most of them would return it to me, but someone might cheat.
But the effort of going to try to do something like this
without being certain that you're going to be getting
something that's going to be on the exam,
it just doesn't happen that often.
So I think now if it comes out that we'd
give the same exam year in and year out,
people will look at it.
So we won't do that again.
Other observations.
This is kind of the psychological observations.
Active learning is a lot more fun for the students.
You could see even in that video when they stand up,
there's a lot of energy in the classroom.
You could hear the buzz.
People are not sleeping.
Co-teaching, if you can do it, is tremendous fun.
I can't get away with anything with John standing there.
I'm a little more generous to him.
I let him make mistakes.
Students like getting to know their teachers.
This is a really important value at MIT.
A number of students graduate from MIT
without knowing enough teachers well enough
to get three letters of recommendation.
Here, we're walking around, were meeting them,
both of us like students, and like talking to students.
And they appreciate it, they do get to know us,
we have to write a lot more letters of recommendation
than we ever did before.
There's a downside to everything.
The students like the targeted reading.
In fact, Glenda Stump, who is with the teaching and learning
laboratory here and has done surveys of our students
and interviews with our students, the report-- and this
was interesting to us-- they like the reading more than they
like watching videos.
Videos take longer.
The targeted reading is just what they need to know.
We're telling them, this is what we expect you to learn,
and it doesn't take as much time as a video.
And finally, students really love the pset checker.
It gives them a very comfortable feeling.
So finally, looking forward, Glenda
is continuing to study what we're doing.
At some point, there will be a paper report
that comes out of this.
The course will be on OpenCourseWare and the OCW
Educator, where we'll give-- it's being videotaped here,
so this talk might show up on OpenCourseWare Educator,
to tell other teachers about what we're doing,
as well as present the course materials.
And finally, and this is important,
we've overstaffed the class in the sense of what
the budgets will really allow.
We've had a Davis foundation grant that helps with it.
The math department has supported it.
We have to move to a more traditional staffing.
So actually next year, I'll teach the class by myself,
without John's help.
We'll see what we do.
I'll have to learn how do these sort of demos,
because we leave that to John.
And with just undergraduates and maybe a graduate student
in the class.
This year John didn't say it, but we
have actually another instructor who's
a statistician in the class with us--
Peter Kempthorne-- at all times.
And then, equally important for a project like this
is making a transition to the next teacher.
We're going to have to convince someone else
that they want to teach like this, and they want to do it.
Part of our strategy will be, after three years,
to have such a compelling set of materials
that is so easy for them to use, that it
won't feel like as much work.
So that's what we're looking forward to in the future.
Finally, come visit.
We have had visitors.
We like it when people come to visit.
If you send us a note, we'll tell you
what's a good day to visit.
If not, you can take your chances and just show up.
We're in the basement of the Stata Center
in the TEAL room, which we should have mentioned.
This is in part an outgrowth of the physics TEAL project.
PROFESSOR 2: It's a huge room.
There's plenty of space.
So you don't have to worry it'll be full.
PROFESSOR 1: You saw in the video it's a big room.
Stata, Tuesday, Thursday and Friday at 1:00 to 2:30.
Come for all or part of it.
PROFESSOR 2: And I should say that Tuesday and Thursday are
really the days, our sort of standard flipped days.
And then Fridays are when we do that studio with R,
and they have the computers.
So it's a bit different.
And of course some weeks they have exams.
So you might not want to come on an exam day.
That won't be as interesting.
So you might just let us know when you're going to show up.
AUDIENCE: What's the size of your [INAUDIBLE]?
PROFESSOR 1: We have about 60 students.
Right now the room could comfortably hold, I think,
90 people.
So the class actually had about 31,
32 students two years ago when we took it over
as a lecture course.
Last year that bumped up to the low 40s.
The first time there was a flipped,
and now we have 60, or maybe 58 the second time
through, so we're hoping that we can keep that going.
AUDIENCE: Do you find that it's more work to teach the class--
PROFESSOR 1: No.
I find it's less work once you're in the class.
If you have the slides prepared and-- you
spend a lot of time walking around just
schmoozing with students.
And working-- no, it's not.
It's easier, it's more energizing,
I'm never tired when we're done, I never
feel that I've just droned on for 40 minutes
and not modulated my voice at all.
So it's not more work to teach.
It's more work to prepare, I should say.
You have to be fully prepared.
There's no thinking about, yeah what
is this topic on the way to the lecture,
and giving the example, and being
confident that you'll be able to work it out the next line
before you get there.
AUDIENCE: You might have heard that we
don't have the bodies to staff like this, knowing
how much money we had.
So I guess I'm wondering, how does
the need for staffing scale with enrollment, if you 90 students
show up?
PROFESSOR 1: The physics department
has been doing this for a lot of years.
They have one instructor in the room
and typically a lot of graduate students
who would be teaching a recitation otherwise.
So you can staff it in that way, with one instructor
and graduate students.
Undergraduates, good undergraduates,
are also very good.
It's a little harder to find students
who are free Tuesday, Thursday and Friday.
I think maybe next year, because Friday is just the studio,
and there's less of a need, they do less on Friday.
If I get students just on Tuesday and Thursday,
we'll do it.
But there's a lot of very good undergraduates
who are quite helpful.
AUDIENCE: [INAUDIBLE] a lot of the grading is automatic.
Is that the only thing [INAUDIBLE]?
PROFESSOR 1: No.
The good--
PROFESSOR 2: No.
It's not automatic at all.
PROFESSOR 1: The psets are graded by graders on paper.
PROFESSOR 2: Undergraduates.
PROFESSOR 1: Undergraduate graders.
That's the same as if we taught it as a lecture.
You have to hire them.
It's easier to find graders because they
don't commit to being in a certain place
at a certain time.
AUDIENCE: I asked you this at lunch.
Did you get feedback from students
that help you refine it or make you think about the things that
maybe weren't so good and things that were especially good?
PROFESSOR 1: Yes.
So the first year we did the studio in MATLAB,
and it's not MATLAB's fault.
PROFESSOR 2: It wasn't MATLAB's fault.
PROFESSOR 1: It's not MATLAB's fault. It was our fault.
It was really tedious.
We did a lot of tutorials on using the software,
and so it was click here, click there.
You have students at very different levels.
MIT students are very good.
If you show them a program, a script--
you don't write complicated scripts with a lot of logic.
It's typically just a series of commands
that you can get out of this-- they're
able to see what's going on and use it.
So it's based on feedback from them.
Part of it is just being in the room
and realizing how dull it was.
But also from talking to students.
We changed that.
What else?
We did get feedback on others but I'm
forgetting what they say.
PROFESSOR 2: But we got feedback fact
that it's good to have the full week's materials posted
before the week starts, which was very
challenging the first year, because we were often writing
it like at that exact moment.
This year we've corrected that.
As far as using the software, I think exactly what Jerry said,
plus I'll add that we had to think,
we thought more of what our goals were,
with them interacting with the software.
And really it was less about making
them experts in using MATLAB or R,
and more about leveraging the software as a way
to increase their understanding of the material.
And from that point of view, it's
not quite as necessary to have them build up
complicated things from scratch, but rather
to be able to understand something
we've written well enough that they
can change components, play with it, visualize things,
and so we've really moved in that direction.
And also, another piece of feedback we got
was that students really appreciate
real-world applications, especially lot of them
are pre-med, they want to see medical research papers,
they want to see data that speaks to them.
And so Peter Kempthorne, for example,
has a background in the financial world,
so he's helped us create some studios centered around stock
prices moving around.
The one we created last week was a studio where, basically, they
used a two-dimensional Bayesian updating
to simulate looking for Malaysia flight 370
under the Indian Ocean by putting a sensor grid.
PROFESSOR 1: And it was found the next day.
Or they got that ping.
PROFESSOR 2: That's right.
Right?
They're not going to know how to make this sort of heat map
representation or a contour plot representation of the posterior
of updating based on what sensors detected, and so on,
but that's OK.
We can instead give them the framework,
they can play with it.
We can say, well, what happens if instead
of the black box being right in the middle of this grid
of sensors, it's up here.
And then they realize after playing with it,
oh, actually it takes longer to converge
and at first, the probability sort of spread out this way,
and why is that, oh, because it's
about the same distance left to right.
Those things were coming naturally to them.
Didn't mean they had to understand
how to do this complicated syntax,
but it was a benefit of getting these visuals right away,
being able to play around.
AUDIENCE: And relevant to what was
of interest to them because their field or something--
PROFESSOR 2: Well in this case, it was relevant to the New York
Times report the night before, which
said that they were deploying sensors behind ships.
So we talked about how would you modify
this to make it more realistic based on what they're actually
doing, which isn't so hard.
And they got the idea, hopefully,
that this stuff is useful.
PROFESSOR 1: So these are details.
I think part of the point is we're always getting feedback
from the students, because that's
part of what happens in a classroom, the kind
of formative assessment John mentioned.
One of the things we've done, and it differs
from what TEAL does, is-- a question
is, how long do you let them work on a problem?
And through their feedback, through talking to them,
and through observation, we've kind of converged on how long
you go with a problem.
And I think one of the differences with TEAL
is we give them a longer time to solve problems
than they do in TEAL, and they do express
an appreciation for that.
How did you learn to work effectively
in the TEAL classroom?
Did you observe others?
Did you have special training?
PROFESSOR 1: No.
We went to Peter Dourmashkin's class, we talked with Peter.
The day before our first class, we went into the room
and we rehearsed it down to the minute.
And then we just sort of tried it over and over again.
We were lucky the first day-- partly
I think because of the rehearsal-- was so successful.
Haynes, you probably remember how high
we were flying after that day.
And so--
PROFESSOR 2: The level of preparation,
quickly deteriorated after.
PROFESSOR 1: No, no, no.
We don't do it down to the minute.
We're more confident in 10-minute intervals,
but we're well prepared.
PROFESSOR 2: The room is, I mean there's a little control
panel you just need to learn how to make the screens go
up and down, or make a video camera go on or off.
Make the lights go up or down.
It's not that that's challenging,
but it definitely took a lot of trial and error
to optimize what's the best way to use those technologies,
and that was experimentation, we got better as we went.
PROFESSOR 1: I don't think the technology was as much.
But things like how long do you do a problem.
In the first year, I'd be off in the corner
talking with students and John would come and say,
you know, it's been a long time.
It's time to stop.
And I think maybe it happened in reverse on occasion.
PROFESSOR 2: It's a really common problem
with active learning is that you do have groups that
will go at different speeds.
One advantage with having enough staffing
is that you can try to target those groups,
get people to about the same point.
There are always going to be winners and losers
in any approach you take to teaching a class.
We're not claiming that what we're doing
is better for everyone.
But just this feedback you get as you're doing it, dynamically
in the hour, it really helps.
It really helps you, so know how to reach them.
AUDIENCE: Did you have a schedule of we're
going to do this for 10 minutes, we're going to do this for 15.
Did you have [INAUDIBLE]?
PROFESSOR 1: We have it broken down by slides.
And so the slides do it.
PROFESSOR 2: We have a slide presentation
that is prepared ahead, so it does give you an order.
PROFESSOR 1: And at this point, you
learn when you prepare lectures.
At this point we know that 16 slides we can get through,
20 slides we're not going to get through.
PROFESSOR 2: We should also say that--
PROFESSOR 1: We fall still into the mistake of thinking,
oh, we could do it.
We could do it.
But you never do.
PROFESSOR 2: One thing we also do that students, I think,
appreciate is that you're still not going to get every student
to understand the problem they just did, necessarily,
in the confines of the class time.
And there might be times where it doesn't make sense
to go through every detail of a problem,
because it wouldn't be that efficient, because most
students basically got it.
And so the version of the slides that we post right before
and use in the class is different from the updated
version we post after class, which
includes complete careful solutions to every problem
that we did in class, so students know
that that's there for them.
They might feel like they're missing out.
They want to see an expert solve the problem the right way.
Why should they have to come up with that way completely
themselves.
And so we capture some of that by putting that in there.
PROFESSOR 1: But you should say, we usually do solve it
for them afterwards.
Most problems we give them the expert, nice, succinct view.
And I think that's an important part of this.
Students want to feel that they're
getting some of your expertise.
AUDIENCE: You mentioned students preferred
the targeted questions reading to online lectures.
I was wondering what students felt
was so good about the targeted reading
or what it was that was so bad the online lectures.
PROFESSOR 1: I don't think they'd say
the online lectures are bad.
Online lectures take longer to watch if you try to MOOC.
And how many people in this room have tried.
The videos are long.
I mean you always are looking to try
to speed it up, find the two times speed up, et cetera.
They can take a long time.
The targeted-- the fact that the reading is targeted,
the fact that we wrote it, and it
covers what we expect them to learn,
is, I think, the main thing they like.
There's clear signals in there, we think this is important.
Not we said read chapter five and you have to figure out,
by listening to us, which parts of chapter five are important.
PROFESSOR 2: And I mean those are
skills that as you go higher up in your education at MIT,
you might be able to do more for yourself,
and you should learn those skills.
But I think the level of this course,
I mean many of these students, they're
strong mathematically by national standards of course,
but maybe within MIT, if they're life science majors,
they're not as confident as some of the math majors
in physics majors.
It helps them to have a little more guidance
in what the expectations are.
What do you do for small groups where
there's a group that's gotten done,
and everyone else is still working?
Do you have extra stuff for their enrichment?
PROFESSOR 1: You can walk around and try to challenge them.
Most groups it's rare that there's
people who have understood-- particularly
say the first time we work through an idea--
have understood it completely.
So there's always questions.
You could say, have you thought about this,
or what if you did that?
AUDIENCE: How do you form the groups
and do you change them during the course of the semester?
PROFESSOR 1: No.
We let them form themselves.
I think there's a debate in the education community--
maybe Jenny can speak to this-- about whether it's better
to form the groups, for you to form them
by some sort of heterogeneous ability, or to let them form.
We feel that, first of all, we don't
know their ability coming in.
Second, people like autonomy.
And so allowing them to form groups makes them comfortable.
There are students who are resistant to this idea
of active learning coming in.
We win them over quickly, and part of it
is that we're generous, say, in that sense.
AUDIENCE: What do you if you see in one group
there's a dominant member or there are
people who are sitting back?
PROFESSOR 1: You know it doesn't happen that much.
Groups of three, they tend to work together.
These are largely juniors and seniors at MIT.
A few sophomores.
They have become better at being students
and making sure they're getting what they want.
There are some groups where you come
and you try to draw people in.
It tends to be the groups that are overall weaker, where
you see people hanging back.
But those are exactly the groups where you go to first,
and you spend more time with them.
AUDIENCE: You mentioned the flipped component
of the course.
So, can you give a little bit more idea
about your expectations time-wise and material-wise
and you coordinate it?
I guess that they have these readings, maybe 10 pages,
for every class?
Or week?
PROFESSOR 1: Twice a week.
Tuesday and Thursday.
So prior to Tuesday and Thursday,
they need to do reading, about 10 pages.
It's a very fast 10 pages.
We use a lot of white space.
AUDIENCE: What are the problem sets?
Is there one problem set per week?
And also one more follow-up question
on that, what's the turn-around for the assessment
and feedback for those psets so that they could hopefully
learn for the next one?
PROFESSOR 1: They have one pset per week,
except in weeks where we have an exam and vacations.
So usually, let's say, one per week, with some variations.
PROFESSOR 2: So the psets, the way we structure it, right?
They're learning say some new material
Tuesday, Thursday, studio on Friday.
And the pset we have due Monday.
And the idea is that by turning it on Monday,
they still then, Monday night, can
prepare for Tuesday's class.
We try to get the graders to turn back
the pset by the end of the week, by Friday.
I'm not convinced that getting that problem set back
is actually helping that many students in the end.
Like I said, half the students don't even pick them up.
PROFESSOR 1: But it is nice for the students that want it.
So it's been good.
The graders get it back by Friday.
PROFESSOR 2: Oh yeah, the graders, they've done that.
AUDIENCE: How many questions do you expect them to work through
in that pset.
PROFESSOR 1: OK.
So the piece set is just the challenging part
of a problem set we would ask in a regular lecture class.
They've done more of the drill problems
in class, and somewhat on the reading questions.
So five or six questions, which is
about what we would have on what we call the part two
part of a regular lecture class, what they would
see in 18.01 or 18.02 or 18.03.
AUDIENCE: About the grading of the problem sets.
One thing that a lot of education research shows
is that if there's a delay between the act of doing
the problem and the feedback, they
don't pay any attention, which is what you're saying.
What if you force them to actually think about it
again by grading the problem set themselves.
So you could say, everyone upload their problem
set, and then you get somebody else's problem set,
and you have to then grade it with your solution.
PROFESSOR 1: I think it's a great idea.
One of my questions about it would be two-fold.
The students are not going to like it, especially when you
start them doing it.
And is it the best use of their time?
I don't know what the research shows on that.
AUDIENCE: We're writing [INAUDIBLE]
the solution set is probably worth their time reading it.
Otherwise--
PROFESSOR 1: Oh I see.
If they have to grade they have to read our solutions.
AUDIENCE: And really understand them,
and also learn what counts as a good argument and what doesn't.
[INAUDIBLE]
PROFESSOR 1: I can tell you this.
[? Analea ?] [? Bronte ?], who I teach with in the experimental
study group, who teaches physics there,
she did try to do this at some point,
and she did learn that students become good at grading it.
You don't have to worry about that.
Over time with your feedback, they
learn to grade pretty close to what you would grade.
The questions.
She would give them a rubric but she no longer does it.
There's a lot of resistance to it.
AUDIENCE: By the students.
PROFESSOR 1: By the students.
PROFESSOR 2: You would hope that a little bit of that
occurs before exams.
But I'm not sure.
AUDIENCE: Are you collecting data more
than you would in a regular class?
[INAUDIBLE]
PROFESSOR 2: Do you mean like the sort of data coming in
from the students.
AUDIENCE: [INAUDIBLE].
PROFESSOR 1: Here's the data that we have.
There's something called the statistics concept inventory,
and we had them last year do a pre-test.
And we have them do a post-test, but we got to it too late,
so we couldn't assign it to them.
It was in the last week of class where
you can't assign things at MIT.
So we didn't get a lot of compliance here.
This year they did the pre-test, and the post-test
will be an assignment for them to do.
So we'll be able to measure how well they improved
on the concept inventory.
It's not a perfect match for what we do.
It has no Bayesian component to it.
But we'll get a measure there.
The other is what Glenda has done
with surveys and interviews.
And finally there are analytics on MITx,
so we can see how long they've spent at least with the PDFs
open, how many tries they took at various questions.
We haven't tried to analyze that data yet, but we could.
MODERATOR: OK.
I think so we won't have people starting to get up and leave.
Thank you guys.
I think it strikes me as it's one of those really
in the trenches talks.
You're actually doing it and you can hear the width
from the questions this time.
We sort of moved from a sort of theoretical construct,
that you could do it this way, what's
actually happening around.
A lot of people are thinking about it
and just, thank you so much for coming.