The following content is provided under a Creative
Commons license.
Your support will help MIT OpenCourseWare
continue to offer high quality educational resources for free.
To make a donation or to view additional materials
from hundreds of MIT courses, visit MIT OpenCourseWare
at ocw.mit.edu
JANET RANKIN: All right, great.
I looked over some of the Mud cards we had from last time.
And there were some really good questions,
some really good points, and one lingering one
from the previous time that I forgot to address.
Someone said, could we please just
move the desks into an arrangement where
we could see each other.
It was a fabulous suggestion.
Today we can't do it.
But I'm going to try to do it in the future whenever
the logistics allow, because I think you're absolutely right.
A little bit more of a U, or a circle,
would be great for this class.
So thank you for that suggestion.
Another one was that we didn't get to motivation theory.
And I think I mentioned this last time,
the idea that there's a huge amount of stuff that's
been written and published on motivation
and theories of motivation.
And I'd be more than happy to discuss this
as the semester goes on.
And it might be a great topic for one
of those lunches that somebody was going to organize,
one of those informal lunches that someone
was going to organize.
Or maybe we can carve out some space
at the end of the semester and pop it in, take something out
and pop it in, because it's really fascinating.
It's really interesting, but it doesn't quite
fit anywhere in particular.
So thank you for bringing that up.
And keep bugging me about it.
The other thing is someone asked if all those learning
theories that we discussed were equally valid.
And yes, they are equally valid.
I mean, there are situations where being behaviorist
could help.
There are situations where being constructivist could help.
It depends on what you're trying to have students learn.
So you want to make sure that you tailor the way that you're
teaching to support what it is you want students to learn.
And we'll get into that in a lot more detail
today when we talk about developing learning outcomes,
and then also when we talk about tailoring
your instructional activities to support
those learning outcomes.
So this class, this particular class today
is a balance between developing course design, developing
intended learning outcomes, which
for me is at the heart of it.
If you can come up with some good intended
learning outcomes, you basically have designed your course.
And I'll show you how.
I mean, you can't just ignore it, but that's the hard work,
and then the act of writing constructing a syllabus.
So once you've got those learning outcomes,
you have an incredible anchor for the rest of your course.
And there was just a very timely haiku.
I put it on the wiki.
But there's this Twitter feed, academic haikus,
which if you want to waste any time and read them,
it's a way to waste time.
But there was one here, "that question you have?
I know where the answer is-- on the syllabus.
So it was, like I said, it came out yesterday.
I thought it was pretty timely.
And it's true.
If you write a good syllabus, it should really clear up
a lot of, not the content, not the understanding level issues
students have, but the issues with respect
to when's the test, do we have to know this,
how will this be graded, what's important?
All those things should be in your syllabus.
So that's why that's there.
So here's a little slide, a graph
that I saw a few years ago, and I
thought it's a really good way to start this discussion.
So on the y-axis we have sort of an assessment, a proclamation
of students understanding.
They got it, they didn't get it.
And I think the word in French is [FRENCH].
So for some reason I find it very cool to say [FRENCH].
Got it.
I don't know.
So whether they get it, like whether after the tests,
you've given a test, and you go, man, that student really
got it.
That student really gets this material,
and whether they passed the exam or failed the exam.
So the size of the circle is more or less
the number of people that fall into those quadrants.
So anybody surprised by this plot?
Yes, Rachel.
AUDIENCE: [INAUDIBLE]
JANET RANKIN: OK.
All right.
Well, there's the segue.
So we want, we don't want the green,
but we more or less expect the green, right?
If you get it, you pass the exam.
If you pass the exam, you get it.
That's that, and we like to have lots of people
have that happen too.
And then we also acknowledge that somebody
isn't going to get it, and they're not
and they're not going to pass the exam.
And that kind of maybe makes us sad, but it's not surprising.
And then anybody can kind of have
a bad day or kind of a fluke.
So sometimes, sure, somebody will fail the exam,
and actually really understand the material.
Generally speaking, that doesn't happen,
and then there's often a reason for it.
But it's that red circle that Rachel
mentioned that is troublesome, at the least, troublesome.
So why might that happen?
Why so many red circles?
Yeah, Dave.
AUDIENCE: Because the test was set up for basic memorization,
or you could figure out with a lucky guess
with multiple choice.
So you could easily pass just with good test-taking skills
or a good memory.
JANET RANKIN: Excellent.
So you pass with sort of good test taking skills.
Now I want us to hang on to that.
There's an underlying assumption in Dave's comment.
So I'd like to hang on to that thought,
but let's hear a few other conjectures and see.
Rachel, why do you think you might have passed it?
AUDIENCE: So often I feel like TAs are either
really good or really terrible when
it comes to the exam reviews, and they're mainly good.
They often told us too much of what
was going to be on the exam.
And so then you kind of just knew
what the test was going to be if you could well on it
if you [INAUDIBLE].
JANET RANKIN: OK.
That's interesting.
Yes, Gordon.
AUDIENCE: Yeah, it's possible that [INAUDIBLE]
tests [INAUDIBLE].
When it comes to true life situations and something
that [INAUDIBLE] knowledge that was
gained from that material, then [INAUDIBLE].
JANET RANKIN: OK.
So Gordon is sort of kind of flirting with this idea of you
as the instructor, this real life knowledge,
this is what it means.
They have to have this practical working knowledge for us
as the instructors to say they got it, right?
But the exam isn't measuring it.
And that's exactly what, that's what Dave was saying as well.
This idea that the exam is measuring one thing,
but it's not really measuring whether they got it.
When we go home and think what does
it mean to be a good material scientist,
or what does it mean to be a good chemist,
we don't think, oh, it means that you
can define what a mitochondria is,
or you can balance a simple equation.
And yet, we put that on the exam.
It's also a bit of the idea that students
are good at taking tests.
They're good at gaming systems.
They're good at listening to TAs or TAs may have not given
out appropriate information.
So it comes from both sides.
It comes from the students and it comes from the professors.
But you have control over this.
You shouldn't be writing an exam that people
can get a good grade on, but then when you go home
and say, oh, they got a good grade on this test,
but they didn't get it.
You need to define what it means to get it in your field,
to get this subject, whatever it is that the test is about.
And then you need to write a test that measures that.
And too many times we don't do it.
We're kind of rushed, or we don't really know.
We haven't really thought about how to measure
whether somebody gets it.
So the first step in all this is to define what it
is that means that they got it.
And that's the first step, and that's the hardest step.
But once that's done, the rest is easier,
and it will save you from this.
OK?
OK.
So that's the topic of today's class.
And if you needed a little more motivation,
I'd like you to read this paragraph,
and I'd like you to tell me what it's talking about.
But I'm just going to give you about 2 and 1/2 minutes
to read it.
And then we'll discuss what it means.
What do you think this is talking about?
Yes, David.
AUDIENCE: Yeah, I think it stresses
about how we can arrange by say, maybe schedules or activities
that put them into groups.
Then maybe if we form a group, how we plan [INAUDIBLE].
JANET RANKIN: OK.
So arranging schedules or activities.
OK.
Something else?
AUDIENCE: When you're planning, how
to go about teaching a module, [INAUDIBLE].
JANET RANKIN: Planning teaching.
What else?
Anything?
AUDIENCE: It's like a mailroom operation or something.
JANET RANKIN: Mailroom.
Something from the back, maybe?
You know you're smiling.
AUDIENCE: I couldn't think of anything.
JANET RANKIN: All right.
Well, then don't say anything.
AUDIENCE: The first is the course content, the course
outline, and what you are going to teach
the students, [INAUDIBLE].
JANET RANKIN: OK.
So students, grouping students--
Adam, right?
Adam.
OK.
So content, arranging content.
Anybody in the back?
Flyer?
Yes, Gordon.
AUDIENCE: I think you talked about sorting materials.
JANET RANKIN: Sorting materials.
So we'll go with maybe sorting.
So if I tell you-- so these are all great guesses.
They're as good as what I'm going
to tell you the answer is.
So this was written to describe the process
of doing your laundry.
Like doing the wash.
You take the clothes and you put, maybe
you put the dark clothes in one bin and the white clothes
in another bin.
And then you fold them and you put them away.
It's impossible.
This happens almost every time I show
this little piece of writing.
People don't generally get it.
They get the fact that it has to do with sorting,
and some planning, some arranging things,
but they don't get that it's about the laundry.
If you skim it-- you don't have to totally reread it--
but if you skim it with the idea that it's
about doing the laundry, you probably
can understand it better, maybe.
So now I'm going to ask why did I show you this now,
today, before the third, the beginning of the third class?
Yes.
AUDIENCE: I think it's sort of about arrangement of course
content [INAUDIBLE].
JANET RANKIN: So it could be about how you arrange
the content in the syllabus.
[? Hina. ?]
AUDIENCE: I think it's easy to describe methodology
as a person more in a different field.
But if there's no context for the audience about what
this is, it doesn't matter that you've laid out everything.
No one knows [INAUDIBLE].
JANET RANKIN: Right.
It's very detailed at a certain level,
but it's still really hard to understand, right?
Any other ideas?
Yes.
AUDIENCE: I think it's interesting.
There's a lot of value judgements in it.
Like, it's really easy.
And one can never tell with comments like that.
Because I'm trained to figure out what they're saying,
it's actually just frustrating me more,
because I don't know what they're talking about.
And it's really easy--
JANET RANKIN: All right.
So that's a great point.
So we're talking about writing learning outcomes,
about organizing the course around writing
learning outcomes.
If you're using phrases, or you're
using language that's just, there's
just like throwaway words in there or value judgments,
it's not doing your students any good, right?
And also, that if you have this beautiful course that's
beautifully organized even, wonderfully organized,
but if you don't tell them what it is you're
trying to do with it all--
what it is they're going to be able to do
with it all, whatever it is you expect them to do--
they're not going to be able to follow you.
So imagine a procedure as fairly straightforward
as doing the laundry.
Now imagine, OK, you're going to be
able to select the right, the appropriate method
to integrate equations, so methods of integration,
by separation, by card or substitution or whatever.
So if you don't tell them what it
is you're expecting them to do, even
if you're very good at explaining what it is they're
going to do, they won't get it.
So this is my sell for why when you write a learning outcome,
you have to write a learning outcome that's clear
and that's understandable to the students, that doesn't have
a lot of extra words and that tells them
exactly what they're supposed to do.
So I have some learning outcomes for today
and I hope that they're clear.
You'll be able to describe the components
of the constructive alignment and backward design processes.
And those, constructive alignment anyway,
was in the pre-class reading.
You'll be able to classify the content of a course
that you might teach.
So what do I expect students to do with this content,
to be able to do with this content?
You'll be able to create measurable, specific, and
realistic learning outcomes for a class you will teach,
and you'll be able to state the components of a syllabus.
So I feel that those are pretty understandable.
You may disagree with me, or maybe you want to wait.
So constructive alignment, it was in the pre-class reading,
right?
Anybody want to just give a brief summary of it?
I mean it's kind of up here, but in your own--
AUDIENCE: I think constructive alignment
is about aligning assessment, method of assessment,
with course content.
[INAUDIBLE] assess [INAUDIBLE] type of assessment
so that what the teacher has in mind, the course
content or the content I want them to get,
to test it with the assessment process.
Then we would also do the assessment,
and then come up with the intended outcome.
JANET RANKIN: Right.
They'll achieve the learning outcomes.
And if you do this right, you get rid
of that red circle on that graph.
You get rid of students who did well on the assessment
but don't really know what's going on,
because you have aligned the assessment
with what it was you wanted them to know or be able to do.
So that's important.
I just think this is a very, very compelling image
to keep in your head, that these three things have to be linked.
And as I said before, this, you will
see that the top part, writing the learning outcomes
is the tricky part, it's the hard part.
It's the part that requires a lot of thinking and work.
When you get a good learning outcome,
the other two things almost take care of themselves.
OK?
Did anybody have any, does anybody
have any questions about something
from the reading associated with constructive alignment
or some other idea from the reading?
Yes, Gordon.
AUDIENCE: I just wanted to make a contribution.
I think that this constructive alignment is one of the things
that the intended outcome should measure.
So I think maybe that's confusing [INAUDIBLE]
and also how we can test and be sure and confident.
JANET RANKIN: Right.
Right.
I think the measurability part for scientists and engineers
is kind of intuitive.
Like why would you say something if you couldn't
tell whether or not it was true, or not
from a scientific point of view.
What's the point of saying, making
a proclamation about what somebody
should be able to do if you can't measure it in the end?
So it's comforting to say, OK, I'm
going to make learning outcomes that are measurable.
But it also can be a bit challenging.
But we'll do that.
We're going to do that today.
Some other comment?
Nina.
AUDIENCE: I guess as far as [INAUDIBLE]
Do people write learning outcomes that
have no constructive alignment process,
like choosing to write learning outcomes,
are you going through the prcoess?
It seems decoupled a little bit in the reading,
but it seems like you can't have one without the other.
JANET RANKIN: You owe it to yourself--
and I don't know what people do, and I guarantee you
people write learning outcomes without thinking about
whether they can assess them or measure them,
I guarantee that happens in real life--
you owe it to yourself to try very hard not to do that.
OK?
And I think the important thing to do is, often we say,
oh, my god, this is the best assignment.
This is the greatest question.
It's this project, and they're going to do this,
and they're going to whatever.
They're going to build a race car.
And if you sit down and say, well, what
do I want them to get out of that?
If you can't articulate it in a way that's consistent
with your class, consistent with the course
as it sits in this series of their courses,
or as it sits in the institution,
then you either need to rethink the assignment,
rethink the project, or think a lot harder about what
learning outcomes that activity advances.
That's a very good point.
It's really kind of iterative.
And I think it might be easier to see with this diagram.
This is another-- so John Biggs is from the UK.
And so there's sort of the UK camp of people
and they like John Biggs.
And then there's Wiggins and McTighe.
Grant Wiggins and Jay McTighe are from the US.
They wrote primarily for the K through 12 audience,
but it's completely applicable.
And they're from the US, so they have kind of a US following.
It's just how it works.
It's essentially the same thing.
It's identify the results, so the intended learning outcomes.
Figure out how you're going to get them there.
So the black is Wiggins and McTighe and the red is Biggs.
So plan the learning experiences that
are going to support them, achieving the learning
outcomes, and really help you measure whether they got them
or not.
And then determine what evidence will
be that acceptable evidence that they achieved the learning
outcomes and what are the assessments.
So real often we do have this project that we want.
Oh, this is a great problem.
Often in thermo-- I taught a lot of thermo-- there's like, oh,
they're going to do this double loop, double reactor
cycle with the refrigerant and a heating cycle.
And it's going to be coupled and they're
going to use the output from one for the input in the other
and all this crazy stuff.
Like, this is a great assignment.
And you kind of, I know this is a great assignment.
If I can't back up from this assignment--
whether or not I put it here or here--
if I can't back up and say after a successful completion
of this assignment, you will be able to describe
the components of the Brayton cycle,
you will be able to define the increase in entropy
for a system with x boundaries, If I can't articulate
those things, then I need to either change the assignment,
or, well, then I need to change the assignment.
OK?
But you can start with the activity,
and then back up and say, well, what
learning outcomes does that support, and then,
am I OK with that?
Is that relevant to this class?
So it's often a very iterative, the arrows kind of go this way,
but it's often iterative.
So I had asked that you come to class with some topics
from a class you're likely to teach.
That was on the assignment page.
If not, start thinking now.
And so you want to have, for most classes, most
undergraduate classes, there's a bunch of topics.
You might have 10 topics, 20 topics.
And I don't care right now about the scale of the topics,
whether they're really big.
You probably don't want to go with like first law,
second law.
That's probably a little bit too big,
but on the order of 5 to 15 topics.
Jot down the topics.
And then I have a chart here.
So the chart has two sides, and the columns
are Bloom's Taxonomy, which you don't
have to worry about that specifically,
but you can look at the words that are associated with it.
And then the second column is what students should
be able to do with each topic.
So let's say my topic is first law for closed systems.
So I might think about, OK, do I want
them to be able to define, to just state
the first law for closed systems?
Do I want them to be able to use the first law for closed
systems in order to solve certain kinds of equations?
Do I want them to be able to derive the first law?
Do I want them to be able, whatever it is.
So you're going to look at the columns, the categories
on the left side, and you're going
to think about your topic.
And you're just going to decide which box in the middle column
to put the topics into.
You don't have to worry about writing anything.
You're just going to distribute the topics to the level of,
it's really cognitive processes that are associated
with the things on the left.
Remember, understand, apply, analyze, evaluate, and create.
So we'll take about five minutes.
And you just do that by yourself.
And just fill in that middle column
of the table with the topics from your course.
Do you want more explanation, Rachel?
It's starting to make you think about this part
of the beginning circle.
But you don't want to write any sentences yet.
But it's starting to help you think about what are they
going to do with these topics.
What do I want them to do?
Is it remember, or is it create something using these things?
Where is it?
Those are two ends of the spectrum.
And in your chart, the things that at the top
are the more remember, right?
It's more about kind of a recall information.
And then the things in the back of the chart
tend to be more higher order cognitive processes,
like synthesize, apply, create.
Yes.
AUDIENCE: [INAUDIBLE]
JANET RANKIN: So don't worry about the third column.
Don't worry about the last column yet.
We're going to do that later.
So all you have to do is just say
where each topic falls loosely.
OK.
Does everybody have a chart?
OK.
AUDIENCE: Do you want us to write
as many topics as possible?
JANET RANKIN: Write as many topics
as possible, but distribute them through the chart.
Right?
Not everything is going to be remember, right?
So some things will be remember, some things will be apply,
some things will be analyze, some things will be create.
OK?
If you have any questions, just raise your hand.
All right.
So what we're going to do is I'm going
to ask you to just keep those with you.
We're going to do another activity later
on using those as a starting point,
but I want you to just keep that sorting with you.
Any comments or questions about the experience,
about what you just did?
Was it easy?
No.
It can sometimes be a challenge.
Well, what the heck you were supposed
to do with this anyway?
What do I really want them to do with it?
This is the beginning of that process whereby
you define exactly what it is you want students to do,
know or be able to do.
And then you can start to think about how
you're going to assess whether or not they can do it or not.
But it's very different if I were
to say I want everybody to be able to state
the first law of thermodynamics versus I
want you to be able to use the first law of thermodynamics
to define chemical, mechanical, and thermal equilibrium
in a simple composite system.
That's different.
That's very different.
So that's what we're starting to do.
So now we have kind of the seeds.
We thought about which topics kind of we want to have,
what we want to do with each of the topics.
But now we really have to turn them into learning outcomes,
into real statements.
By the end of this class, you will
be able to, by the end of this course, you will be able to.
And just as an aside, remember that sometimes you'll
notice that for every class, every session we have,
I have learning outcomes.
But there's also overarching learning outcomes
for the course.
So I will use them interchangeably,
but they'll be at a slightly different scale.
The ones for the class are a little bit more fine grained.
As was mentioned in the reading, and as Gordon mentioned,
they have to be specific, measurable, and realistic.
If they don't have those properties
they're not going to be particularly useful, or as
useful.
All right.
And we'll see some examples of that in just a sec.
So here's what they're not.
They're not topics.
So you came with topics.
You will leave today with learning outcomes.
But topics are not learning outcomes.
The first law of thermodynamics is not a learning outcome.
You need a verb with it.
What do you want me to do, what do you
want your students to do with the first law
of thermodynamics?
What do you want them to know about the first law
of thermodynamics?
So they're not topics.
They're not things that you will do.
It's easy to say, I could have a list a mile long.
I will talk about the first law of thermodynamics.
I can check that off pretty easily.
I mean I just did.
I just talked about the first law of thermodynamics.
It says nothing about what you learned about it, what
your students learned about it.
So it's not about you.
Get over it.
That was a joke.
It's not value statements.
As Adrian pointed out, some of these think this is easy,
this is good, this is useful.
Any of those things don't belong in a learning outcome.
And it's not your hopes and dreams for your students.
I mean, ultimately, it is.
You hope that they will be able to use the first law
of thermodynamics, da, da, da.
But it's really about what they're doing,
about what they're going to do.
So, no.
Here's a little example.
This is just from like an earth and planetary course.
Students will understand plate tectonics.
How's that?
Is that OK for a learning outcome?
Why not?
AUDIENCE: It's too broad.
JANET RANKIN: It's not very specific.
What the heck about plate tectonics?
What else?
AUDIENCE: Measurable.
JANET RANKIN: Thank you.
OK, the measurable part.
This word understand, strike it from your brains.
It's a property that we love.
We love understand.
But you can't flippin' measure it.
You can't go in there and measure it.
You can measure lots of things that tell you
whether students understand, but you
can't measure understanding.
And I think we'll see that with some more examples.
I don't think anybody is earth and planetary here.
So how about this one?
Students will be able to interpret
unfamiliar tectonic settings based
on information on volcanic activity and seismicity.
Again, I don't think any of us are geologists.
But the idea that I've told, or the person
has told us what we're going to use
to interpret these situations, and then we're
going to make predictions based on those interpretations, I
mean, that's an assumption.
So it's specific and now it's measurable.
Even though we're not geologists,
I suspect we can think of how we could
write a test question that would ask students to do this.
And then we could see whether they did it or not.
So that's measurable.
And probably, by doing this, by having students do this,
we get a pretty good measurement about whether they
understand plate tectonics.
But it's much more specific and it's much more measurable.
From here, this is from the University of Minnesota,
which has a nice website on learning outcomes.
They're not from your point of view, as we mentioned.
They're not what you're going to do.
I'm going to introduce students to the major turning
points and processes in North American history.
You can say anything you want, but it doesn't mean
the students learned anything.
So it's not about you.
It's what they're going to do.
They're going to list and describe
the turning points and processes in North American history.
I'm going to create an understanding
of the formal constructs of physical design.
That one is like, this one is, it's
like, I don't even know what it means, right?
Create an understanding of the formal constructs
of physical design.
I don't even know what it means.
And so it's like that laundry example.
Your students aren't going to know what it means,
and it's not going to help them.
It's not going to help them access the material.
So the ones on the left are all teacher-centered
and they're not very useful.
The ones on the right are student-centered
and they are generally useful.
But there's some things wrong with the things on the right.
AUDIENCE: Understand.
JANET RANKIN: Understand, yes.
So what would you write instead of
understand the formal constructs of physical design, whatever
the heck that means?
What might you use?
AUDIENCE: Identify.
JANET RANKIN: Identify.
OK.
That might be like a list, right, or identify or list,
pick it out of the lineup kind of thing.
This is a nice one.
Explain to an intelligent non-expert, an INE.
And so explain this to somebody who's not in the field.
If you could do that, it probably means that you get it.
Somebody I know used to say, explain this
to my four-year-old brother.
Explain-- what was it-- angular momentum,
to my four-year-old brother.
I would argue if you can explain it to a four-year-old kid,
you probably understand it.
So you can use that.
Use formal constructs to design something.
I mean, that's kind of an inversion.
How about the one on the bottom.
Understand gender, race, ethnicity, socioeconomic class,
understand how they have shaped Americans' lives.
Completely unmeasurable, right?
So what would you write instead?
AUDIENCE: Describe.
JANET RANKIN: Describe how, sure.
Anything else?
AUDIENCE: [INAUDIBLE] it seems like you're
trying to get [INAUDIBLE].
JANET RANKIN: Right.
So that's a great point, is that we
don't know where this faculty member put this on their table.
We don't know whether it really just states
some ways that, list some ways that these things have affected
Americans, North Americans' lives, or Americans' lives--
which is kind of a remember thing, just make a list--
or whether it's do some analysis on some situation.
We don't know that, but you as the instructors
get to decide that for your classes.
So that's an important thing.
You get to call all the shots.
You get to make the decision.
We can only guess about what this person meant,
and therefore, his or her students
can only guess about what he or she means.
So you have on your table this hierarchy of Bloom's Taxonomy.
It's often shown in this triangle, this pyramid.
There's a handout which has exactly the same strata on it.
And it just has some handy dandy verbs associated with it.
But Benjamin Bloom came up with this classification
of cognitive processes way back when, in the 1950s.
And it is this idea that it is this pyramid, that
at the base of the pyramid are these ideas like remember.
And those are basic cognitive processes, remember, list.
And then it moves up the pyramid to things that require a higher
level of cognitive processing.
Rather than just pulling facts out of storage
and stating them, you're going to do something
with those facts.
So as you go higher and higher up the pyramid,
note that he chose--
really unfortunately-- to call the second level understand.
However, there are words that can help you.
If it really is understand that you want to measure,
there's words that describe that a little bit better.
They're in your table, and they're
also in that second table that I handed out.
So arrange, list, label.
For understand-- describe, relate, recognize, explain,
those things.
Gordon.
AUDIENCE: Just a little [INAUDIBLE].
When it comes to understand, maybe
if you put it like [INAUDIBLE].
JANET RANKIN: Understand to what?
AUDIENCE: Understand to an [INAUDIBLE].
Maybe that's what we can measure.
Understand to this level or to this [INAUDIBLE].
JANET RANKIN: But how, can you give me a specific example?
AUDIENCE: I wish I could go back to the last slide [INAUDIBLE]
can describe [INAUDIBLE].
JANET RANKIN: OK.
So I see where you're going.
Does someone have a suggestion?
So think about that laundry.
Think about that description of doing a laundry.
Can somebody take what Gordon said
and get where he wants to go, but get there in perhaps
a more direct way.
Katherine.
AUDIENCE: [INAUDIBLE]
JANET RANKIN: Exactly.
So if you're going to do that, and I
see where you're coming from, understand
so that you can describe, in order to describe,
or to the level of describe.
Just forget the understand part, and just
say describe, or define, or whatever it is you want.
But yes, I mean, that first step is saying,
what do I mean by understand?
And when you say that, then it generally rewrites itself.
And so the more direct you can be, generally, the better.
And keep that laundry example in your head.
Yes.
AUDIENCE: You're asking a student [INAUDIBLE]?
JANET RANKIN: So I would say, well, let's just open that up.
What do you think about that, of asking students?
So do you understand this stuff about learning outcomes?
AUDIENCE: I think as teachers, when
you, in the front of the class, you're teaching students
and you say do you understand, they say yes.
But you see from psychology you know who doesn't understand.
JANET RANKIN: So David says as I ask you
that, do you understand?
Most people are going to say, hey, I understand, sure.
Don't think I'm an idiot.
I understand.
Right?
So I completely agree.
Students are not likely to say they don't understand.
What else?
AUDIENCE: That's difficult for the teacher to [INAUDIBLE]
JANET RANKIN: Well, so do you want
to say something about that?
AUDIENCE: Yeah, I agree.
I think the part that we do understand
as nonspecific is the learning understanding [INAUDIBLE].
As an instructor you have to know, understand
what you're trying to get at, which
I think is where the clicker question is beautiful,
because you know already what you're
trying to evaluate that they're catching in class in real time,
if you can see who's getting at the nuance of what
you're teaching.
JANET RANKIN: Yes.
I think that's an excellent, excellent point,
that many of these activities that we do,
these small group activities, or the clicker
questions or other activities that students
are doing in class are helping them learn.
But they're also formative assessments of whether or not
they get it or not, whether or not they understand.
And you as the instructor can see it.
If I walk around, if I listen, if I
look at who answered what question on the clickers,
or on the raising your hand, multiple choice questions,
whatever methods I used here, I get a better,
a much better sense of who understands
or doesn't understand.
If all I'm doing is lecturing I can't tell.
And I can ask do you understand, but they're not
likely to tell me.
Which is kind of the other point is that sometimes--
and I think you were alluding to this, Gordon--
that sometimes students don't know whether they understand
or not.
So we need to give them the opportunities to find out
whether they understand or not.
And arguably, if you're just lecturing
and you're a great lecturer--
you're eloquent, you don't stumble,
the writing on the board is awesome, all this stuff--
students will think they understand maybe
when they don't, because they're not
having to confront any misunderstanding,
and it all seems very nice.
So these activities where you actually have students
do something, say something, where
you ask them a specific question, that's
when you can determine whether or not they understand or not.
And because they are not likely to say
they don't understand, either because they just
don't want to admit it, or because they really don't know
whether they don't understand.
So you need to give them measurements to find out
whether they get it or not.
Yes.
AUDIENCE: Sometimes, for example lecture and the activity
has [INAUDIBLE].
For example, we have lectures and the teacher just
[INAUDIBLE] and then we have expectations
as we do exercises.
So do you recommend that we also do some activities
during the lecture or just [INAUDIBLE]?
JANET RANKIN: No, do them during lecture.
I mean, the model that we're trying to do here
is the model that you should do even if you have a hundred
or 200 people in your class.
You can use clickers.
You can use pair share discussions.
You can do all sorts of things to break it up,
and it's effective.
There's a paper coming up for next week,
a meta-analysis, active learning strategies,
how they actually, students learn more in a lecture
if you actually ask them to talk to each other,
break up, share, et cetera.
And I think when we talked about misconceptions, remember
that example with the coin?
How many students got it, could draw the free body
diagram before and after?
I mean, before and after in a straight lecture,
there's hardly any learning gain.
But if you incorporate active learning,
the learning gains go way up.
So that's Bloom's Taxonomy.
And an interesting take on Bloom's Taxonomy,
there's a woman, Kathy Schrock, who--
it's a nice website and the slides
we posted so you have the link--
but she decided, she has this theory that it's not
really so pyramidal.
That everything really is dependent,
these cognitive processes are dependent on each other
at a certain level.
And so she's drawn it as a gear, where everything
sort of supports the act of creating,
which is up at the top.
But the other ones are sort of around the side,
but they all contribute to the ultimate cognitive
process of creating.
So that might be a nice way to think about it.
And then she's taken the specific words, like analyzing,
and said, OK, what's an instantiation
of analyzing, outlining, deconstructing, organizing,
structuring, surveying, whatever?
So it's the same idea, it's just a different graphical
representation.
So what you've done is you've taken your topics,
put them in those cells in the table.
And I just want to point out kind of how as the instructor
you have a lot of power about where
you're putting those, where you've decided to put those.
And it may be different for a first year undergraduate
class, or a second or third year graduate course.
It could be the same topic, but you're
going to put them in a different category.
So that's why when I asked you to bring a topic,
it's for a specific class.
It's for a specific group of students.
So let's say we take the idea of interstitial sites.
So I have a crystalline structure,
it's got an order of periodicity of atoms.
And then I have spaces between the atoms,
those are the interstitial sites.
And so I could say you'll be able to identify
the interstitial sites.
So I could say where's the tetrahedral site
in a face-centered cubic or something, whatever?
And students just really have to find it.
I could say calculate the maximum size
of an interstitial atom that could occupy
that space, an ion that could occupy
that space in a particular crystal structure.
So those are two very different levels,
but it's the same topic.
X-ray diffraction, I could say apply Bragg's Law
to calculate d-spacing.
So that's very much a turn the crank kind of thing.
I know the equation for Bragg's Law,
I stick things in, I solve for the unknown.
I could then also ask you to index unknown diffraction
patterns, meaning identify an unknown crystal structure based
on the diffraction pattern.
That's a completely different activity,
completely different set of cognitive
processes than just using Bragg's Law.
But you have to decide what it is.
And arguably, if all you've had students do during the class is
sort of problems that they applied Bragg's Law and they
solved for d-spacing-- so they solved for the incident angle
or they solved for the wavelength or whatever it is--
you really can't throw this at them on the exam, right?
Or conversely, if you spent the whole time, they've indexed,
they've been in the lab indexing unknown diffraction patterns,
and then this is the question you ask them on the exam.
Well, then you have people that perhaps get it without really,
perhaps pass the exam without really getting it.
So that's just something to keep in mind.
The other thing to note is let's say
this is my intended learning outcome--
calculate the maximum size of the interstitial atoms
in a variety of crystal structures.
How do I know whether students get that or not?
What's that?
AUDIENCE: It's easy.
JANET RANKIN: It's very easy.
I give them a crystal structure and I say
calculate the maximum size of the ion that
can fit in the space.
So I have figured out how, the measurement is easy, it's cake.
It's totally straightforward.
Now I haven't told them, OK, I'm going to give you this crystal
structure and I'm going to ask you
this spacing, this ion size.
I haven't given it away, but I have told
them what's expected of them.
And my colleague wrote these for VSEPR theory.
It's the same topic, and it's different.
Identify the common geometric shapes
found in simple molecules.
Explain the assumptions of this theory.
Apply the theory to predict 3D structures.
Compare and contrast the geometry of a certain molecule
as predicted by two different theories.
Evaluate the accuracy of each theory
for a particular set of compounds,
and then create some recommendations.
So it's the same topic, but you decide
what it is that students should be able to do.
OK?
So I have some, just a little exercise for us.
So they should be specific, measurable, realistic.
I'm going to put them up here and we're going to say,
you're going to tell me what's wrong with them.
So number one, t-tests are like a statistical analysis.
So is anybody a t-test kind of person?
We can skip that one.
How about gaining appreciation for the use
of linearization techniques?
What's wrong with it?
AUDIENCE: It's not specific.
JANET RANKIN: It's not specific.
AUDIENCE: It's not measurable.
JANET RANKIN: It's totally not measurable, exactly.
I mean, how do I know that you appreciate it?
Don't do it.
All right.
Great.
Have an intuition for the most effective method of integration
for a given problem.
AUDIENCE: It's vague.
It's not measurable.
JANET RANKIN: So I've heard it's not specific,
it's not measurable, and it's not vague,
I mean, and it's vague.
So right, so have an intuition, I cannot measure that.
I have not figured out a way to measure intuition.
I can tell whether you can do it, right,
but I can't tell you whether you have an intuition for it.
Well, how would you rewrite it?
Actually, how would you rewrite 2?
AUDIENCE: Use [INAUDIBLE]
JANET RANKIN: So it could be use the techniques.
Another suggestion?
AUDIENCE: Make something using the evaluation techniques.
JANET RANKIN: Calculate something.
Make a first pass estimate using a particular linearization
technique.
AUDIENCE: Describe when you would use a linearization
technique.
JANET RANKIN: Describe when you would
use it, describe why you would use it, all of those things.
AUDIENCE: List those areas where [INAUDIBLE]
JANET RANKIN: List the areas where it's useful, yes.
Great.
All right.
For number 3, the gain an intuition,
we decided it's vague, it's not measurable.
So how would you rewrite it?
AUDIENCE: Compare.
Compare the areas integration technique [INAUDIBLE]
Solve a particular problem using different kind of [INAUDIBLE].
JANET RANKIN: OK.
So solve a problem, and then we need a little more.
Did you have something?
AUDIENCE: Identify the most effective method of--
JANET RANKIN: Let's hear what Rachel had to say.
Rachel.
AUDIENCE: Yeah.
Learn how to identify the simplest [INAUDIBLE].
JANET RANKIN: Right.
So you probably don't want to say learn how.
You would say be able to identify.
Right?
Gordon.
AUDIENCE: [INAUDIBLE]
JANET RANKIN: Right.
I mean, you could just say integrate--
AUDIENCE: [INAUDIBLE]
JANET RANKIN: Integrate a problem using whatever.
Select a technique to integrate a particular equation.
And again, those things end up being more measurable.
So 4, provide problem solving tools and strategies.
What's wrong with that one?
AUDIENCE: It's too wide.
JANET RANKIN: It's too wide.
AUDIENCE: It's instructor centered.
JANET RANKIN: It's instructor centered.
What else?
AUDIENCE: But it's measurable.
JANET RANKIN: Well, I can measure that I provided it.
Here, I provided these handouts, right?
OK.
Check.
It happened.
Whether or not you could do anything with them,
I don't know at this point, right?
That's the problem.
AUDIENCE: It doesn't tell us what kind of problem
and what kind of tools we're going to use.
Just who gets to choose the strategies.
JANET RANKIN: Right.
Right.
It's very, very general, totally not specific.
So what could we do?
I mean, we don't know what this instructor had in mind,
but what could you do if this was your class?
What would you say?
AUDIENCE: What about develop instead of provide?
JANET RANKIN: Develop.
And again, it would have to be student centered,
so the student will be able to develop problem
solving tools and strategies.
That's for x, exactly.
We want to make it a little more specific and a little more
focused, because that's pretty broad.
I mean, I have developed a problem
solving strategy that lets me get out
the door without tripping.
Does that mean I satisfied your learning outcomes?
Probably not.
So in order to do something--
Use thermodynamics to solve engineering problems.
AUDIENCE: That's not specific as well.
JANET RANKIN: It's not specific at all.
AUDIENCE: Engineering is [INAUDIBLE].
JANET RANKIN: Yeah.
I mean, it's crazy broad on a number of fronts, right?
I mean, it's like engineering is huge.
Thermodynamics is pretty darn big, too.
So which part of thermodynamics, which part of engineering?
Build an SAE race car.
AUDIENCE: [INAUDIBLE]
JANET RANKIN: What?
AUDIENCE: It's not realistic.
JANET RANKIN: Right.
It's pretty specific.
Build a functioning SAE race car.
It's very specific, and I can totally
think of the metric of how I would measure
whether students could do it.
I would just ask them to build the race car.
But they're never going to do it in a class,
in one class period, in one course.
It's unrealistic, exactly.
Learn to use Laplace transforms to solve
differential equations.
AUDIENCE: [INAUDIBLE]
JANET RANKIN: It's funny.
I mean, you can imagine the instructor
saying you will learn to use that.
But let's go, Katherine, work your magic with this one.
AUDIENCE: [INAUDIBLE]
JANET RANKIN: Right.
You're my go-to person for when you want to get rid
of all the extra words.
I'm going to call on you.
Right.
Use Laplace transforms.
That you can measure.
You can give them an equation and see
if they can use Laplace transforms to solve it.
And yes, you might want to be a little more
specific about the kinds of differential equations they
use, and know how to upper diagonalize a matrix.
Katherine.
AUDIENCE: .
Upper diagonalize a matrix.
JANET RANKIN: Upper diagonalize a matrix.
That's all you have to say, right?
Forget the extra words.
And that is really, I have just written the assessment question
for that.
I know it's totally measurable.
I give you a matrix and I ask you to upper diagonalize it.
I haven't given it away.
I haven't told you you will upper diagonalize this matrix,
right?
Memorize these equations that you're
going to need to write down in order to upper diagonalize it.
But I have told you that that's what's expected of you.
AUDIENCE: I have a comment on it.
It's pretty useful when a student
is trying to evaluate the use of the class.
For example, the career fair tomorrow [INAUDIBLE].
There's a million [INAUDIBLE] to get jobs.
OK, how are they going to evaluate me?
And if you could come in and say,
this happened and this is what we
did, as opposed to having, or have them
evaluate me on some random knowledge.
And then I don't know that I could respond,
but I do know stuff.
So this is really helpful in kind of a broader sense.
JANET RANKIN: Right.
Great.
No, I'm glad.
And I think it's so true.
And people often think, well, you're giving it away.
But you're not giving anything away.
You're just telling them the expectations.
You're not giving them the problems,
you're not teaching to the test, but you're just telling them
what they're going to be able to do at the kind of high level,
or medium level.
You're telling what they're going to be able to do.
I will say-- so ABET, the Accreditation Board
for Engineering Teaching--
every accredited engineering program has to list,
they have A through K outcomes.
And if you look it up, ABET--
and I'm sure you see us as accredited--
if you look up, they have to, they
used to do a lot more bean counting.
But what they would do is the outcomes
would be critical thinking and problem solving skills,
and they were a little bit broad.
But every department that got accredited
had to show how various courses contributed to the outcomes.
So somewhere in your department should be a pretty detailed
list of how 6 double 00 whatever, did this.
Or 6 yada, yada--
I know your grad students-- but did this.
The problem is they don't have to do it
for grad student courses.
But it might be helpful if you looked at that, because I would
imagine that sometimes the recruiters would
be looking at that information, so that might be a commonality.
But every program in the US that's accredited has to have,
show at least how they've addressed those outcomes.
So you might want to take a look at that.
But yeah, if it was a little more explicit
it would certainly be better.
And I'm glad it's helpful.
So I think we can see, you can start
to see how there are better and worse learning outcomes,
that if they're not measurable they're not so useful.
If they are measurable, they're quite useful.
So now what I'd like to do is revisit the worksheet.
And what we can do is we're going
to take about five minutes where you just sit quietly
and write out specific intended learning outcomes.
So you'll be able to upper diagonalize a matrix,
you will be able to use Laplace transforms
to solve this type of differential equation,
whatever you've written in this column.
And note that if you put it in this column,
let's say you put it in the apply column,
then when you write the learning outcome it's going to be you
will be able to interpret data using blah, blah, blah.
You will be able to model the vibration, model a car
suspension system using linearization techniques.
They're going to be whatever, wherever you put it in the box.
These are some of the verbs, these
are the verbs you're going to use in your learning outcome.
Does that make sense?
And I have even more verbs for you if that's helpful.
Did I pass those out?
You got those, right?
AUDIENCE: Yeah.
JANET RANKIN: So there's more verbs.
It's the same thing.
I mean, it's consistent.
So take five minutes by yourself and do that.
And then you're going to get into pairs
and you're going to discuss.
You're going to trade worksheets and give each other feedback.
All right?
So five minutes of quiet and then we'll pair up.
So if you can pair up, you don't necessarily
have to pair up with people that are sitting next to you.
You might want to get a different perspective,
so feel free to get up and move around.
You can talk to somebody that wasn't in your group.
So maybe if you go back and talk with Alex, maybe.
We need one group of three.
Oh, are you a group of three?
So now just trade sheets of paper with your partner,
or if you're a threesome you can figure out how to do it.
And then review the learning outcomes critically.
Guys, just one sec.
So just review your partner's learning outcomes critically.
Make sure that they're measurable, realistic,
and specific.
If your partner has used the word
understand, help him or her work through that
to get a better word.
So you're going to be a critical friend here.
And then you guys can discuss how to improve it.
And then I'm going to pass around some flip chart paper
and each one of you will write, maybe
just write your name at the top, and write two or three
learning outcomes on the flip chart paper.
And then we can stick them up around the room,
and then everybody can see everybody else's,
all learning outcomes.
[INTERPOSING VOICES]
JANET RANKIN: Was there a question?
AUDIENCE: Oh, no.
I just think--
AUDIENCE: I think [INAUDIBLE].
JANET RANKIN: How are you guys doing?
Doing OK?
[INTERPOSING VOICES]
JANET RANKIN: So you can use this zone if you need it.
As you're looking at other people's
learning outcomes, think about how clear,
how easy it would be, or not easy,
to come up with an exam question, or a project,
or an assessment or measurement of whether
or not the student reached, attained the learning outcome.
You can put them up here, Ina and Shau.
OK.
And as I said, make sure that you've read everybody else's.
I'm going to say a few words about them.
I think we may have a record.
This is like the first time ever that no one
has used the word understand, which I think
you're all to be commended for.
So David, make sure you don't write understand.
You build me up just to knock me down.
AUDIENCE: [INAUDIBLE].
JANET RANKIN: You can, or you don't have to.
It doesn't matter.
Does anyone have any comments, either about the exercise
itself, or about somebody's learning outcomes,
that you see around?
Again, the measurement, the thing
you want to look at when you look at your own learning
outcomes and those of others, is are they specific, measurable,
and realistic?
And would you know if a student achieved
those learning outcomes?
So I would say about 90% of these are very specific,
which is great.
I mean, there's some incredibly specific ones here.
The more specific, the easier it is to measure.
So that's one thing.
That's a useful thing.
You want to make sure you don't get so specific
that there's only like one question you could ask.
So that's the balance.
Sometimes you can't tell that unless you're
in the discipline.
Do we have other comments about any of these learning outcomes,
something you don't get, something that
looks particularly interesting?
Yes, Gordon.
Perhaps you can sit down if you're done reading.
AUDIENCE: I'm looking around and I'm seeing [INAUDIBLE].
I don't know how it's going to be.
How are we going to test that?
JANET RANKIN: Well, I guess you have to come up with a reaction
that you're confident they haven't seen before.
So if you can do that, then you can ask them to propose
a mechanism for it, I suppose.
If the person that wrote it wants to--
I know who wrote it-- but if the person that wrote it
wants to talk a little bit about it, that would be great.
If not--
AUDIENCE: OK, So a lot of people are
going to just try to memorize each specific outcome.
But it's easier to just identify your activity group
and find patterns within them.
So if you can try [INAUDIBLE] They can't just [INAUDIBLE].
JANET RANKIN: Right.
And this is a great point.
I mentioned it to one of the groups.
But sometimes we'll say explain how blah, blah, blah, works.
And that's a great thought, that, oh,
they're going to be able to explain it, right?
But many times they can fall back
on a memorized explanation or sort
of kind of a canned procedure.
So if you really want them to be at the level of explain,
you will have to kind of make some things up
to get them out of, to get them away
from the ability to just pull something completely
from memory.
So this is an example of how you might do that.
Other observations or questions?
AUDIENCE: I also [INAUDIBLE] something that all [INAUDIBLE].
JANET RANKIN: So perhaps the person
that wrote it wants to comment on it.
AUDIENCE: OK.
So [INAUDIBLE].
I think what we have, just [INAUDIBLE].
So we write and compile.
So you write a code and then you compile it.
JANET RANKIN: Right.
And this is a great example of how
having people that aren't in the field
read them can help clarify things for your students, who
are by default not experts.
So perhaps to an expert, it totally
makes sense, of course, code.
But to somebody that's not an expert, maybe that's confusing.
Maybe the use of code twice was confusing.
To an expert, it's totally readable.
It's totally understandable.
Perhaps to a novice it isn't.
You're laughing.
AUDIENCE: It's funny, because we're both [INAUDIBLE].
JANET RANKIN: Yeah, that was kind of random.
Well, that's a perfect example, right?
It's a perfect example of how getting maybe other people that
aren't in your field to look at them
can help make them more accessible to your students.
OK.
This is great.
Oh, Hina.
AUDIENCE: I had, it's on a different topic.
But really, I'm talking about buzz words
in learning outcomes.
I think with engineering, you're really
excited about telling people about real world problems.
I guess that's the reason why.
I guess it's more of an opinion question.
As you write your learning outcomes I think in some ways
it's better to be specific about the real world problems
that they're going to attack.
Because I've seen that on syllabuses all the time.
I'm very excited about it.
I think it's a way to teach people in your class.
It's going to help them know what tools they're getting.
JANET RANKIN: Right.
I mean, real world problems in particular
really do kind of encapsulate a whole pile of hopes and dreams.
So it may help to be a little more specific.
You can use it in your learning outcomes,
but then maybe describe it in a little more detail
in your syllabus, the kinds of real world problems,
or why we care about real world problems.
They're messy.
They're ill-defined.
What about them is so important?
Try to minimize the buzz words.
But at some level, they may not know about something
until they're completely through with the course,
and you may have to use that word.
So there's a bit of a balance.
What I'm going to do before, I'll
probably bring all these back to my office,
take pictures of them, and put them up on the wiki
so you'll have a record of them.
I wanted to just as we go out here,
my point is really that the hard part
is these learning outcomes.
And that once you write the learning outcomes,
as we said before, you kind of know
how you're going to test whether students got them,
and you know what you're going to do to help them get them.
The syllabus itself is just an articulation of that.
It's a description of the course,
an articulation of the learning outcomes
stated, like the laundry example, with the title.
This is what you're going to do.
By the end of the class this is what you'll be able to do.
It's kind of a promise that that's what you're going to do.
It motivates students to take the class,
to stay in the class, to engage with the class.
It keeps you on track.
So at the beginning of the semester
maybe I'll be very clear, but four weeks in,
and you're a little bit fuzzy, because you've been dealing
with all sorts of stuff.
And you kind of go, what the heck.
What's this in here for?
Well, if you've lined everything up at the beginning using
your learning outcomes as the anchor,
then you don't have to worry about that.
You just follow the map.
So it really, really helps you and the students
get through the semester in a logical way.
And it tells the students what they
can expect of you, what you can expect of them.
So it's motivational, structural,
and it's evidence, hopefully of what they did, but certainly
of what you hope to do with them.
So for the post-session assignment,
you'll have to write up more learning outcomes
from your course.
You can use these as the basis for sure.
Hopefully, that table will help you as you write more of them.
And I will go in and give you feedback.
I want to make sure that everybody knows that you should
read each other's postings.
So a lot of you had great examples from the first class
about how you would use these learning
theories in your courses and how you teach.
They were fabulous examples.
Make sure you read those postings of others.
That's one reason why it's up on a wiki and not just
you handing it in to me.
So you have the Mud cards.
Fill them out if you have any more information that you want.
And otherwise, we'll see next week.