The following content is provided under a Creative
Commons license.
Your support will help MIT OpenCourseWare
continue to offer high-quality educational resources for free.
To make a donation or to view additional materials
from hundreds of MIT courses, visit MIT OpenCourseWare
at ocw.mit.edu.
JULIAN SHUN: Good afternoon, everyone.
Let's get started.
So welcome to the 11th lecture of 6.172.
It seems that there are many fewer people
here today than on Tuesday.
[LAUGHTER]
All right.
So today we're going to talk about storage allocation.
And it turns out that storage allocation
is about both allocating memory and also freeing it.
But in the literature, it's just called storage allocation,
so that's the term we're going to use.
And whenever you do a malloc or a free,
then you're doing storage allocation.
So how many of you have used malloc or free before?
So hopefully all of you, since you needed it
for the projects and homeworks.
So the simplest form of storage is a stack.
And in a stack, you just have an array and a pointer.
So here we have an array, which we call A,
and then there's some portion of this array that's
used for memory, and the rest of it is free-- it's not used.
And then there's a pointer, sp, that
points to the end of the used region in this stack.
And if you want to allocate x bytes on the stack, all you do
is, you just increment this sp pointer by x.
And then, of course, you should also check for overflow
to make sure that you don't actually
go off the end of this array, because if you
do that then you'll get a segmentation fault.
But actually, nowadays, compilers
don't really check for stack overflow
because your stack is usually big enough for most
of your program, and when you do get a stack overflow,
you'll just get a segfault and then you go debug your program.
So for efficiency reasons, the stack overflow
isn't actually checked.
OK?
And then it returns a pointer to the beginning of the memory
that you just allocated, so that's just as sp minus x.
So that's pretty simple.
And in fact, this is how the C call stack works.
It also uses a stack discipline.
So when you call a function, you save local variables
and registers on the stack, and you also
save the return address of the function that's
calling another function.
And then, when you return, you pop things off the stack.
So you can also free things from the stack.
So what you do is, you just decrement sp
by x if you want to free x bytes.
So here we just decremented sp by x, and everything
after sp is now considered to be free.
And again, if you're careful, you
would check for a stack underflow.
But again, the compiler usually doesn't do this
because if you do have a stack overflow,
there's a bug in your program, and you'll get a segfault,
and you should go fix it.
So allocating and freeing in a stack
takes constant time because all you have to do
is manipulate the stack pointer, so it's pretty efficient.
However, you have to free consistently
with the stack discipline.
So the stack has limited applicability.
Does anybody see why you can't do everything
with just a stack?
So what's one limitation of the stack?
Yes?
AUDIENCE: [INAUDIBLE]
JULIAN SHUN: So it turns out that you can actually
pass a compile time constant to make the stack bigger
if you wanted to.
There's actually a more fundamental limitation
of the stack.
Yes.
AUDIENCE: You can only read things
in the reverse order in which you allocate them.
JULIAN SHUN: Yeah, so the answer is that you can only
free the last thing that you allocated,
so there's no way to free anything
in the middle of this used region here.
You have to free the last thing here
because the stack doesn't keep track of the objects
in the middle of this used region.
So there's limited applicability,
but it's great when it works because it's very efficient,
and all of this code can essentially be inline.
You don't have to have any function calls,
so it's very fast.
And it also turns out that you can
allocate on the call stack using this function called alloca.
It's actually not a function.
It's just a keyword that the compiler recognizes,
and it will transform it to instructions
that manipulate the stack.
However, this function is now deprecated
because it turns out that the compiler is actually
more efficient when you're dealing
with these fixed-size frames if you just allocate
a pointer on the stack that points to some piece of memory
on the heap.
But nevertheless, if you want to allocate on the call stack,
you can call this alloca function,
but you should check that is doing the right thing
since it's now deprecated and the implementation is compiler
dependent.
So what's another type of storage besides the stack?
So you can't do everything with a stack.
So what else can we use?
Yes?
AUDIENCE: Heap.
JULIAN SHUN: Yes.
So we also have the heap, which is more general than the stack.
So a stack looks very nice and tidy,
and it's very efficient to use the stack,
but it doesn't work for everything.
So that's why we have the heap.
And a heap is much more general, but it's very messy.
It's very hard to organize this and work with it efficiently.
And for the rest of this lecture,
I am going to be talking about how
to manage memory in the heap.
And I found these pictures on Stack Overflow,
so maybe they're biased towards stacks.
[CHUCKLES] OK, so how do we do heap allocation?
So let's first start with fixed-size heap allocation,
where we assume that all of the objects that we're dealing with
are of the same size.
In general this isn't true, but let's just
start with this simpler case first.
OK, so as I said earlier, if you use malloc and free in C,
then you're doing heap allocation.
C++ has the new and delete operators,
which work similarly to malloc and free.
They also call the object constructor and destructor,
and the C functions don't do that.
And unlike Java and Python, C and C++ don't provide a garbage
collector, so the programmer has to manage memory him
or herself, and this is one of the reasons for the efficiency
of C and C++, because there's no garbage collector running
in the background.
However, this makes it much harder
to write correct programs in C because you
have to be careful of memory leaks, dangling pointers,
and double freeing.
So a memory leak is if you allocate something
and you forget to free it and your program keeps
running and allocating more and more stuff
but without freeing it.
Eventually, you're going to run out of memory,
and your program is going to crash.
So you need to be careful of memory leaks.
Dangling pointers are pointers to pieces of memory
that you have already freed, and if you
try to dereference a dangling pointer,
the behavior is going to be undefined.
So maybe you'll get a segmentation fault.
Maybe you won't see anything until later on in your program
because that memory might have been reallocated for something
else, and it's actually legal to dereference that memory.
So dangling pointers are very annoying when
you're using C. If you're lucky, you'll get a segfault right
away, and you can go fix your bug,
but sometimes these are very hard to find.
There's also double freeing.
So this is when you free something more than once.
And again, this will lead to undefined behavior.
Maybe you'll get a segfault, or maybe that piece of memory
was allocated for something else,
and then when you free it again, it's actually legal.
But your program is going to be incorrect,
so you need to be careful that you don't free something
more than once.
And this is why some people prefer
to use a language like Java and Python
that provide these built-in garbage collectors.
However, these languages are less efficient
because they have a general-purpose garbage
collector running in the background.
So in this class, we're going to use C
because we want to be able to write the fastest programs as
possible, so we need to study how to manage memory.
And there are some tools you can use
to reduce the number of memory bugs you have in your program.
So there's memory checkers like AddressSanitizer and Valgrind,
which can assist you in finding these pernicious bugs.
So AddressSanitizer is a compiler instrumentation tool.
When you compile your program, you pass a flag,
and then, when you run your program,
it's going to report possible memory
bugs you have in your program.
And then Valgrind-- it works directly off the binaries.
You don't need to do anything special when you compile it.
You can just pass your binary to Valgrind,
and if there is a memory bug, it might find it.
But Valgrind tends to be slower than AddressSanitizer,
and it tends to catch fewer bugs because it
knows less about the program than AddressSanitizer.
And AddressSanitizer sees the source code of the program
and has more information, whereas Valgrind just
works directly off the binary.
Also, don't confuse the heap with the heap data structure
that you might have seen before in your algorithms or data
structures courses.
So these are two different concepts.
The heap data structure in your algorithms course
was a data structure used to represent a priority queue,
where you can efficiently extract the highest priority
element, and you can also update the priorities
of elements in the set.
And this could be used for algorithms
like sorting or graph search.
But today we're going to be talking
about another heap, which is the heap that's
used for storage allocation.
So don't get confused.
So any questions so far?
OK.
All right.
So we're going to first start with fixed-size allocations,
since that's the easier case.
So we're going to assume that every piece of storage
has the same size.
Some of these blocks are used, and some of them are unused.
And among the unused blocks, we're
going to keep a list that we call the free list,
and each block in this free list has a pointer
to the next block in the free list.
And since this memory is unused, we
can actually use the memory to store
a pointer as part of our storage allocator implementation.
There's actually another way to do fixed-size allocations.
Instead of using a free list, you
could actually a place a bit for each block saying
whether or not it's free, and then when you do allocation,
you can use bit tricks.
But today I'm going to talk about the free list
implementation.
So to allocate one object from the free list,
you set the pointer x to be free.
So free is pointing to the first object in this free list.
Then you set the free pointer to point
to the next thing in the free list,
so this is doing free equal to the next pointer of free.
And then, finally, you return x, which
is a pointer to the first object in the free list.
So here's an animation.
So x is going to point to what free points to.
You also need to check if free is equal to null,
because if free is equal to null,
that means there are no more free blocks in the free list,
and the programmer should know this.
You should return a special value.
Otherwise, we're going to set the free pointer
to point to the next thing in the free list,
and then, finally, we return x to the program,
and now the program has a block of memory that I can use.
There is still a garbage pointer in this block
that we pass back to the program because we didn't clear it.
So the implementation of the storage allocator
could decide to zero this out, or it can just
pass it back to the program and leave it up
to the programmer to do whatever it wants with it.
So in the latter case, the programmer
should be careful not to try to dereference this pointer.
OK, so how about deallocation?
So let's say we want to free some object x.
What we do is, we just set the next pointer
of x to be equal to free, so it's
going to point to the first thing in the free list.
And then we set free equal to x.
So right.
So now free is pointing to x, and this x object
that we wanted to free now is a pointer to the first object
in the original free list.
So pretty simple.
Any questions on this?
So this sort of acts like a stack
in that the last thing that you've freed
is going to be the first thing that you allocate,
so you get temporal locality in that way.
But unlike a stack, you can actually
free any of the blocks and not just the last block
that you allocated.
So with a free list, allocating and freeing
take constant time because you're just
adjusting some pointers.
It has good temporal locality, because as I said,
the things that you freed most recently
are going to be the things that are going to be allocated.
It has poor spatial locality due to external fragmentation,
and external fragmentation means that your box of used memory
are spread out all over the place
in the space of all memory.
And this can be bad for performance
because it can increase the size of the page table,
and it can also cause disk thrashing.
So if you recall, whenever you access
a page in virtual memory, it has to do address translation
to the physical memory address.
And if your memory is spread out across many pages
in virtual memory, then you're going to have a lot of entries
in the page table, because the page table is storing
this mapping between the virtual memory address of the page
and the physical memory address of the page.
So this can complicate the page table,
make it less efficient to do lookups in it.
And then if you have more pages than you
can fit in your main memory, then this
can cause disk thrashing because you have to move pages
in and out of disk.
The Translation Lookaside Buffer or TLB can also be a problem.
Does anybody know what a TLB is?
Yes?
AUDIENCE: A cache of the result of translating
from virtual memory to physical memory.
JULIAN SHUN: Yeah, so the TLB is essentially
a cache for the page table, so it
will cache the results of the translation
from virtual memory addresses to physical memory
addresses for the most recent translations.
And looking up a translation in the TLB
is much more efficient than going through the page table.
And if you have a lot of external fragmentation,
then you have a lot of pages that you might access,
and this means that when you go to the TLB,
it's more likely you'll get a TLB miss,
and you have to go to the page table
to look up the appropriate address.
So that's why external fragmentation is bad.
So let's look at some ways to mitigate
external fragmentation.
So one way to do this is to keep a free list or a bitmap
per disk page, and then when you want to allocate something,
you allocate from the free list of the fullest page.
So you sort of skew the memory that's
being used to as few pages as possible.
And when you free a block of storage,
you just return it to the page on which that block resides.
And if a page becomes completely empty--
there are no more items that are used on that page--
then the virtual memory system can page it out
without affecting the program performance because you're not
going to access that page anyways.
So this might seem counterintuitive.
Why do we want to skew the items to as few pages as possible?
So let's look at a simple example
to convince ourselves why this is actually good for dealing
with external fragmentation.
So here I have two cases.
In the first case, I have 90% of my blocks on one page
and 10% of the blocks on the other page.
In the second case, I have half of my blocks on one page
and half on the other page.
So now let's look at the probability
that two random accesses will hit the same page.
So let's assume that all of the random accesses
are going to go to one of the two pages.
So in the first case, the probability
that both of the accesses hit the first page is going
to be 0.9 times 0.9, and then the probability that they
both hit the second page is 0.1 times 0.1,
and if you sum this up, you get 0.82.
That's the probability that both of the random accesses
are going to hit the same page.
In the other case, the probability
that both of the accesses hit the first page
is going to be 0.5 times 0.5.
The second page is also going to be 0.5 times 0.5,
so that sums to 0.5, and that means
that there's only a 50% chance that two random accesses are
going to hit the same page.
So in the first case, you actually
have a higher chance that the two random accesses hit
the same page, and that's why we want to skew the items as much
as possible so that we can reduce
the external fragmentation.
Any questions?
OK, so that was fixed-size heap allocation,
and obviously you can't use that for many programs
if you're allocating memory of different sizes.
So now let's look at variable-size heap allocation.
So we're going to look at one allocation scheme called
binned free lists.
And the idea is to leverage the efficiency of free list
and also accept the bounded amount
of internal fragmentation.
So internal fragmentation is wasted space within a block,
so that means when you allocate possibly
more space than you're using, then
there's some wasted space in there.
So in binned free lists, what we're going to do is,
we're going to have a whole bunch of bins,
and each bin is going to store blocks of a particular size.
So here I'm going to say that bin k holds
memory blocks of size 2 to the k,
so I'm going to store blocks of sizes powers of 2.
So why don't I just store a bin for every possible size?
Does anybody know why?
Why am I rounding up to powers of 2 here?
AUDIENCE: You'd have too many bins.
JULIAN SHUN: Yes, if I wanted a bin for every possible size,
I would have way too many bins, and just the pointers
to these bins are not going to fit in memory.
So that's why I'm only using bins that store
blocks of size 2 to the k.
And now let's look at how I'm going
to allocate x bytes from a binned free list.
So what I'm going to do is, I'm going
to look up the bin for which I should take a block from,
and to get that, I'm going to take the ceiling of log base x.
This is log base 2, so recall that lg is log base 2.
If that bin is nonempty, then I can just
return a block from that bin.
However, if that bin is empty, then I
need to go to the next highest bin that's nonempty,
and then I'm going to take a block from that bin
and then split it up into smaller chunks
and place them into smaller bins.
And then I'll also get a chunk that is of the right size.
So for this example, let's say I wanted to allocate 3 bytes.
The ceiling of log base 2 of x is 2, so I go to bin 2.
But bin 2 is empty, so I need to look for the next bin that's
not empty.
And that's going to be bin 4.
And I'm going to split up this block into smaller powers of 2.
So in particular, I'm going to find
a nonempty bin k prime greater than k
and split up a block into sizes of 2 to the k prime minus 1,
2 to the k prime minus 2, all the way down to 2 to the k.
So I'm going to split it into sizes of all the powers of 2
less than 2 to the k prime and greater than or equal to 2
to the k.
And I'm going to actually have two blocks of size 2 to the k,
and one of those will be returned to the program.
So here I'm going to split up this block.
I'm going to place one of the smaller blocks in bin 3, one
of them into bin 2, and then I also have another block here
that I'm just going to return to the program.
So any questions on how this scheme works?
OK, and if there are no larger blocks that exist--
so that means all of the bins higher
than the bin I'm looking at are empty--
then I need to go to the OS to request for more memory.
And then after I get that memory,
I'll split it up so I can satisfy my allocation request.
In practice, this exact scheme isn't used,
so there are many variants of this scheme.
So it turns out that efficiency is
very important for small allocations
because there's not that much work performed
on these small pieces of memory, and the overheads
of the storage allocation scheme could cause a performance
bottleneck.
So in practice, you usually don't go all the way down
to blocks of size 1.
You might stop at blocks of size 8 bytes so
that you don't have that much overhead,
but this does increase the internal fragmentation
by a little bit, because now you have some wasted space.
And then-- one second-- and then you
can also group blocks into pages,
as I said before, so that all of the blocks in the same page
have the same size, and then you don't
have to store the information of the size of the blocks.
Yes.
AUDIENCE: How do you--
JULIAN SHUN: Yeah, so there are two commands you can use.
One is called mmap and the other one is called sbrk.
So those are system calls.
You just call that, and then the OS will give you more memory,
and then your storage allocator can use it.
Yes?
AUDIENCE: They don't have to use something like this in order
to implement those?
JULIAN SHUN: No, the standard implementation of malloc--
internally, it uses these commands, mmap and sbrk,
to get memory from the OS, so the OS just gives you
a huge chunk of memory.
It doesn't split it up into smaller blocks or anything.
That's up to the storage allocator to do.
It just gives you a big chunk of memory,
and then the storage allocator will break it up
into smaller blocks.
There are similar commands where you
can free memory back to the OS when
you're not using them anymore.
AUDIENCE: Can you explain the paging thing [INAUDIBLE]??
JULIAN SHUN: Yeah, so what I said
was that you can actually keep blocks of different sizes
on different pages, so then you don't actually
have to store the size of each block.
You can just look up what page that block resides in when
you get the memory address.
And then for each page, you have one field
that stores the size of those blocks on that page.
So this saves you the overhead of having
to store information per block to figure out its size.
Yeah.
AUDIENCE: --changing the size of the blocks.
JULIAN SHUN: Yeah, so I mean, if you
do change the size of the blocks,
then you can't actually use this scheme,
so this is actually a variant where you don't
change the size of the blocks.
If you do change the size, then you
have to change it for the entire page.
Yeah, so there are many variants of memory allocators out there.
This is just the simplest one that I described.
But it turns out that this exact scheme isn't the one that's
used in practice.
There are many variants.
Like some allocators, instead of using powers of 2,
they use Fibonacci numbers to determine the different bins.
Yeah.
Any other questions?
So you'll actually get a chance to play around
with implementing some allocators in project
3 and homework 6.
So let's briefly look at the storage layout of a program.
So this is how our virtual memory address
space is laid out.
So we have the stack all the way at the top,
and the stack grows downwards, so we have the high addresses
up top and the low addresses below.
Then we have the heap, which grows upward,
and the heap and the stack basically
grow towards each other, and this space is dynamically
allocated as the program runs.
Then there's the bss segment, the data segment,
and the text segment, which all reside below the heap.
So the code segment just stores the code for your program.
So when you load up your program,
this code is going to put your program into this text segment.
Then there's this data segment, which
stores all of the global variables and static variables,
these constants that you defined in your program.
These are all stored in the data segment,
and when you load your program you also
have to read this data from disk and store it
into the data segment.
Then there's the bss segment this segment is
used to store all the on initialize variables
in your program, and this is just
initialized to 0 at the start of your program,
since your program hasn't initialized it,
so it doesn't matter what we set it to.
And then the heap-- this is the memory
that we're using when we're calling malloc and free.
And then we have the stack, which we talked about.
So in practice, the stack and the heap
are never actually going to hit each other because we're
working with 64-bit addresses.
So even though they're growing towards each other,
you don't have to worry about them actually
hitting each other.
And another point to note is that if you're
doing a lot of precomputation in your program,
for example generating these huge tables of constants,
those all have to be read from disk when
you start your program and stored in this data segment.
So if you have a lot of these constants,
it's actually going to make your program loading
time much higher.
However, it's usually OK to do a little bit of precomputation,
especially if you can save a lot of computation at runtime,
but in some cases it might actually
be faster overall to just compute the things in memory
when you start your program, because then you
have to read stuff from disk.
So here's a question.
So since a 64-bit address space takes over a century
to write at a rate of 4 billion bytes per second,
we're never effectively going to run out of virtual memory.
So why don't we just allocate out of virtual memory and never
free anything?
Yes?
AUDIENCE: If you allocate a bunch
of small things in random places,
then it's harder to update than a large segment?
JULIAN SHUN: Yeah, so one thing is
that you have this issue of fragmentation.
The blocks of memory that you're using
are not going to be contiguous in memory,
and then it makes it harder for you to find large blocks.
So this is called external fragmentation,
which I mentioned earlier.
So if you do this, external fragmentation
is going to be very bad.
The performance of the page table
is going to degrade tremendously because the memory that you're
using is going to be spread all over virtual memory,
and you're going to use many pages,
and this leads to disk thrashing.
So you have to do a lot of swaps of pages in and out of disk.
Your TLB hit rate is going to be very low.
And another reason is that you're also
going to run out of physical memory
if you never free anything.
So one of the goals of storage allocation
is to try to use as little virtual memory as possible
and to try to keep the used portions of the memory
relatively compact.
Any questions so far?
OK, so let's do an analysis of the binned free list storage
allocation scheme.
So here's a theorem.
Suppose that the maximum amount of heap memory in use
at any time by a program is M. If the heap is managed
by a binned free list allocator, then the amount
of virtual memory consumed by the heap storage
is upper bounded by M log M. Does anybody
have an intuition about why this theorem could be true?
So how many bins do we have, at most?
AUDIENCE: [INAUDIBLE]
JULIAN SHUN: Right.
So the number of bins we have is upper bounded by log M,
and each bin is going to use order M memory.
So let's look at this more formally.
So an allocation request for a block of size x
is going to consume 2 to the ceiling of log base 2
of x storage, which is upper bounded by 2x,
so we're only wasting a factor of 2 storage here.
So therefore, the amount of virtual memory
devoted to blocks of size 2 to the k is at most 2M.
And since there are at most log base 2 of M free lists,
the theorem holds just by multiplying the two terms.
So you can only have log base 2 of M free lists
because that's the maximum amount of memory you're using,
and therefore your largest bin is only
going to hold blocks of size M. And it turns out
that the bin free list allocation scheme is theta
of 1 competitive with the optimal allocator,
and here an optimal locator knows all of the memory
requests in the future.
So it can basically do a lot of clever things
to optimize the memory allocation process.
But it turns out that the binned free list is only
going to be a constant factor worse
than the optimal allocator.
This is assuming that we don't coalesce blocks together,
which I'll talk about on the next slide.
It turns out that this constant is 6,
so Charles Leiserson has a paper describing this result.
And there's also a lower bound of 6, so this is tight.
So coalescing.
So coalescing is when you splice together smaller blocks
into a larger block.
So you can do this if you have two free blocks that
are contiguous in memory.
This will allow you to put them together into a larger block.
So binned free lists can sometimes
be heuristically improved by doing coalescing,
and there are many clever schemes
for trying to find adjacent blocks efficiently.
So there's something called the buddy
system, where each block has a buddy that's
contiguous and memory.
However, it turns out that this scheme especially, the buddy
system scheme, has pretty high overhead.
So it's usually going to be slower
than just the standard binned free list algorithm.
There are no good theoretical bounds
that exist that prove the effectiveness of coalescing,
but it does seem to work pretty well in practice
at reducing fragmentation because heap storage tends
to be deallocated as a stack or in batches.
So what I mean by this is that the objects that you free
tend to be pretty close together in memory.
So if you deallocate as a stack, then all of them
are going to be near the top of the stack.
And when you deallocate in batches--
this is when you do allocate a whole bunch of things
that you allocated together in your program.
For example, if you have a graph data structure
and you allocated data for the vertices all at the same time,
then when you deallocate them all together,
this is going to give you a chunk of contiguous memory
that you can splice together.
OK, so now let's look at garbage collection.
This is going to be slightly different from storage
allocation.
So the idea of garbage collection
is to free the programmer from having to free objects.
So languages like Java and Python,
they have built-in garbage collectors,
so the programmer doesn't have to free stuff themselves,
and this makes it easier to write programs
because you don't have to worry about double
freeing and dangling pointers and so forth.
So a garbage collector is going to identify and recycle
the objects that the programmer can no longer access
so that these memory objects can be used for future allocations.
And in addition to having a built-in garbage collector,
you can also create your own garbage collector
in C, which doesn't have a garbage collector.
So if you have an application, you
can actually create a special-purpose garbage
collector that might be more efficient than
a general garbage collector.
Yes?
AUDIENCE: This is the previous topic, but why [? INAUDIBLE ?]
order of M memory?
JULIAN SHUN: Why is it not order M?
AUDIENCE: Yeah.
JULIAN SHUN: Because for each of the bins,
you could use up to order M memory.
So if you don't do coalescing, basically, I
could have a bunch of small allocations,
and then I chop up all of my blocks,
and then they all go into smaller bins.
And then I want to allocate something larger.
I can't just splice those together.
I have to make another memory allocation.
So if you order your memory requests in a certain way,
you can make it so that each of the bins has order M memory.
OK, so for garbage collection, let's go over some terminology.
So there are three types of memory objects, roots,
live objects, and dead objects.
Roots are objects that are directly
accessible by the program, so these are global variables,
things on the stack, and so on.
Then there are live objects, which
are reachable by following the roots via pointers,
and then finally, there are dead objects,
and these objects are inaccessible via sequences
of pointers.
And these can be recycled because the programmer can
no longer reach these dead objects.
So in order for garbage collection to work in general,
you need to be able to have the garbage collector identify
pointers, and this requires strong typing.
So languages like Python and Java have strong typing,
but in C, it doesn't have strong typing.
This means that when you have a pointer
you don't actually know whether it's a pointer.
Because a pointer just looks like an integer.
It could be either a point or an integer.
You can cast things in C. You can also
do pointer arithmetic in C. So in contrast,
in other languages, once you declare something
to be a pointer, it's always going to be a pointer.
And for those languages that have strong typing,
this makes it much easier to do garbage collection.
You also need to prohibit doing pointer arithmetic
on these pointers.
Because if you do pointer arithmetic
and you change the location of the pointer,
then the garbage collector no longer
knows where the memory region starts anymore.
In C, sometimes you do do pointer arithmetic,
and that's why you can't actually
have a general-purpose garbage collector in C that works well.
So let's look at one simple form of garbage collection.
And this is called reference counting.
The idea is that, for each object,
I'm going to keep a count of the number of pointers referencing
that object.
And if the count ever goes to 0, then that
means I can free that object because there
are no more pointers that can reach that object.
So here, I have a bunch of roots.
So these are directly accessible by my program.
And then I have a bunch of objects
that can be reached via following pointers
starting from the root.
And then each of them have a reference count
that indicates how many incoming pointers they have.
So let's say now I change one of these pointers.
So initially, I had a pointer going to here,
but now I changed it so that it goes down here.
So what happens now is I have to adjust a reference counts
of both of these objects.
So this object here, now it doesn't
have any incoming pointers, so I have to decrement its reference
count.
So that goes to 0.
And then for this one, I have to increment its reference count,
so now it's 3.
And now I have an object that has a reference count of 0,
and with this reference counting algorithm,
I can free this object.
So let's go ahead and free this object.
But when I free this object, it actually
has pointers to other objects, so I also
have to decrement the reference counts of these other objects
when I free this object.
So I'm going to decrement the counts.
And now it turns out that this object also
has a reference count of 0, so I can free that, as well.
And in general, I just keep doing this process
until the reference counts of the objects
don't change anymore, and whenever I encounter an object
with a reference count of 0, I can free it immediately.
And the memory that I freed can be recycled.
It can be used for future memory allocations.
So questions on how the reference counting
procedure works?
So there's one issue with reference counting.
Does anybody see what the issue is?
Yes?
AUDIENCE: What if it has a reference to itself?
JULIAN SHUN: Yes.
So what if it has a reference to itself?
More generally, what if it has a cycle?
You can't ever collect garbage collect a cycle when
you're using reference counts.
So here we have a cycle of length 3.
They all have a reference count of 1,
but you can never reach the cycle by following pointers
from the root, and therefore, you
can never delete any object in the cycle,
and the reference counts are always going to be non-zero.
So let's just illustrate the cycle.
And furthermore, any object that's pointed to by objects
in the cycle cannot be garbage collected, as well,
because you can't garbage collect the cycle,
so all the pointer is going out of the objects in the cycle are
always going to be there.
So there could be a lot of objects downstream
from this object here that can't be garbage collected,
so this makes it very bad.
And as we all know, uncollected garbage stinks,
so we don't want that.
So let's see if we can come up with another garbage collection
scheme.
So it turns out that reference counting is actually
pretty good when it does work because it's
very efficient and simple to implement.
So if you know that your program doesn't
have these cycles in them among pointers,
then you can use a reference counting scheme.
There are some languages, like Objective-C,
that have two different types of pointers, strong pointers
and weak pointers.
And if you're doing reference counting
with a language with these two types of pointers,
the reference count only stores the number
of incoming strong pointers.
And therefore, if you define these pointers inside a cycle
to be weak pointers, they're not going
to contribute to the reference count,
and therefore you can still garbage collect.
However, programming with strong or weak pointers
can be kind of tricky because you
need to make sure that you're not dereferencing something
that a weak pointer points to because that thing might
have been garbage collected already,
so you need to be careful.
And C doesn't have these two types of pointers,
so we need to use another method of garbage collection
to make sure we can garbage collect these cycles.
So we're going to look at two more garbage collection
schemes.
The first one is called mark-and-sweep,
and the second one is called stop-and-copy.
So first we need to define a graph abstraction.
So let's say we have a graph with vertices V and edges E.
And the vertex at V contains all of the memory
objects in memory, and the edges E are
directed edges between objects.
So there's a directed edge from object A
to object B if object A has a pointer to object B.
And then, as we said earlier, the live objects are the ones
that are reachable from the roots,
so we can use a breadth-first-search-like
procedure to find all of the live objects.
So we just start our breadth-first search
from the roots, and we'll mark all
of the objects that can be reachable from the roots.
And then everything else that isn't reached,
those are available to be reclaimed.
So we're going to have a FIFO queue,
First-In, First-Out queue, for our breadth-first search.
This is represented as an array.
And we have two pointers, one to the head of the queue
and one to the tail of the queue.
And here let's look at this code, which essentially
is like a breadth-first search.
So we're first going to go over all the vertices in our graph,
and we're going to check if each vertex v is a root.
If it is a root, we're going to set its mark to be 1,
and we're going to place the vertex onto the queue.
And otherwise, we're going to set the mark of v to be 0.
And then while the queue is not empty,
we're going to dequeue the first thing from the queue.
Let that be u.
Then we're going to look at all the outgoing neighbors of u.
So these are vertices v such that there
is a directed edge from u to v. We're going to check
if v's mark is equal to 0.
If it is, that means we haven't explored it yet,
so we'll set its mark to be 1, and we place it onto the queue.
And if the neighbor has already been explored,
then we don't have to do anything.
So let's illustrate how this algorithm works
on this simple graph here.
And for this example, I'm just going
to assume that I have one root, vertex r.
In general, I can have multiple routes,
and I just place all of them onto the queue
at the beginning, but for this example,
I'm just going to have a single root.
So I'm going to place it onto the queue, and the location
that I place it is going to be where the tail pointer points
to.
And after I placed it on the queue,
I increment the tail pointer.
Now I'm going to take the first thing off
of my queue, which is r, and I'll explore my neighbors.
So the neighbors are b and c here.
Both of them haven't been marked yet, so I'm going to mark them,
and I'm going to indicate the marked vertices
with shaded blue.
And I'll place them onto the queue.
Now I'm going to take the next thing, b.
I'm going to check its neighbors.
It only has a neighbor to c, but c is already on the queue.
It's already marked, so I don't have to do anything.
Now I dequeue c, and c has neighbors d and e,
so I place them onto the queue.
d doesn't have any outgoing neighbors,
so I don't to do anything.
Now when I dequeue e, it has neighbors f.
When I dequeue f, it has a neighbor g,
and when I dequeue g, it doesn't have any neighbors.
So now my queue is empty, and my breadth-first search procedure
finishes.
So at this point, I've marked all
of the objects that are accessible from the root,
and all of the unmarked objects can now
be garbage collected because there is no way I
can access them in the program.
So the mark-and-sweep procedure has two stages.
The first stage is called the mark stage,
where I use a breadth-first search
to mark all of the live objects.
And the sweep stage will scan over memory
to free the unmarked objects.
So this a pretty simple scheme.
There is one issue with this scheme.
Does anybody see what the possible issue is?
Yes?
AUDIENCE: You have to scan over all the [INAUDIBLE]..
JULIAN SHUN: Yeah, so that's one issue, where you have
to scan over all of memory.
There are some variants of mark-and-sweep
where it keeps track of just the allocated objects,
so you only have to scan over those instead
of the entire memory space.
Besides that, are there any other possible issues
with this?
Yes?
AUDIENCE: This also requires that you [INAUDIBLE]
strong typing.
JULIAN SHUN: Right, so let's assume
that we do have strong typing.
Any other possible limitations?
Anybody else?
Think I called on--
yeah.
AUDIENCE: [INAUDIBLE] reference counting,
you can see the object that has a reference to it,
whereas for here you can find everything that would not be
garbage collected [INAUDIBLE].
JULIAN SHUN: Yeah, so for the scheme that I described,
you have to look over all of the things that
don't have references to it.
So that is another overhead.
So those are all issues.
Good.
The issue I want to get at is that
the mark-and-sweep algorithm that I presented here
doesn't deal with fragmentation.
So it doesn't compact the live objects
to be contiguous in memory.
It just frees the ones that are unreachable,
but it doesn't do anything with the ones that are reachable.
So let's look at another procedure that
does deal with fragmentation.
This is called the stop-and-copy garbage collection procedure.
At a high level, it's pretty similar to
the mark-and-sweep algorithm.
We're still going to use a breadth-first search
to identify all of the live objects.
But if you look at how this breadth-first search is
implemented, is there any information
you can use here to try to get the live objects to be
contiguous in memory?
Does anybody see anything here that we can use
to try to reduce fragmentation?
Yes?
AUDIENCE: [INAUDIBLE]
JULIAN SHUN: Yes, so the answer is
that the objects that we visited are contiguous on the queue.
So in the mark-and-sweep algorithm,
I just place the IDs of the vertices on the queue,
but if I just place the actual objects onto the queue instead,
then I can just use my queue as my new memory.
And then all of the objects that are unreachable
will be implicitly deleted.
So this procedure here will deal with external fragmentation.
So let's see how this works.
So we're going to have two separate memory
spaces, the FROM space and the TO space.
So in the FROM space, I'm just going to do allocation
and freeing on it until it becomes full.
So when I allocate something I place it
at the end of this space.
When I free something, I just market as free,
but I don't compact it out yet.
And when this FROM space becomes full,
then I'm going to run my garbage collection algorithm,
and I'm going to use the TO space as my queue
when I do my breadth-first search.
So after I run my breadth-first search, all of the live objects
are going to appear in the TO space in contiguous memory
since I used the TO space as my queue.
Right, and then I just keep allocating stuff
from the TO space and also marking things
as deleted when I free them until the TO space
becomes full.
Then I do the same thing, but I swap the roles
of the TO and the FROM spaces.
So this is called the stop-and-copy algorithm.
There is one problem with this algorithm
which we haven't addressed yet.
Does anybody see what the potential problem is?
Yes?
AUDIENCE: If nothing is dead, then you're
copying over your entire storage every single time.
JULIAN SHUN: Yeah, so that's one good observation.
If nothing is dead, then you're wasting a lot of work
because you have to copy this.
Although with the mark-and-sweep algorithm,
you still have to do some copying,
although you're not copying the entire objects.
You're just copying the IDs.
There's actually a correctness issue here.
So does anybody see what the correct this issue is?
Yes?
AUDIENCE: So maybe the pointers in the TO space
have to be changed in order to point to the new [INAUDIBLE]..
JULIAN SHUN: Yeah, so the answer is
that if you had pointers that pointed to objects in the FROM
space, if you move your objects to the TO space,
those pointers aren't going to be correct anymore.
So if I had a pointer to a live object before
and I moved my live object to a different memory address,
I need to also update that pointer.
So let's see how we can deal with this.
So the idea is that, when an object is copied
to the TO space, we're going to store a forwarding
pointer in the corresponding object
in the from space, and this implicitly marks
that object as moved.
And then when I remove an object from the FIFO
queue in my breadth-first search, in the TO space
I'm going to update all of the pointers
by following these forwarding pointers.
So let's look at an example of how this works.
So let's say I'm executing the breadth-first search,
and this is my current queue right now.
What I'm going to do is, when I dequeue an element
from my queue, first I'm going to place the neighboring
objects that haven't been explored yet onto the queue.
So here it actually has two neighbors,
but the first one has already been placed the queue,
so I can ignore it.
And the second one hasn't been placed on the queue yet,
so I place it onto the queue.
And then I'm also going to--
oh, so this object also has a pointer
to something in the FROM space, which I'm not
going to change at this time.
But I am going to store a forwarding pointer
from the object that I moved from the FROM space
to the TO space, so now it has a pointer
that tells me the new address.
And then, for the object that I just dequeued,
I'm going to follow the forwarding
pointers of its neighbors, and that will give me
the correct addresses now.
So I'm going to update the pointers by just following
the forwarding pointer.
So the first pointer pointed to this object,
which has a forwarding pointer to this, so I just make a point
to this object in the TO space.
And then similarly for the other pointer,
I'm going to make it point to this object.
So that's the that's the idea how
of how to adjust the pointers.
One question is, why can't we just adjust the pointer
is when we enqueue the object?
So why do I have to adjust the pointers
when I dequeue an object?
Yes?
AUDIENCE: Because we haven't processed its neighbors yet.
JULIAN SHUN: Yeah, so the answer is
that, when you enqueue object, you don't actually
know where your neighbors are going
to reside in the TO space.
And you only know that when you dequeue the object,
because when you dequeue the object,
you must have explored your neighbors,
and therefore you can generate these forward pointers.
So any questions on this scheme?
So how much time does it take to do the stop-and-copy procedure?
So let's say n is the number of objects
and the number of pointers I have,
so it's the sum of the number of objects and number of pointers.
How much time would it take to run this algorithm?
AUDIENCE: [INAUDIBLE]
JULIAN SHUN: Yeah, so it's just going
to be linear time because we're running a breadth-first search,
and that takes linear time.
You also have to do work in order
to copy these objects to the TO space,
so you also have to do work proportional
to the number of bytes that you're copying over.
So it's linear in the number of objects,
the number of pointers, and the total amount of space
or copying over.
And the advantage of this scheme is
that you don't actually need to go over the objects that
aren't reachable because those are going to be implicitly
deleted because they're not copied over to the TO space,
whereas in the mark-and-sweep procedure,
you had to actually go through your entire memory
and then free all the objects that aren't reachable.
So this makes the stop-and-copy procedure more efficient,
and it also deals with the external fragmentation issue.
So what happens when the FROM space becomes full?
So what you do is, you're going to request a new heap
space equal to the used space, so you're just
going to double the size of your FROM space.
And then you're going to consider the FROM space to be
full when the newly allocated space becomes full,
so essentially what you're going to do
is, you're going to double the space,
and when that becomes full, you're going
to double it again, and so on.
And with this method, you can amortize
the cost of garbage collection to the size of the new heap
space, so it's going to be amortized constant overhead
per byte of memory.
And this is assuming that the user
program is going to touch all of the memory that it allocates.
And furthermore, the virtual memory space
required by this scheme is just a constant times
the optimal if you locate the FROM and the TO spaces
in different regions of virtual memory
so that they can't interfere with each other.
And the reason why it's a constant times the optimal
is because you only lose a factor of 2
because you're maintaining two separate spaces.
And then another factor of 2 comes from the fact
that you're doubling the size of your array when it becomes full
and up to half of it will be unused.
But it's constant times optimal since we're just
multiplying constants together.
And similarly, when you're FROM space
becomes relatively empty--
for example, if it's less than half full--
you can also release memory back to the OS,
and then the analysis of the amortized constant overhead
is similar.
OK, any other questions?
OK, so there's a lot more that's known and also unknown
about dynamic storage allocation,
so I've only scratched the surface of dynamic storage
allocation today.
There are many other topics.
For example, there's the buddy system for doing coalescing.
There are many variants of the mark-and-sweep procedure.
So there are optimizations to improve the performance of it.
There's generational garbage collection,
and this is based on the idea that many objects are
short-lived, so a lot of the objects
are going to be freed pretty close to the time
when you allocate it.
And for the ones that aren't going to be freed,
they tend to be pretty long-lived.
And the idea of generational garbage collection
is, instead of scanning your whole memory every time,
you just do work on the younger objects most of the time.
And then once in a while, you try
to collect the garbage from the older objects
because those tend to not change that often.
There's also real-time garbage collection.
So the methods I talked about today
assume that the program isn't running when the garbage
collection procedure is running, but in practice, you
might want to actually have your garbage collector running
in the background when your program is running,
but this can lead to correctness issues
because the static algorithms I just described
assume that the graph of the objects and pointers
isn't changing, and when the objects and pointers are
changing, you need to make sure that you still
get a correct answer.
Real-time garbage collection tends to be conservative,
so it doesn't always free everything that's garbage.
But for the things that it does decide to free,
those can be actually reclaimed.
And there are various techniques to make real-time garbage
collection efficient.
One possible way is, instead of just having one FROM
and TO space, you can have multiple FROM and TO spaces,
and then you just work on one of the spaces at a time
so that it doesn't actually take that long
to do garbage collection.
You can do it incrementally throughout your program.
There's also multithreaded storage allocation
and parallel garbage collection.
So this is when you have multiple threads running,
how do you allocate memory, and also how do you collect garbage
in the background.
So the algorithms become much trickier because there
are multiple threads running, and you
have to deal with races and correctness issues
and so forth.
And that's actually a topic of the next lecture.
So in summary, these are the things
that we talked about today.
So we have the most basic form of storage, which is a stack.
The limitation of a stack is that you can only free things
at the top of the stack.
You can't free arbitrary things in the stack,
but it's very efficient when it works because the code is
very simple.
And it can be inlined, and in fact this
is what the C calling procedure uses.
It places local variables in the return address
of the function on the stack.
The heap is the more general form of storage,
but it's much more complicated to manage.
And we talked about various ways to do allocation
and deallocation for the heap.
We have fixed-size allocation using
free lists, variable-size allocation using
binned free lists, and then many variants of these ideas
are used in practice.
For garbage collection, this is where
you want to free the programmer from having to free objects.
And garbage collection algorithms
are supported in languages like Java and Python.
We talked about various ways to do this reference counting,
which suffers from the limitation
that it can't free cycles.
Mark-and-sweep and stop-and-copy-- these
can free cycles.
The mark-and-sweep procedure doesn't
deal with external fragmentation,
but the stop-and-copy procedure does.
We also talked about internal and external fragmentation.
So external fragmentation is when your memory blocks are all
over the place in virtual memory.
This can cause performance issues like disk thrashing
and TLB misses.
Then there's internal fragmentation,
where you're actually not using all
of the space in the block that you allocate.
So for example, in the binned free list algorithm,
you do have a little bit of internal fragmentation
because you're always rounding up to the nearest power of 2
greater than the size you want, so you're wasting up
to a factor of 2 in space.
And in project 3, you're going to look
much more at these storage allocation schemes,
and then you'll also get to try some of these in homework 6.
So any other questions?
So that's all I have for today's lecture.