mikeash.com: just this guy, you know?

Posted at 2014-11-07 16:04 | RSS feed (Full text feed) | Blog Index
Next article: Friday Q&A 2015-01-23: Let's Build Swift Notifications
Previous article: A Brief Pause
Tags: braaaiiiinnnssss fridayqna memory zombie
Friday Q&A 2014-11-07: Let's Build NSZombie
by Mike Ash  

Zombies are a valuable tool for debugging memory management problems. I previously discussed the implementation of zombies, and today I'm going to go one step further and build them from scratch, a topic suggested by Шпирко Алексей.

Review
Zombies detect memory management errors. Specifically, they detect the scenario where an Objective-C object is deallocated and then a message is sent using a pointer to where that object used to be. This is a specific case of the general "use after free" error.

In normal operation, this results in a message being sent to memory that may have been overwritten or returned to the kernel. This results in a crash if the memory has been returned to the kernel, and can result in a crash if the memory has been overwritten. In the case where the memory was overwritten with a new Objective-C object, then the message is sent to that new object which is probably completely unrelated to the original one, which can cause exceptions thrown due to unrecognized selectors or can even cause bizarre misbehaviors if the message is one the object actually responds to.

It's also possible that the memory hasn't been touched and still contains the original object, in a post-dealloc state. This can lead to other interesting and bizarre failures. For example, if the object contains a UNIX file handle, it may call close on a file descriptor twice, which can end up closing a file descriptor owned by some other part of the program, causing a failure far away from the bug.

ARC has greatly reduced the frequency of these errors, but it hasn't eliminated them altogether. These problems can still occur due to problems with multithreading, interactions with non-ARC code, mismatched method declarations, or type system abuse that strips or changes ARC storage modifiers.

Zombies hook into object deallocation. Instead of freeing the underlying memory as the last step in object deallocation, zombies change the object to a new zombie class which intercepts all messages sent to it. Any message sent to a zombie object results in a diagnostic error message instead of the bizarre behavior you get in normal operation. There is also a mode where it rewrites the class and then frees the memory anyway, but this is typically much less useful since the memory will typically get reused quickly, and I'll ignore that option here.

To write our own zombies implementation, we need to hook object deallocation and build the appropriate zombie classes. Let's get started!

Catching All Messages
If we make a root class without any methods, then any message sent to an instance of that class will go into the runtime's forwarding machinery. This would seem to make forwardInvocation: a natural point to catch messages. However, that one happens a bit too late. Before forwardInvocation: can run, the runtime needs a method signature to construct an NSInvocation object, and that means that methodSignatureForSelector: runs first. This, then, is the override point for catching messages sent to a zombie object.

Dynamically Allocated Classes
In addition to the selector that was sent, zombies also remember the original class of the object. However, there may not be any room in the object's memory to store a reference to that original class. If the original class had no additional instance variables, then there's no space that can be repurposed for storage. The original class must therefore be stored in the zombie class rather than in the zombie object, and that means the zombie class needs to be dynamically allocated. Each class which has an instance that becomes a zombie will get its own zombie class.

The next question is where to store the reference to the original class. It's possible to allocate a class with some extra storage for things like this, but it's somewhat inconvenient to use. An easier way is to simply use the class name. Since Objective-C classes all live in one big namespace, the class name is sufficient to uniquely identify it within a process. By sticking a prefix on the original class name to generate the zombie class name, we end up with something that's both descriptive on its own and can be used to recover the original class name. We'll use MAZombie_ as the prefix.

Method Implementations
Note that all of the code here is built without ARC, since ARC memory management calls really get in the way here.

Let's start off with a simple method implementation, which is an empty one:

    void EmptyIMP(id obj, SEL _cmd) {}

It turns out that the Objective-C runtime assumes that every class implements +initialize. This is sent to a class before the first message sent to the class to allow it to do any setup it needs. If it's not implemented, the runtime sends it anyway and hits the forwarding machinery instead, which isn't helpful here. Adding an empty implementation of +initialize avoids that problem. EmptyIMP will be used as the implementation of +initialize on zombie classes.

The implementation of -methodSignatureForSelector: is a bit more interesting:

    NSMethodSignature *ZombieMethodSignatureForSelector(id obj, SEL _cmd, SEL selector) {

It retrieves the class of the object and that class's name. This is the name of the zombie class:

        Class class = object_getClass(obj);
        NSString *className = NSStringFromClass(class);

The original class name can be retrieved by stripping off the prefix:

        className = [className substringFromIndex: [@"MAZombie_" length]];

Then it logs the error and calls abort() to make sure you're paying attention:

        NSLog(@"Selector %@ sent to deallocated instance %p of class %@", NSStringFromSelector(selector), obj, className);
        abort();
    }

Creating the Classes
The ZombifyClass function takes a normal class and returns a zombie class, creating it if necessary:

    Class ZombifyClass(Class class) {

The zombie class name is useful both for checking to see if a zombie class exists and for creating it if it doesn't:

        NSString *className = NSStringFromClass(class);
        NSString *zombieClassName = [@"MAZombie_" stringByAppendingString: className];

The existence of the zombie class can be checked using NSClassFromString. This also provides the zombie class so it can be returned immediately if it exists:

        Class zombieClass = NSClassFromString(zombieClassName);
        if(zombieClass) return zombieClass;

Note that there's a race condition here: if two instances of the same class are zombified from two threads simultaneously, they'll both try to create the zombie class. In real code, you'd need to wrap this whole chunk of code in a lock to ensure that doesn't happen.

A call to the objc_allocateClassPair function allocates the zombie class:

        zombieClass = objc_allocateClassPair(nil, [zombieClassName UTF8String], 0);

We add the implementation of -methodSignatureForSelector: using the class_addMethod function. The signature of "@@::" means that it returns an object, and takes three parameters: an object (self), a selector (_cmd), and another selector (the explicit selector parameter):

        class_addMethod(zombieClass, @selector(methodSignatureForSelector:), (IMP)ZombieMethodSignatureForSelector, "@@::");

The empty method is also added as the implementation of +initialize. There's no separate function for adding class methods. Instead, we add a method to the class's class, which is the metaclass:

        class_addMethod(object_getClass(zombieClass), @selector(initialize), (IMP)EmptyIMP, "v@:");

Now that the class is set up, it can be registered with the runtime and returned:

        objc_registerClassPair(zombieClass);

        return zombieClass;
    }

Zombifying Objects
In order to turn objects into zombies, we'll replace the implementation of NSObject's dealloc method. Subclasses' dealloc methods will still run, but once they go up the chain to NSObject, the zombie code will run. This will prevent the object from being destroyed, and provides a place to set the object's class to the zombie class. This operation gets wrapped up into a function to enable zombies:

    void EnableZombies(void) {
        Method m = class_getInstanceMethod([NSObject class], @selector(dealloc));
        method_setImplementation(m, (IMP)ZombieDealloc);
    }

We can then put a call to EnableZombies at the top of main() or similar, and the rest takes care of itself. The implementation of ZombieDealloc is straightforward. It calls ZombifyClass to obtain the zombie class for the object being deallocated, then uses object_setClass to change the class of the object to the zombie class:

    void ZombieDealloc(id obj, SEL _cmd) {
        Class c = ZombifyClass(object_getClass(obj));
        object_setClass(obj, c);
    }

Testing
Let's make sure it works:

    obj = [[NSIndexSet alloc] init];
    [obj release];
    [obj count];

I chose NSIndexSet semi-arbitrarily, as a convenient class that doesn't hit CoreFoundation bridging weirdness. Running this code with zombies enabled produces:

    a.out[5796:527741] Selector count sent to deallocated instance 0x100111240 of class NSIndexSet

Success!

Conclusion
Zombies are fairly simple to implement in the end. By dynamically allocating classes, we can easily keep track of the original class without needing to rely on storage within the zombie object. methodSignatureForSelector: provides a convenient choke point for intercepting messages sent to the zombie object. A quick hook on -[NSObject dealloc] lets us turn objects into zombies instead of destroying them when their retain count goes to zero.

That's it for today. Come back next time for more frightening tales. Until then, keep sending in your suggestions for topics.

Did you enjoy this article? I'm selling a whole book full of them. It's available for iBooks and Kindle, plus a direct download in PDF and ePub format. It's also available in paper for the old-fashioned. Click here for more information.

Comments:

Rick at 2014-11-07 17:31:27:
Cool. Sounds like there's a couple of weak points, though: 1. The CF bridging weirdness you mentioned. 2. Zombify requires a class' dealloc implementation to call super. Could one swizzle dealloc instead? You'd need a place to catch the loading/registering of every class to swizzle its dealloc, though.

Also, this article is a week late. October is for zombies... Now we need articles on NSTurkey. Or NSPumpkinSpiceLatte, or something... :)

Avi Drissman at 2014-11-07 17:40:53:
If you want to see a different implementation of zombies, including a treadmill, one that has been hardened by shipping it for years, check out the Chromium version:

https://chromium.googlesource.com/chromium/src/+/master/chrome/common/mac/objc_zombie.mm

Disclosure: I'm on the Chrome team, though I never worked on the zombie implementation.

Konstantin Anoshkin at 2014-11-07 20:52:42:
"If the original class had no additional instance variables, then there's no space that can be repurposed for storage."

Technically speaking, the granularity of malloc() on Darwin is 16 bytes, which is a minimum space occupied by an Obj-C object (barring evil custom +alloc implementations, of course). The isa variable on a 64-bit system occupies 8 bytes, which still leaves 8 bytes for a pointer to the original class. In other words, that's a great opportunity for optimization for 99.9% cases on all currently supported Apple systems. Just my 2 cents.

mikeash at 2014-11-08 02:29:53:
Rick: The CF stuff is a big problem, and one that NSZombie actually shared for many years. The requirement to call super is related, and I don't think NSZombie will catch that either. Once you start doing custom allocators it's hard to hook stuff reliably. You could swizzle dealloc, but how will you stop the memory from being deallocated? You still need to run the rest of the dealloc code for things to work properly. And you're right, I originally meant to get this article out last week but didn't quite make it!

Konstantin Anoshkin: You're right, we could safely assume there's enough storage for another pointer in the real world. But I didn't like to make that assumption, partly just because it's messy, and partly because there's nothing that says Apple couldn't decide to use a more fine-grained allocator for objects that don't have any extra ivars. In a real implementation, assuming you're only running it for debugging and you check with malloc_size before you start overwriting stuff, it would be fine.

Brent Royal-Gordon at 2014-11-09 21:10:23:
Rather than dynamically installing +initialize and -methodSignatureForSelector:, couldn't you just write an MAZombie base class and have all of your individual zombie classes subclass it? It seems like that would reduce the amount of runtime magic you need to write for each zombie class.

Also, I believe this implementation of zombies won't call -.cxx_destruct, which is used by ARC. Would it be a good idea to call that in ZombieDealloc?

Connor Sadler at 2014-11-11 11:29:03:
I enjoyed the tags ;]

bob at 2014-12-04 09:06:23:
I wonder if it would be useful to create a zombie that doesn't prevent deallocation -- it would still catch incorrect message sends for a while, up until the memory gets overwritten. And since it doesn't leak, it can be used in production. What do you think?


Comments RSS feed for this page

Add your thoughts, post a comment:

Spam and off-topic posts will be deleted without notice. Culprits may be publicly humiliated at my sole discretion.

Name:
Web site:
Comment:
Formatting: <i> <b> <blockquote> <code>. URLs are automatically hyperlinked.
Code syntax highlighting thanks to Pygments.
Hosted at DigitalOcean.