mikeash.com: just this guy, you know?

Posted at 2009-01-16 22:35 | RSS feed (Full text feed) | Blog Index
Next article: Friday Q&A 2009-01-23
Previous article: Friday Q&A 2009-01-09
Tags: fridayqna ipc
Friday Q&A 2009-01-16
by Mike Ash  

Happy Friday to everyone, and welcome back to another Friday Q&A. This week I'll be taking Eren Halici's suggestion to discuss the various ways to do interprocess communication on OS X.

IPC is an interesting and sometimes complicated topic, especially on OS X, which has a veritable zoo of IPC techniques. It can be hard to decide which one to use, and sometimes hard to even know what's available.

OS X is a funny mixture of mach and UNIX so you end up with IPC mechanisms from both:

Those are all what I would call system-level functionality, things which are either provided directly by the kernel/libSystem, or which are thin wrappers around them. OS X also provides a bunch of higher-level IPC mechanisms at the framework level:

So which one is right for you? Well, it all depends on what you're doing. I've used nearly every one of these to accomplish different things over the years. You'll have to see which one fits your problem best, and I hope the above gives you a good place to get started.

That wraps things up for this week's Friday Q&A. Come back next week for another exciting installment. Did I miss your favorite IPC mechanism? Don't understand the point of one of them? Did I miss the whole point of one of them? Post your comments below.

Remember, Friday Q&A is driven by your suggestions. I'm much easier than Public Radio: I don't ask for your money, just your ideas, but your contributions make this possible. Post your ideas in the comments, or e-mail them. (Reminder: I will use your name unless you tell me not to.)

Did you enjoy this article? I'm selling a whole book full of them. It's available for iBooks and Kindle, plus a direct download in PDF and ePub format. It's also available in paper for the old-fashioned. Click here for more information.

Comments:

Erik at 2009-01-17 09:55:51:
I'm curious about the unreliable behavior you described regarding DO, and why it's worthy of investigation in spite of this. Perhaps more detail on this could be a topic for a future Friday.

arwyn at 2009-01-17 13:50:16:
I think your higher-level IPC section should have included a blurb about MIG. Most of the system uses it as a high level* way to automate the mach minutia, far more than any of the other methods you listed.

* high level being a somewhat relative term

Jonathan at 2009-01-17 19:33:24:
An interesting review of the possibilities. In just over a year of Cocoa coding I have found myself using all of these methods of IPC with the exception of FIFO's and shared memory. There is no getting away from IPC in modern coding.

My first port of call was DO, which looks great on paper put doesn't seem to stack up too well in practice - I would hesitate to use it in anger for large scale network communication. It doesn't win any prizes in the cross platform parade either, but then neither does the .NET remoting technology.

AppleScript et al are undeniably nasty, whether you look at them at the individual event level, at the OSA component level or at the actual script level. It is however pervasive and powerful.

Christopher Lloyd at 2009-01-17 22:52:08:
MIG? might as well cover RPC's with rpcgen too :)

has at 2009-01-17 23:26:18:
"Generally it's best to avoid AppleScript and simply send the corresponding raw Apple Events instead, either directly or through a mechanism like Scripting Bridge."

While I completely understand the sentiment, as a long-time application scripter I feel this is still a little unfair. It's true the language itself has many significant faults - unpredictable and ambiguous syntax being the biggest of them (even William Cook, one of the original designers, now admits this was a mistake). However, even after fifteen years the AppleScript language still provides the most elegant and reliable API for creating and sending Apple events to other applications.

(It also provides the only OSA-based language component worth a damn, although IMO that's as much the fault of the OSA API as other language developers. But I digress.)

While the format string-based AEBuild* functions are fine for performing fairly simple tasks from C/C++/ObjC, they become very verbose for more complex tasks. And while Scripting Bridge may provide a higher-level API, it heavily obfuscates the actual Apple events mechanism (far more than AppleScript does), reducing functionality, confusing and misleading users as to what's going on, and causing various incompatibilities with many scriptable applications. Even appscript, which must've been through a few thousand hours of design and testing by now, still occasionally chokes on some application commands that would work flawlessly in AppleScript.

The other thing to remember is that an awful lot of what AppleScript gets blamed for isn't actually its fault at all, but the fault of individual application developers who don't adequately design, test and/or document their Apple event APIs. In turn, some of that blame can be placed on Apple in general for not providing application developers with sufficient tools and guidance in the first place.

has at 2009-01-17 23:43:00:
BTW, while I've never used DO myself, I'm disappointed to hear that it has problems too. IMO Apple should develop a new, conventional object-oriented IPC system - something simple, efficient and reliable. Perhaps something along the lines of the traditional browser DOM, which is what most developers and users seem to expect and prefer anyway. Apple event IPC - which they've never gotten to work completely right and most likely never will - could then be quietly moved to legacy status and eventually retired completely, thereby curing all its ills. Then again, if they can't get DO right either... :/

dan at 2009-01-18 00:20:32:
Another nice thing about Mach ports is that you can send a memory buffer as part of a message (OOL data). The buffer is remapped into the target process with copy-on-write semantics, so basically you can send over an arbitrarily large chunk of data between processes with virtually no overhead. The Mach APIs will even deal with deallocating the memory for you if you tell it. Amit Singh's book has details.

I'd also be curious to hear what problems you've had with DO.

Suits at 2009-01-18 00:32:24:
Wow, I had to write some code on windows and it's great writing a CreateRemoteThread. After writing that code i moved over to Mac. I never looked into the mach API's but now i will give them a lookover

Pau Garcia i Quiles at 2009-01-18 03:03:36:
No mention of DBUS?

http://www.freedesktop.org/wiki/Software/dbus
http://dbus.darwinports.com/

It's not included with Mac OS X but it's available and it's used extensively by the Mac OS X port of KDE4 ( http://mac.kde.org/ )

mikeash at 2009-01-18 05:16:42:
Erik: Thanks for the suggestion. To go over it quickly here, DO tends to be unreliable under really heavy use, with complicated objects, or when used remotely. It also tends to deal unreliably with timeouts. It's still worth investigation because if your needs are simple, those problems mostly go away.

arwyn: Yep, MIG (stands for Mach Interface Generator for those who don't know and want to find out more) is definitely worth checking out if you want to look more into the mach side of things.

has: I strongly disagree and think it's completely fair. AppleScript is so bad that I went as far as writing my own Apple Event builder library (AEVTBuilder) to avoid using it in one application. I don't know how you're getting anything to work flawlessly in AppleScript but whatever secret you have, I'd like to know what it is. My experience with AppleScript is that it's a frustrating exercise in trial and error for even the most trivial of tasks. It's useful for banging out quickie scripts to get some apps to do what you want as a user. As a developer, my opinion is that AppleScript the language is 100% useless and should simply be avoided. AppleScript support in your app can still be worthwhile but it hurts pretty hard.

Pau Garcia i Quiles: The reason I didn't mention DBUS is perfectly answered by the beginning of your second question: "It's not included with Mac OS X". To be frank, IPC mechanisms which don't ship with the OS and which require installing daemons do not interest me.

mikeash at 2009-01-18 07:52:37:
I see the asshole crowd is out now. Thanks very much, 74.79.232.49 for that delightful security test, but I'm afraid I'm going to have to delete both the attempted script injection and the very long string of nonsense now.

has at 2009-01-18 09:28:16:
mikeash: "I strongly disagree and think it's completely fair. AppleScript is so bad that I went as far as writing my own Apple Event builder library (AEVTBuilder) to avoid using it in one application."

And I wrote appscript so that I could control AppleScriptable applications from Python (and later Ruby and ObjC). More precisely, I spent the first year trying to make appscript work better than AppleScript... and then I then spent the next four years just trying to make the damn thing work as well as AppleScript.

Along the way, I learnt a valuable lesson: you cannot out-AppleScript AppleScript, and thinking you can just gets your butt kicked. It's pretty much impossible to create a high-level Apple event bridge that works better than AppleScript does - even if you get the basics right (which early versions of appscript didn't, and most others never have), you still have a ton of problems to deal with: almost every application available today is developed and [hopefully] tested against AppleScript, with the result that many of AppleScript's own unique quirks are relied on by those applications. For a high-level bridge to work really well, it not only needs to speak Apple events fluently, it also needs to replicate AppleScript's own particular way of speaking them.


"I don't know how you're getting anything to work flawlessly in AppleScript but whatever secret you have, I'd like to know what it is."

No secret, alas (otherwise I'd patent it and be rich). Just lots of hard work. Although it's still less work than is needed to script applications equally well from Python/Ruby/ObjC, since at least with AppleScript you have a very well established community and lots of existing example scripts and documentation to help you. (And believe me, it took me several years and much poking from the likes of Matt Neuburg to suck it up and admit this.)


"My experience with AppleScript is that it's a frustrating exercise in trial and error for even the most trivial of tasks."

Oh, I quite agree. AppleScript syntax causes real problems: in order to provide high-level readability (i.e. anyone can look at existing AppleScript code and get a fair idea of what it does) it sacrifices low-level semantics (i.e. it's a real bear to understand how it does it). e.g. I recently discussed this at http://macscripter.net/viewtopic.php?pid=108997

However, the majority of problems in figuring out how to get any particular scriptable application to do what you want it to do are due to the schlonky implementations and crappy documentation provided by individual application developers. And these deficiencies affect all application scripters equally; so regardless of what language you use, you'll find it "a frustrating exercise in trial and error".

...

Like I say, I think Apple should replace the whole lot with a conventional, dumb-as-rocks object-oriented IPC system, as that's what most users - and most Apple engineers - seem to want anyway, but realistically I don't expect them to do so any time soon as that would be an outright admission that they screwed up the first time around.

In the meantime, sure, blame AppleScript for mistakes it is responsible for (e.g. syntax soup). However, blaming it for everybody else's screw-ups as well - crappy application scripting interfaces and documentation; faulty Apple event bridges, etc. - only serves to let the real culprits off the hook. And that ultimately doesn't serve anyone's interests.

Jamie Hope at 2009-01-19 13:30:23:
"DO is really cool technology and it's the sort of thing that tends to blow people's minds when they come to Objective-C from lesser languages such as Java or C++."

I've never used DO, but your description of it sounds very much like Java's Remote Method Invocation or Jini. Except that Jini *does* allow you to specify the IPC mechanism to allow for encryption or whatever.

Damien Sorresso at 2009-01-19 17:44:36:
It's worth noting that anything dealing with Mach under the covers will invariably have to deal with Mach bootstraps. A naive use of CFMessagePort is okay in most cases, but on Leopard, each user has his own Mach bootstrap, and each console session (be it SSH or the GUI console) has its own Mach bootstrap underneath there. Therefore, you cannot expect, having registered a DO server in the GUI (aka Aqua) bootstrap, that a client in the SSH (aka StandardIO) bootstrap will be able to look up the name.

I'd highly recommend checking out the Daemonomicon for further details.

http://developer.apple.com/technotes/tn2005/tn2083.html

Also, if you're going to use Mach, don't bother using the mach_* APIs. Learn to love MiG. Even XNU exports its interfaces via MiG, and there are very few clients of the mach_* APIs. MiG has evolved over years and years to encompass various corner cases in its generated code that would be very difficult for just about anyone to deal with their first time through. And when using MiG, keep in mind that, to do really interesting stuff, you're going to have to delve into the implementation at some point. MiG doesn't do a very good job of maintaining a wall of separation between interface and implementation, so don't feel too dirty about it.

More on MiG...

ftp://ftp.cs.cmu.edu/project/mach/doc/unpublished/mig.ps

Also, for simple, synchronous communication over a socket, you can also check out BetterAuthorizationSample.

http://developer.apple.com/samplecode/BetterAuthorizationSample/index.html

It has a nice library for sending commands to a server over the wire as serialized CFDictionary's, making it very easy to create a rich, synchronous protocol for client/server communication. For asynchronous communication, you'd have to look elsewhere though. I had written a library to do what BetterAuthorizationSample's did using asynchronous means by way of CFRunLoop's, but I can't release it for various reasons.

It's also worth pointing out the most basic difference between Mach and Unix domain sockets. Mach is a message-based protocol, while sockets are streams. So when you get a Mach message, it's guaranteed that it's the whole message. With sockets, you have to work out things like dealing with incomplete messages, whether the kernel buffer has been filled, etc. Mach takes care of all that for you. The downside is the the loss of portability, lack of documentation and misunderstandings that come from poor terminology choices on the parts of the original Mach developers. But for local interprocess communication, Mach is (in my opinion) the better option once you have mastered it.

The deal-breaker for Mach is if you need to pass file descriptors between processes. Mac OS X does not currently support passing file descriptors via Mach messages. Nor can it pass Mach port rights over sockets. Those two worlds are separate in this respect.

I'd also like to note that IPC is particularly important on Mac OS X, especially on the daemon side of things, because it's the mechanism for expressing dependencies when writing a launchd daemon or agent.

By the way, Mike, your comment system doesn't handle umlauts very well. :)

mikeash at 2009-01-20 00:48:01:
Thanks for the comment. I'm aware that the comment system falls down with non-ASCII characters, but the python/MySQL combination has defeated my every attempt to fix the problem. Maybe I'll try again soon, in the meantime, sorry, only ASCII accepted here.

bil at 2016-03-03 21:07:36:
Comments here sound like xpc is the way to go for IPC


Comments RSS feed for this page

Add your thoughts, post a comment:

Spam and off-topic posts will be deleted without notice. Culprits may be publicly humiliated at my sole discretion.

Name:
Web site:
Comment:
Formatting: <i> <b> <blockquote> <code>. URLs are automatically hyperlinked.
Code syntax highlighting thanks to Pygments.
Hosted at DigitalOcean.