The Continuing Evolution of Darwin – Apple’s BSD Unix Has Lots of Promise, and Lots of Rough Edges

By Jeffrey Carl

Boardwatch Magazine
Boardwatch Magazine, September 2000

Boardwatch Magazine was the place to go for Internet Service Provider industry news, opinions and gossip for much of the 1990s. It was founded by the iconoclastic and opinionated Jack Rickard in the commercial Internet’s early days, and by the time I joined it had a niche following but an influential among ISPs, particularly for its annual ranking of Tier 1 ISPs and through the ISPcon tradeshow. Writing and speaking for Boardwatch was one of my fondest memories of the first dot-com age.

There’s a new BSD out there, and it’s unquestionably the odd member of the family. “Darwin” is the BSD Unix core of Apple’s next generation OS, MacOS X (see page 39 for the lowdown on OS X). While MacOS X (which includes Darwin) is a proprietary commercial product, Darwin by itself is open-source and available for free.

So why should you care (or think about deploying it)? First, it’s being developed for both PowerPC and Intel x86 architectures. Second, it’s being developed not only by its user community by also by Apple – which, like ‘em or hate ‘em, has considerable manpower and money to throw at the project. 

Lastly, Apple is scheduled to start shipping MacOS X on new Macintoshes in January 2001, as well as making it available for current machines – meaning that there will probably be a couple million boxes out there with Darwin at their heart within a year. While that’s no guarantee that all of your favorite software will rush to support it or optimize for it, it will certainly have the user numbers to make it worth noticing.

What is Darwin?

To get a feel for where Darwin is and where it’s headed, I asked Ernie Prabhakar, Apple’s Developer Platforms Product Line Manager.

Q: Can you give me a brief history of Darwin (including its roots in NextStep/ Rhapsody/Mac OS X Server)?

Ernie Prabhakar: Darwin is based on the Mach/BSD technology that traces its roots back through Mac OS X Server, Rhapsody, OpenStep for Mach, NextStep, and ultimately the original Mach 2.0 work at CMU.  In all the NeXT/Apple products, this UNIX core provided the support for advanced GUIs and development environments.  Over time, we’ve evolved from BSD 4.3 to BSD 4.4, and from Mach 2.0 to Mach 2.5 and ultimate Mach 3.0 (based largely on OSF MK 7.3). 

Q: Who is Darwin’s intended audience of consumers?

EP: Darwin is targeted at Macintosh developers who want to leverage the power of open source, as well as Open Source developers who want to enjoy the performance, consistency, and advanced GUI available on the Apple platform. Certain large customers (e.g., research universities) will take advantage of the open core to customize it to their environments, as well as helping students and researchers studying open source.

Q: What is the relationship between Darwin, its BSD kernel, and the Mach microkernel? 

EP: Darwin is the entire product, including a hybrid Mach/BSD kernel plus assorted utilities (mostly from FreeBSD) and developer tools (primarily from the GNU project). 

Q: How does using Mach affect its characteristics or performance?

EP: Very little. We use a microkernel architecture but a monolithic implementation, so Mach and BSD normally run in the same address space.

Q: Where does Darwin come from? Which parts are built around each of the different BSDs and/or NextStep?

EP: The current 1.0 version of Darwin actually shares very little code with the original NextStep implementation.  The Mach microkernel is actually based on the OSF code used in MkLinux.   The BSD kernel is based on BSD 4.4Lite, primarily the FreeBSD 3.2 distribution, with a healthy dose of NetBSD.

Q: What does Darwin add to the mix? Are there any innovations new to *BSD in Darwin?  

EP: For an existing BSD developer, Darwin has the advantages of:

• a powerful kernel extension mechanism and I/O Kit for writing drivers

• traditional Apple technology (HFS+, AppleTalk)

•  a well-funded commercial developer with a single well-supported reference release 

•  a very cool optional (though proprietary) GUI called Aqua (i.e., Mac OS X) 

Q: Is there any other OS which Darwin is trying to match or beat in features?

EP: Darwin’s top priority is making Mac OS X the world’s best personal computer operating system.  Beyond that, we want to help our developers turn Darwin itself into a fully-functional standalone open source operating system, compatible with FreeBSD and comparable to Linux. 

Q: Will Darwin include features like FreeBSD’s sysinstall or ports tree?

EP: Actually, we already ship Darwin with the Debian packaging mechanism.

Q: Is Darwin POSIX-compliant? How well should existing open-source apps (especially server apps like Apache, sendmail, qmail, PHP, etc.) compile on it or port to it?

EP: Most BSD and GNU stuff compiles out of the box. The major limitation used to be pthreads, but that was pretty much fixed in Darwin 1.0. We have a very active developer community that is ensuring third-party UNIX software is becoming “Darwin-ready” which usually means just defining the appropriate #defines and makefiles.

Q: Does Darwin bring any particular optimizations or improvements that would make it good from an ISP/Internet server perspective?

EP: Well, we have a dedicated team of networking engineers focusing on giving it best-of-breed performance integrated with Apple hardware. As well as a really cool GUI for administration (if you buy Mac OS X). Other than that, we use generally the same code as FreeBSD and NetBSD, which is already pretty thoroughly optimized.

Q: Are there any reasons you could think of that someone might choose to use Darwin instead of FreeBSD/NetBSD/etc.?

EP: Several: 

1.  We will be shipping Darwin (as Mac OS X) on millions of machines next year, giving developers an enormous potential user base.

2.  You have the option of purchasing a very cool UI (Mac OS X).

3.  We have excellent support for the latest high-performance PowerPC hardware 

Q: How much of a priority is the x86 port of Darwin?

EP: A lot of Darwin developers are excited about being able to use the exact same open source OS on both PowerPC and Intel hardware, so we’re doing what we can to help.  We demonstrated some major progress at WWDC [the Apple World Wide Developers’ Conference], and I think the community is gearing up to finish the work on their own.

Where is Darwin Now?

As of this writing, the current version of Darwin is 1.0.2. Darwin currently boots on Intel hardware, but there are a number of unresolved issues right now which keep it from being genuinely functional. Apple provides a binary release and installer for Darwin on the PowerPC architecture (located at http://www.publicsource.apple.com/projects/darwin/release.html), and it current supports G3/G4 Macintoshes as well as a few older desktop models. The PowerPC version sits entirely inside a single UFS or Apple HFS+ disk partition (current limitations of the Darwin booter limit the boot volume size to 8 GB).

Once you install Darwin, you can tell pretty quickly that most of Apple’s efforts are (understandably) going towards making Darwin work with MacOS X, rather than on its own. (In fact, most of the Darwin projects in its CVS repository are live and being used by the Apple MacOS X team at the same time as the Darwin community.) This is most obvious in the fact that the Darwin-specific documentation and install/setup goodies included with the OS are about zero – Apple is working on the MacOS X GUI to include these things.

When you install Darwin 1.0.2, you’ll boot and be dropped off into a tcsh shell. From there, you’re on your own – don’t expect any sysinstall or LinuxConf-style goodies to help with setup and configuration. For most experienced administrators, this wouldn’t pose too much of a problem, except that Darwin has a lot of its files stashed in MacOS X-centric locations that take a while to puzzle out. For example, don’t bother looking in /etc/httpd or /usr/local for a httpd.conf file – you’ll find it as /Local/Library/WebServer/Configuration/apache.conf.

At this point, Darwin is certainly more of a toy for developers and “early adopters” than an OS worth implementing for an Internet server. OpenSSH hasn’t been finished up yet, nntpd and some other server staples seem to be MIA, there are plenty of device drivers missing, and there’s plenty of work yet to be done on the Intel side. Documentation is still pretty sparse, but some good stuff is developing out there, including a helpful “Unofficial Darwin FAQ” at http://www.synack.net/darwin/faq.html. You can keep up on Darwin’s progress by subscribing to any of the Darwin mailing lists at Apple (http://www.publicsource.apple.com/projects/mail.html) or, if you’re lazy like me, checking out project leader Wilfredo Sanchez’s updates to his Advogato page (http://www.advogato.org/person/wsanchez/).

Darwin is very new, and time will take care of many of its current deficiencies (many of the ones I’ve described will almost certainly be fixed by the time you read this). Despite its shortcomings, Darwin 1.0.2 is a fully functional BSD Unix system, and the default install includes several of the third-party packages you’d hope to find like Apache, Sendmail, Perl and BIND. Ports for XFree86 (much of the original work on this port was done by id Software’s John Carmack), OpenSSL and other common tools are available in source form from the Darwin CVS repository. 

Where is Darwin Going?

Darwin’s promise lies not in where it is now, but where it will be. Darwin can catch up on plenty of things by grabbing code from the other BSDs, and it is likely only a matter of time before Darwin develops rough feature parity with some of your favorite BSDs. The Apple Darwin team has said that future releases of Darwin will try to track FreeBSD; Jordan Hubbard was one of the speakers at the BSD session at May’s Apple World Wide Developer Conference. The IOKit driver framework that Apple has created for MacOS X has a lot of potential for (relatively) easy device driver development. 

The real promise lies in the possibility that the large installed user base of MacOS X systems that will be out there will drive developers (especially commercial ones) to bring their server apps natively to a BSD-based system for the first time. Of course, porting something to MacOS X isn’t the same as porting to BSD; most commercial applications coming to MacOS X will use its API rather than the Darwin/BSD API. Nonetheless, even if some of these apps which currently exist for [insert your least-favorite commercial Unix here] are ported to the Darwin layer of MacOS X, it will be a victory for BSD. 

The close relationship between Apple’s Darwin team, the Darwin community, and the FreeBSD community is an excellent sign for Darwin’s future. While it may not be much more than a curiosity right now, Darwin is a welcome addition to the free *BSD family, and will definitely be something to watch in the next year.