Darwin Evolves


By Jeffrey Carl


In a Nutshell: DarwinOS, the core of Apple’s just-released MacOS X, is open-source and available as a free download. While it inherits many characteristics from current BSD Unixes, it’s also radically different – revealing its true identity as the direct descendant of the ahead-of-its-time NeXTSTEP operating system. Darwin has some rough edges and missing documentation, but offers some very interesting possibilities. Unfortunately, the fact that it’s intended for Apple hardware will limit its appeal to many ISPs and sysadmins.


By the time you read this, Apple’s MacOS X should be on the street – the first-ever consumer OS with a Mach microkernel/BSD Unix core. That core is called Darwin, and it’s an open-source project – available under the BSD-ish Apple Public Source License – that can be downloaded for free (MacOS X, which includes the new Mac GUI and lots of other goodies, costs $129).


I’ve talked about Darwin here before, when it was in its earlier stages, but wasn’t able to go into many specifics about what Darwin was really like for a Unix admin. Now that the version that ships with MacOS X 1.0 is here, let’s take a look at this remarkable OS change for Apple.


The Politics Behind Darwin

Darwin is the triumph of NeXT over Apple. The GUI-only, outmoded classic MacOS of Apple’s last 17 years is gone, and a tough, sexy Unix core has replaced it (although MacOS X includes a compatibility layer to run existing MacOS apps). To understand why – and to understand why a Unix admin might be very interested in an Apple OS (other than the “research project” MkLinux) for the first time in many years – it helps to know the politics behind the situation.


In case you haven’t been paying attention to Apple’s corporate soap opera for the last few years (or, more likely, just don’t care), here’s the short version. Apple was at its lowest depths in late 1997, having wasted more than six fruitless years trying to create a modern operating system to replace MacOS (which was still building on foundations laid in 1984). Then-Apple CEO Gil Amelio instead looked to buy a company that already had an advanced OS, and settled on NeXT, the failing company run by Apple’s exiled cofounder Steve Jobs. The brilliant, mercurial, occasionally tantrum-prone Jobs was instrumental in building Apple from Steve Wozniak’s garage into an industry giant. But in 1985, he was fired by his own board of directors, essentially for being a major jerk to everyone within a 50-yard radius. It was a very nasty “divorce,” and Jobs left in a huff to found NeXT.


The NeXTSTEP (later called OPENSTEP) operating system which powered NeXT’s computers was years ahead of its time – but shipped on underpowered, grossly overpriced computers (sound familiar?) and was rejected by the marketplace. NeXTSTEP was based on Unix and the Mach microkernel, and included a radical new GUI and a pretty cool object-oriented development framework.


Microkernels (of which Mach is the most famous example) provided a much more elegant OS design than most OSes used (and still use), by moving everything but memory management and other lowest-level tasks out of the kernel. However, the overhead of this elegant and scalable design provided reduced performance in many “everyday” situations (like replacing a simple Excel spreadsheet with a full relational database), and microkernels were sidelined as academic curiosities. The NeXT object-oriented development kit was very advanced, but required knowledge of the relatively obscure Objective-C language and was largely ignored as well.


For Apple’s $400 million purchase of NeXT in 1997, they got not only the company’s OS but CEO Jobs as well. Then-Apple CEO Gil Amelio thought he was getting a valuable figurehead/consultant in Steve Jobs. But an ex-Apple employee who knew Jobs better than Amelio did predicted to Apple CTO Ellen Hancock that “Steve is going to f*** Gil so hard his eardrums will pop.” Sure enough, on July 6 1998, Gil Amelio was forced to resign and Steve Jobs once again was in charge of Apple. (Hancock resigned when Amelio did, and went on to become president of web hosting and datacenter giant Exodus.)


The rest is history. In the foreground, Jobs was introducing the fruit-colored, legacy-free  iMac to the world, sparking Apple’s sales resurgence. In the background, nearly all of Jobs’ loyal NeXT troops were assuming the top posts at Apple and changing the company’s technology and direction.


In 1999, the hype about Linux and open source was at its height, and Apple felt the pressure to join the crowd. Since BSD and Mach – which formed the core of NeXTSTEP – were already open source, it wasn’t hard for the normally ultra-proprietary Apple to take the step of officially open-sourcing the core of the forthcoming MacOS X. The NeXTSTEP core of MacOS X officially became “Darwin,” and a developer community of Apple engineers, Macolytes and Unix hackers began to form around the project. Along the way it saw contributions from Quake designer John Carmack, a basic port to Intel x86 hardware, and a “1.0” version that has evolved significantly as MacOS X neared release.


It’s noteworthy to keep in mind how much of a departure Darwin is from the old MacOS. It’s as if the “House that GUI Built” was taken over by the Unix geeks from Carnegie-Mellon and handed over to the hacker community for safekeeping. NeXT was the “ugly step-child” of the BSD family that nobody else noticed; and now, it will claim more users than the rest of the family combined – probably in less than six months.


Getting In-Depth with Darwin

After all that, it’s time to get under the hood of Darwin as a Unix admin would approach it. Many reports have claimed erroneously (at times, I have said this as well) that Darwin was basically “FreeBSD on Mac.” In fact, it’s a bit of updated Mach, a bit of traditional Free/Open/NetBSD, and a lot of NeXTSTEP flavoring.


Darwin uses the Mach microkernel, but attempts to solve some of its performance problems by moving much of the BSD functionality into the same “kernel space” (rather than in “userland” as a pure microkernel would). As such, it merges the two worlds in a way that is designed to keep the architectural elegance of a microkernel design while minimizing the performance overhead that microkernel process scheduling causes.


The first thing a BSD Unix admin will notice upon logging into Darwin is its familiarity. The default shell is tcsh, and nearly all of the customary /bin, /usr/bin/ and /usr/sbin utilities are there. In addition, you’ll find vi, pico, perl, sendmail, Apache, and other typical goodies. Ports to Darwin of popular Unix open-source Unix apps – from mysql to XFree 4.0.2 are proliferating rapidly. From a typical user’s perspective, it’s almost indistinguishable from *BSD.


It’s only once you become root and muck around in the system’s administration internals that you start to notice what makes the system a true child of NeXTSTEP. You’ll notice in /etc (actually a link to /private/etc; see below for more information) that /etc/resolv.conf doesn’t contain what you would expect. Nor does /etc/hosts, /etc/group or /etc/passwd. Why? It’s because Darwin wraps many of the functions contained in separate system and network configuration files in *BSD into a single service (inherited from NeXT) called NetInfo. Why is this useful rather than just annoying?


In *BSD and Linux, a number of different services derive their information from separate text configuration files – each with its own syntax and options. There’s no global preferences database – for example, any application that doesn’t automatically know how to read all the items in /etc/resolv.conf can’t find out what name servers your computer is using, or a program that can’t parse the syntax of /etc/passwd doesn’t know which users are on your system.


Somewhat like the Microsoft Windows Registry, Darwin’s NetInfo provides a database of network and user settings that can be read by any other NetInfo-aware application. NetInfo supersedes the information in the traditional /etc/* configuration files, as well as being favored by system services. NetInfo is consulted not only by MacOS X-native applications, but also by traditional BSD/Unix applications as well (making it much easier to port these apps to Darwin). The Apple engineers have accomplished this by hooking a check into each libc system data lookup function to consult NetInfo if it’s running (by default, it’s only “off” in single-user mode).


MacOS X’s GUI provides graphical tools for manipulating the NetInfo database; in Darwin, this can be done using the niutil and nicl commands (use man niutil and man nicl to see the syntax and options; it’s interesting to note that these man pages are dated from NeXT days). NetInfo can also “inherit” its settings from a “parent” NetInfo server, so you can create one server which has everyone’s account information on it, and all of its client machines will have their login info, network setup, et cetera (imagine a “family” of servers where users can interchangeably log in with the same accounts, settings, etc.).


Like NetInfo settings, application preferences are stored in a global XML database; they can be manipulated from the command-line defaults program. Typing man defaults from the command line will give you an idea of how its structure works.


One area that has been changed since the MacOS X Public Beta is that it is no longer necessary to reboot or log out/log back in to change network settings. Anyone who has used *BSD/Linux on mobile computers, or changed network profiles in Windows NT will appreciate this difference.


Darwin/MacOS X includes a /private directory at the root filesystem level, which includes the normal BSD /etc, /var, /tmp and the Darwin /cores and /Drivers). It appears that (in behavior inherited from NeXT) this directory is filled with info that is machine-specific, so that the rest of the filesystem can be booted from a parent network server. This clearly plants the roots for Darwin or MacOS X-based systems to serve as “thin clients.”


As for performance, recent builds of Darwin work admirably on mid-range Mac hardware. The only real complaint I have about Darwin (my gripes with the MacOS X user interface could fill a small book, however) is its woeful lack of documentation. While many BSDs suffer a similar problem, their user communities have had time to fill in the gaps; the changes that Darwin makes to the traditional BSD model are largely known only to the Darwin community and old-school NeXT gurus.


The closer I look at it, the more Darwin (and MacOS X) is appearing to take up where Novell left off in the race to compete with Windows NT/2000 in the corporate network space. It might be possible that Apple is looking to go head-to-head with Windows Whistler/XP while nobody is looking. And any victory in that space for *nix is something to be cheered.


Darwin’s Dilemma

Unfortunately, most of the work that has gone into Darwin thus far (understandably) has been in developing it for recent Apple hardware. While an Intel x86 port exists (but is still as of this writing in embryonic stage for hardware support), and older PowerPC Mac hardware support will undoubtedly extend over time, the current release of Darwin is not officially supported for Mac hardware older than G3 Power Macs. This sadly eliminates (at least for now) using Darwin to make a great server out of any old Mac hardware you have sitting on a shelf.


When you buy a new Mac, you’re paying for things (like full MacOS X, an ATI Radeon or nVidia GeForce video card, optical mouse, etc.) that you aren’t going to care about as a server admin. While new Mac systems are surprisingly powerful considering their CPU clock speed (I would sooner compare a G4 to an UltraSPARC III than a Pentium III), you still won’t get the same performance dollar-for-dollar as you would with commodity x86 hardware and a free OS.


As a result, buying a new Mac just to make into a Darwin server simply isn’t worth the money. If you love the GUI tools of MacOS X, it may be worthwhile (I’m personally salivating over the new Titanium PowerBook G4); otherwise, it still doesn’t make bottom-line dollars and sense to purchase a new Mac as a Darwin server.


As Darwin expands its processor base, this may change. In the meantime, it’s well worth your while to keep an eye on the Darwin project, and to get to know it, since some of its features are well worth adopting by the other BSDs and Linux.


Some of proceeding items stem from a series of articles that BSD guru Matt Loschert and I wrote for the BSD news site Daemon News (see www.daemonnews.org/200011/ and www.daemonnews.org/200012/ for excessively long and drawn-out versions). ;-) For great info on Darwin, you can skip its home page (www.opensource.apple.com) and go straight to Darwin Info (www.darwinfo.org). Also, check Xappeal (www.xappeal.org) and Apple’s Darwin lead Wilfredo Sanchez’s updates page (www.advogato.org/person/wsanchez). As always, let me know about any comments, corrections or suggestions you have, and I’ll publish them in a future column.