By Jeffrey Carl
Boardwatch Magazine was the place to go for Internet Service Provider industry news, opinions and gossip for much of the 1990s. It was founded by the iconoclastic and opinionated Jack Rickard in the commercial Internet’s early days, and by the time I joined it had a niche following but an influential among ISPs, particularly for its annual ranking of Tier 1 ISPs and through the ISPcon tradeshow. Writing and speaking for Boardwatch was one of my fondest memories of the first dot-com age.
In a Nutshell: Linux’s day as a scalable server is here. The long wait for Linux kernel 2.4 made its release seem somewhat anti-climactic, so many of its new features have gone largely unnoticed. Although many of the changes were available as special add-ons to 2.2.x kernels before, the 2.4 kernel wraps them all together in a neat package – as well as integrating a number of great new features, notably in the networking, firewall and server areas.
Bringing Everybody Up to Speed
If you’re at all familiar with the Linux kernel and its upgrade cycle, you can skip the next several paragraphs and go on to make fun of the technical inaccuracies and indefensible opinions in the rest of the column. Everybody else should read these introductory paragraphs, and only then should they go on and make fun of the rest of the column.
The Linux kernel is the foundation of the Linux operating system, since it handles all of the low-level “dirty work” like handling processes and memory, I/O to drives and peripherals, networking protocols and other goodies. The capabilities and performance of the kernel in many ways circumscribe the capabilities and performance of all the programs that run on Linux, so its features and stability are critical.
In Linux, odd-numbered “minor” version numbers (like x.3) are unstable, developmental versions where adding cool new features is more important than whether or not they make the system crash. They are tested in the developmental kernels and once the bugs are worked out, they are “released” as the next highest even-numbered minor version (like x.4) kernel, which is considered stable enough for normal users to run. Linux kernel 2.2.x was the “stable” kernel (while kernel 2.3.x was the developmental version) from January 1999 to January 2001, when the new stable kernel became the very long-awaited 2.4.x (as of this writing, the most recent version was 2.4.2).
What’s New, Pussycat?
The changes since Linux kernel version 2.2 largely reflect the expansion of Linux as it comes to be used in an ever-wider variety of hardware and for different user needs. It wraps in features required to run on tiny embedded devices and multi-CPU servers as well as traditional workstations. Improving Linux’s multiprocessing capabilities also requires cleaning up a lot of other kernel parts so they can make use of (and also not to get in the way of) using multiple processors. To expand Linux’s acceptance in the consumer marketplace, it includes drivers for a large number of new devices. And to hasten Linux’s acceptance in the server market (especially on the high-end), it has enhanced its networking performance – notably in areas where earlier benchmarks had shown it losing to Microsoft’s Windows NT/2000.
With high-end server vendors (most notably IBM) embracing Linux, they have pushed for the kernel to include the features that would make Linux run reliably on high-end hardware. In addition to all the CPU types supported by Linux 2.2, the Intel ia64 (Itanium) architecture is now supported, as are the IBM S/390 and Hitachi SuperH (Windows CE hardware) architectures. There are optimizations for not only the latest Intel x86 processors, but also their AMD and Cyrix brethren as well, plus Memory Type Range Registers (MTRR/MCRs) for these processor types. Support for Transmeta Crusoe processors is built-in (as you would expect from Linus Torvalds being an employee of Transmeta). Whereas kernel 2.2 scaled well up to four processors, 2.4 supports up to 16 CPUs.
As part of the “scalability” push, a number of previous limitations have been removed in kernel 2.4. The former 2 GB size limit for individual files has been erased. Intel x86-based hardware can now support up to 4 GB of RAM. One system can now accept up to 16 Ethernet cards, as well as up to 10 IDE controllers. The previous system limit of 1024 threads has been removed, and the new thread limit is set at run time based on the system’s amount of RAM. The maximum number of users has been increased to 2^32 (about 4.2 billion). The scheduler has been improved to be more efficient on systems with many processes, and the kernel’s resource management code has been rewritten to make it more scalable as well.
Improved Networking
Kernel 2.4’s networking layer has been overhauled, with much of the effort going into making improvements necessary for dealing efficiently with multiprocessing. Improved routing capabilities have been added into Linux by splitting the network subsystem into improved packet filtering and Network Address Translation (NAT) layers; modules are included to make backward compatibility with kernel 2.0 ipfwadm and 2.2 ipchains-based applications available. Firewall and Internet protocol functions have also been added to the kernel.
Linux’s improved routing capabilities make use of a package called iproute2. They include the ability to throttle bandwidth for or from certain computers, to multiplex several servers as one for load-balancing purposes, or even to do routing based on user ID, MAC address, IP address, port, type of service or even time of day.
The new kernel’s firewall system (Netfilter) provides Linux’s first built-in “stateful” (remembering the state of previous packets received from a particular IP address) firewalling system. Stateful firewalls are also easier to administer with rules, since they automatically exclude many more “suspect” network transactions. Netfilter also provides improved logging via the kernel log system, automatically including things like SMB requests coming from outside your network, the ability to set different warning levels for different activities, and the ability to send certain warning-level items to a different source (like sending certain-level logging activities directly to a printer so the records are physically untouchable by a cracker that could erase the logfiles).
The system is largely backward-compatible, but it now allows Netfilter to detect many “stealth” scans (say goodbye to hacker tool nmap?) that Linux firewalls previously couldn’t detect, and blocks more DoS attacks (like SYN floods) by intelligently rate-limiting user-defined packet types.
Under kernel 2.2 (using a model that is standard across most Unix variants), all Unix network sockets waiting for an event were “awakened” when any activity was detected – even though the request was addressed to only one of those sockets. The new “wake one” architecture awakens only one socket, reducing processor overhead and improving Linux’s server performance.
A number of new protocols have been added as well, such as ATM and PPP-over-Ethernet support. DECnet support has been added for interfacing with high-end Digital (now Compaq) systems and ARCNet protocols. Support for the Server Message Block (SMB) protocol is now built-in rather than optional. SMB allows Linux clients to file share with Windows PCs, although the popular Samba package is still required for the Linux box to act as an SMB server.
Linux 2.4 has a web server called khttpd that integrates web serving directly into the kernel (like Microsoft’s IIS on WinNT or Solaris’s NCA). While not intended as a real replacement for Apache, khttpd’s ability to serve static-only content (it passes CGI or other dynamic content to another web server application) from within the kernel memory space provides very fast response times.
Get On the Bus
Linux’s existing bus drivers have been improved as part of the new resource subsystem, plus significant improvements and new drivers (including Ultra-160!) for the SCSI bus support. Logical Volume Manager (LVM), a standard in high-end systems like HP/UX and Digital/Tru64 UNIX that allows volumes to span multiple disks or be dynamically resized, is now part of the kernel. Support is also there for ISA Plug-and-Play, Intelligent Input/Output (I2O, a superset of PCI), and an increased number of IDE drivers.
The device filesystem has been changed significantly in kernel 2.4, and the naming convention for devices has been changed to add new “name space” for devices. These device names will now be added dynamically to /dev by the kernel, rather than all potential device names needing to be present beforehand in /dev whether used or not. While backward-compatibility is intended, this may interfere with some applications (most notably Zip drive drivers) that worked with previous kernel versions.
New filesystems have been added (including a functional OS/2 HPFS driver, IRIX XFS (EFS), NeXT UFS supporting CD-ROM and NFS version 3). Support for accessing shares via NFSv3 is a major step forward, although Linux volumes will still be exported using NFSv2. Linux’s method for accessing all filesystems has been optimized, with the cache layer using a single buffer for reading and writing operations; file operations should now be faster on transfers involving multiple disks.
For the Masses
There are, of course, a large number of updates to Linux that are primarily oriented towards the desktop (rather than server) user. A generic parallel port driver has been added which enables abstract communication with devices; this can be used for things like Plug-and-Play (PnP) polling or splitting the root console off to a parallel port device (like a printer). The new Direct Rendering Manager (DRM) provides a “cleaned-up” interface to the video hardware and removes the crash-inducing problem of multiple processes writing to a single video card at once.
There are a wide variety of new graphics and sound card drivers (including support for speech synthesizer cards for the visually impaired). The UDF filesystem used by DVDs has been added, and infrared (IrDA) port support is also included. Support for Universal Serial Bus (USB) has been added but isn’t yet perfect (although IMHO, whose USB implementation is?) PCMCIA card support for laptops is now part of the standard kernel rather than requiring a custom kernel version, but an external daemon will still be required for full support. FireWire/i.Link (IEEE 1394) support is there, as well as IEEE 802.11b wireless (Apple’s “AirPort,” Lucent’s “WaveLAN”).
Probably the most far-out “consumer”-level enhancement is that kernel 2.4 has added support for the rare infrared RS-219 standard, a management interface used by specialized remote controls for Mobil and Amoco station (and some others) car washes! With the optional xwash software package, this can actually be used (on a laptop) to send signals for a “free” carwash.
I’m kidding about that last one.
Is Anything Missing?
The 2.4 kernel itself does not have encryption technology built into it; that’s probably a wise decision, based on the various cryptography regulations of countries worldwide that might actually make it prohibitive to export or import the Linux kernel. Unlike the 2.2 kernel which included Java support automatically, you must specifically include it when building a 2.4 kernel.
Although Journaling File System (JFS) efforts have been underway for a while, their maturity was not sufficient to include in kernel 2.4. JFS systems – a major requirement for true mission-critical servers – record (“journal”) all of their operations (analogous to a transactional database), so advanced data recovery operations (such as after a crash or power loss during read/write operations) are possible. See IBM’s open-sourced JFS project (http://oss.software.ibm.com/developerworks/opensource/jfs/?dwzone=opensource) for more information and software availability.
For Mac Linux users, support for newer Mac Extended Format (HFS+) disks has not yet been added. As of this writing, the NTFS (Windows NT/2000 file system) driver can read but not write data from within Linux. Alas, support for Intel 8086 or 80286 chips is not present either.
Lastly, you should immediately assume that things that worked with kernel 2.2 will always work with 2.4. Changes in the device filesystem and the block device API (block devices are non-serial objects; i.e. devices like hard disks or CDs that can have any sector on them accessed randomly rather than receiving input in order) may break compatibility with some existing drivers.
Getting In-Depth with Linux 2.4
In this column, I’ve only been able to touch the surface of the new functionality available in Linux kernel 2.4. The “definitive” (most frequently quoted) analysis of 2.4 kernel changes is an ongoing set of posts to the Linux kernel-developers list by Joe Pranevich. The (currently) most recent version can be found at http://linuxtoday.com/stories/15936.html.
There’s also a good “kernel 2.2 vs. 2.4 shootout” with specific application test results and details at http://www.thedukeofurl.org/reviews/misc/kernel2224 and upgrade instructions for kernel 2.2.x systems at http://www.thedukeofurl.org/reviews/misc/kernel2224/5.shtml.
For an excellent overview of Linux 2.4’s new firewalling capabilities, see the article at SecurityFocus (http://securityportal.com/cover/coverstory20010122.html). For great information on the new network system’s routing capabilities, check the HOWTO (http://www.ds9a.nl/2.4Networking/HOWTO//cvs/2.4routing/output/2.4routing.html). A detailed article on the new security features in 2.4 can be found at http://www.linuxsecurity.com/feature_stories/kernel-24-security.html.
For a more in-depth overview of the general features, read the IBM DeveloperWorks kernel preview part 1 (http://www-106.ibm.com/developerworks/library/kernel1.html) and part 2 (http://www-106.ibm.com/developerworks/library/kernel2.html).
For an interesting comparison between Linux 2.4 and FreeBSD 4.1.1 (ignoring many of the advanced features of the new Linux kernel and concentrating on common tasks), see Byte Magazine’s article (http://www.byte.com/column/BYT20010130S0010).
For the kernels themselves and the most recent change logs, visit http://www.kernel.org/pub/linux/kernel/v2.4/. For ongoing news about Linux 2.4 and life in general, see http://slashdot.org.