{"id":183,"date":"2001-03-25T12:32:41","date_gmt":"2001-03-25T20:32:41","guid":{"rendered":"http:\/\/www.jeffcarl.com\/?p=183"},"modified":"2020-07-08T19:04:32","modified_gmt":"2020-07-09T02:04:32","slug":"the-web-server-first-aid-kit","status":"publish","type":"post","link":"https:\/\/www.jeffcarl.com\/index.php\/2001\/03\/25\/the-web-server-first-aid-kit\/","title":{"rendered":"The Web Server First Aid Kit"},"content":{"rendered":"\n

By Jeffrey Carl<\/strong><\/p>\n\n\n\n

\"Boardwatch
Boardwatch Magazine, March 2001<\/figcaption><\/figure><\/div>\n\n\n\n

Boardwatch Magazine was the place to go for Internet Service Provider industry news, opinions and gossip for much of the 1990s. It was founded by the iconoclastic and opinionated Jack Rickard in the commercial Internet’s early days, and by the time I joined it had a niche following but an influential among ISPs, particularly for its annual ranking of Tier 1 ISPs and through the ISPcon tradeshow. Writing and speaking for Boardwatch was one of my fondest memories of the first dot-com age.<\/em><\/p>\n\n\n\n

It\u2019s a sad fact that most system administration learning is done in the minutes and hours after you say the words, \u201cWow. I\u2019ve never seen something get broken that<\/em> way before.\u201d Learning to be a sysadmin means that you discover how to fix all the problems that pop up, until you find a problem you\u2019ve never run into before. Then you scramble to learn how to fix that<\/em>, and you\u2019re fine until the next new Unidentified Weird Thing\u2122 happens. And so on.<\/p>\n\n\n\n

Fortunately, about 90 percent of Unix web\/mail\/etc. server problems can be discovered or fixed with just a few tools \u2013 much like 90 percent of all household repairs can be done with a screwdriver, a wrench or a baseball bat. Knowing just a few likely trouble spots and troubleshooting tools can help you resolve a lot of that unidentified weirdness without getting so frustrated that you want to rip the hard drives out of the computer and make refrigerator magnets out of them.<\/p>\n\n\n\n

The key here is that unlike cars or girlfriends, everything that goes wrong on a Unix system happens for a clearly defined reason. While that reason may sometimes be freakish or undocumented, it\u2019s almost always one of a few fairly common issues. <\/p>\n\n\n\n

So, with that in mind, we\u2019re going to take a look at the Handy Tools and the Usual Suspects \u2013 the top commands and tools to use, and common places to look that will at least shed a clue on most server problems. I\u2019m going to use FreeBSD as the example system \u2013 but most other BSDs and Linux can use the same tools, even if they act slightly differently or are located in a different place in the filesystem.<\/p>\n\n\n\n

The Handy Tools<\/h2>\n\n\n\n

\u2022 When a server is responding slowly, you need to figure out whether the problem is on the server or in the network. After you\u2019ve logged in to the server and become root, your first stop should be uptime<\/strong>.<\/p>\n\n\n\n

The important part of the information it provides is the server\u2019s load averages \u2013 shown for the last one, five and 15 minutes. If the load average is high (above two or three), the most likely cause of the slowness is one or more \u201crunaway\u201d processes or some other processes extensively utilizing the system. If the load average isn\u2019t high, then you\u2019re probably looking at a networking issue that is slowing access to the server.<\/p>\n\n\n\n

\u2022 If you\u2019ve found that a high load average is the likely culprit, turn to top<\/strong>. The top command lists the server\u2019s process in order of CPU and memory utilization.<\/p>\n\n\n\n

By default, top shows the top 10 processes, or you can use it in the form top N, where N<\/em> is the number of processes you wish it to show. If you have one or more \u201crunaway\u201d processes (like the tcsh process shown above \u2013 most likely from an improperly terminated login session), you can quickly identify it and issue a kill or kill \u20139 (which effectively means \u201cI don\u2019t care what<\/em> you think you\u2019re doing, just shut up and go away\u201d) command to the process ID number (PID) of the runaway.<\/p>\n\n\n\n

\u2022 For a more complete listing of the processes that are running on your computer, use ps<\/strong>. The ps \u2013auxw command (on BSD-based systems; ps \u2013ef on System V-based systems; the ps on most Linuxes will accept either) will show all system processes owned by all users, whether active or background.<\/p>\n\n\n\n

You can use this to find any active process and get its PID if you need to \u201cre-HUP\u201d or kill it. You can find processes for a single server by using ps in combination with the venerable grep<\/strong>, such as finding all Apache processes by using ps \u2013auxw | grep httpd | grep \u2013v grep. Compare the number of web processes to the server\u2019s \u201chard\u201d and \u201csoft\u201d limits (the \u201chard\u201d limit is set when Apache is compiled; the \u201csoft\u201d limit is set in the [apache_dir]<\/em>\/conf\/httpd.conf file for recent versions) to the number of active processes. If those numbers are close to being equal, consider either upgrading your hardware or reconfiguring\/recompiling Apache with higher limits.<\/p>\n\n\n\n

\u2022 If you\u2019re worried that a user on your system is running an unauthorized program, hacking the system or otherwise foobaring things, then w<\/strong> is a simple check.<\/p>\n\n\n\n

The w command lists active users on the system and what they\u2019re doing. If any of them are performing unauthorized activities, simply kill that user\u2019s shell and use vipw<\/strong> to either give them a password (the second colon-separated field, immediately after the username) of \u201c*\u201d or assign them a shell (the last field of each user\u2019s line) of \/sbin\/nologinuntil you have sorted out the what they were doing and whether it violated your policies. A kill \u20139 may be necessary for \u201cphantom\u201d or \u201czombie\u201d processes that were left running after improper logouts.<\/p>\n\n\n\n

\u2022 If your problem is a crashed or non-starting Apache webserver, use the built-in apachectl<\/strong> command to work out the issue. It\u2019s generally installed in the bin subdirectory of the Apache installation; if this isn\u2019t in your shell\u2019s command path, you may need to specify the full path to this command. Aside from the basic apachectl start and apachectl stop commands, one of the more useful options is the apachectl configtest command, which performs a basic evaluation of Apache\u2019s httpd.conf configuration file (where almost all options are specified for Apache 1.3.4 and later). <\/p>\n\n\n\n

Unfortunately, apachectl is notorious for providing \u201cokay\u201d readings when some configuration problems are still present (most notably when a directory specified for a virtual host is not found or not readable, which causes Apache to fail). For these situations, you\u2019ll need to consult your Apache error logs (see below). Also, apachectl consults the file \/var\/run\/httpd.pid to find its originating process; if this PID is different, the apachectl stop command won\u2019t work. In these cases, find the httpd process owned by root using ps (this will be the \u201cparent\u201d Apache process) and kill that process.<\/p>\n\n\n\n

\u2022 Your first tool for diagnosing whether a problem may in the server\u2019s network connection rather than on the server itself is ping<\/strong>. Using ping to test the connection to a server is a common test, but some problems (such as an error in duplex or settings between a server and its switch) may not show up using ping normally. If a ping to a server appears normal but you suspect a network error is involved, try using ping with larger-than-normal packet sizes. The default size of the data packet used by ping is only 56 bytes, but many errors will only show up when large ping packets (2048 bytes or greater) are used. Use the \u2013s flag with ping to specify a larger packet size (use the \u2013c option to specify the number or \u201ccount\u201d of pings to send).<\/p>\n\n\n\n

With large packet sizes, a longer-than-usual round-trip time is normal, but excessively long times or packet loss are good indicators that there is a network configuration problem present. Try sending large ping packets for at least a count of 50, and compare the results with a long-count ping with normal packet sizes.<\/p>\n\n\n\n

\u2022 If a network misconfiguration between a server and its switch (or router) is possible, then you\u2019ll want to To show the status of your server\u2019s network connections, use netstat -finet<\/strong>. Netstat will show you which ports are open or your server or which services are active, as well as what foreign host is connecting to the port or service in question. <\/p>\n\n\n\n

If you\u2019re concerned that your server is being attacked across the network, this will generally show up in excessive usage of the memory that the kernel has allocated to networking. To find this out, use the \u2013m (memory buffer, or \u201cmbuf\u201d) flag for netstat. If you find that normal services like httpd aren\u2019t heavily burdened but the percentage of memory allocated to networking is still high (90 percent or more), consider shutting down network services or ports that are open and may be being attacked or misused.<\/p>\n\n\n\n

\u2022 If a network issue is the likely cause of your problem, use ifconfig<\/strong> (the interface configuration command) to check how the NICs (Network Interface Cards) on the server are set up.<\/p>\n\n\n\n

You can ignore the lo0 (loopback) interface; what really matters are the settings for your server\u2019s NIC(s) as specified by their driver type. These will show its IP address(es), netmask, duplex and speed, as well as which driver is in use. <\/p>\n\n\n\n

Very frequently, a server which otherwise boots up and appears fine but has a problematic or nonexistent network connection can be fixed with a check of its network interface configuration. Double-check the options set for your default ifconfig startup settings in the file \/etc\/rc.conf (at least in recent versions of FreeBSD). Frequently, a slow network connection is the result of a NIC configured for a different speed or duplex than its switch\/router port, especially when \u201cautosense\u201d options are set but fail for whatever reason. This can frequently be remedied by resetting the connection with a simple ifconfig down [interface]<\/em> [options]<\/em> OPTIONS followed by an ifconfig up [interface]<\/em> [options]<\/em> command.<\/p>\n\n\n\n

\u2022 Weird errors with files or services may sometimes be caused by a full hard drive (preventing the system from writing logfiles or other operations). Use the df<\/strong> command to show your server\u2019s mounted partitions and their available capacity.<\/p>\n\n\n\n

\u2022 A whole nasty horde of seemingly inexplicable problems are caused by simple issues with file permissions. In these cases, the humble ls<\/strong> command can be your best ally. Using ls \u2013l will show you the permissions settings for files in any directory. Common issues include missing \u201cx\u201d (executable) permissions on CGI scripts or applications, or directory permissions which don\u2019t allow \u201cr\u201d (reading) or \u201cx\u201d (entering). <\/p>\n\n\n\n

The Usual Suspects<\/h2>\n\n\n\n

\u2022 When bizarre things are happening, the system logfiles<\/strong> are the first place to check. Under BSD, you\u2019ll find these in \/var\/log; the first place to look is \/var\/log\/messages., where syslog deposits all the messages that aren\u2019t specified to go into another logfile. In fact, the entire \/var\/log directory is home to the messages for different services \u2013 from telnet\/SSH or FTP logins to SMTP and POP connections to system errors and kernel messages. <\/p>\n\n\n\n

Checking these files can often provide the answers to 90 percent of \u201cI can\u2019t do X<\/em>\u201d messages from desperate system users. Check \/etc\/syslog.conf to see where the syslog daemon is sending the errors it receives; check the config files for individual applications or services to see which logfiles they\u2019re writing to.<\/p>\n\n\n\n

\u2022 If the webserver won\u2019t start, but there aren\u2019t any clues elsewhere, immediately look at the  webserver logfiles<\/strong>. Using Apache, these are generally located in the file [<\/em>apache_dir]<\/em>\/logs\/error_log or something similar. Even if apachectl runs and fails while printing a simple message like \u201chttpd: could not be started\u201d (this message is the winner of the \u201cFrontPage Memorial \u2019Duh\u2019 Award for Unhelpful Error Handling\u201d three years in a row), the problem will almost certainly be logged to Apache\u2019s errors file.<\/p>\n\n\n\n

For problems with specific virtual hosts on a server, check wherever their logfiles are located. This is generally specified inside that domain\u2019s <Virtual Host> \u2026 <\/Virtual Host> directive in the httpd.conf file. If no error logfile is specified for that virtual host, then errors will be logged to the main Apache error file.<\/p>\n\n\n\n

Getting a Second Opinion on First Aid<\/h2>\n\n\n\n

Of course, all of the above are merely a few recommendations derived from my experience; if you have found other \u201cFirst Aid Tools\u201d or \u201cUsual Suspects\u201d that you rely on for server administration, please let me know at me@schnell.net<\/a> and I\u2019ll include them in an upcoming column. <\/p>\n","protected":false},"excerpt":{"rendered":"

By Jeffrey Carl Boardwatch Magazine was the place to go for Internet Service Provider industry news, opinions and gossip for much of the 1990s. It was founded by the iconoclastic and opinionated Jack Rickard in the commercial Internet’s early days, and by the time I joined it had a niche following but an influential among … Continue reading The Web Server First Aid Kit<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":22,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[8,66],"tags":[],"class_list":["post-183","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-boardwatch-writing","category-tech"],"jetpack_featured_media_url":"https:\/\/www.jeffcarl.com\/wp-content\/uploads\/2020\/04\/bwatch.gif","_links":{"self":[{"href":"https:\/\/www.jeffcarl.com\/index.php\/wp-json\/wp\/v2\/posts\/183","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.jeffcarl.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.jeffcarl.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.jeffcarl.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.jeffcarl.com\/index.php\/wp-json\/wp\/v2\/comments?post=183"}],"version-history":[{"count":1,"href":"https:\/\/www.jeffcarl.com\/index.php\/wp-json\/wp\/v2\/posts\/183\/revisions"}],"predecessor-version":[{"id":184,"href":"https:\/\/www.jeffcarl.com\/index.php\/wp-json\/wp\/v2\/posts\/183\/revisions\/184"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.jeffcarl.com\/index.php\/wp-json\/wp\/v2\/media\/22"}],"wp:attachment":[{"href":"https:\/\/www.jeffcarl.com\/index.php\/wp-json\/wp\/v2\/media?parent=183"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.jeffcarl.com\/index.php\/wp-json\/wp\/v2\/categories?post=183"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.jeffcarl.com\/index.php\/wp-json\/wp\/v2\/tags?post=183"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}