From about 7 a.m. to 10:50 a.m., the main Yakima Herald-Republic Web site was unavailable. Vaguely ominous messages such as “Service Temporarily Unavailable” or “Rails application failed to start properly” greeted our readers. (Warning: We’re about to get geeky with our technical descriptions.)
Early this morning, the master DHCP server for the building went offline. This server provides network addresses and lookup services for all machines within the H-R’s offices, including both desktop computers and the web server. Without these addresses, most computers will automatically switch to using a local address (rendering them incapable of talking to other computers on the network until restarted), or assume that the lookup request was lost and continue to retransmit requests (which ties up network traffic).

The DHCP server was not fully restored until 9:30 a.m., at which point most desktop computers in the building were able to recover. However, the Web server depends on a connection to a separate computer that hosts the news articles themselves. While we were able to restore both computers to working status, they were unable to communicate with each other. The problem was finally solved with a complete reboot of the Web server, which is not our standard procedure.

As we move forward with upgrading and improving portions of our site, we are keeping an eye on these behind-the-scenes issues in addition to the visible elements such as the new navigation system. One of our main goals in our future redesign plan is to streamline this process via which news articles are published to the Web, so that we can eliminate as many points of failure as possible.

- Scott