“Meltdown” causes disruption in Novell service

Jason Garber

The campus map displays the buildings that were affected by the network outage. The outage hit the Science Center and the Library the hardest, where a Novell connection is required to log into classroom and lab computers.

The catastrophic failure of a fiber optic switch over the weekend inconvenienced users and forced Information Systems technicians to work more overtime than usual to restore network connectivity to eight EMU buildings.

Sometime around noon on Saturday a Cisco 3508 eight-port fiber optic switch in the Hartzler Library failed, cutting off network service to eight buildings at the southeast end of campus. A student in the Conflict Transformation Program student lab was the first to notice.

"I got a call saying, 'Hey, is the system down? I can't log in,'" said Jack Rutt, Director of Information Systems. He stopped by on his way home to check it out. "When I got to the CTP lab, sure enough, absolutely none of the computers could find the server. I thought, 'Well, maybe there's something wrong with this building,' but in the back of my mind I thought, 'or maybe it's worse.'"

The problem was in fact much worse. Looking at the network's online diagnostic diagram, Rutt discovered a major branch of the network was offline. "When I came in, this whole branch was red," he said Monday afternoon, pointing to the diagram. "I think the power supply [of the switch] had a meltdown. We didn't actually open it up but we peeked inside and there was gooey stuff in places it ought not to be," he said with a chuckle.

Rutt contacted the manufacturer, Cisco Systems, to request that a replacement switch be sent. EMU's current support agreement with Cisco, however, only requires parts to be sent by FedEx the next business day, so the replacement unit didn't arrive until Tuesday morning. In the meantime, IS technicians used spare two-port switches to jury-rig connections to six of the eight affected buildings.

Though professors' and students' computers can operate offline, classroom and lab computers require a successful Novell login to start Windows, rendering them inoperable when the network is down. The Science Center lab was locked during the outage and professors had to find alternatives to using classroom computers.

"I had a video ready in case I couldn't access my PowerPoint but they had it up and running in time," said history professor Mark Metzler Sawin. "Kudos to IS for fixing it so quickly."

The temporary switch was not installed in time for Biology Professor Roman Miller's 9 a.m. class, however, so he gave a PowerPoint presentation from a laptop.

Two buildings, Marketing Services and the Advancement Division, could not be reconnected until the replacement switch arrived because of their priority and the network topology. Staff in these offices found it hard to do their work without their network connection.

"I can't do much at all" said graphic designer Mike Eberly. "I had one project for a Mennonite organization I absolutely had to have in today [Monday]." He made the deadline by taking his files on disk to the library.

Last summer, IS upgraded 17 "legacy" hubs and switches and installed Cisco managed gigabit switches, which diversified connections to the south end of campus. "Up until that time, if this same thing had happened, the houses along College Avenue would have been affected too-Sociology, Bible, and Preparatory Music."

The failed switch had been working nonstop since it was installed in June of 2000. To upgrade to Cisco's four-hour response time would cost $1,500 for each of EMU's two 3508 switches and $4,000 for a larger one in the Campus Center.

"It was felt that we didn't have anything that either next business day or through the weekend would warrant spending substantial sums of money, probably somewhere on the order of $4-10,000 [per year] that would basically just gain anywhere from the weekend of 72 hours to a weekday of 20 hours," said Rutt.

For that money, IS could purchase computer upgrades for a small department, two or three upgraded computers for the digital media lab, or a projection system for a classroom.

EMU network patrons in general seem pleased with the mix of reliability and new equipment IS has chosen and their quick responses when network problems occur. Experts say the goal of technology is to make itself invisible to the user. Failures remind us it's there. Quipped Byer, "I think they just created the problem so they could fix it and look good. I'm a Conspiracy Theorist!"