Tech blog

IT managers and analysts are expressing surprise at the footing of time it appears to be taking officials at the City of San Francisco to regain full control of the city's FiberWAN network after a disgruntled network administrator allegedly locked access to it by resetting administrative passwords to its switches and routers.

Skip to next paragraph

• CIO - Business Technology Leadership
• Computerworld - News & Top Stories on Software, Hardware, Windows & VOIP
• Infoworld - Get Technology Right
• Network World - Software, Hardware, Electronics & Gadgets
• The Industry Standard

With few details publicly released put on what exactly happened, many are suggesting that the network lockout and the city's answer to it point to a abortion to implement and manage essential principle security controls.

Terry Childs, a network administrator at San Francisco's Department of Telecommunications and Information Services (DTIS) was arrested on July 13 for allegedly tampering with the network, which carries almost 60% of the city government's traffic. He is also alleged to have planted network devices that enabled illegal remote access to the FiberWAN network.

He was jailed last week on US$5 million bond after refusing to divulge the passwords he had used to block access to the network. Child, 43, pleaded not guilty to the charges against him at a hearing in San Francisco Superior Court last week. He is scheduled for a bail hearing tomorrow. If convicted on all charges, Childs faces a maximum of seven years in prison.

As of late Monday, the city's efforts to repair the network still remains a "work in progress" according to Ron Vinson, deputy director of the San Francisco Department of Telecommunications and Information Systems (DTIS). Despite around-the-clock efforts with vendors and staff the city is still only in the process of determining "what level of unlawful access or what level of tampering" Childs was responsible for, Vinson said.

According to Vinson, the WAN is operating normally, but the city has not yet regained full administrative of all routers. What remains unclear also is whether IT officials have discovered all of the network devices that Childs is alleged to have illegally installed on the WAN, Vinson said. He did not say what exactly the network devices were or which they did, but claimed that Childs had gone to great lengths to hide them from detection. "We have 60 plus departments that are clients of ours," Vinson said and the task now is to find out if Childs managed to install the devices at some of those departments, he said.

"That is why we need to do a system-wide analysis of where we are with access. We don't know what he had access to," Vinson said. He added that the focus right now is on damage containment. "We want to make sure we maintain full operability if [Childs] was to be released on Wednesday," and were to try to gain illegal attack to the network, he said. "We want to make sure we are up and running."

Vinson however did not provide any details on what exactly Childs did or the extent to which he may have compromised the network. He said that the reasons for the slow recovery would become apparent once those details were publicly released.

Meanwhile, news of the city's continuing struggles, combined with a relative lack of publicly available details on the sort of exactly happened, is fueling questions and theories about what may have happened in some quarters.

"I am completely floored that it is taking with equal reason long to reinstate access to the equipment," said Jim Kirby, senior network engineer with Dataware Services, a Sioux Falls, SD-based service provider. "Unless they have some crazy uptime requirement that prevents them from rebooting gear, it's hard to understand." In in the greatest degree cases, he said, passwords can be reset with a reboot and some keyboard combinations.

goal, he added, with some networking gear from Cisco Systems Inc. it is practicable to sending out a command that renders password recovery impossible, formation it necessary to send the equipment back to Cisco to be "re-flashed to new." "Even so, it's solely a matter of replacing locked gear with new gear created from the backup configurations they surely must have somewhere. One box at a time, if needed," Kirby said. The apparent fact that San Francisco city officials took longer than 48-hours to restore access says that they were not following "even basic network administration standards," Kirby said.