When Hurricane Katrina made landfall in the early morning hours of August 29, 2005, Theresa Burton, technology coordinator at the Pass Christian School District in Pass Christian, Miss., was confident that her district was prepared for the onslaught. In the hours before the storm, Burton dispatched her staff to follow the plans laid out in the district’s disaster recovery plan. One person backed up the district servers onto tapes. A group of others unplugged computers and put them up on desks, covering the equipment with thick plastic tarps.
In previous storms, the district’s measures had proven to be sufficient. This time, however, during the worst storm in more than 40 years, no amount of disaster planning would have saved the day. Because Katrina caused a storm surge that leveled everything along the Mississippi coast, water reduced two of the district’s schools to rubble. Barton remembers receiving word that water had crested over the roof of one of her schools. Everything inside the school was ruined.
“Technically, we had an elaborate disaster recovery plan to help us deal with what we thought would be the worst-case scenario,” she says now, looking back. “At the end of the day, though, what we didn’t have was a sense that any storm ever could be this bad. We simply were not prepared.”
Unfortunately, the story from Pass Christian is familiar to other technology leaders in hard-hit states like Mississippi, Louisiana, and Alabama. Across the board, technologists say they had basic disaster recovery plans—then note that these plans were decimated along with the rest of their district IT assets. What went wrong? How do these districts plan to learn from their mistakes? While many are still answering questions like these, some are already looking ahead.
Communication Failures
Hands down, the biggest problem for Gulf Coast districts during Hurricane Katrina was communication. A case in point is the Gulfport School District in Gulfport, Miss. Terri Burnham, director of technology, says that her pre-storm plan called for a system administrator to back up all servers and take the tapes with him. But the administrator was forced to evacuate, and because the phone lines were down, Burnham had no way of checking to make sure the tapes were safe.
“As soon as the communication system went out, our entire disaster recovery plan went with it,” says Burnham, who notes that the district lost all of its servers, routers, and data backups. “Without having the luxury of confirming that our [administrator] had made the backup, we had to assume he did, but he didn’t.”
Donna Torres, programs director at Bay St. Louis-Waveland Schools in Bay St. Louis, Miss., experienced similar frustrations. Bay St. Louis is a city right on Mississippi Sound, exposed to water on three sides. Just as Katrina’s storm surge leveled buildings in Pass Christian, so, too, were several facilities in the Bay St. Louis-Waveland district destroyed. School had started a week earlier, and Torres had just installed $72,000 worth of new computers in two labs. Within minutes, all of these new computers and the labs that contained them were washed out to sea.
As Torres and her employees were evacuated north, they all left home with their district cell phones. Because all the cell phone towers were knocked out, communications were completely cut off. Torres says it took her two weeks to find other district administrators, and even then, she was only able to do so through the State Department, which directed her to the district superintendent. Today, six months later, the district has limited communications that run over a cellular network because the cost of laying new cable is still too high.
“You really take communications for granted,” she says. “I know we did—as we put together our disaster plan, it never occurred to us that we might not be able to find each other to take stock and discuss how badly we were hit.”
Thinking Quickly
Luckily, unreliable communications didn’t plague all of the districts victimized by Hurricane Katrina. Carl Gaines, district technology coordinator at the St. Bernard Parish Public School District, dispatched his computer services manager to back up the district’s payroll data late in the evening of August 28. While the manager managed to get all of the backups, after the hurricane hit New Orleans it took the duo nearly a week to reconnect and recreate the system in East St. John, where the manager had settled after evacuating.
While the computer services manager was headed for East St. John, Gaines backed up the district Web site and took the files with him when he evacuated New Orleans for Baton Rouge. Though Gaines wasn’t able to communicate with any of his employees, he figured the site had gone down when the parish flooded, and delivered the backup to Transformyx, a Baton Rouge company that had volunteered to host the site free of charge. Within days, the Web site was back online.
“There was a service disruption, but we were relatively prepared for it,” Gaines says. “I think everybody agrees it could have been a whole lot worse.”
Jeff Mendoza, IT manager at New Orleans Public Schools, agrees. The night before the storm, Mendoza and a cadre of network administrators headed to the district data center in an office building in downtown New Orleans to shut down all of the critical computers. Next, the team ran an extra data backup, a practice that usually didn’t happen on weekends. Finally, team members locked the data backups in a fire-proof cabinet, and pushed all of the computers to the back corner of the room, away from the nearest windows.
These disaster recovery efforts paid off. Though the data center remained relatively untouched during the storm, a leak in the building’s ceiling resulted in a small pool of water in one corner of the room. But because all of the mission-critical machines had been moved to one side, Mendoza says the puddle was nowhere near damaging any important data. What’s more, when Mendoza and a handful of colleagues were finally able to return to the facility the day after the storm to assess damage, they found the backup tapes in the very same cabinet where they had left them, in perfect condition.
“Ideally, I think we would have liked to store those tapes off-site instead of putting all our eggs in one basket and leaving them there,” he says. “Now we know this is something we need to focus on the next time we put our plan to the test.”
Lessons Learned
Mendoza has already made these improvements to his disaster recovery plan; every week, the district sends its data backups to Iron Mountain, an international records management company with an office up the freeway in Jefferson, La. Other New Orleans districts have learned similar lessons, and are planning to send data off-site as well. Gaines, the technology coordinator in St. Bernard Parish, says he’s considering another alternative: data mirroring, where certain servers automatically copy themselves to other servers across the country once or twice a day.
In Mississippi, where many districts have been crippled completely, it will be more difficult to apply immediate changes to disaster recovery plans, but technology coordinators say it’s the very least they can do. Torres, the programs director at Bay St. Louis, says her district has invested in special Optiplex GX620 desktop computers from Dell that come with a carrying case for easy transport at a moment’s notice. This way, she says, if teachers need to evacuate the district immediately, they don’t have to worry about backups because they can take computers with them.
“It’s more a preventative measure than it is protective,” she says. “From here on out, just in case the backup doesn’t work, we’re taking everything with us when we evacuate.”
Terri Burnham says she, too, is making equipment more portable to facilitate transportation in the event of an evacuation, but she is also redoubling efforts to train teachers how to back up data themselves. After the hurricane, every teacher was given a flash drive and extensive instructions about how to use it. She says district officials hope that by instilling a culture of backing up data on the individual user level, they’ll create an armada of data backups to rely upon should the formal disaster recovery plan falter again.
Another strategy of Gulfport’s is to decentralize networking across the board. Before the hurricane, the district ran all of its computing operations from a central facility—a building that was leveled when the first storm surge hit. Down the road, Burnham says, each school will have a server from which it can run a Local Area Network (LAN). This way, instead of having one pain point in the event of a catastrophic failure, data at each school can be stored and protected separately, improving chances for at least some of the data to survive.
“If we’ve learned anything over the last few months, it’s that you can never be too careful in protecting your data,” says Burnham. “If I’d have to single out a theme for our disaster recovery plan down the road that would definitely be it.”
Matt Villano is contributing editor of School CIO.
