Today’s award goes to the operator(s) of a data center in Sweden. A tip o’ the hat to reader Snoggeramus for sending me the link.
Having worked in the information technology industry for a decade or so, rising from computer (mainframe) operator, through programming and systems analysis, to manage a department and then be a director of a small IT company, I’m pretty familiar with commercial computer operations. This was entirely preventable, and should have been foreseen.
A loud sound emitted by a fire suppression system has destroyed the hard drives of a Swedish data center, downing Nasdaq operations across Northern Europe.
The incident took place in the early hours of Wednesday, April 18, and was caused by a gas-based fire suppression system that is typically deployed in data centers because of their ability to put out fires without destroying non-burnt equipment.
These systems work by releasing inert gas at high speeds, a mechanism usually accompanied by a loud whistle-like sound. With non-calibrated systems, this sound can get very loud, a big no-no in data centers, where loud sounds are known to affect performance, shut down, or even destroy hard drives.
The latter scenario is what happened on Wednesday night, as the sound produced by the errant release of the inert gas destroyed hard drives for around a third of the Nasdaq servers located in the Digiplex data center.
. . .
A Digiplex spokesperson told Bleeping Computer that Nasdaq only rents space in the data center, and uses its own equipment. Nasdaq said there weren’t enough servers in the whole of Sweden to replace the destroyed ones, and had to import new machines.
There’s more at the link.
Yes, loud noises can be devastating to computer disks. Have you ever seen a really loud woofer at full volume on the back shelf of a car? The speakers are vibrating in and out, shaking the entire vehicle. Do that to a disk drive while its heads are reading or writing data, and they’ll crash into the disk surface, scratching it and damaging the read/write heads. Bye-bye, disk drive. If Nasdaq had to import servers, because “there weren’t enough … in the whole of Sweden to replace the destroyed ones”, that must have been a very loud noise next to a very large number of server units.
Unfortunately, even though they should know better, sometimes that sort of known problem is overlooked when other priorities are pressing. I remember when a new fire suppression system was fitted to the mainframe computer center of a South African oil company, where I was employed at the time as a computer operator. I looked at the emergency masks, designed to allow operators to exit the room in the event of a fire. They were all smoke inhalation masks, designed to take particulates out of the air so one could breathe freely. I pointed out to the Operations Manager that halon, the gas used in our new fire suppression system, actually made it impossible to breathe at all. It was as if all the oxygen had been removed from the air. In such circumstances, particulate filters would do nothing at all to save our lives. Smoke or not, we needed something to breathe! The offending masks were replaced with respirators within a day, each with a small self-contained cylinder of oxygen, enough for up to five minutes, to let us get out alive. We called that an improvement . . . again, something that should have been foreseen, but was overlooked due to pressure of other factors.
Peter
Back in the '70s, I was a reactor operator at a test reactor facility. Our sister reactor (designed in the '60s) had an incident when a ventilation upset activated the CO-2 fire suppression system. The ventilation upset caused the control room to be positively pressurized with respect to the surrounding area. Couple that with doors that opened inward thus the operator could not open a door against the air pressure. The CO-2 system had a 20 second delay and warning horn before CO-2 was released. Fortunately for all concerned, the operator got the attention of some other folks and was able to escape with about two seconds to spare. Shortly after that, all of our occupied spaces with CO-2 systems had them replaced with Halon. The Halon used was non-hazardous and at a concentration that allows occupancy for an hour or so. Yeah, Murphy is an optimist.
Halon dumps are famously loud but shouldn't happen until after the power to the server room has been cut for a few seconds. That gives the drive heads time to park off the platters to avoid just this problem.
We've known about this issue for 10+ years at least, the fact that this server room was effectively destroyed says it wasn't managed properly in the first place.
Digiplex will be paying a lot for the replaced equipment, and I will be quite surprised if they survive much longer.
Nasdaq can afford to use solid-state disks only, why haven't they done so?
Yep, Murphy strikes again… sigh
hmmmm. yep. halon. bromofluoromethane in the can supercharged with nitrogen to blow the stuff thru the system. effective. not good for the ozone layer.definitely not good breathing gas.
FE1301 on the other hand is effective as a fire suppressor at about a what, 5% concentration? at that level it won't cause harm for about 15 minutes depending on body mass and lung efficiency. Fenwal used to market FE1301 systems for computer centers. but then nothing is perfect. fenwal has sensors that detect pre-combustion byproducts.just striking a match would set it off. used by lockheed to protect the cargo compartment of C5A cargo planes.
CO2 displaces O2 and will kill you dead right there(DRT) but will put down the fire also. it has its uses.
i have serviced several nitrogen inerting systems but the problems with it are the same a CO2 systems. and expensive dewars for the LN2.
i have never seen a warning horn driven by the agent or discharge. the horns i have seen have all been electrical and with visual indicators. see a blinkie light, hear a low pitch horn? run for the exits! learn something new every day!
When the government banned Halon fire extinguishers, but allowed existing ones to remain, Boeing basically cornered the market on them. They are all over the factory.
The most amazing thing I ever saw there was one of those giant wheeled extinguishers on the factory apron, containing 150 POUNDS of Halon.