It's research that's so valued they don't have a battery backup and don't page a grad student or two when the power goes off in the middle of the night. It's so important they don't explain the problem to people who have access to the lab saying "hey. look at this sign over here."
RPI does have emergency generators and UPS for their campus. The sign was on the fridge so that someone wouldn't do something crazy, like pull it away from the wall and unplug it. Someone going to the circuit breaker and killing the power to the whole area is not something any reasonable person would do or expect others to do, the same as I wouldn't expect the cleaners to "turn off the lights" by smashing the bulbs with a broom handle. It isn't exactly reasonable to come in after the fact and say "of course someone might try to turn off the lights by smashing the light bulbs. Obviously, you should have had cages around the lights to prevent this. How negligent of you."
Over 100 years I would expect 2 failures to overlap multiple times especially if you are willing to let one slide.. I'd have to assume they were getting paged when it failed and they failed to set a new bound to resume paging for a worse failure. Anything else could have been their second failure that caused an unacceptable temperature change instead of a cleaner misreading a breaker panel that apparently only needed one failure per 25 years.
Not the most scientific cleaners meet not the sharpest planners.
Just watch a few CSB videos (among others) on YT to understand how some companies are run, when it comes to functional safety and ergonomics.
This is unfortunately not taught everywhere apparently.
Long story short, some companies would have done exactly the same management for something that would cause Armageddon.
In the 80s in Paris, a train crashed when some train driver thought he was helpful by responding to an alarm and fixing the brakes himself instead of calling maintenance, and actually turned off the brakes because a poorly design switch made it look like the circuit was open.
Plot twist, he was found to be not guilty of anything because company procedure were not clear.
So a mere freezer probably went under the radar of management.
In this case, a power loss could have been accidental as well. What if a fuse went out, or some unrelated piece if equipment caused an upstream GFCI to trip ?
How can you trust a fridge and its power supply to protect 1M dollars for 20+ years, if 20 years worth of samples were destroyed? Power was just bound to be lost at some point in time, over a period of a few decades.
(It even happens in datacenters)
And some UPS would cost up to 2000$ ?
Up to 500 more for the warning system to send a text ?
IMO, functional safety should be a staple in university.
> How can you trust a fridge and its power supply to protect 1M dollars for 20+ years, if 20 years worth of samples were destroyed? Power was just bound to be lost at some point in time, over a period of a few decades. (It even happens in datacenters)
This very point makes the entire thing so ridiculous I'm not convinced this isn't the setup to outright fraud. Like a failing storeowner who starts making customers with children traverse the artisanal glasscraft section to get to the toy section. "Oopsies! You owe me $1M."
I can't say this is about money (it's a university), but suspect maybe this project was doomed anyway and now its coordinator is staging sabotage to save face.
The only thing that gives me pause is the thought that it's a little weird the cleaner was messing with the circuit breakers at all. The explanation offered makes little sense.
The cleaner cut the power at the circuit breaker. It isn’t like they had an extension cable running through the middle of the room that they tripped over. In what world is that equivalent to “making customers with children traverse the artisanal glasscraft section to get to the toy section”? The RPI campus does have UPS and emergency generators because it is important, so the whole lab is protected from power failures. They put a sign on the fridge for the short period of time that the alarm was going off while waiting for repair just in case someone wanted to do something crazy, like pull the fridge out and unplug it. You can’t account for every single thing someone might do. What if instead of going to the circuit breaker to cut the power, he opened the fridge and pulled all the samples out so he could clean inside? Or did just unplug it? or something else stupid?
I’m pretty sure there are a lot of businesses that would lose a lot of money if staff members started flipping off random circuit breakers to various systems and pieces of equipment.
> You can’t account for every single thing someone might do.
yes you can. It's called security, and it has been a booming industry since at least 4 decades. People will do stupid things because they think differently, have different things in mind,...
Never wonder "if" they will do it, but "when"
> if staff members started flipping off random circuit breakers
If they are accessible to them then they shouldn't cause damage.
Just lock the cabinet if nobody should touch them.
A lock is 30$, isn't it?
A tag with a phone number for maintainance is 5$ ?
> They put a sign on the fridge
So they knew they were 1 switch away from disaster, human intervention or not.
I hope it had proper SIL rating, if they are mad now it failed.
Either humans are nice and well intentioned but stupid, and it's a shame to blame them when they just tried.
Or they're really evil, and then you need to protect the system from them.
Anyways, it's stupid to blame workers afterwards.
UPS should have been directly between the fridge and the wall, without switch in the middle. And it should have warned staff. If not, they had accepted the risk.
The fridge was also already malfunctioning and required maintainance (hence the temperature alarm)
So I'm surprise there was no monitoring whatsoever, just a sign, and now they are mad about its failure.
It's not consistent.
They already had an alarm that was defective that tracked the fridge temp and would alarm when it went off. What would a battery backup would have done if the fridge itself is defective?
Worth reading to answer these questions. Alarm was not defective. Repairs were delayed and this happened while waiting for a repair. It wasn't expected that samples would be damaged due to a slightly dropped temperature for 1 week. Most major flaw here was a lack of functional safety/training to avoid this sort of situation.
Can't help but think RPI shares some culpability here if they allowed anyone to throw that switch. Surely electrical codes allow for the protection of such breaker switches.
Dating myself here, but reminds me of the episode of the '70s TV program "Salvage 1" about a scrap dealer (played by Andy Griffith) who builds a rocket from parts he finds. On their maiden voyage (I think) the cleaning guy unplugs the guidance computer so that he can plug in his vacuum cleaner.
Reminds me of the company that sequestered private keys. Some idiot VP went to the vault, took out the master list and email'd it to a friend who had forgot their key. Gutting the entire value of the company and ruining it forever.
People do the damndest things. Critical stuff needs a little extra care put in place or some fool will inevitably do the thing-that-must-not-be-done.
Sometimes I have no confidence that a mature civilization is even possible if it's composed of human beings.
For every one of us monkeys who says "hey, y'all...watch this!" and does something insanely stupid, there are always at least a couple who take a step back, make sure we're out of the blast radius, and think "this guy is going to make a great warning to the others"...thus preserving the species for another day.
So they keep millions of dollars worth of samples in a single freezer in a single location?
The loss of the samples was an inevitability and regardless of who pulled the plug - a cleaner, a power cut or a faulty compressor they are solely responsible for the loss.
Assuming a cleaner would read a note attached to a freezer, given they no doubt have to see lab notes all over the place - and probably have instructions not to move or read the notes - is completely delusional.
IMO the blame falls squarely at the feet of the researchers and RPI and at worst the cleaning company should agree to terminate the contract (thus losing the $1.4 million per year deal they had).
I mean, looking at a site now that sells cryogenic freezers, only one model goes colder than -86C, and this was going down to less than -110C. this is expensive and specialized equipment. I understand not having a second one but they needed more safety and redundancy on this one.
The article mentions Fahrenheit (-112f), which means it was -80C, so the first one you found would have been "good enough". A quick google finds a -80C freezer for $14000 [0] a fraction of the $1million they've "lost".
While I agree RPI screwed the pooch on this one; I am surprised for $14k the damn power switch isnt under a locked panel/switch cover given what it is used for.
Apparently they had a sign that explained how to mute the alarm. I think that’s going to work against them. They were aware that the alarm could trigger erroneously and didn’t fix the issue.
Why does that matter? They knew it was broken but didn’t explicitly notify the cleaning company to avoid the fridge. Like what is the argument exactly? “I knew this was broken and didn’t tell you about it. Now you owe me money because the broken thing interfered with your work.”
The sign was specifically for the cleaner, explaining the problem, plus instructing them not to clean the area. It also told them how to mute the alarm sound if it was annoying.
In hind sight it would have been better to pre-contact the cleaning company and make sure the cleaner in question understood the situation.
But that's only from knowing after the fact that the cleaner was hmmm... a friggin idiot. ;)
Do you really think the cleaner looks at every thing posted in the lab as he cleans? Very unlikely. Agreements require a meeting of the minds. The only idiots I see are the ones that lost decades of research.
As far as I'm concerned killing the breaker to mute an annoying alarm is malicious. While there certainly seems to be laxness in their precautions I think the primary blame goes to the cleaning company.
The system was already in a failing state if an alarm was going off. It was being neglected. Letting your car alarm blare 24/7-- that's malicious. Anyone attempting to disable it is an aggrieved victim. You're reversing the roles of victim and offender.
If the system was that vulnerable, area denial is step 1. Evacuation/remediation is step 2-3. Putting up a sign saying both "stay away" and "push this button" (that some Honduran or Dominican or whoever may not even be able to read in the first place) is well-intentioned but misguided. It's not the cleaner's fault they walked into a minefield. This wouldn't even be up for discussion if this were an open elevator shaft or arcing electrical panel. A problem existed-- fault is not attributable to whoever next encounters it.
Right now, I have a degraded zpool that's lost 3/6 disks-- one more failure and it's toast. Putting my NAS on the circuit connected to the light switch (no UPS!), then blaming my wife for destroying all of my life's work when she so much as walks into a dark room is just sadistic irresponsibility. Teenage girls do this shit. It's bully logic.
How about if you stick a sign over the top of the light switch saying "Please don't turn this off, there's an emergency with the NAS and I'll lose all the data if this switch gets turned off"?
If she still turns it off anyway, you'd be forgiven for being a tad unhappy about it... ;)
RPI clearly had the information and obligation to understand the nature of their problem and to implement an effective solution. Turns out they were unable to do so. One might say that the scientists were not trained on how to implement effective operational procedures, believing instead that a piece of paper with a message on it would be effective. Their claim that a cleaning company should have been able to anticipate their failure and do a better job at operational assessment than RPI seems like a losing argument.
