This reminded me of James Hamilton's (the AWS datacentre guru) writeup of the power outage at the 2013 SuperBowl. Specifically how the default configuration of backup generators is often to protect themselves, even if that means shutting down the things they are meant to keep running. In my own experience running a reasonably sized datacentre I was often surprised at how UPSs and generators reacted to adverse conditions. One thing with power that you don't want is to be surprised. But it's difficult to test this equipment so it's mostly a learn as you go experience.
Some military equipment, especially naval, has something called a "battleshort" where electrical protections can be disabled during combat, where losing equipment could be more dangerous than electrical hazards. It's interesting to think about whether this could be applied to DC environments, but honestly, I think few DCs actually have this kind of risk aversion for downtime. Alternatives like precautionary replacement of generators that have been oversped probably come at too high of a cost, not to mention that many of those false-positive-prone protective devices are there to reduce risk of fire.
> According to Allied Ordnance Publication AOP-38-3,[1] a NATO publication, a battleshort is "The capability to bypass certain safety features in a system to ensure completion of the mission without interruption due to the safety feature." It also says, "Examples of bypassed safety features are circuit overload protection, and protection against overheating".
> For example, the electrical drives to elevate and traverse the guns of a combat warship may have "battleshort" fuses, which are simply copper bars of the correct size to fit the fuse holders, as failure to return fire in a combat situation is a greater threat to the ship and crew than damaging or overheating the electrical motors.
> Battleshorts have been used in some non-combat situations as well, including the Firing Room/Mission Control spaces at NASA during the manned Apollo missions — specifically the Moon landings.
As a side note, with great apology to those who may have lost their lives due to the consequences of real-world battleshorts - this is actually a really interesting analogy for the "war room" scenarios we find ourselves in at startups. Risking damage to morale and productivity can be acceptable if the alternative is irrecoverable loss of the startup's reputation. But this also can't be a sustained state of affairs - these types of procedures are meant for a battle, and risk compounds if you don't return to a steady state. Perhaps adding "battleshort" to our lexicon would make it clear that "doing things that don't scale" is often necessary but not without its costs.
> Perhaps adding "battleshort" to our lexicon would make it clear that "doing things that don't scale" is often necessary but not without its costs.
... And 6 months later marketing people will have commandeered it, without regard to the original meaning, and they’ll be asking people to battleshort so they can really be agile and lean in on providing air cover via the updated sales deck.
Doubly so because I used to be in the US military but now live in a different country, and it's crazy to see how those terms and acronyms infect a lot of US business culture.
A decade ago I would have been fine, even glad, that I could wrap myself in verbiage that would jive with poor career choices I made at 18. Now it just feels incongruous or jingoistic.
Most of military equipment is built to be able to operate outside the safety bounds, because the enemy in a battle is actively trying to push it outside these safety bounds and outright destroy it. Burning down your afterburner chamber and winning a fight is better that preserving it and losing the whole aircraft.
In a datacenter though your bets are usually much lower. Nobody is going to die, or lose many millions, if some of your equipment shuts down to prevent it from being damaged, catching a fire, etc.
There are though high-stake situations where you want exactly that: spend the entire amount of the resources of certain hardware to prevent a loss of life, or of untold millions, when you are powering a surgery chamber, or a large stock trading operation.
People who realize that do over-provision and pay top dollar for that when they can afford that. I remember that when a major fire occurred in one of the skyscrapers in the financial district of NYC, traders of a particular financial company were evacuated with their laptops into helicopters on the roof, and ferried to a spare office across Hudson river, in NJ. To minimize the impact of that, they connected to the corporate network via their phones as hotspots, and kept trading while airborne.
Of course one cannot hope to pull such an operation off without extensive preparation and likely regular drills.
Not preparing to a black swan event during a high-stakes event, like translation of a Superbowl match, sounds like either not having enough paranoia which is professionally required, or, more likely, as a cost-cutting after a wrong assessment of risks.
That sounds a lot harder than having half of each team in two separate buildings. If one needs to shut down the secondary team just takes the full load...
> (2) Think about hiring twice as much senior programmes, just in case. Good day traders are even more expensive.
Day traders are expensive? I thought a lot of those folks keep what they kill, and when they screw up they don't get to eat. "Coffee for closers," etc.
Electrical codes in the US are very clear that they care much, much more about preventing fires than preventing loss of power. The one exception is the fire pump that provides water pressure to the sprinkler system, since if that's running there's already a fire.
The owner of the datacenter's aversion to downtime is not going to be a concern to the building inspector or the fire marshal.
Fire supression systems basically run their own electrical code.
At one of my previous workplaces I learned a bit about our local firecode in relation to electrical code. Mostly that firecode supersedes electrical code. The fire alarm is running of a 230V line (standard) without grounding (non-standard) via a special line from the upstream power transformer that has higher amperage tolerance.
All the fire detectors are powered and wired from this at 40-70V, no ground or if there is a ground, it's grounded at the control panel of the fire alarm system.
In the event of a fire, the entire system is rated about 10X over it's standard ratings. The wires that are allowed to handle 1 Amp before are now allowed to go to 10 Amps, just in case there is a short this might burn it out and allow other equipment to operate.
I've seen some systems that are "optionally fused", they have an internal relais that's held by external power and if the fused lines fail, it automatically switches to unfused power, where the alarm system can evaluate the condition and decide to go back to the fused line if power returns. That's on top of having it's own backup batteries. Some of the more modern systems have watchdogs systems built in so that in case the fire alarm computer is on fire, it won't switch to the unfused line once the computer is no longer operational.
I think you will find that those circuits use isolation transformers and have no path to earth normally -- if you touch active and normal building earth there is no circuit.
At to amperage ratings, don't you just mean that they have a 10x safety factor? Probably using current limiting supplies that can handle a short circuit. Fire circuits (and the feed from the main panel) should use fire rated cabling -- generally mineral insulated copper sheathed cables. Most of the standards (NFPA 75/76) are moving away from being prescriptive to being risk or performance based though, so it is possible they used normal conduit if the overall risk was small.
Incidentally, hospitals also make extensive use of isolation transformers for wet areas (like emergency / ICU), to provide a local point for resetting breakers (electronically), identifying which point is tripping, etc, see IEEE 602-2007. Much better filtering for medical equipment too. You also tend to see positive retention plugs (in the US, green dot UL 817).
The cables have fairly high ratings but in the even of a short circuit, the ratings have a safety margin of about 10x that the cable can last about 6 hours under, above the normal rating under standard conditions (ie, not fire).
Many years back I had the opportunity to visit one of those data centres at the heart of Europe's financial operations. In case of fire, you had a minute or so to get out, because then the place is pumped full of argon. I guess that's a DC variation of your "battleshort"...
Worked on a project in late 80s for an Italian bank. We implemented a SWIFT (financial) interface (on Stratus for reliability reasons). A few fun and games:
- The Stratus supposedly had fault tolerant duplexing of boards - pull one out and the other keeps working. A famous demo where local consultant pulls out the only board which wasn't duplicated!
- The machine room was also fire proofed with Halon gas (if I recall correctly), but due to security concerns the door was locked from inside by operators when present (luckily as programmers we weren't expected to work inside with doors locked!). Luckily also, it was my first exposure to "paired programming"/"paired operating" as in there were invariably 2 of them.
- A couple of the operators became too interested in how to work things, and wrote a quite comprehensive manual. They were "moved on" within the bank, because for security, management only wanted "people who could follow instructions", not "who can think for themselves".
- the whole headquarters building was in a guarded compound, where ultimate protection against "red brigade terrorists" (https://en.wikipedia.org/wiki/Red_Brigades) were the armed guards. We used to meet them in the ground floor cafe every mid-morning, with a revolver on their hip, while they drank an espresso with brandy chaser. Really inspired confidence that they would be there to protect us should the need arise!
> They were "moved on" within the bank, because for security, management only wanted "people who could follow instructions", not "who can think for themselves"
And this is why we cannot have nice IT security practices.
Interesting, one of the guys that worked on my Uni's DC told us that the fire suppressor system is interlocked: to enter the DC, you need to open a door using a key. Upon opening the door, the fire extinguishing system is disabled, and is only enabled when the door is closed. The people working in the DC are supposed to keep the key on themselves. Also, they used Halon 1301 instead of Argon as the gas.
Yeah, you can't install new Halon 1211 systems, but it's still out there. New systems are usually FM-200 or Inergen, which I believe can use the same plumbing as Halon.
Older systems, before Halon was invented, are typically CO2 or Argon. They require a lot more agent to accomplish the job, but they're very simple.
I worked briefly in a CO2 datacenter. It dated from the mainframe era, and had some gorgeous mementos on the walls. There were SCBA packs in the hallway outside, and everyone got a brief safety lecture before being allowed into the facility. The point I remember was basically "The alarm will sound intermittently for twenty seconds, then go solid for five, before discharge. If it goes off, do not stop to save your files, just run for the door. If you're not outside yet when the solid tone sounds, hold your breath and run on what's in your lungs."
Apparently the CO2 is considerably nastier than Halon to get a lung-full of...
Yes, carbon dioxide is not biologically inert like Halon is. It's the signal your body uses to say it's out of oxygen. You will definitely react negatively to getting a lung full of it.
For reference: It's not a lack of oxygen that causes the sensation of asphyxiation. It's a surplus of CO2. It'd essentially be the gas-based equivalent of waterboarding.
Amusingly, tetrafluoroethane and pentafluoroethane are themselves being phased out in refrigeration applications because of their sky-high global warming potential, thousands of times worse than CO2.
What would they do in case of injured/incapacitated people in there? Seems conceivable that some things that could start a fire might cause that as well.
We use something similar on turbocharged drag (grudge) racing cars.
The boost controller has a "scramble" button which overrides the preset boost targets and allows you to run a higher (and unsafe) boost level as long as the button is pressed.
This is useful when you're in the middle of a $10,000 race and you're losing by a few feet.
Yes, definitely real! Although what you're probably thinking of is nitrous oxide injection. One popular manufacturer of nitrous oxide systems is "NOS", which was featured heavily on the Fast & Furious movies.
The "scramble" button I'm talking about is for "boost", which is the amount of positive air pressure inside the intake manifold.
Both "boost" and nitrous can dramatically increase power by essentially getting more oxygen into the combustion chamber, but nitrous is a much more dramatic effect.
All motorcycle mechanics have a sign "speed costs money, how fast do you want to go". It isn't a joke, most of them can get as much power out of a 500cc motor as industrial engines get out of 19 liters. The catch is that you have to rebuild the engine every 10 hours of operation, while the industrial engine will run non-stop for many years.
Sure they do, run away engines are a real problem. They will sometimes start using their own oil as fuel, and at that point you better run. (modern engines have safety air shutdowns)
A320 and B737 had special thrust bump modes for short fields like KSNA. Going back many years to the P&W JT8D you could operate in the red provided you did it for only a certain amount of seconds and logged it down.
There's a maximum power mode selectable on the FADEC interface, which gives you at least 30 minutes of engine life - any use means the engine needs total overhaul.
This mode is pretty much reserved for go around or post V1 takeoff when one engine on twin engined airliner had failed.
Indeed. I forget where I read it, but it provides that power increase for (IIRC) some small double digits number of seconds before you need to throttle back, or the engine eats itself. If you are able to make it back to base, and that WEP wire is broken, the engine needs a rebuild before it flies again. Definitely an impressive power boost though.
The MiG-25 also have something similar where the engine redline is marked around Mach 2.8, but there's no system stopping the pilot from keeping it maxed to somewhere above Mach 3 but damaging the engine in the process.
Funnily enough I believe the SR-71 has the opposite problem, where the engines would merrily go above Mach 3.5 with no issue but the airframe and everything attached to it would get torn apart by the intense heat if the pilot tried to go any faster
No, SR-71s are still limited by engine temperature. They measure the Compressor Inlet Temperature, but I suspect that actual limitation is the turbine inlet temp. That said, it'll merrily try to go above that without issue if you don't throttle back.
Additionally, there's the Mach Cone. The shock wave off the nose forms a cone shape, with the angle determined by the speed of the aircraft. From the nose to the wingtip forms an angle of about 17.5 degrees, which corresponds well with the max speed from the CIT of Mach 3.3.
That both methods of determining max speed agree shouldn't be surprising. Skunk Works was filled with good engineers.
MiG-31 has limiter on engine controls connected to airspeed and mach number indicators, because the engine will happily run above Ma 3, with no issues.
The airframe, however, will get bent from temperature and you better hope the missile cooling loops don't fail.
The power isn't dialed down at lower speeds in order to provide thrust for maneuvering and climb.
I work in the power generation industry. We have a lot of primary/secondary as well primary/standby systems.
The cardinal rule is - test everything. Test it when you install and commission it, test it when part of the system changes, test it periodically during maintenance shutdowns, test it when you have a convenient time to do so without losing production (such as an outage to a different system that necessitates your system being offline).
Test components individually - at the factory and in the field - and test systems end-to-end.
Test with the most adverse possible conditions, not the most optimal. Test beyond your normal operating envelope.
When I participate in design reviews as a maintenance engineer my primary line of inquiry is: how will I test this stuff during operations? Has it been designed in a way that makes testing impossible without an expensive outage? Can workers safely gain access to components that need routine testing without having to de-energize other, unrelated parts of the system?
Now not every industry has the budget available to test to the extent that we do. But even in our industry, I often see a penny-wise, pound-foolish approach of "assume this thing works perfectly from the factory then act surprised years later when an abnormal condition occurs and it fails".
If something was not tested, it may not work. It might be for a very simple reason (like mis-configuration of a single setting during installation) or a very complex reason. But either way, wouldn't the company rather know their equipment doesn't work -before- putting it into service than after?
Apropos of this, in TFA's case they apparently never tested "rip the battery out while the system is running". (There are other, harder-to-test ways for a battery to fail, but that seems like a obvious and easily-tested failure mode.)
That's a good point about testing. I think the key point is where the testing is happening. I'm sure the UPS (and other power equipment) is tested by the manufacturer for many different failure scenarios. The issue is that to the end user, the response is often surprising. So I'm sure to the APC engineer, having the unit shut down when the battery fails (according to some metric) but while there is still mains power available is something that they tested. But the consumer hasn't tested a battery failure, and in fact it may be difficult to test the exact failure mode (low voltage etc) without access to some spare, old, batteries. So when it happens and you're surprised and you call the manufacturer, they (almost always) will say that it operated according to how it was designed and tested. It just didn't do what you wanted or expected. At least that was always my experience doing a post mortem on a datacentre outage.
The telco Central Office I worked in had weekly tests where we'd run the generator for an hour. Everyone's desk computer required a UPS as they were on utility. Comically after a few years you'd have to get a new UPS at your desk as theyd die from continual use.
Most of the COs I worked in did monthly generator "dry runs" without putting load on it, and then every 6 months they'd do a "drawdown test":
Disable the turbine. Kill AC power (turn off all the rectifiers and HVAC) and let the office run on battery for a while. Closely monitor the batteries, and the temperature in critical areas.
After a good while on battery (an hour or two, if I recall), walk leisurely over to the turbine and enable it. Let it start and warm up, then transfer the rectifiers and HVAC over to generator power.
Closely monitor the turbine while the batteries recharge, the HVAC while on unusual power, and the humans who are really glad to have air conditioning back.
When satisfied that all is copacetic, transfer things back to utility AC, shut down the turbine, and call the fuel company to top off the tank.
Most of the batteries in these offices were between 10 and 20 years old, with a couple instances of the beautiful cylindrical "Bell Cells" still in service. They were sized to not break a sweat running the whole building, and impeccably maintained, so they didn't tend to fail prematurely.
How do you test diesel generators which back up a nuclear reactor? I understand you do it during the plant downtime, but don’t you require something at the other end of the network that consumes the 1GW you generate?
A combination of routing the power elsewhere in the grid when possible and banks of gigantic (shipping container sized) dummy loads. As other people have noted, you don't need to dissipate the full amount of power all at once, you test the generators individually.
The diesel generators can back up power to safety systems, so closer to MWs than GWs. Some run with left-handed and right-handed engines coupled face-to-face for redundancy; thus you might AFAIK shut down fuel injection to one to prove the other is supplying torque.
You test them live: someone turns off the outside power switch and you watch them start and take over automatically. Everyone is watching the gauges and if there is a sign of trouble you reengage mains power, then fix the problem and retry. This test is done monthly.
I don't have actual knowledge of nuclear power plants, that is classified beyond the public. However hospitals do the above, because if thing dont work on a bright sunny day how do you think they will work when a tornado has not only taken out power, but also filled the er with server trauma cases.
Electrical Engineers would determine the settings for the electrical protection of a generator. Mechanical Engineers would determine the settings for mechanical protection of a motor. To operate the equipment beyond thresholds would result in reduced lifespan or immediate and permanent damage to the equipment. If this is what the customer requires they have to specify that to the Engineer so that when the equipment blows up the Engineer is not liable.
Diesel generators for hospitals and water pumps for fire suppression systems are generally set up with very loose settings as it is clearly worse for a patient to die or a building to burn down than to destroy a machine, and this requirement is in the specifications.
Diesel generators for hospitals and water pumps for fire suppression systems
Those systems also require regular testing by qualified personnel. NFPA 25 requires monthly testing of fire pumps and annual flow testing. For low rise buildings with inadequate municipal water pressure, a water tower might be cheaper and easier. NFPA 99 - Standard for health care facilities, requires emergency generators to be tested 12 times per year.
True, but the easiest test is to pull the main power to the hospital and watch the system automatically take over. If anything fails you know what is wrong. This is scheduled so that those in surgery (or other worrisome situations) have extra help and can signal get back to mains power now.
ideally the hospital’s generator control system or transfer switch is able to start and synchronize the generators and turn up the throttle until no power is being imported from the utility prior to opening the utility feed, thereby transferring the load from the utility to the generators. This test is only performed when there are no surgeries scheduled and it is cancelled if there are any emergencies coming in all of the sudden. If the test is already in progress incoming ambulances with patients requiring surgery are diverted to another hospital.
To finish the test the hospital is synchronized to the utility, the throttle on the generators turned down until no power is flowing through the generator breaker which is then opened.
I’ve built and programmed control systems that do all this with multiple generators, multiple utility feeds, and multiple tiers of loads of different priorities.
That is a good way to test the generator without interruption, but at some point you still need to test the ATS under real world circumstances: unstable or suddenly missing utility power. Throwing the main covers one of those, and the other requires a little more planning and resources.
Honestly I am struggling to remember an instance where we purposefully plunged an operating hospital in to darkness.
You can test the system’s ability to detect loss of power or poor quality power by lifting sensing wires or injecting out of frequency or voltage range power with a test set, and load pickup and load testing we used a load bank. The actual hospital load is fed through bypass breakers during these tests so if the utility went out we would have to manually operate breakers to disconnect the load bank and swing the vital loads over to the generators.
If I am setting up electrical protection on a generator I don’t put a short circuit on the generator, I show the protection relay what it would see by injecting current and voltage with a relay test set and make sure the relay gives the signal to open the breaker when it should.
Greenfield hospital generation sites are straightforward to open the utility feeds since the vital loads are still fed through the old power system.
You can (and should) do all that. However until you actually disconnect from the utility power you don't know for sure that the systems all work. The generator powers a load, but turns out your hospital is more load than you thought. Or the generator starts but the switch to connect the hospital to the generator is broke and you didn't test that...
In the end you WILL test what happens when the hospital plunges into darkness. Would you prefer the first test to be a a bright sunny day when all the staff is ready for something (and you can turn the utility power back on quickly if something fails), or when someone with a backhoe/chainsaw has an accident with your power feed. The latter will happen, I don't know how or when, but at some point in your future trees will take down power lines, backhoes will cut something, major weather events will take out power for days, maybe a large blackout...
You should have course test everything separately often. However if you wouldn't let a random electrician (electrician only because they will know how to not kill themselves) turn off your utility connection with only a day of warning, your system isn't trusted to handle real events which are typically much worse than a clean turn off the switch.
I remember some complication with the elevators not working or homing properly at one place so we must have had an outage. it was probably at 3am, scheduled, and with incoming patients diverted. All our tests are and I’m in the electrical room so they all looks the same to me. An actual outage to vital loads would be done once and then you shut the door and walk away as there was never any question that it was going to work.
FYI where I’ve worked the tiers of loads are metered individually and picked up in order of priority and with a check to verify the combined capacity of the generators which are online is great enough to pick up the load. Likewise loads are shed in order if there isn’t enough generation such as one generator tripping.
I don’t really see that much difference between opening a breaker and bad quality power. It is the same sensing and logic in both cases: is the magnitude and frequency of the phase to phase voltages in the acceptable range or not?
The important part is that whatever bad happens to mains power, the backup systems detect it and properly take over. Bad frequency or power out are both bad situations that the backups need to handle. You need to have confidence that your systems will handle both. If you only pull power and never test bad frequency, one phase missing, brown outs, over voltage, (list not complete because I don't know everything) then you have not done your job. If engineering determines that your test also covers another bad scenario then I'm fine with not testing it.
Electric utilities provide appropriately higher levels of service and communication to hospitals. Hence there are protocols insuring hospitals don’t lose power in the way you are imagining. The people involved are serious competent professionals with extensive relevant experience, training, and education focused on safety and in touch with the specific facts of actual hospitals and utility grids.
There isn't much that they can do about storms knocking down wires, squirrels chewing a wire and knocking out a phase (other than taking down all phases). Which is why hospitals have several backup systems.
My grade school neighbored a hospital. I recall occasional loud rumblings of the backup generators during recess, presumably they were tests. Legend was they came from a WWII submarine! Neat!
Years later I had a tech support job in that hospital and all the PCs had small UPSs to keep them alive for the 30-60 seconds it took the big generators to come online.
I never did get a chance to see the generator room.
e: Another personal anecdote that is more relevant to the topic.
I was working for a VOIP provider back when Hurricane Sandy hit NYC. Part of our production network was housed in a telecom building in Manhattan. We received daily updates on how many floors the flooding was from our equipment and about the fuel deliveries to the building backup generators. Apparently the generators were on the roof? I'd love to know more about the setup in that building because we had zero downtime.
That blog is an amazing insight into internet operations + culture from that time frame. Windows Media for streaming! URLS with perl/script.pl in them! AIM/ICQ! It helps the writer has a bit of a flair for storytelling.
Working in a 911 call center this is how it worked for us. Except we had larger UPS's that could last probably 30-60 minutes in the building and then a large external generator in a small building beside. You'd see stuff like the lights that weren't on the UPS's turn off for a few seconds before the main generator fired up.
My current job refuses to replace old UPSs that have died, because they have a generator to keep the building powered in case of loss of power. But that doesn't keep your PC running for the moments it takes the generator to spin up.
Oh yeah, if we had that where we worked people would definitely be in danger. Most likely all of our systems would be down if we didn't have the UPS+Generator setup. It often takes 5-30 seconds for the generator to kick in and be running at full capacity. Normally 5 seconds, but sometimes there is a bit of a delay.
The other part of the equation is that they're recommending that everybody switch to laptops, which are capable of running for a long time without power. My job involves a lot of long compiles, so I want to stick with a desktop PC to get the maximum compile speed.
Considering how old the software we used was it definitely could run well on a laptop. However it does mean you have to manage the battery life on a lot of different laptops versus one central area.
Don't get me wrong, the company pushes laptops for reasons other than eliminating UPSs. Even a nearly dead laptop battery should tide you over for the minute it takes for the building generator to get online. I don't think managing laptop batteries is a big concern for our IT department.
Everything involved in 911 calls had special tagging to indicate it. Failure to go through appropriate review and approval processes before doing work on tagged equipment would create a Resume Generation Event.
Nowadays, some hospitals, especially those with heating loads, are doing the opposite:
Run a combined-cycle natural gas generator 24/7, and use the grid as your backup. Gets good efficiency. Runs loops on the coolant, oil and exhaust for water/building heat, and can switch over instantly in the event of failure.
WWII submarine generators produced DC, so they couldn't have been used directly as backup power for a hospital. Maybe they were refitted with alternators.
Can't you just flip the circuit breaker or just pull the plug out? Or does a real power outage usually look different to that in some way, like a surge followed by a shutoff?
In our small data centre we have a board with five lights and a key. You turn the key and the mains is cut off. The lights give the status. The UPSs beep for a few seconds "on battery", the genny fires up in the boiler room and when it stabilises, it takes the load. We leave it running for 5-15 minutes, go out and check the fuel levels and top up accordingly. It's diesel so safe to hot refuel. When done, turn the key back and then the UPSs take the load again and then hand back to the mains. The genny runs for another 15 minutes and then shuts down. That should avoid flip flopping.
You then fill in the test form on our wiki and job done. We do it weekly to fortnightly. An ESP8266 based thingie reports status back to the central monitoring (GPIOs and ESPHome -> Home Assistant)
It depends. We built and deployed some IoT devices (controlling and reporting status of lights, HVAC systems, that sort of thing) to a bunch of greenhouses--one of which ran on generator power.
The customer was running the generators right at the maximum capacity, and we'd see really, really odd things on the power lines. Some times a surge like you mention, sometimes changes to the frequency, sometimes changes to the voltage. We discovered all sorts of weird brownout conditions in our devices.
It's difficult in practice to transfer load to generators for test especially if there's load shedding involved as is common in larger facilities (higher-power-consumption items like part of air conditioning capacity are cut off when transferred to generators). To help address this a lot of more critical facilities are equipped with a test load (really just a very large outdoor resistor with active cooling) that allows for running the generators at rated load for test and exercise without actually transferring. Of course, these tests do not always detect problems with the transfer equipment, and if things aren't designed and maintained well the generator may experience load under real conditions that behaves differently from the test load.
Iirc (top of my head), some datacenters go off the grid regularly at peak (provider) load times and get more favorable rates for doing it. I guess it ensures your fuel doesn't go stale and all your transfer systems work.
Good point. Many power companies will give you a nice discount year round if you will disconnect from mains on demand. If you are on the fence about a good backup generator this will more than make up the difference.
You can put your equipment on a voltage relay to make failure modes as simple as flipping a circuit breaker. It will turn off once voltage drops too much and will wait for it to stabilize for specified amount of time to turn it on.
Initially, there's a fault, say, a tree contacting a wire. This conducts a bunch of current to ground, the voltage in part of the network sags a fair bit, and a protective relay senses the overcurrent and opens the circuit. This happens in a second or two, and your power goes out. You may notice lights dimming or flickering in the instant beforehand.
But, many faults are transient. Perhaps that tree branch finished falling, and is no longer against the wire. So the protective relay performs a "reclose" operation, where it turns the power back on for a moment and measures the current. There's normally a huge starting inrush as motor-driven appliances restart, so it waits a moment before sampling the current, and this is the period when the power comes back on for a few seconds. (It's probably pretty nasty power during this interval, too.)
If things are good, hey, you're good! A brief outage while waiting for the fault to clear itself, and then everything's back. The operation will be reported to the dispatch center, and someone may come out to inspect the area in case there's damage.
But if the fault is still there, and after a moment the current is still way higher than it should be, the recloser opens again, and this time it stays open. Gonna need a crew to move that tree, and then they'll manually try again.
If you want a poors man home server just repurpose old laptops. I do with a bunch of them and it's fine. I know laptops are not for that, but they wouldn't have any other use anyway, and it's like a server with an UPS built in.
Most tasks are running scrapers or data wrangling so I don't have to keep my desktop running.
I've been doing this for maybe close to two years, and for now it works. I wouldn't recommend for VMs though.
Also, your router needs an UPS too but it will be fine with a cheap one.
I was considering this but then I stumbled across some ~7 year old micro towers. Those little thin client boxes that were popular for a short time as a thing that can connect a keyboard, mouse and monitor to a corporate network.
Refurbished with a Windows license (if I ever want to use it) was less than a surplus laptop. Power usage seems acceptable but I'll compare with a Kill-A-Watt. I have one on the way and if it works out I'll grab a few more.
I do like the idea of a surplus laptop cluster, mostly because of the built in UPS factor but there's a lot of wasted hardware there, such as the monitor and keyboard. But I suppose that's kind of a feature too.
I have shower thoughts of some kind of custom bladecenter made of old laptops but that's a lot of work for probably zero benefit.
Another upside is laptop chips suck less power but still perform ok in most use cases, especially without a graphical load. The monitor and keyboard allow easy interfacing if needed plus ports are often convenient too. even just eSATA for more storage.
I run a uSFF HP box for my hypervisor and a low power J5005 core in my NAS and am more than happy with the performance let alone minimal power usage. Sure the J5005 system takes ages to update requiring a backup DC/PiHole setup on the HyperVisor but it just works at low cost.
To me the main problem would be the noise. I don't have a dedicated server room and even if me and my wife's bedroom + living room is quite big it'd still be annoying during the night. But I am quite fond of the idea to reuse old laptops regardless.
Can you post a link to those 7 year old micro towers, please?
Yeah noise is why I didn’t go with surplus servers, and the power requirements. Proper servers are actually pretty cheap, less than these tiny towers even. But the power and noise is just too much. Plus less horsepower is actually more interesting to me so my workloads have a hope of hitting scaling limitations.
They’re available in a bunch of configurations and on multiple sites so you can look around for a good deal that fits your needs. I found one with an SSD for additional power savings.
There are advantages to this, but do note that earlier core i* chips had earlier versions of QuickSync that work, it just wasn't as good as new versions (5th gen and newer).
Plex can use nVidia cards for hardware acceleration for encoding.
In regards of power efficiency I'd still be drooling over one of the Xeon D-1600 configurations but haven't found one that's not in a rack form factor yet.
As a router, it's quiet in a desktop environment, unless the CPU is fully loaded. For a bedroom, silent is better than quiet, which could justify investment in a fanless case for mini-ITX. These are not cheap, e.g. Streacom or Akasa.
Broadwell NUCs are available on eBay and Newegg has a matching Akasa Plato X case for $100. With an SSD, that has no moving parts. But only dual core, single NIC.
In most cases the beefiest air cooler + the lowest tolerable fan curve (meaning no overheating) will do the job just fine, especially if the CPU TDP is between 35-65W. Alternatively you can also limit the CPU power usage by disabling turbo or doing other tricks that forces it to run slower, but more efficiently.
Woah these are really cool. $$$ for what amounts to bling in my case but I might still be interested in something like this. The NUCs are great but I don't see many of them on the second hand market. At least not as many as these thin clients.
Different gear, but you’re starting to see HP t630 thin clients on the market for <$100. They are silent and have standard m2 interfaces for storage and sometimes come with a bunch of memory.
Noise can be lowered cheaply on adapted laptops with a little work. Re-paste, cut out the fan grill plastic, keep the HSF assembly clean and fluff-free, run the fans slower. Keep the unit elevated on feet.
Replace the fan with a Noctua-type quiet model bolted on. If the HSF needs that airflow directed, hack a simple shroud (or 3D print a replacement.)
I've been running a 2008 MacBook Pro as a server for 10 years. I've replaced the battery once, about 2 years ago, and one of the fans about six months ago. Other than that, it's been on the whole time.
What OS do/did you run on it? Thinking about repurposing an old 2011 Mac Mini as a server, it should still be more than capable, but since it stopped getting newer macOS versions a few years back it wouldn't get security updates, that kinda worries me.
Just checked, it's actually a mid-2009 MBP and its running 10.11.6 - it's not had a security update since 2018, which yeah, that is a little worrying. Might wipe it and install Ubuntu if there's such a thing as Firewire800 drivers for it.
Well, used to be. As I understand it newer laptops are more and more often using pouch style batteries to keep things thinner by using any available space.
They're just general computing devices. With the right software (and/or some hardware) you can turn them into whatever you want. CCTV, web server, door/gate intercom, electric meter reader/submitter, NAS (maybe with multiple drives using a powered USB hub), Wifi AP, media center, desktop computer, etc
I run a Raspberry Pi 4b, recently upgraded from a 3b+, and I believe that a lot of personal computing could (and maybe should) be run on them. They're quiet, energy efficient, and you could probably run them on a battery if you needed it.
I wouldn't run VMs on it, but I do run docker on it, and that works just fine.
> I think the pi is not particularly power efficient.
While there might be efficiency gains available, RPIs use a very small amount of power compared to a laptop. Idling a RPI4 uses 2.8 watts[0], which not only rates favorably compared to most laptops, but is far below what my charging cell phone uses. Maxed out my RPI can only hit 15W, since that's the maximum power that my official USB C adapter can output.
Now that being said, it does depend on what kind of load you're expecting to handle. If you're constantly maxing out a RPI, it is probably more energy efficient to purchase a larger server than to just keep adding RPIs. But if you're staying well below the theoretical max of a RPI, it'll consume far less than a used laptop.
> I think the pi would benefit from a more robust filesytem layout such as an overlay filesystem to allow continuous writes to /var to go to ramdisk.
This is one of the biggest drawbacks of the RPI; microSD cards cannot handle a ton of writes without frying. I've settled on using log2ram to ensure that logs are only periodically flushed from RAM to the SD card, to extend the lifespan of my cards.
Watts per low traffic website it's very efficient. CPU is generally not pegged at 100%.
For watts per CPU cycle not as much, I agree. Say it takes 10W at full power (I've not measured). That makes it 1/10th of a normal CPU. But it's like 30-40 times slower.
what's uptime like on an Rpi? I want to start working on a home website and have an old rpi laying around. But I've read that their reliability isn't great since they use SD cards. Any advice?
What I've done with my RPi 4 is attach an external SSD to it and mount it to /home and /var. This reduces writes to the SD card dramatically. Lengthening the SD card's life. I have serious uptime on it, almost a year (since I got an RPi 4 basically). It only shutdown with power failures (only one, which was entirely my own fault) and system maintenance (OS upgrades and other tinkering).
I've also imaged a version of that SD card, so it would be easy to just swap out and probably get things running again without much hassle.
I'm sure there are more fancy solutions with overlayfs, making the SD card read-only except for upgrades. Though I couldn't find a nice resource on it at the time.
My Home Assistant is running on a Raspberry Pi 3B+. It has been running on the same SD card for two years now, with a large sqlite database and with debug logging enabled for Z-Wave so a fair amount of writing to the card.
It has only been down when the power has been out, and so far no issues with the card. It is however an A1 or A2 class card (I forget which), so supposedly design with applications in mind.
It should also be noted that a lot of issues with the Pi is due to poor USB cables / power supplies. I had another Pi that kept crashing every few days, until I measured the USB "charger" cable I was using for it. Turns out it had a resistance of almost exactly 1 Ohm. So if the Pi drew say 1A, that would be 1V loss in the cable... After I swapped cables it has been rock solid. A key point here is that a lot of SD cards do not like losing power while being written to. Avoid that and a quality SD card should have a long life.
Just dont have heavy writes on sdcard. I use external hdd for it.
I have a running rpi for more than half a year. No issues whatsoever. But again - I have a very stable electricity and I dont remember last time it was cut not due to planned maintenance.
My rPi3, running Arch, has an up time of close to 3 years on the same SD card. It runs various Docker containers including 1 for MiniFlux. It's mostly idle but when I need it, it's been reliably there for me. Maybe, I've been lucky.
I don't have numbers, stats handy but in my couple of years experience with Rpis - both regulars and the smaller Ws - the SD cards fail well before the Rpis do.
Just because I already have it and it's low-powered. I prefer starting projects out with 0-cost methods, and I already have an rpi laying around somewhere
That is a good point. However one of my goals/constraints is that I want my home site to be able to connect to other personal devices, e.g. a NAS (another WIP project). My ideal vision is a simple site running on my network, backed up on some kind of cloud storage.
Any suggestions? Though I'm a software engineer, I've worked mostly in app dev rather than infrastructure setup. This project is a way for me to learn more about the latter. The rpi is the easiest and cheapest option, but I have wondered if it wouldn't make more sense to use AWS or something like that. I wouldn't mind paying a few bucks a month, but more than that and I'd rather wait until I've learned a bit and know what my specific needs are. I've also considered an actual computer, but figure it's a waste of power when my initial work will probably just be figuring out the server config, HTML pages, etc.
>That is a good point. However one of my goals/constraints is that I want my home site to be able to connect to other personal devices, e.g. a NAS (another WIP project). My ideal vision is a simple site running on my network, backed up on some kind of cloud storage.
You could host everything on your home network but there are a couple of things you should probably know before you start: 1) Some ISP do not give you static IP, meaning its extra work
2) Security - Ideally your home network has some kind of segregation, you keep your trusted devices on one, IoT or not patchable on another for example, with firewall blocking connections from dirty network to clean one. Having externally facing website potentially having direct access to personal devices and services might be an issue from security point of view.
You can sign up on AWS/Azure or any other cloud provider for free and get free credits, which might be enough for you to learn. Feel free to ask further questions if you need to
Why not? I have a laptop with a 4th gen i5 that has VT-x. I've upgraded it to 16 GB RAM and an SSD. For now, it's a Spotify connect player but I'm contemplating running a couple VMs on it too, like pi-hole and whatnot.
Sure, that’s still reasonable, but it’s not young.
I’m also using similar class hardware for my VM server at home. Works well enough for home workloads, and for everything else there’s the cloud on demand.
I have an thinkpad x220 with some core-duo on it and 4GB Ram, can't remember, and it really was struggling with VMs. Didn't really cared much to check the specs. Just tried some debian VMs and it was less than satisfying.
Been running a 2010 MBP for the last few months hosting bitwarden, a samba file share, a DNS server, and a ton of other stuff. It's been pretty great so far, so much so that I dug out another old laptop and set it up as a pi-hole for a friend.
Have you any suggestions for increasing the amount of storage one could address? I can't find any Thunderbolt 2 or FireWire enclosures so I'm sadly limited to the two USB3.0 ports. Right know my plan is to resort to putting a high-capacity 2.5" hdd inside the MBP and adding a usb hdd in addition to the one I already have. But that doesn't strike me as ideal for some reason.
Yeah I just repurposed a 2015 MBP as a desktop now that it is painfully heavier than my 2018 12" Macbook. I just found out that it could do 4K60 through the display port, and as a desktop machine it's great.
Then the next level of slipping down the chain is the headless server or timelapse photo taker. :)
This has served me well for a year ($80 Dell laptop w/ an i5 from Craigslist) but I did just splurge on a Synology NAS and an APC UPS because having multiple external hard drives plugged into the laptop was quite dicey. The laptop is still a Plex server but now I have peace of mind for the data.
I'm curious, with shared servers at Digital Ocean available for $5 or $10 per month, why run a home server anymore? What is the value to having it local?
I've been considering this calculation for a year or so now. I spend an average of $20/mo on small VMs for testing (including the occasional $1/hr VMs for a couple of hours for heavy loads).
I thought I should replace my 2012 Mac Mini (HTPC) with a modern Hackintosh and 6c/12t or 8c/16t and do my testing workloads on that (or even just use my current 2c/4t for small workloads). But it's just so easy to spin up DO droplets within seconds and do so programmatically. Plus you can snapshot and create droplets from those, and turn on backups if you need to.
Traders still seem to be the most gullible when it comes to buying unnecessary hardware.
They really think their prosumer workstation is going to give them an advantage.
I guess thats one silver lining about Robinhood’s proliferation, many people know its good enough. Gullible traders still misread options settlement UI but seems there is a selective evolution at play to make that less common too.
Given how unoptimized frontends for crypto exchanges tend to be and how rapidly the trading volume could spike in these markets, actually it might make a difference.
* Back-UPS: inverter offline, expect a 15-20ms power transfer
* Smart-UPS: interactive, inverter is always online 2-5ms trasnfter
* Symmetra and above: inverter online, voltage conditioned, 0ms transfer
Most desktops will work just fine with the Back-UPS line. The processor is practically sleeping. Workstations should be on a Smart-UPS because the PSU capacitor won't have the capacity for a brownout with a Xeon-class CPU and multiple GPUs.
> Workstations should be on a Smart-UPS because the PSU capacitor won't have the capacity for a brownout with a Xeon-class CPU and multiple GPUs.
Also make sure it's rated for the load. My UPS went "lol nope" when I lost power while I was gaming, because I had forgotten about it when I went from single mid-range GPU to SLI top-end GPU...
Lots of users have desktops, monitors, chargers, and even heaters plugged into our UPSes. To them it's just a chunky power strip. They're lucky if they can squeak out more than 5 minutes on battery.
Hold-up time is one of these things (besides long full-load life) that separates cream from crop in power supplies. High-grade ATX power supplies don't just promise 25+ ms hold up time, but actually deliver.
Some better review outlets test for that, since you can't rely on PSUs actually managing the spec-required 17 ms (I think that's the number, something in that ballpark)
I had 3 hard drives (likely) fail because of brownouts regularly occurring at a residence when high-amperage devices kicked on (power tools, air con, hairdryers, etc). The line would drop into the 80-90v range (120vac) which wasn't enough to turn anything off, but you could notice the lights "flicker." After a voltage conditioning UPS was added, haven't had a hdd failure
For situations where a UPS is too expensive or impractical you can also buy pure voltage regulators that use some kind of autotransformer, like the APC Line-R. I worked for an institution for a while that had every laser printer on one, there had been some bad experiences with losing laser printers to overvoltage incidents but they generally draw too much current to go on a UPS. I've also seen them used for A/V equipment.
I have the next step up from this, the 1500 VA. I assumed the UPS under my desk was the Smart-UPS series but I can see it is the Back-UPS Pro. I haven't been in the office in months!
Looking at the schematics of some old models of BackUPS I found online the output relay needs to be powered in order to supply output from mains. Combine this with slightly funky internal power topology where +12V for internal logic comes mostly directly from the battery and it is pretty clear why battery with shorted cell(s) causes the thing to completely shutdown (probably even due to thermal shutdown of the LM317 in charging circuitry).
It's more complicated than that.
There are Smart-UPS online. Thats double conversion also 0ms transfer
Some of the "newer" online models will have a green mode, which although double conversion green mode introduces a transfer time (~2ms, basically relay lift time).
Backups are also square wave, or stepped sine
Smart are Sinewave output.
This is specifically APC.
Also remember there are tare losses (copper / magnetizing and conversions), so a UPS just sitting there can use up to 30W.
Back-UPS stepped sine wave has about 3 square steps above or below zero, so it's a very rough approximation of a sine wave. Adequate for many kinds of equipment, but ugly on a scope.
There is, or used to be, a pin on the Smart-UPS line that you could tie high (or low?), to tell the device to operate in "Online" mode (inverter always active, line power continuously tending the battery). APC also sold it preconfigured that way as a separate, more expensive, product. :)
There's also a trick to jump-starting an unplugged-but-charged Smart-UPS. I've used this for several hours of remote power in inconvenient locations.
Quality devices, but test them regularly. The switch from idle to high-draw can be violent, and it takes out weak components. Similar to tungsten lightbulb filaments.
Oh, I know very well it's complicated. But that is the 500ft explanation I give to the beancounters, "why are we using these expensive UPSes when we can get them for $100 from Office Depot."
I'm most familiar with the APC line because that is what I use.
> the PSU capacitor won't have the capacity for a brownout with a Xeon-class CPU and multiple GPUs.
I'm not sure I buy that. The motherboard/GPU backplane VRMs alone should be able to buffer 20ms on these devices. Would love to see a test of this kind of thing.
I live in a place where blackouts are very infrequent (perhaps a couple per year)
Ouch. Does anyone know where the author lives that a couple of times a year is considered "infrequent?"
Decades ago I lived in a rural area of America where we'd have power outages two or three times a year because of storms. But I've lived in more than a dozen cities since then, and the power has only gone out twice. Once about a decade ago during a tornado, and then again last year when the power company replaced a transformer across the street.
I ask because it's my impression that electric service has gotten much more reliable over the years. There are still access issues where I live now (believe it or not, thousands of Americans don't have access to the electric grid at home), but reliability seems improved. Am I living in a bubble when it comes to electricity?
I live in florida. And in an area where most power is overhead.
Storms during the rainy season usually means i experience cuts. Usually 1-2 times a month for 3-8 hours ea from May-Nov
Hurricanes are usually expected to be 3+ days at minimum of outage. Most recent ones were closer to 14 days.
I have a couple gennys, and interlock kits with 30a hookups on my house during these times. I can run 1 genny on the whole house, or even 2 since i have a sub-panel with another interlock kit.
The only thing i CANT run is my AC. But we have 2-3 window units/portables to get by.
FWIW running generators for days at a time is NOT cheap. on my setup its about 20-30 bucks a day in fuel alone.
Lived in Florida for about two years. We’d lose power due to the neighborhood transformer exploding a couple of times a year. That must have gotten expensive real quick.
The quality of the residential wiring in the original half of the house was not friendly to my hard disks. I lost four in rapid succession. Got a newer UPS, line interactive, had better luck after that until we moved out of state.
(That month of disk meltdown made me love ZFS. Didn’t lose any data.)
| FWIW running generators for days at a time is NOT cheap. on my setup its about 20-30 bucks a day in fuel alone.
What type/size of generator are you running? Might be worth taking a look at an inverter unit next time; they're much quieter and sip gas. Can't stress the quiet part enough - you can hold a conversation a few feet away easily.
I have looked at inverter generators. But they seem harder to find in larger wattages/capacities. I dont know that i trust Champion engines for long runtimes (days)
I currently have a 7kw (8750 surge) and a 6kw unit. they are dewalt gennys with Honda engines. I really only would trust a B&S, Honda or Kohler engine i feel like.
I think my next purchase will either be a bigger inverter or just a 14-17kw contractor genny and switch to double conversion Battery Backups.
Not quite sure. But what i have does work too. Just that my current generator is approaching a decent number of hours (400-500).
Floridian here as well. Our lines are buried, but the feed is above ground so if a tree takes that out, a couple thousand or so of us have no power for hours. I've thought about a generator, but I can usually keep working on battery (laptop and phone) so, I've not yes seriously considered it.
Even with my roof (which is east/west facing) the cost to just install panels would be 20-30k. And my city has a 1:1 buy back...And even assuming i sell them back power it would take a decade to recoup the costs...and by then the panels would be less efficient. And this is with buying 95% panels.
but then during an outage....
So you need something to store it... Short of rednecking a series of deep cycle batteries the Tesla Powerwalls are really the only big option.
And those arent cheap either. To run my house for 2-3 days would be in another 10-30k
I can do all of that with 2x7kw gennys and some ancillaries. With a 14-17kw i could probably run my AC (which have hard start caps on them) Even at 30 bucks a day in gas im still well south of even 10k.
Next step up would probably be a active standby generator with transfer switch and giant Propane tank to fuel it. That would be about 10-15k once permitting and all is done. And that adds complexity and cost. But it would mean i dont have to manually flip breakers and cutover. (as it is, my computers are just set to talk to a NUT server, and shutdown in the event i dont cutover within 10 minutes.)
And during storms supply chains are strained. Its hard to source diesel and LP and sometimes even petrol (in fact i warned a previous company of this, and we almost had to shutdown due to fuel levels). And you have to maintain a contract with LP providers and usually rent a tank from them.
With petrol and my little standby units i can source it myself (3-4 am is the best time during runs on gas), even siphon from one of our cars.
I may spring for a full kit standby one day. But for me, it costs about 2k in generators. Another 1k to have an electrician install the interlock kits. And i can service/manage the gennys myself. swap them out while i do maintenance during extended runs etc.
They are panels with slight manufacturing defects that you can buy and wont have a full lifetime or near the normal efficiency. So are less expensive as well.
They used to sell them last time i was looking into it, which was about 3 years ago.
I've had at least a hundred times more power outages due to protection devices tripping (usually true positives) than the grid going down, so spending 20 minutes to figure out which socket would be the best choice to hook all the communications stuff up to is a much better investment than buying an UPS. FWIW, I had more downtime with the UPS than through actual grid outages because the UPS didn't like a battery hotswap and powered off, or because the batteries started to gas off hydrogen sulfide, which required disconnecting them immediately, but the UPS then powered down (without a battery) and couldn't start without one, either, so it required rewiring a little bit to get going again.
Edit: My UPS is a line-interactive rack-mount APC unit, which as I understand things don't actually contain a power supply, so everything runs from the battery voltage and is supported by the charger in mains operation, but without a battery inserted the charger doesn't work, because its control circuit is powered by the battery.
It is highly variable depending on the location. I know of two electrical grids in the Bible Belt, less than 10 miles apart. One has a power outage every year or two. The other, 10x or more per year is not uncommon.
I've lived in many different states and cities, and "it varies" is about the best you can say. Areas I would expect to be relatively problem free, end up being some of the worst offenders (looking at you, El Paso County, Colorado).
Germany here. I'd say from personal experience maybe once decade. That includes living in a village of ~3000 for eight years. One time I remember was caused by an excavator where it lasted several hours, the other instances were rather short.
I guess not having earthquakes, floods or hurricanes helps.
The other items at least in the US midwest would be ice storms (ice coating power lines making them heavy and branches getting too coated and falling on power lines) and thunderstorms (50+ mph wind coming through bringing down tree branches on to powerlines) or lightning strikes hitting power transformers.
It's a lot more common in Europe to have buried lines for anything after the high-voltage national backbone, so the grid is much less susceptible to weather.
Where I lived in Sweden, 30% of the network was already underground, but after a large storm in 2005 caused long-lasting power outages, they invested heavily and are now up to 70% underground.
| Am I living in a bubble when it comes to electricity?
Your electricity service may not be as reliable as you think - some utility power can be quite "dirty" with high levels of distortion. There's a lot of infrastructure between you and the power company that can cause problems. A good UPS will not only protect against outages, but will filter/condition utility power.
I live in Chicago, in an older apartment building. I haven't had a full on outage in the past few years, but during major storms I have seen some flickering and other issues. I just looked at the readout on the UPS, and so far it's kicked in 5 times this year. I have a pretty sizable NAS (40TB) and a beefy workstation - the $200 I spent on an APC BR1500MS is well worth the protection.
Safety is another consideration. My parents live on their hobby farm a few hours away - they are the only house at the end of a mile long road with older power lines. I put a similar UPS to keep their alarm/phone/network online. If there's a medical alarm, fire, carbon monoxide, water leak, etc the alarm can still alert the monitoring company, their internet/VoIP landline can fail over to cellular, and we can get alerted that something is up. Again, the peace of mind alone is well worth the $200 (plus $100 for a battery every few years).
Forgot to add, I was living in Virginia during the June 2012 derecho blew through during a period of 100 degree weather (https://en.wikipedia.org/wiki/June_2012_North_American_derec...). I was without power for an entire week - even in a nice subdivision with buried power lines, in a city of 100K+ people. UPS won't help for that, but it did automatically shut down the NAS and kept my modem/wireless router going for a few hours. Bought a 2,000 watt inverter generator for a few hundred dollars. If you have somewhere to store it, highly recommend purchasing one. Extremely quiet (~50db), sips gas (1 gallon/12 hours), and enough power to keep a fridge, small window AC, and lights going. Even managed to run the dishwasher. https://wenproducts.com/collections/inverter-generators/prod....
Agreed, compared to most developed countries, a couple times/year would be extremely high. But, it's not relatively high for America. I remember living in the midwest and every major storm usually resulted in power lost to somebody. Many of my neighbors had (Generac) generators that would kick in automatically when the power went out. They're so common there that home depot stocks them on the floor.
In Southern California, we rarely get rain, let alone major storms and we still lose power 1-2 times/year.
On the other hand, I lived in a number of areas across Canada and power outages were quite rare despite the harsher climate.
I have a UPS mostly because my laser printer sometimes trips the stupid arc fault circuit breaker[0] that gets false alarms incredibly easily. Since my apartment is new construction it has AFCI on every circuit. The TV also used to trip it and maintenance replaced that circuit’s breaker and it’s stopped, and I’d probably get the other replaced too if not for COVID requiring all our maintenance requests to be emergency-only.
AFCI’s are such a horrific scam. Imagine spending thousands of dollars to fill a breaker box with less reliable breakers.
I had AFCI trip on my fridge after renovating my kitchen while we were away for a weekend and ruined the brand new floors when the ice box melted. So infuriating.
I’ve since pulled the AFCI’s out of all mission critical circuits and sold them on eBay.
Just an absolutely disgusting example of regulatory capture.
Here in San Francisco's Mission district the power used to go out every time it rained heavily (2-3 times a year for a period of 5 years or so.) Seems to have been fixed a year or so back. Heavy rain in SF isn't even all that heavy, though the roads do become almost impassable since they're not built with a gradient.
In other cities I've lived in power cuts were a once every two or three years thing and you could still drive on the roads when it rained.
My power goes out all the time in Denver. In the summer, with no adverse weather. The grid here is a joke compared to most of Florida which is FAR more resilient.
Few times a week in summer and twice or so weekly in winters here (India). It's not that uncommon in developing countries to have outages. Many of them are also arbitrary (not related to infrastructure failure) but saving cost, political reasons, social events, etc.
When our office was in Santa Monica, we'd get short (10 second - 2 minute) power outages several times per year (5-10 times), and real 1/2 day long outages about once or twice per year. Several days we were told to go work from home instead.
Canada here, metropolitan area. Power outages happen about once a decade, usually as a result of a wind storm. I have had zero server downtime due to power outages, but UPS / battery failures occur at about 4x the outage rate.
I have power failures a few times a week up to a few hours each time, living in a capital city of an European Union country. Having power failures a couple of times per year is "extremely infrequent".
Over the years I have lost a lot of confidence in UPS units which do not use double conversion topology.
I have a fairly expensive line-interactive unit that can be tricked into doing bad things just by power cycling my laser printer which is plugged into the same circuit (but not the actual UPS). One day this caused me more trouble than it was worth WRT lost work. I would have been better off plugged directly into the grid and just taking the 100 millisecond voltage dip at the PC's power supply. This is one area where buying high quality PSU can go a long way. High wattage Seasonic units can ride through some pretty nasty conditions before needing any outside help.
I also have a cheaper passive-standby modified-sinewave unit which has never misbehaved in terms of trigger conditions, but it is hot, noisy and otherwise does not inspire any confidence when it's running on its inverter. I would also never dream of plugging anything into this unit that uses a non-switching power supply.
Again, I am faced with the reality that you get precisely what you pay for. I always assumed double conversion was expensive paranoia never for home use, but over time this has proven out just like everything else when looking at the value equation of technology.
I too use double conversion UPS. Liebert Vertiv GXT4 in my case. The power in my neighborhood is sloppy and the power company refuse to replace the transformer until it goes out entirely. The neighbor has a septic pump that creates nasty spikes. I've had multiple APC UPS literally catch on fire. It's always the tower models with the real time voltage regulators that catch on fire. The floor power strip models have been rock solid, but they would beep every time the neighbors pump kicked on. I tried power conditioners, filters, isolation transformers, but in the end, it was the double conversion UPS that cleaned everything up and the Liebert has provided reliable power ever since. Some day I want to go solar, but I would have to move somewhere else first.
Man. Ive got like 3 of those suckers all of them with fairly decent usage. Been needing to swap them out. Two of them are laying on their side stacked.
> just by power cycling my laser printer which is plugged into the same circuit
So I encountered in a previous IT helpdesk job a laser printer connected to one of the tower APC units, with a wattage display.
Laser printer (it was a Ricoh Aficio SP 4100-something MICR printer for checks IIRC) was connected to the battery backup set of receptacles for some odd reason.
Any time the laser printer would start up, or run, and I presume charge the fuser, it would briefly spike up to 900w and often set off an alarm. It was rated for 550w. I think sudden high loads like that could be damaging to the unit.
I would not attempt to connect a laser printer to a UPS at all, even if not connected to backup battery--IMHO it's in the same category as high-current devices like toasters and hair dryers. Many commercial copiers I've seen have some type of large surge protector but no UPS.
Laser printers plugged into UPS's aren't great (and then there was my previous boss who destroyed a pretty large unit by repeatedly boiling a kettle from it during a power cut), but it's worth pointing out that the OP had his laser plugged into the same circuit as the UPS but not onto the battery - I took that to mean it's on the same radial/ring/breaker rather than connected to the UPS at all.
I'm guessing the surge current to the laser was causing the UPS to register a brownout and potentially not behave correctly at that point.
My Cyberpower 1500VA actually says not to plug laser printers into the Battery Load side. It's rated for 1500W, but you know it won't work when the manufacturer says it.
My own UPS system is focused around keeping the telephone system and one DSL modem up and running through 2 days without power. That way if there is a bad thunderstorm there is no panic about working at home, I can use a laptop and tablets and communicate as much as I need.
I have a PC tower server and realized off the bat that it would not be feasible to supply it with backup power for any worthwhile period of time. If the power goes out it will crash, but it runs on ext4 and ZFS and odds are very good it will come back when the power does and put itself back together.
I’m with you on zfs (although it is a pain to restore overwritten metadata, that’s not a typical symptom of power/controller failure) but please change ext4 to jfs or xfs. Xfs has a bad rep from the early naughts in terms of fs corruption but actually addressed the underlying issues and today both those options are equally great.
I’ve tested all three extensively with unreliable media (raspberry pi kernel dev testing on an sd card) and ext4 was the absolute worst.
All. Even in the tests where the SD cards never physically failed, their different write patterns and flushing guarantees exposed underlying issues in the file system. You just don’t encounter them as often when working on systems that typically align with the assumptions the fs developers incorrectly made.
NILFS2 or F2FS, the latter is much more heavily tested as it is now the default on a number of extremely popular Android phones.
SD cards don’t perform wear leveling so you really need a file system that minimizes/distributes block rewrites, and a log-structured filesystem will do that by never rewriting old data (until there is no space left).
You should also consider about SSD's reliability without PLP capacitor. NAND flash is erased by block and it runs GC silently so possibly even non-touched data could be corrupted if power loss.
I wonder that most consumer SSDs are aimed to power loss safe (by implementing like CoW?) even without capacitor but I don't know the truth.
> I wonder that most consumer SSDs are aimed to power loss safe (by implementing like CoW?) even without capacitor but I don't know the truth.
The flash translation layer in a SSD looks a lot like a journalling or log-structured filesystem, and without power loss protection capacitors you get similar data integrity guarantees: unexpected power loss may mean loss of recently written data (still in the drive's write buffers, and maybe stuff in partially-written NAND pages), but shouldn't cause catastrophic loss of data. You generally have more data at risk from being buffered in main system RAM by the OS than you do from the volatile/unprotected buffers in the SSD itself.
The SSD should never be erasing a block as part of a garbage collection process until the live data from that block has been committed to a new block.
Is there a term for UPS supplies for this purpose?
I want one for work to keep the network and internet devices running, but it needs about 30 Watts continuously for say 24 hours, not hundreds of Watts for 30 minutes like a UPS for a server.
I presume a "trickle" power UPS would need to care about power efficiency?
I don't need days of power, but I wanted a few hours. I ended up getting a Belkin Battery Backup REV B, BU3DC001-12V off of ebay and modified the barrel connector to fit my cable modem and router. I can get at least 3 hours (I haven't tested it longer than that). I'll probably get another while they're still available.
As for my server, I have it on a UPS big enough to allow it to shut down (with some slack for battery degradation). That's all I really need.
If you want to run a handful of low current DC devices for an eternity and in a rock-solid manner, just get a DC-DC converter for each and a few batteries+charger. You can keep the devices permanently looped in on the battery while it is on trickle charge. Think of it as a car's electrical system in your network cabinet, replacing the alternator with grid power.
If we are just talking about cable modem + wifi router + NAS, you cant be looking at more than 30 watts. Buck/boost DC-DC converters are upwards of 98% efficient, so you don't have a bunch of losses with that extraneous inverter/rectifier stage. All of this stuff can be purchased and assembled with minimal fuss.
That's why I went with the Belkin, it isn't a "normal" UPS. 120V comes in and is stepped down to 12V, but there's no up-conversion the other way; the output is 12V (up to 3A) to a barrel connector. I bought a Y so I could energize my two devices (after making sure combined they don't exceed 3A).
The cheap UPSs aren't very good and don't work well or at all with modern PCs. I've had several of these from Cyberpower that haven't let me down. Far more reasonably priced than APCs equivalents.
Seconding Cyberpower. Even their cheaper ones do better than BackUPS and don't fail to pass mains power along if the battery fails a check. The PFC Sinewave you linked do some nice power conditioning. It's a big help for me when my house switches to backup generator.
(Why yes, I live in California, land of multi-day power outages. Why do you ask?)
I like Cyberpower for home A/V racks, and yes, that PFC Sinewave line.
Generally the rack mount models, such as CyberPower OR1500PFCRT2U PFC Sinewave UPS System, 1500VA/1050W, 8 Outlets, AVR, 2U or CyberPower OR2200PFCRT2U PFC Sinewave UPS System, 2000VA/1540W, 8 Outlets, AVR, 2U.
Very happy, and stopped having to replace sensitive flat screens in bad power zip codes (trees vs. lines every other storm, such as for the last week after Isaias).
Same here, I've had no issues with the 1500VA version. It's line interactive so should still work if the battery dies, but I can't say that I've tested it.
I don’t know if their newer revisions are any better, but I bought a CP1500PFCLCD in 2011. When the battery died from old age, it cut power to all of the battery backed receptacles until I power cycled the UPS. When the charger failed a few years after that, it did the exact same thing.
But the main point in the article is worthy: UPS manufacturers have been skimping somewhat on functionality, features and quality over the past few years...
No, the article is wrong: the low end UPS devices are bad on quality and functionality and lead-acid batteries in UPS devices is a lot shorter than some people would expect.
I still have a few APC Smart UPS's bought in 1999-2000 running in my house; at the same time I had a few cheapies that broke after a few years, sometimes in spectacular ways (electronic exploded melting part of the battery case); I will not name brands, it is not worth.
Since 2000 I changed cases full of batteries, usually every 2-3 years. The good thing with the APC Smart series is that it works even if it gives an alert that the battery is bad, but you will get a few minutes of power instead of 30-60 minutes (my server & NAS is very low power, 5% of the UPS capacity). I also set the server to shutdown when the battery is reaching 50%, so I never deplete the batteries. At the same time, the UPS on the water pump is usually running on batteries as much as it can, so I need to change these batteries after 2 years. I have both the server and the water pump in the basement, the temperature is around 16-20 degrees Celsius (winter-summer), when I was keeping the server and the UPS in the house the battery life was at half (~18 months). The heat and the deep discharge cycles are effectively killing batteries.
I had the same type of battery on my motorcycles, one lived about 8 years and a second one 6 years, but in very different conditions: it was warmer, but almost no discharges.
Deep discharge is a factor, but not a big one, heat has very little effect, it only affects voltages at which battery will be considered overcharged or deeply discharged as specs are usually all for room temperature.
I've used UPSes that discharge batteries no lower than 10.5 (no deep discharge) and only once in a couple of years and that charge them to and keep them at 13.6 volts under stable room temperature all year. All batteries lost most of the capacity after 3 years, two almost all of it, one had like 40% left. In five years only that one was still surviving short minutes outages, while initially was able to do a few hours. For comparison, I had one battery just laying around for 6 years in the same room without touching it and it lost only like 30%.
It turns out that APC intentionally overcharges the batteries to maximize runtime during the 2-year target battery lifetime, at the cost of killing the batteries relatively young. And the overcharged voltage drifts upwards over time.
I have a horror story with 700VA APC ES-B700G, I bought it for my NAS in case I would be updating its operating system not to have it turned off by random power loss.
I had issues with power supply to my apartment once in 2 years or something like that.
This model has "controlled by master" outlets which I did not use because I needed only backup lines connected to battery. That "controlled by master" thing would turn on once in 3 months without reason.
Then one day there was no power loss and UPS flipped and turned off power to my NAS and to my devices. Battery was good power was there but switches flipped and my NAS was turned off without warning. There was no system updates, maybe just normal writes to the disk.
Turned out my HDD WD-RED 1TB died because of that one flip. It would have been better for me not to have a UPS...
It sounds like you used a Back-UPS unit instead of a Smart-UPS, which I’ve heard switches to battery 10x faster than the Back-UPS.
After learning this I won’t use a Back-UPS for anything where meaningful data loss might happen. They’re great for lights, cable modems, WiFi routers, etc., though.
Thanks, it was just a faulty unit that I trusted too much.
I was doing turn off tests with my NAS attached and it was switching correctly and NAS was not going down, I had couple RPis connected also not going down and notifications from NUT were in logs. Where my NAS was NUT server and RPis had NUT client.
It was that one day when UPS malfunctioned in a bad time. Where my electricity provider was more reliable than UPS I owned. So bad risk management on my side.
What you want is any type of UPS that is line-interactive (Smart UPS or BackUPS Pro), those convert line voltage to 12V and back to line voltage with the battery in between instead of having to switch the battery into the circuit. Basically means that it switches (almost) instantly and they don't depend on external voltage or frequency.
I had an APC UPS battery age out on me, after only a couple of months after purchase. Back-UPS XS 1500.
I called their customer support, and they asked about the kind of equipment I had plugged into it.
My main NAS motherboard had died, and I hadn’t gotten around to the full rebuild that was going to require. So I had ended up with about five or six cheap USB external drives, with those “wall wart” DC power supplies, strung along a couple of power strips that were plugged into the UPS. That plus a 27-inch iMac.
It was generally not much power - not much surge to spin up, or to power up the Mac, and of course at idle was less than 100 Watts.
APC Customer Support said that the power strips might actually be weirding out the battery.
They sent me a replacement battery at no charge, and suggested that I go with a Smart-UPS, with their Power Distribution Units if I needed more outlets.
I had always thought the rack-mount PDUs were a silly expensive replacement for a no-name power strip. But I had long had a keen interest in this sort of IT equipment fail, after witnessing some spectacular crazy failures in corporate, and losing the manuscript while I was writing a book on small-office IT practice.
So I bought a basic PDU, and a Smart-UPS rated at something like 700 VA.
That has been rock solid. Even better than a larger setup I had with some big CyberPower systems.
I’m willing to go along with the cargo-cult of PDUs now. I would be quite surprised if there were any significant difference among the basic PDUs of well-known brands.
I still have those USB drives. They’ve lasted about as long as the hamsters, not as long as the dogs.
"APC Customer Support said that the power strips might actually be weirding out the battery."
I think it's more likely that you purchased a guerilla-remanufactured unit that was properly sealed and packaged to look brand new (with hologram stickers and anti-tamper seals, etc.) but had an old, already dead battery that was reconditioned just enough to last a few cycles after purchase.
I have 4-5 hours outages every day, I use 2 APC ups to keep my modem and router up, while I still work with my laptop.
It works perfectly for me, these APC devices have given me at the very least 1500 hours of continuos internet access where I could work or entertain myself during the outages over the last 4 years, so I think the 180$ per unit it cost, was a pretty good investment in my case
I was very pleasantly surprised when my power went out one day, but my UPSes managed to keep the Modem/Router and my PC running, and I got to finish my multiplayer game :)
If the cable provider offers digital voice, they're required to have backup power for their infrastructure for at least 24 hours (or 8 hours under older regs) to allow for 911 calls. The power injectors used for cable amplifiers usually just have a couple of lead-acid batteries in them.
Two things about Lead-Acid or AGM based UPS's (regardless of UPS topology or design):
1.) check your battery health annually and replace bi-annually. I don't care about the battery quality, an always-online UPS is very hard on lead-based batteries. they're float charged basically all the time, and deep discharged during outages. if you're in a power failure prone area (i.e. more than one per year), invest in a generator too in order to avoid stressing the batteries with full discharges.
2.) check the battery float voltage with an actual volt meter to make sure the UPS is calibrated correctly. APC UPS's in particular, prior to and post the Schneider Electric buyout, had a lot of really poor cost-savings decisions applied to their enginnering. this included using a carbon film resistor-based voltage divider to measure the battery voltage on much (or all) of their Back-UPS and Smart-UPS range. if you don't know anything about resistor technology, carbon film resistors (especially cheap ones!) change in value due to temperature, humidity, and age, usually drifting up in resistance. this is a bad thing when you need a stable voltage reading...
oh and on APC's cost cutting measures; there's the fun time they stopped using nickel plating on the bus bars in their Symmetra UPS line. they switched to tin plating instead. apparently none of them had heard of tin whiskers. :)
Some APC UPSs will check battery health monthly by switching to battery, and letting them run down. You will want to disable this feature as it takes 6-8hrs to recharge to full, leaving a massive random gap in coverage. This also puts undue wear on the batteries as they are only good for a dozen or two cycles.
Another fun fact I discovered the hard way is that many APC UPSes refuse to go back into bypass when they are supplied by a small generator with a dirty waveform.
We had quite a few APC units in the 1990s and 2000s, and I had the impression the older ones were easier on batteries. We were on a three year replacement cycle until Y2K, and after that we found it necessary to check the batteries every six months or so in the newer units.
> I live in a place where blackouts are very infrequent (perhaps a couple per year), but occasionally the mains drops out for only a second or two. I suspect these very short dropouts occur when substation switchgear operates, but have no way of being sure. Anyway, with a server running 24/7 I obviously wanted protection against any loss of the mains supply.
> So I thought I had covered all bases, and, indeed, the UPS proved useful on several occasions. I would quite often be on a work trip and receive an e-mail from the server informing me that mains power to the UPS had been lost, then another e-mail soon after informing me that mains power to the UPS had returned. Only once did the power cut last longer than the battery capacity, and the server was shutdown automatically.
It sounds like the UPS is doing exactly what he wants it to do. This is exactly why I bought a UPS. I would have power transients sometimes and rebooting my entire network (with NAS and servers) was annoying, and I didn't want it to go out when I was gone.
His main complaint is he bought a standby UPS where the battery failed, it turned off the UPS (which is understandably frustrating), and now replaces the battery every three years. Looking at the white paper referenced, it even says to go for Line Interactive for servers. I'm also curious how the person knew that power wasn't cut when he was away?
So other than buying the wrong type of UPS (which at face value looks to be a valid complain based on research)....why isn't a UPS as useful as they'd think?
I ran into an extremely perplexing problem a few years ago where shutting down one server connected to a UPS would kill the entire network until the server was brought up again. The server was not providing any kind of networking at all, no DNS, DHCP, routing, etc.
After wasting a lot of time remotely trying to diagnose this, it was obvious what was happening once I got on site.
It turned out that once this server was shut down, the UPS didn't have enough power draw to power the other equipment attached to it, so a switch that was plugged into the same UPS would shut down as well. For some reason it had failed in such a way that it required a certain level of draw for it to function at all.
Some UPSes have a special outlet on them that when it's not drawing power, it intentionally shuts off the other ports. It's meant for power saving, but I've seen more than one person get tripped up by this functionality.
That reminds me of a "feature" in many of the consumer grade UPS units on the market now. I don't think it's enabled by default, but they do have an option to power down the UPS when the primary outlet is no longer seeing any power consumption.
>a standby UPS where the battery failed, it turned off the UPS
I assume this would be because the power is always flowing through the battery/inverter? Wouldn't that kill the battery a lot faster? Although it'd make the switch from mains to actual battery power basically a non-event from the output's perspective.
Now that I think about it, going 120/240->low voltage dc->120/240->12/5/3.3 is incredibly silly. IIRC there are 48 (or -48?) DC power supplies out there, mostly for data center use. Does anyone make an ATX version that you could hook up to a UPS that has a DC output?
Using a DC auxiliary power supply for backup is common in telecommunications equipment and by extension common in network appliances - "redundant power supply" for network equipment typically means a built-in AC power supply and then DC connections for an external aux power supply that provides battery, often called an RPS.
This isn't common for servers though which, if equipped for redundant power, have two AC power supplies. I think this just comes down to the higher current draw usually associated with servers, running even 48VDC power can become costly and impractical when you need to supply say a couple thousand watts per rack, so the extra space and thermal load of an AC power supply in each unit becomes preferable.
The downside is that while redundant power supply is usually a "standard" feature on switches and routers (that is, they always have the connection whether or not you buy an RPS is up to you), you need to specify and pay extra for dual AC power supplies up-front in servers, so you don't get as smooth as an upgrade path. That said in datacenter environments you often buy servers, racks, and power systems all at once so it's not as much of a concern.
There's a similar tradeoff that exists around bottom-of-rack UPS and central (building or area) UPS---bottom-of-rack UPS tends to be more expensive, higher maintenance, higher thermal load, etc for large installations, so usually large installations use one (or two for A+B power) large central units which may be hybrid between different technologies like flywheel and battery (and you could view the transfer to generators as a "third stage" of a central UPS system), but it also makes your cabling a little more complicated and you have a bit of an "eggs in one basket" situation. Central UPS generally have internal redundancy so their risk of failure is low, and often a bypass device where if a non-recoverable failure of the UPS is detected a contactor "shunts" the entire UPS removing it from the circuit so you don't lose power due to a UPS failure, but that doesn't necessarily save you when the HVAC pours water directly into the UPS control cabinet causing widespread failure of the control electronics... a situation that I have somehow seen twice. I bet on bigger installs you can get an external bypass with some sort of health monitoring though? If HP/Tandem taught is one thing it's that you can always through more redundancy into your very special basket.
Standby UPS topologies like the APC Back-UPS lines keep the battery out of the loop until a loss of power from the AC input is detected, at which point the inverter kicks on the the transfer switch takes the feed from there.
This is a design choice from APC, money is on the control electronics being powered off the battery meaning a dead or missing battery means the entire unit is non-functional. Making the unit resilient to this probably costs as much as a line-interactive unit, however, so you might as well just tell people to get a better unit than make a middle-ground solution.
I have a line-interactive APC and it most definitely is just a UPS-shaped brick without a battery. Might have changed in newer models, I wouldn't know.
My APC BackUPS Pro (supposedly line interactive) runs fine without battery (ran it with a tab from shipping still in the case), might depend on the setup. Generally line interactive makes little sense why it wouldn't work without battery since the power rail is shared.
As a person who has never bought a UPS, this behaviour of turning off power when the battery dies even though there is mains power is unintuitive, and I would have most likely hit the same issue. I see the post title as a PSA, in that if you happened to buy a standby UPS without knowing better, it might not be that useful if you very rarely have power interruptions but the battery keeps dying every couple of years.
Trust me.....having infrequent power outages like that are very very annoying. Everything turns off and I have to spend a non-trivial amount of time bringing my network back up (first router, then NAS (encrypted to I need to hand jam in the key), then servers that depend on NAS (also encrypted)). What eventualy got me to get a UPS was during a storm, it happened twice within an hour! This isn't even talking about if I were away, now my servers are down and I have to do it when I get home.
That UPS was well worth it to put a stop to that happening.
> I'm also curious how the person knew that power wasn't cut when he was away?
Most homes have something with a mains-powered, non-battery-backed clock. You get home and the microwave or the oven is flashing 12:00 and you know there was a power cut.
> So other than buying the wrong type of UPS
This isn't a failure of 'type' of the UPS - it's a design fault deliberately left in by the device manufacturer for market segmentation.
It would be trivial for them to keep the power outputs on if the battery failed but the mains remained on. Leaving it out is a business decision, hobbling their home product so it doesn't steal market share from their small-business product.
The hum is caused by cheap or warn lamination on the transformer. I recently "fixed" an older APC by opening it, removing the large transformer, coating the fins in nail polish, and adding rubber washers to the mounting screws.
When the power goes out on an APC UPS, just press and hold the "On" button until it chirps. Alarms are silenced.
Now imagine the sound at the APC factory, when the power goes out and thousands of UPSes on the charging racks start beeping in sync...! (this is not a hypothetical)
There is a way to turn off the beeps on the APC UPS line (at least the ES 700 I have), but it involves installing software that comes with it onto a Windows machine, connecting to the UPS via USB and changing the settings.
It's enough of a pain to do that rather than configure all 3 UPS devices in my house, I've only ever gotten around to doing it on one.
Can't say I've noticed the electrical hum, except during a blackout. But I don't have mine in a bedroom.
> The loud can't turn it off beeps when power is out.
That depends on the UPS, it seems.
I have a pair of Back-UPS 1500s with the external battery packs. They have a button on the front-panel for disabling/enabling the audible alarm (just hold it down for 2s, IIRC).
You can silence the alarm by pulling the battery connection. Obviously, that's inconvenient if you want the UPS to be useful again as soon as the power comes back.
I personally like the white noise from electronics, so I can't help you there.
The explanation of the failure was very strange to me. I know there are three main UPS types - offline, line interactive, and online. Offline types have the battery disconnected until there is a power outage and only use it when needed. Line interactive and online use the battery more often to absorb power spikes, brownouts, and other anomalies. These two types also use the battery more and need replacements more often than offline UPS.
I have been using the same offline UPS and battery for the last 7 years and it is still working fine - a few days ago it handled a 15 min power outage with 50% of the battery.
Can someone explain why offline UPS can fail if the battery dies? Is it by design or a manufacturer specific issue (e.g. only APC brand ones do this)?
Theres a few more types than that. But thats the gist of it.
"online" is generally a double conversion UPS.
These are generally the best for clean power to the electronics. Since its always supplied by the battery. Its also often one of the hardest on the battery.
Line-interactive are probably most common. They are pretty solid, but since there is a delay for the cutover, surges and really dirty spikes can make it through to the equipment. So say a lightning strike or REALLY bad surge on an overloaded generator can get through, whereas on a Double conversion it may just trip the fuse or breaker. Also running a whole home or standby generator on these line interactive can make them trip constantly since the generators often dont
A) run at 60Hz (ie: mine runs closer to 63Hz)
B) run with a pure sine wave
You can get inverter generators to help with this. But thats a cost too. And otherwise the solution is to "de-tune" the UPS sensitivity.
I de-tune this on mine. Its been fine, running on gennys during the rainy months for hours and every know and then...days at a time.
I have had to swap out power supplies more often..But even then, usually thats after like...5+ years of runtime.
I was just pricing a UPS for home office use this week as we have frequent outages where I am but became a little stuck when I realized that to get a UPS that won't barf when my generators come online moments after an outage, I'd need a double-conversion UPS, which is $$$$. A little stuck for a reasonable home office option given this wrinkle.
I mean. Im using line interactive ones and have for 4-5 years.
I have to de-tune their sensitivity but they work.
But it does stress the CPU power supply caps a little more and shorten their life. But the caps on the board are fine, and havent had an equipment failure outside of a PSU...And ive run literal weeks on genny power.
But that stress is more due to Hz being at the threshold and artificial stepped sine wave i think. Voltage fluctuations when load kicks up certainly doesnt help.
because he gets emails from the unit when it happens.
I had a script in a raspberry pooling my UPS each minute asking if the unit was running on AC or battery, and dropping that info into a file when it changed. I know the software/daemon can also do something like that.
Yes, but in this case he wouldn't have gotten an email since the UPS failed right when the power went out.
I actually had the exact same thing happen to me a few months ago. A power flicker so brief that it didn't even reset any of my clocks, but it caused one of my cheapo standby UPSes to give out.
One thing I always wonder about these home server is this. When you are leaving house for a long vacation, is it really safe to leave some electrical appliances working? Aren’t people afraid of fire hazard. What if there is a short circuit while you are away. A smelling short circuit would definitely burn something if left unattended for few days.
For once there was a high voltage fault in my neighbourhood, destroying few lights, all mobile chargers plugged in and microwave left in standby. Had I not been home, and fault lasting more than 5min, things might have gone really bad.
Now on, Whenever I leave my home for long duration, I just empty the Refrigerator and switch the house mains supply off before loving the house.
One thing I always wonder about these home server is this. When you are leaving house for a long vacation, is it really safe to leave some electrical appliances working?
I used to feel this way, but then I realized that the electronic devices don't know I'm away on vacation, or when I've returned, and they're just as likely to start a fire when I'm down the street getting coffee as on the other side of the planet. And when I return, my presence doesn't magically reset some catastrophe timer and everything's all better because I touched "base."
That said, if I'm gone from home long enough that I turn off (or down) the refrigerator, then I'll unplug everything. So I guess I'm still a bit paranoid.
There are failure conditions that are not immediately catastrophic. The reasoning being that if you come home after a work day (or even better, after coming back from your coffee down the street) and see that a device malfunctioned you will attend to it. As opposed to leaving it plugged in and turned on for weeks while you are not there.
A lot of devices will smoke before starting a fire, sometimes for quite a while until temps increase sufficiently for ignition. You would smell the smoke and figure out the culprit.
Turning off everything is still worthwhile while you are away. This is just risk management. It is a small risk (maybe you have faulty appliances or house wiring? maybe you live in an earthquake or tornado area? etc) but the mitigation is also not very inconvenient.
That said, if you can have a smoke detector pinging you (or a similar device - Amazon's cameras have a microphone that can alert you to alarms) it would be better. As you point out, stuff could happen while you are temporarily outside the home.
Short of being home and reacting immediately with a fire extinguisher when the smoke alarm is triggered I certainly agree with you that there's little that being away changes w/ respect to the electrical fire risk profile and potential for loss. Whether you're gone for a week or gone for 15 minutes makes little difference in responding to a fire.
Responding to a water-related disaster is a different story, though, and worth bringing up.
You might want to consider turning off your water at the main if you're going to be away from home for an extended period. I was fortunate in that the only water-related failure I've had personally was a ruptured water heater leaking into a floor drain of a 1st floor utility room. I know others who haven't been so lucky (having clothes washer hoses rupture, fixtures fail, etc) and have had water running inside their home for an extended period (several days, in one case). Until someone notices the local municipality reading excessive usage, a neighbor noticing water running out a patio door, etc) the damage continues and, to add insult to injury, you're also getting charged for the water / sewer usage.
I can answer this as I’m currently on a two-week holiday and I have a small homelab running at home.
First and foremost: fire alarms. Buy them, and use plenty of them.
Secondly: let your neighbors know you’re on vacation. This implies a certain amount of trust with your neighbors, but it’s good to be on good terms with them for more reasons than just this.
Thirdly: I get notifications if a server goes down or a fire alarm is triggered.
With all this in place, I don’t worry at all. I may be somewhat ignorant or oblivious to certain risks, but in general I trust the safety net I just described.
I find it surprising that service panels don’t have more extensive protection features. I would like my main breaker to trip under any of these conditions:
- Sustained overvoltage. Any overvoltage lasting too long to be handled by a surge protector should trip the main breaker.
- Sustained undervoltage. Some devices don’t like this.
- Phase error. A two phase main should detect incorrect phase-to-neutral voltages and incorrect phase-to-phase voltages. A three phase main should also detect incorrect phase sequence. This should also catch some cases of a disconnected neutral.
- Excessive neutral to ground current. In the US, for reasons I personally strongly dislike, every “service” has a connection from ground to neutral, usually in the meter box or the main panel. Various errors can cause substantial current to flow through this connection and can be dangerous. For example, if a house and the utility both have excellent grounds but the house’s neutral feed breaks, all the voltages can be close to correct and the house’s neutral current will return to the utility through ground. This is dangerous and will not be detected by common equipment. (A whole-house residual current detector would satisfy this, too. I think these are used in Europe.)
Before the pandemic I spent ~1/3 of my time away from home anyway (work during the week, either out camping or otherwise out and about for at least 1 in 3 weekends). I certainly didn't unplug everything every time I left for work. I don't really see a significant difference in risk profile for a week long vacation.
Electrical fires inside stuff that isn't the cheapest straight-from-china import shit are incredibly rare. Most fires causes are A) very old wiring that has never been attended to but loaded to the max B) people messing with mains wiring and causing a fire (e.g. bridging thermal cut off switches in cable drums) and the occasional person drilling into a cable in the wall and hitting it just-so that PE is untouched but L and N are getting just cozy enough to start a fire without tripping the line breaker.
Like the author, I rarely see power interruptions that last more than a second or two (unless there's a major snowstorm or something).
It's kind of annoying and I'd love to see a solution that isn't just "get a battery based UPS" -- maybe something capacitor based, enough for say 15-30 seconds?
I guess I could get a flywheel [0], but I don't think it'd fit in my basement.
At the reasonably sized datacentre that I ran we had a voltage conditioner that also had "sag protection". It was an HPC shop and we decided that it wasn't worth the money to put the compute on UPS (just storage and core servers but the Top 500 machines ran off mains) but we still had 10s of millions of dollars worth of hardware that a lightning strike etc could fry.
The conditioner provided a constant voltage to the servers and could correct "sags" where the voltage would drop, and it could keep things going for a few seconds if the power went off completely. As you say this took our total outages down from probably 10 a year to maybe 1 or 2 - most outages are quite short. Although looking at the graphs on this thing, there were pretty frequent surges and sags coming from the street, like when you see light bulbs flicker in your house.
One other thing we noticed was that if there was a short interruption, all of the HP servers could survive for a second but the Dells turned off right away. Different power supplies store up some amount of power locally.
Looks like it's capacitor-based, removes the need for an inverter, and even has a clever way of communicating to the host, by taking over the power switch connector.
Unfortunately, I can't seem to find a price or where to buy. Would love to see more momentum around products like this, and marketing directed at home systems.
EDIT:
Thought of one more system that uses capacitors to limp to a clean shutdown--the Unifi Cloud Key Gen2.
Again though, the power solution isn't packaged for consumer use.
Though, maybe it'd be amusing to try to hack together a NAS out of Cloud Key Gen2 Pluses... there's a hard drive bay in each one ;D
Back in the late 90s I had a PC motherboard that had a capacitor for exactly this. I could (if i was quick) change which outlet the pc was plugged into without loss of uptime
I also remember a green pc set up which used one of the low power x86 clones (Via?) in the 90s which used the bios battery double as a mini ups similar to that.
I ended up in India during the lockdown. Power outages are frequent and so are voltage fluctuations. Most people have a generator. Most houses haves 3 phase power coming in. Basically anything over 3 kvA the utility guys put in 3 phase at 415V - 3 hot and a neutral. About 3 weeks ago we lost neutral which resulted in double voltage both my MBP chargers got fried including the long extension wires which are super useful. It also took out an AC, a refrigerator, 2 air purifiers. Anything connected to the BackUPS did survive including the 65" oled tv. Luckily we managed to fix all of fit with simply replacing the PCBs. Being stranded with my laptops would really have sucked.
Since I am stuck here til next July I started fixing some of this. More than anything else power conditioning was the single most important thing. Anything with any reasonable switch-mode power supply will be fine on a line interactive UPS, most people don't need to spend $ on an online UPS - esp if your power is fairly clean. We had old locally made generator so I finally replaced that with a nice single phase 25kVA generator. The alternator has voltage regulation. For the mains power I was told most people just put a Servo Stabilizer but I didn't want any moving parts so found an IGBT based static stabilizer instead.
Get phase monitors for your condensing units. There’s nothing more annoying than loosing a compressor due to your poco’s incompetence when the compressor would be just fine if something locked it out as soon as the power got weird.
I did a compressor change out on a package system last year because while we apparently sprung for all sorts of other “options” on the machine apparently the line was drawn on a $48 phase monitor. Our cheaper machines had them and rode out our service being single phased, and the package units stage two was thankfully off at the time; but we still had few thousand dollars of compressor needlessly converted to a piece of scrap metal.
I'm not sure line-interactive UPSs will guarantee a change in behavior in and of themselves since they are still stand-by UPS's but instead of cutting over to battery when in an over or under-voltage situation they attempt to correct it first.
For critical loads where I don't want interruption I prefer online or double-conversion UPS's. Basically they have two inverters instead of one. The load runs off of battery constantly on the first inverter, and the second continuously charges the batteries. There is no cut over time when there is mains power interruption since the loads run from batteries.
But I hadn't considered this dead battery scenario so that's another thing to add to the checklist of things to ask about. It seems that there should be a bypass mode if there is mains power to supply power to the load too all the time.
I've found UPS's to be a mixed blessing - when the power does glitch they can save your bacon but the battery maintenance issue can cause their own outages. If I lived in Florida with their legendary lightning I'd just put up with it but I used to have UPSs on everything and these days I have really dropped down where I have them since we get maybe an outage a year and glitches are equally infrequent too.
Where I live (in the UK) the power is reliable enough (one outage every 3-5 years) that I only tend to spec UPSs for devices with dual PSUs - one raw mains, one UPS-protected. I've had more UPS failures than mains failures in the past decade... and mains power will normally come back up, starting the equipment automatically, while failed UPSs generally require manual intervention to fix, and consequently much greater downtime...
> While I was away from home on a long work trip, suddenly I could no longer connect to my server and I had not received an e-mail from the server informing me of any problem. Luckily it was near the end of my trip so I was not too inconvenienced. When I arrived home I found that the UPS was sounding an alarm and was not supplying power to the server even though there was mains supply to the UPS. It transpired that the UPS battery had suddenly died without warning and could no longer hold a charge, and this had happened while there was mains supply to the UPS, i.e. there had not been a power cut while I was away. Fortunately there was no loss of data on the server; I was able to run fsck during boot-up.
And this is why professional servers have two, redundant, power supplies.
If you had two power supplies, the UPS would fail, but you'd continue to function off of mains voltage.
UPS can fail. Mains can fail. Two power supplies hooked up two both powering your server redundantly means you'll only fail if both fail simultaneously.
That's not really a proper way to deal with it. Proper UPS for a home server or any standalone server or appliance has to hook up either after ATX power supply in parallel and provide all the ATX voltages or hook up directly into a power supply to save a bit on extra converters. This way UPS failure won't affect anything and such UPS will be much more efficient.
It's just that consumer home UPSes are almost scam level products designed to profit from naive people. They are both not improving things for an average user and killing batteries after like a year of operation (lead acid batteries can last for like a decade when not subjected to those UPS chargers).
My cheapo UPS certainly improved things for me when I lived in an apartment where the entire unit minus kitchen appliances was wired to a single 15A circuit. I tripped the breaker semi-regularly by forgetting to turn off the AC before using the vacuum cleaner or toaster oven or whatever.
Also, surge protectors wear out and should be replaced before they lose the ability to protect.
This will probably occur at a different time than when the UPS wears out, so it could be beneficial to keep them separate so you can replace the surge protector only.
You don't need an UPS to protect against surges. In fact, UPSs aren't surge protection devices; they may filter a few of those, but they should be protected too.
I recall from my years as a small business sysadmin that some of the Smart-UPS systems that have a serial connector for management access require a special serial cable. If you plug in a “standard” serial cable it cuts power to the whole UPS unit and your whole server rack loses power even if the batteries are fully charged. That was a bad day.
Indeed. I, too, learned that lesson the hard way many, many years ago! I also know several other folks who learned that the same way that you and I did.
Although I'm a bit "old school" and generally prefer serial (RS-232) ports, this is one case where the change to USB really was a huge improvement!
My experience has been that inexpensive UPS units create more problems than they solve in the long-run. Spending more money up front to get a corporate-production-grade unit is worth the money.
One of the annoying things I’ve seen with the Back-UPS range from APC is that they all seem to have an audible alarm when it’s running on battery that cannot be turned off. The UPS will beep periodically until you either turn the UPS off or until mains power returns. This is so annoying that you can find instructions and videos online on how to perform a surgery on the UPS to disconnect the buzzer from the rest of the circuitry.
Maybe APC really wants to drive users of its lower tier (and cheaper) UPSes mad with the beeps and force them to buy the more expensive Smart-UPS range that comes with a built-in option to turn off audible alarms.
Well, audible alarms do have a place in such systems to inform the person that the connected devices or systems are running on a limited battery capacity. But forcing that at all times is a big annoyance.
"But forcing that at all times is a big annoyance."
Agreed. The cyberpower devices do let you "silence" the alarm, but even with the alarm "silenced" the UPS will begin rapid beeping as the battery nears depletion.
During an extended power outage years ago I used my Back-UPS to keep my phone charged while I watched Netflix over cellular. I ended up wrapping the thing in a whole bunch of blankets and towels to muffle the obnoxious beeping. Huge oversight that there isn't any button to silence the alarm, there are plenty of reasons someone would want to do it.
I meant to tear it apart and remove the beeper after that but never got around to it before the battery made it useless anyway. TBF it was a few years old at that point so the battery giving up wasn't unexpected.
As we were growing, we slowly converted a telco hardware server room into a real server room. Knocked out a wall to get enough room volume for a mini-split AC, removing the sprinkler head, moving the servers... And replaced all the wiring with plugs higher on the wall so that flooding wouldn't result in an electrical fire.
So when it came time to move our staging database over to the new wiring, the head of IT unplugged the UPS from the wall, the UPS complains like UPSes are wont to do in such situations, and in the time it took him to untangle the cable and plug it into the new circuit, the UPS dies. Bad batteries that the health checks didn't detect. Unplanned shutdown of our shared database. Whups.
And that is when management was taught about redundant power supplies, and periodic UPS maintenance.
I also have an APC Back-UPS. Mine are the Back-UPS 650 model, I have two of them. Both emit an annoying LOUD high-frequency capacitor whirring noise that is audible to my ears even when the loud AC intake fan is blowing in the same room. I'm actually shocked any company would ship a product in this condition, unless all their customers have hearing problems. It's a power strip... it's supposed to be silent. Not like a computer.
If my batteries die, I'll just replace them with non-battery backup strips. Not worth the trouble for me living in a city where power goes out maybe once per year on average for 10 minutes, and I don't have servers that need to stay online anymore.
- failed battery caused APC to stop operating even with power (just as in the original post)
- stopped charging battery even though battery was still good (blow electrolytic cap. When I into the circuit that capacitor was rated for 14V on a 12V trace, ie no guard-band. Seriously, skimping on maybe $0.01 total parts cost to leave a system at risk of fire??? Who the heck signed on on that decision???)
- couple of data center APC UPSes caught fire. On the plus side the emergency systems kicked in quickly enough that the only damage were the UPSes.
Anyway, that was a few too many stupid failures. APC is on my "never ever buying from this company ever again" list.
I was doing some researches about this very topic a couple of weeks ago and saw a post on Speciesworks (still trying to find the post). In that post, one Eaton rep replied that the ups device would shut down once the battery died to protect the devices connected to it.
Teah, it's just this simple. This is just a byproduct so all those companies can boast their insurance coverage on damaged equipment you connected to their UPS. And guess what, most of the time you won't be able to get this insurance even if your stuff got fried, saw a few stories like these from recent Slickdelas posts of UPS.
Generally line interactive is less preferable since they provide less protection from surges than a double conversion setup. Big price difference though so line interactive is usually a reasonable compromise.
For a home setup you aren’t going to be spending the money a double or delta conversion unit costs, so line-interactive is the only sane choice.
You really don’t want line-interactive gear when dealing with larger loads either, as the transformer will just cause extra current draw during a voltage sag. This is why you don’t see (many of) them above 3-5KVA.
it appears the powerwall moves the line frequency up from 60hz to as much as 65 hz as a signal to the solar power inverters and that can trip up some UPSs which monitor 60hz as an indicator of "good power"
"I live in a place where blackouts are very infrequent (perhaps a couple per year)"
I live in a fairly rural place (Somerset, UK) In this particular part of the county, outages are approaching "per demi-decade". A recent boiler room explosion in a BT phone exchange caused my office to lose power for about 10 hours but my home, which is about a mile away, was fine. I kept the genny topped up at regular intervals. The many but small UPSs we use keep things running for the 10-20s that the genny needs to fire up and settle.
At home I have several small UPSs to keep things running for about 45-60 minutes. The esxi (Dell T320) in the attic gets shutdown after five mins so that everything else up there gets more time.
Anyway, its all about risk assessment, monitoring and understanding your gear. A Back-UPS is not the same as a Smart-UPS - output waveform and response to load etc. Also, your UPS should be self testing every two weeks. I have several APC Smart UPS 1500VA and others. The 1500 jobbies are something like 15 years old but on their third set of batteries at least. When apcupsd whines that they are getting on a bit we buy new batteries and swap them in on-line.
My golden rule of thumb is do not buy a battery from a firm that you do not recognise. Do not skimp and save pennies over something that can explode or simply be rubbish.
Buy an automatic transfer switch. Plug one input into a UPS, plug the other into either the mains or another UPS (preferably a different brand but identical power). I resorted to this after having multiple APC and Cyberpower batteries fail killing power (even with a line-interactive APC). I run my network (cable modem, router, primary eero AP) off this, no servers. Means we can typically stay online even when the power glitches for an hour or two after a storm.
The one I bought uses standard NEMA cords and outlets: "CyberPower PDU15M10AT Metered ATS PDU" https://www.amazon.com/gp/product/B00NEHUX08/ — it’s overkill for my use case but was the smallest option I could find. I don't have any of the control modules (provide snmp/http APIs).
I found this guide by Cyberpower to be a clear and concise explanation of the topologies of UPS offered and their benefits. Yes it's content marketing, but it's good.
With all of the recent advances in Lithium-type battery tech (e.g. 18650) it seems like a market opportunity to create a new, modern, small, smart UPS.
These low-end APC dinosaurs still use lead acid, and in my experience (with a solid utility over the past decade) outages are just as likely to be caused by battery failure than some kind of actual power outage.
If anyone wants to build something better, happy to chat further.
Main advantage of lithium-ion over lead acid is lithium-ion is more energy dense so you can have a smaller size at a smaller weight. Important if you're building cars, but the price premium probably isn't worth it if you're building another office box at commodity pricing.
Edit: Since we're armchair-building for a predictable, high-uptime application, failure modes might have an interesting advantage though. Any battery engineers know if lithium ion has more predictable or measurable failure modes? Yes, we've all seen the exploding hoverboards... I'm assuming theoretically well-built electronics.
It could possibly target the "Apple-esque" product category by using a smaller physical footprint to emphasize sleek looks, perhaps with a small sharp readout. The small size could be leveraged further by only attempting to provide power for a short amount of time, say 10 minutes, emphasizing and requiring USB connectivity to shutdown the protected computer at minute 9.
Lithium batteries have a much longer lifespan than lead acid, which would eliminate the need to replace UPS batteries every few years. They are also more power dense, so if the goal is just to keep a machine running long enough for a clean shutdown, you can get away with a smaller (energy and size) battery for the same load.
Portable power packs are crazy cheap these days. You can get a 20,000mAh pack for $45 quite easily. They are almost always Lithium Polymer based, and are very small and light. I have had one sitting in my bag for almost a year now and it still has most of its charge.
What I don't understand is why it would be so hard to just scale those power packs up a little bit and slap a normal 110 volt A/C plug on them? Wouldn't that essentially be a super efficient, light weight UPS?
I don't understand how lead acid has an advantage in recharge cycles? My lithium batteries can be discharged and recharged over 1000 times before they wear out, and they hold a charge for ages. My UPS with lead acid batteries, however, is practically useless after a single full discharge, and the batteries have to be replaced practically every year.
There is a reason we don't see lihion car batteries, or fork lift batteries as a default. (Yes, Tesla's are li-ion, but that's a scale where the weight savings begins to matter. Iirc those battery packs can't deliver peak amperage quickly or many times in their life, e.g. ludicrous mode.)
Additionally, li-ion has non-neglegable disposal costs, so I don't think it's fair to count that solely again lead acid batteries.
Automatic Transfer Switches feel like a requirement for using UPSes to me now. Either that or redundant power supplies. I've had too many UPSes, both the BackUPS and the higher end SmartUPS, cause more downtime than they prevented, over around the last decade.
When we moved offices 3 years ago I pulled our UPSes out, because all the outages the previous 5 years were caused by the UPSes.
But then your ATS becomes another potential point of failure. If you have so little trust in your UPS that you need an ATS behind it, I think you are probably better off without the UPS's.
In my last job I had around 100 APS SmartUPS's (mostly 3KVA) in various wiring closets and in the ~3 years that I was there, none of them failed unexpectedly, but we had to replace a few batteries and/or entire units when they failed their monthly self check.
Sure, but I've never had one of those ATSes fail. For a sample size of ~15 over a decade. Where I've pretty much had every APC BackUPS and SmartUPS (mostly 1500RM 2U, some 3000 3(?)U, some 2200 floor units) drop power at some point. That includes being very liberal with battery replacement, the last one was <3 months with a new battery. I would say that they seem to be reliable within 3 years, It's outside of that 3 years that I've had problems.
Correct. This is for our dev/stg environment and desktops (where they aren't laptops, which are nice because they have their own UPS :-). But, our city has all underground power infrastructure, and the number of outages we've had has been very, very small. We do have BBUs on RAID adapters, but we also rebuild almost all our machines every other day, and have backups of everything as well.
Dell latitude laptops have OEM chargers that can run on 10-16VDC. The E6520 I use as a server takes about 6W to run at idle. Grab a 12V battery and a 5A SLA Charger, run your router/modem/server directly off the battery.
UPS's take about 10W to just power the inverter. This gives up a lot of the battery capacity not to mention the losses from converting your power 2 times.
Anyone used those new LiPo UPSes? Wondering if they have a longer useful life than the lead acid, but haven't used them. Thought about getting a PowerWall for our new office, but they don't start providing power for 3-5 seconds after the utility power goes out.
If you're ever inclined, lead acid batteries can be easy to repair and diagnose lifetime for - depending on the type. Sealed ones are harder but not impossible. Open ones take a little more maintenance but have long lifetimes and practically indestructible if you service them often.
The most common cause of death is sulfate build-up, which can be prevented (and often reversed) with a homebrew circuit[1].
I work on these all the time. Problem is normally cheap charger. Try and find a ups that puts load on the batteries everyonce and a while. Good luck. This is hard and never done. We got sick of UPS sitting there for sometimes years and then when needed - batteries collapse and the whole thing comes crashing down.
I have worked on UPS that are in charge of maintaining power to DP systems. When they fail you get things like oil rigs pulling off station costing millions. The company I work for just finished putting batter powered thrusters on an oil rig. First one ever.
PSA: When it comes time to replace your APC UPS batteries, check if you have a "Batteries Plus" (or similar) franchise in your area (in the U.S., at least).
You can purchase replacement batteries from them for roughly half the price!
In Europe, I buy the batteries from the factory and they ask for the old batteries in return; this way the price is lower and I have no problems recycling the old batteries, they are shipped to the factory (a few hundreds km away), dismantled for parts (lead, mostly) and used to build new ones.
"All UPS systems require regular intervals of maintenance. The availability of a system configuration is dependent on its level of immunity to equipment failure, and the inherent ability to perform regular maintenance, and routine testing while maintaining the critical load. As the configuration goes higher on the scale of availability, the cost also increases." (APC White Paper 75)
If your devices are low power like a NUC, recycled laptop, DSL/fiber modem or NAS they'll likely accept 12V as input (or 12-19V for most Intel NUC for example).
For these I wonder if instead of a UPS: buy two LiFePo 12V batteries with included BMS, two battery charger (AC to 12V DC), so you get a redudant 12V bus where to plug your devices.
If above 12V is needed add two DC-DC converters eg 12 to 19V to add a 19V bus. Same for 5V.
It shoud work and support failure of any one of the parts?
I weigh my UPS battery every year and compare it to the original weight that I had printed with a label maker. Over the battery lifetime (2-3 years), there had been around a dozen switchovers, and the only reason of failure I could think of is the electrolyte evaporating because of a warm interior of the UPS.
This gives me an idea, of how much water is lost, and at a certain cutoff, I just replace the battery.
If a power outage is long enough for the battery to go flat, I need to power on things gradually, or my UPS will act as if overloaded. The stuff I've plugged in barely amounts to 280W. It's a home lab, so no much harm done, but also, so much for unattended recovery from failure.
It's good for brief outages caused by thunderstorms, though, so still better than nothing.
The article correctly and importantly points out that there are many types of UPS. Most recently, I got a production-grade APC SMX1500RM2U for home.
Another thing to be aware of, if your PC has a PFC power supply, is that it might reboot or have undefined behavior on event of the switchover to battery, unless the UPS is designed to avoid this.
After reading this, I checked what topology my UPS is using and was happy to see it's line interactive. I have two CyberPower CP1500PFCLCD units, and am approaching the one year mark. It looks like the batteries are even hot-swappable, which I'm sure will come in handy when replacing them.
I've always used line-interactive aka double-conversion UPS. Both because of the battery failure issue, and to protect equipment from crappy generator power. Basically there are rectifier and inverter circuits, with batteries on the DC side.
I did a lot of research before I bought a UPS. The constant that I saw across various brands was that these batteries will die after a few years without any warning. They have to be replaced proactively. No if’s and’s or but’s.
FYI for the APC Back-UPS "Pro" - the "Pro" model's topology is line interactive. (Or at least my Australian 1500 version is, as well as the other "Pro" models I checked.)
I suggest building your own UPS with strictly DC outputs. Most of what you'll be plugging in runs on DC anyway. This does mean you'll need to dig into some of your equipment to find the DC access point, but on other equipment you'll only need to cut the wall wart off the end of the cord.
You can choose your own battery and battery maintenance equipment for maximum longevity, and choose your own switched power supply with different tap points for different DC voltages.
This is simpler and more reliable than AC line-interactive UPSes because synchronizing an inverter with wall-socket power is not required. Yes, you need some EE skills to build it, but the end result will be far more reliable than a cheap off-the-shelf UPS.
Yep, and once you start using custom DC UPSes the biggest issue will be battery chargers that kill lead-acid batteries. Things are better with LiFePO4 these days, but for lead-acid I haven't found a solution except for literally hacking a pulse charger and battery management on top of arduino.
I've had good luck with solar charge controllers for RVs, e.g. [0]. You have to drive them with DC of course but they're designed to be driven by very poorly-regulated DC like that coming from a PV panel. So a cheap AC/DC converter (either switched or linear) driving one of these things should preserve your batteries. Most of them also provide a DC load output which gets automatically switched over to the battery if the input DC (or in the case of a UPS, the input AC) fails.
UPSs, in my experience, quite often have or develop faults that defeat their purpose. We just threw one out that would cut power entirely during self-test, but only during self-test, not during an actual outage. I've seen their battery monitoring circuits fail such that they report a battery as good until power is cut and then die immediately. The other way also happens, reporting all batteries as bad even though they might be brand new, so you're never sure when to replace them.
I've come to the conclusion that if you are going to use a UPS, your equipment should always be plugged in to at least 2 UPSs, either via redundant power supplies or an ATS PDU.
Kind of a lame article. He is complaining about his $50 consumer grade UPS. The minimum any server needs is the Apc SMT750 "Smart-ups". It takes two batteries, cheap ones last about 3 years, the replacements from APC can last between 5-7 years. But the unit can supply mains power with no battery connected and the batteries can be hot swapped easily from the front of the case. These are $300 new, but you can find them used on ebay between $50-$100.
tldr; when a cheap UPS battery is failing the UPS will fail even when the main power is on.
I got personal experience with this failure. Maybe the thinking is that in case of a power outage you don't want to find out that all your UPS devices fail at the same time. But in this case a 24 hour warning would have been much better.
.png){kind=link}
https://perspectives.mvdirona.com/2013/02/the-power-failure-...