Good Practices for Mission Critical Electrical Power Reliability
[December 2020] When the power goes out and the generator comes on, usually there is a sigh of relief. But that relief can vanish in seconds if the whole plant does not come back up. The goal for the technical department is to prevent that from happening.
It is no secret. Your facility may be operational, but failure of sensitive electronics can cripple your ability to conduct business.
Indeed, there is much more than solid emergency power to consider.
Guarding Against Failure
If computers, transmitters, or other equipment depend on cool air for their day-to-day survival, how well can they continue to operate when your one air conditioner malfunctions as collateral damage to a main disaster?
That is why maintenance protocols must extend right down to the wires and cables that link all of your sensitive electronics.
For example, computers, server racks, and communications terminal equipment need to be secured and mounted so they cannot fall in case of an earthquake or easily be flooded out.
This includes assuring that adequate service loops are in place for the wiring, so such earthquake and/or other stresses will not compromise the connections.
Emergency Power
UPS, or Uninterruptible Power Supplies, are now – or should be – a fact of life in the communications and information workplace.
A UPS will either sense when the main fails or, using a double conversion setup, completely isolate your gear from the commercial power grid.
Attention is needed, though: from a small UPS that provides service for a personal computer to giant units that can power an entire facility, they all have one thing in common: batteries.
Battery Backup
Without good batteries, a UPS is nothing more than an expensive door stop.
UPS batteries have a finite life span so, logically, they must be tested regularly. The UPS should go “on line” – under load – to test it. Some units test themselves automatically, even having software to quantify the charge. However, as batteries age, this can be misleading. Therefore, many engineers routinely pull the UPS AC plug out of the wall for a manual test.
Some UPS applications require hours of power, while some only need several minutes.
More Than Just Power
While a UPS provides emergency power when the AC dies, many also are programmed with another electronic agenda: Protect the devices plugged in to it from what the UPS thinks is “Bad Power.”
What is Bad Power? For many reasons, power can become polluted when the alternating current sine way is distorted. Sometimes this can cause component failure. Other times the UPS may think it is not 60 Hz AC, and refuse to connect it to the batteries.
Many diesel generators in emergency service are not sized for the load they have to carry, or they do not have proper power factor correction. Computers and other devices with switching power supplies can distort AC power waveforms. The result: Bad Power.
Normally, after a UPS has come on line, it should go back to sleep to recharge its batteries when the emergency generator picks up the load or city power is restored. However, if an intelligent UPS senses the AC power equivalent of poison, it stays on, or it may cycle on and off. Thus, even though your generator is operating, the UPS battery can run down and cause a failure.
Check the Whole System
Your best defense is to test your entire emergency power system at least once a month under full load, and do proper preventive maintenance on all batteries.
Many problems can be corrected by reprogramming the UPS. Consult your UPS manufacturer or service provider to see if your UPS can be adjusted to be more tolerant. All UPS cycling cannot be avoided, especially if heavy loads like air conditioners switch on and off line.
And, be prepared to replace the batteries before they no longer hold a full charge. Depending upon the size and experience, some facilities change them on a regular basis, planning it as a regular maintenance expense.
The Care and Feeding
of Emergency Generators
Some people still believe starting an emergency generator with no load is an adequate weekly test. This is wrong thinking, and it can lead to many problems.
If your generator is diesel-driven, running it repeatedly with no load or a light load may lead to Wet Stacking. The effect it that when the generator is asked to come online to power a full equipment load, deposits that build up during no-load tests prevent it from developing full power under load.
You must test with the load your diesel is expected to carry during an emergency. If not possible, obtain a device called a load bank that can simulate a full load. The generator can then be hooked up to the load bank for extended “hard runs” several times a year, burning off the deposits that had built up.
Keep it Healthy
Most experts recommend doing a complete generator health check every six months.
Generators with engine block heaters put special stress on fittings and hoses. Vibration loosens bolts and fatigues fittings, wires and connectors. While some vibration is normal, the smoother the generator runs, the longer it will operate without needing major part replacement.
During major overhauls and generator work, you may want to make sure you have a local source for reliable portable power. High-power diesel generators on wheels are readily available to supply field power. For sources of temporary power, consult the Resource Directory in the Disaster Resource Guide, or online at: www.riskandresiliencehub.com.
Why You May Need Two
Mission dictates need. Need dictates reliability. If your facility is super-critical – let us just say you want and need to stay on the air – you may decide to install a second generator.
If the design budget permits, a second or even third emergency generator is a realistic insurance policy. During the August, 2003 US/ Canadian power outage, a Toronto office high-rise lost not only primary power, but generator power as well, leaving tenants in desperate straits.
If you are designing a facility you are told must never fail, consider a redundant and parallel-wired UPS. And, if you are installing a new diesel, engines over a certain size may need licensing by your local air quality management district. Permits must be obtained to construct and store fuel in an underground tank.
Pay Attention to the Fuel
Unlike wine, fuel does not get better with age.
Stored fuel gets old. Fuel begins to break down. Gum and varnish can form. Certain algae can grow. Fuel additives can extend the useful storage period and prevent algae growth. A filtering system and a planned program of fuel cycling also mitigate these risks.
Your maintenance tech should submit fuel and crankcase oil samples periodically for lab testing. The fuel report will let you know if your storage conditions are acceptable. Furthermore, the crankcase oil report might find microscopic metal particles: an early warning of a major failure. Another reason to test: underground fuel tanks can spring a leak and introduce water into the fuel and create for you a costly hazardous materials cleanup project.
Assessing Consumables
Any running generator will consume crankcase oil, and possibly even radiator coolant during extended runs.
You should know your generator’s crankcase oil consumption rate so you can add oil well before the engine grinds to a screeching, non-lubricated halt. Similarly, water-cooled generators must be checked periodically to verify there is enough coolant. You should have enough oil and coolant to get the facility through a minimum of one week of constant duty.
If possible, install external crankcase and coolant reservoirs to eliminate the need to stop the generator and check the oil and coolant levels.
Reliable Transfer Switches
Many facilities use an automatic transfer switch to sense when utility power fails, start the generator, and switch it on line.
When utility power is restored, most transfer switches wait a prudent amount of time and automatically switch back. These transfer switches contain parts and connections which need at least annual checks.
A qualified electrician can perform a simple test to verify high current connections in the switch are tight and corrosion-free. It uses a hand-held infrared heat sensor at each terminal to detect inappropriate heat. This is also a good way for qualified electricians to check all AC and battery power connections on your site right down to each circuit breaker and each battery terminal.
Air Handling Systems
Both people and equipment can crash when they overheat!
Clean, cool, dry and pollution-free air in generous quantities is critical for modern communications facilities and their staff. This bears attention: if you occupy a high-rise – or plan to do so – you may find you do not have your own air system. Plus, many building systems have no air handling backup, do not provide nights and weekend supervision, and may have less than reliable maintenance support.
Your best protection is to get the exact terms for air conditioning nailed down in your lease. You may wish to consider adding your own backup system – a costly but essential strategy if your building air supply is unreliable or has no back-up.
Several rental companies specialize in emergency portable industrial-strength air conditioning. An emergency contract, arranged before an event, for Heating Ventilating and Air Conditioning (HVAC) that can be invoked with a phone call could save you hours or even days of downtime. Consider buying a portable HVAC unit if you are protecting a super-critical facility.
Wherever the cooling air comes from, there are times you need to make sure the system can be forced to recirculate air within the building, temporarily becoming a closed system. Smoke or toxic fumes from a fire in the neighborhood can enter an open system. Toxic air can incapacitate your personnel.
Forcing the air system to full recirculation using fire-dampers could avoid or forestall calamity.
Electromagnetic Pulse Protection (EMP)
The EMP phenomenon associated with nuclear and some chemical explosions can disable many electronic components in a computer-based device or communications system. EMP energy can enter any component or system coupled to a wire or metal surface directly, capacitively, or inductively. Some chemical weapons can produce EMP, but on a smaller scale.
Any site that depends on radio frequency equipment, including AM, FM and TV Transmitter facilities need EMP protection. If you are responsible for such a site, you should discuss EMP protection tactics with a knowledgeable consultant before installing protection devices for your radio frequency (RF) circuitry. EMP devices such as gas discharge tubes can fail in the presence of high RF voltage conditions and disable facilities through such a failure.
The late Glen Clark authored a discussion about EMP in terms of broadcast facilities. You can find it here.
The Federal Emergency Management Agency (FEMA), now part of the Department of Homeland Security, has been involved in EMP protection since 1970 and is charged at the federal level with the overall direction of the EMP program. FEMA provides detailed guidance and, in some cases, direct assistance on EMP protection to critical communications facilities in the private sector.
– – –
Richard Rudman is a regular contributor to The BDR.
He was Director of Engineering for All News KFWB Radio in Los Angeles for 27 years, now operates Remote Possibilities, consults on EAS issues, and is a CalFire vendor for an 85 kWh emergency generator he owns and maintains.
You can reach Richard at rar01@me.com