Power efficiency in the cloud

Experts say average-sized server farms each consume half as much power as it would take to run a typical American city. So it should come as no surprise that such entities are looked on as ripe for energy efficiency measures.

The chance to boost efficiency comes partly from the reality that the servers in most server farms are highly underutilized, and their power distribution systems aren’t optimized. EPRI, the Electric Power Research Institute, says data centers account for 3% of total U.S. energy consumption (including cooling equipment). And the latest estimates of worldwide energy consumption peg data centers as consuming perhaps 1.3% of the world’s total power. Built into that figure is a lot of opportunity for more efficient power use. And this opportunity will probably rise thanks to the developing field of cloud computing.

Cloud computing gets its name from its schematic cloud-like depiction, meant to signify both its far-flung Internet building blocks and a work path for a given job that weaves through server infrastructures in ways that are unknown beforehand. Experts see cloud computing as a core element in minimizing energy consumption. For example, the market research firm Pike Research predicts cloud computing’s continued adoption will be responsible for cutting energy consumption by 31% worldwide from 2010 to 2020. The cloud could also slash data-center expenditures by 38% and reduce greenhouse emissions by 28%. But how?

Through “intelligent efficiency,” power-sourcing schemes that work in tighter step with the server network’s dynamic operating profile. IT designers are counting heavily on ever-improving digital control and management techniques to optimize data flow and job allocation, power (both the server supply and the system’s backup uninterruptible power supply), and thermal management.

The “power-to-Cloud” challenge is a cascade of issues. In his 2011 report Growth in Data Center Electricity Use 2005-2010, Stanford University engineering professor Jonathan Koomey presents data that crystallizes the problem: A typical data center’s utilization is not 100%. The average estimate for a data farm is more like 35 to 45%.
Cloud computing will bump up that utilization rate significantly by optimizing the flow and processing of data in the data center. It ideally lets work distribute more evenly over individual servers in the system even during times of high traffic. It makes efficient use of resources by distributing work to more lightly loaded servers. But there is also a need for devices watching over the computer network to evaluate power consumption in real time.

To make this happen, “Power supply makers need more intelligent chips. There’s otherwise no magic bullet,” says Texas Instruments Inc. product line manager for power supply solutions Brian McCarthy. “If you don’t optimize the entire power chain well in a dynamic world, all the way down to the equipment, it’ll be inefficient, there will be more heat, and you’ll have to install more cooling. If you put intelligent systems in the right areas, you can measure efficiency and do something about it.”

Drawing on digital

Cloud computing facilities want to ensure both that the power available to operate the system will precisely match the load demands of the individual servers in the server farm and that power will be delivered in a way consistent with the lowest losses. Traditional analog methods tend to fall short at both the system and board level in managing power consumption, so designers increasingly turn to digital techniques.

“We concluded from studies conducted in 2003 that analog technology had reached certain physical limits, and to go beyond those limits, something else was required. This was motivation for research into how to combine the best of the analog world with the flexibility of digital,” says Patrick le Fevre, head of marketing and communications at Ericsson Inc., which supports board module development and the like for server supply manufacturers. “The resulting project was composed of two sub-projects: The first to develop power sources able to adjust operational parameters in response to changing load conditions, thus limiting power losses; the second to investigate power architectures that can be optimized to load conditions.”
The digital approach actually breaks down into two areas, digital control and digital management. The former addresses operation of a server’s individual power source. The latter orchestrates the overall operation of the power and data system. Digital control comprises four levels, from basic hardware-centric on-off switching (level one), to software-centric approaches that get involved with the supply’s feedback control loops.

“Digital power control lets a supply or IC dynamically adjust its control scheme as the load or input voltage changes, and do it on the fly,” says TI’s Brian McCarthy. “It affects supply efficiency, transient response, the supply control loop for adjusting dead time, duty cycle, and turning on and off synchronous rectifiers. (This sort of control) is theoretically possible with analog ICs and a microcontroller. But from the perspective of parts and system cost, digital has the advantage.”

The concept of using a digital controller for dc power conversion tends to involve level two and three devices, with the trend towards more software-centric, level-four-type devices. More and more, too, the devices tend to be those compatible with the Power Management Bus (PMBus), an open standard power-management protocol with a fully defined command language that facilitates communication with power converters and other devices in a power system. PMBus compatibility implies a degree of power management. Some of the more recent examples of products in this area include Intersil Zilker Labs’ ZL2101 (regulator with integrated MOSFET), ZL6105 (controller with internal driver) and ZL8101 devices (controller, external driver). These adaptive auto-compensating devices, part of the company’s digital-dc line, are suited to server point-of-load applications. The company’s new ISL6367 six-phase controller provides many of the same advantages for voltage regulator module applications.

Digital management, on the other hand, focuses on overall power system monitoring, the handling of data, and communications. “Looking at the system level, we have the ac/dc supply and UPS, which today tends to have more digital management capability,” says Jim MacDonald, marketing director, Infrastructure Power Product Line at TI Silicon Valley Analog. Thus, digital management techniques coordinate the data center’s power-data profile. “(Digital management techniques) accurately manage the power going to each blade,” says MacDonald. “Power supply monitoring ICs sit at the edge of each server blade and accurately measure how much power they’re using. Digital management functions feed this information to the server host, aggregate the power consumption and send jobs to different servers to balance out power consumption. In so doing they optimize electrical power at the server, rack, shelf, and data-center level.”

Such developments blur the line between digital control and management as systems and chip integration evolve. Consequently, terms used in the two areas become somewhat intertwined. Chips spanning the two roles include TI’s UCD3138, which the company touts as the industry’s most integrated and configurable digital power management controller. The chip contains a 32-bit microprocessor, precision data converters, multiple programmable hardware control loops and communications and protection circuitry, all in a 6×6-mm package. It’s optimized for use in both ac/dc and isolated power supplies. This device implements digital control of up to three independent power-supply feedback loops. It includes several control functions to make supplies more efficient, including sync-FET soft on-off control, dynamic phase shedding, and dynamic frequency adjustment and mode switching.

The company also offers several new similarly equipped controllers for designers who like the benefits of analog chips and the functions of digital configuration and communications. They include the new TPS40422 dual-channel/multi-phase controller, which eliminates the need for 12 external components. As for chips more closely aligned with power management, the company’s LM25066 is a power protection and measuring and management device (voltage, current, temperature data) for each blade in a 12-V server subsystem. Similar devices (LM5066 and LM5064) are available for 48 and –48-V telecom blade subsystems, respectively.
How much of an efficiency improvement can we ultimately expect from digital techniques? It depends on the application. “The dynamic adjustment of the power envelope to load conditions helps reduce energy consumption as well as power dissipation, which will in turn reduce the need for cooling,” says Ericsson’s le Fevre. “Consider server A using a conventional distributed power architecture and server B using optimized hardware combined with an Energy Optimizer series of algorithms (as pioneered by Ericsson). Our experience tells us that this kind of optimization can reduce energy consumption by about 3 to 5%.”

The thermal factor

One byproduct of digital power is easier thermal management. Reduction of waste heat, in turn, boosts data center power usage effectiveness (PUE) -- specifically, the total power consumed (including A/C and other overhead) divided by the power going to the IT equipment.

“Say it takes about a watt of power to cool every watt of power dissipated in a server,” says Chris Young, senior manager for digital power technology at Intersil Zilker Labs. “That is, two watts of power are used for every watt of power dissipated by the server (PUE = 2; the ideal approaches 1). With cloud computing, servers are approaching continuous loading near their peak capacity (versus 10 to 20% loading with non-cloud applications), so thermal management is much more critical than before. Digital control of individual power supplies can help boost efficiency and thus reduce the thermal loading on the cooling systems.”

Thus in cloud computing, “There is a one-to-one correlation between losses and thermal management,” says Young. “If we can cut losses, the thermal management problem diminishes by the same amount.” The savings can hit 10 to 20% by optimizing the intermediate-bus voltage (one of Ericsson’s digital specialties), intelligently adding and dropping phases, and judiciously choosing the controller’s current and/or voltage mode.

Digital control techniques can also simplify the problem of dealing with waste heat by putting energy into cooling only where it is needed. Digital telemetry, for instance, can provide information about the thermal loading anywhere there is a power conversion. “Typically digital point-of-load power supplies report not only voltage and current but local temperature,” says Young. “Local power delivery and temperature information can be used to vary cooling-fan speeds from fan to fan based on the localized need.”

Large data centers require building-scale heat management that typically includes hot- and cold-aisle containment systems, airflow modeling techniques, and the like. “Fluid-flow cooling associated with cold-plates and heat exchangers is becoming standard,” says Ericsson’s le Fevre. “Data centers are increasingly designed to cool only areas where it’s mandatory. Otherwise they apply simpler cooling methods like natural convection.”

Data center operators are also taking a closer look at water-cooling schemes, interesting because water removes heat 4,000 times more efficiently than air. IBM deployed this idea a few years ago in its supercomputers, where processors and other circuit board components get cooled with 60˚C water. Key to this scheme are micro-channel liquid coolers which attach directly to the processor chips. The thermal resistance between the processor and the water is low enough that even hot 60˚C water keeps processors well below their 85˚C maximum. At IBM, the water heats up to about 65˚C and is then routed to another building where it provides hot water heat before recycling back to the supercomputer.

Another twist on liquid cooling is to stick all the servers in a server farm in a bath tub of liquid coolant. That is what Green Revolution Cooling does to reduce cooling energy use by 90 to 95% while also cutting server power by 10 to 20%. The coolant is basically a special grade of nonconductive white mineral oil. The mineral oil has 1,200x more heat capacity by volume than air. That means servers placed vertically in tanks of the mineral oil can be cooled by convection as the oil circulates through the system. The oil is then cooled by a variety of means, such as an evaporative cooler or a radiator.

Finally, cloud computing may eventually run from “zero-energy” data centers. That idea was recently proposed by Hewlett-Packard, which aims to use renewable sources for energy and cooling, including “intelligent energy” techniques to cut total data center power use by 30%.

In other work, Microsoft is developing a different type of tool to cap the power to a data center during peak times. The scheme doesn’t use traditional dynamic voltage and frequency scaling of individual servers in a farm. The problem is these methods don’t respond quickly enough to spikes in power demand, says Microsoft. Instead, the system will be based on an “admissions control” to selectively limit the number of server requests to optimize the performance-versus-power tradeoff. EE&T

The control-management loop

Why is overall server efficiency rising and what does digital power have to do with it? “The raw efficiency of an ac/dc supply at a given operating point is not so much a matter of the controller, but of choosing the right FETs, diode, and various parts,” said Wolfgang Meier, product marketing manager for ac/dc power management controllers at Infineon. “A power supply delivers a given efficiency at a specific operating point, but producing the highest efficiency across all loads is the dc controller’s responsibility.”

“Intel, for instance, is aggressively specifying the amount of intelligence that goes into voltage regulators, and there is a serial interface between the CPU and the voltage regulator,” explains Brian Molloy, head of application marketing for dc/dc power controllers at Infineon. “Intel is constantly modulating the supply voltage to the CPU. They also read temperature from the voltage regulator and other telemetry information such as current and so on. That information helps dynamically manage the power consumed in the system. In addition, we see OEM customers going in the direction of using telemetry info via power supply buses such as PMBus and I2C in the system management of their servers. So the server overall is becoming intelligent about how much power it consumes.”

Where digital control leads, management follows. “Some customers are dynamically managing the whole power server farm, the whole pod,” says Molloy. “They collect the information at the voltage regulator level, at the motherboard level, at the rack level, to dynamically manage power.” Thus Molloy sees even more dynamic management of power by the CPU in Intel’s next few generations.

Economics of power for cloud computing

In simple terms, “cloud computing” is a service that lets users access remote servers, operating systems, and/or databases on demand to run jobs more quickly and economically than possible using their (often limited) in-house resources. Cloud computing is more than just remote file access. It encompasses the idea of running major applications -- CAD and graphic rendering programs are examples -- on far-away servers.

From a business perspective, cloud computing is a system of outsourcing by which numerous independent users can simultaneously share a technical database and computers. They may or may not own either one. The main drawbacks are generally a loss of Internet services due to power issues, no direct control of the system/database, and security concerns.

Cloud computing often calls upon virtualization, i.e., an enabling technology designed to get the most from a server farm’s compute capacity. The terms “cloud computing” and “virtualization” are thus intertwined to reflect both the commercial and technology aspects of the computing infrastructure. The origin of the term “virtualization” is hazy, but by most accounts it stems from the concept of “virtual” servers where a job assigned to the cloud is not assigned to any particular IT asset beforehand.

Cloud facilities can be located nearly anywhere. Because these data centers consume so much power, it pays operators to shop for locations where power costs are low. That’s why some industry observers tout Wyoming, Texas, and Virginia as promising spots for cloud computing having rural areas with bargain energy rates, perhaps one-fifth the cost per kilowatt-hour of urban areas.

Should cloud farms run on dc power?

Here’s what seems like a good idea: Cut rectifier conversion losses by feeding dc to each data center server. From the standpoint of energy efficiency, eliminating the power supply conversion losses embedded in the hundreds of servers at a data center would look like a no brainer.

But not so fast. When it comes to powering data center servers, “You’ll find that the overall efficiencies for ac versus dc are about on a par with each other,” says Peter Panfil, vice president, global power sales, Emerson Network Power. Though it is still a contentious issue, this is basically the same conclusion reached by Green Grid and Schneider Electric. And earlier analyses from Lawrence Berkeley National Labs and EPRI reportedly gave an average 30 and 15% advantage respectively to dc systems.

Panfil dismisses the large percentage differences of the various analyses as an apples-and-oranges comparison. “They often compare a modern dc system to a legacy ac system. But when you look at modern ac versus modern dc systems, they are on a par.” The major argument hinges on the gains accrued by cutting the number of power conversion stages, but it’s an argument not necessarily accepted nor adequately developed.

Thus, the jury is still out, and there are suppliers experimenting with both approaches. Green Datacenter and ABB in Switzerland, for example, just launched what they say is the world’s most powerful dc-powered data center. Their 380-Vdc system, which uses HP servers, claims to be 10% more efficient than a comparable ac-based system. The 380-Vdc systems are the industry’s defacto standard; various schemes propose dc voltages up to 575 V.

An alternative way of boosting server power conversion efficiency is to stick with ac, but use much higher ac voltages, an idea borrowed from the high-tension wires on the utility grid. High ac voltage means lower input currents for a given power level, and thus lower I2R losses.

Despite these novel concepts, most users lean towards traditional ac/dc architectures because they’re already here. And it would take work to make high-efficiency alternatives practical. A high-voltage dc system isn’t normally suited for power transmission over any great distance and usually needs to sit near or at the server site. As for high-voltage schemes, going from 208 to 240 V, for instance, picks up perhaps half a point of efficiency, a lot in a world where supply designers are happy to gain tenths of a point for megawatt-level systems. “Folks are asking us to work on power systems in the UPS world that can deliver as high a voltage as we practically can,” says Panfil. Most often, it’s 240 V line-to-neutral; 415 V line-to-line. As for transport losses, cloud providers that propose going to 575 V line-to-line expect to cut losses by 10 to 20% compared to a 480-V line-to-line (277-V line-to-neutral) system.

Resources

DatacenterDynamics, www.datacenterdynamics.com
Electric Power Research Institute, Palo Alto, Calif., www.epri.com
Emerson Network Power, St. Louis, Missouri,
www.emerson.com
Ericsson Inc., Sweden, www.ericsson.com
Infineon, Milpitas, Calif., www.infineon.com
Intersil Zilker Labs, Austin Tex.,
www.intersil.com/zilkerlabs
Pike Research, Boulder, Colo., www.pikeresearch.com
Schneider Electric, West Kingston, R.I. www.apc.com
Texas Instruments Inc., Dallas, www.ti.com