The Open Rack standard seeks to right some historic wrongs in traditional rack design, set by the EIA 310-D specification, which traces its origins back to railroad signaling relays. EIA 310-D wasn’t designed with data centers and scale computing in mind — it couldn’t, because it was established in the 1950s. EIA 310-D standardized the width between the inner rails in a rack, but left other rack specifications — such as height, depth, mounting and cabling schemes, and connectors — to the manufacturers, each of whom came up with their own proprietary designs. This resulted in gratuitous differentiation in server and rack designs, locking consumers into specific vendors and their implementations.

The Open Rack features a simple design, built with the scale compute space at its core. A slightly taller rack unit, called an OpenU, or OU, is 48mm high (the traditional rack unit is 44.5mm tall), which increases airflow, improving air economization; it also allows for better for cable and thermal management and efficient use of space. Fewer parts and service from the front improves serviceability. Modular design of the IT chassis (anywhere from 0.5 OU to 12 OU high) allows for flexible density within the racks.

Even though it’s a new standard, the Open Rack doesn’t deviate from the 24″ column width, which is driven by standard floor tile pitch. And while the Open Rack’s IT equipment space is 21″ wide, it can be adapted to accommodate existing 19″ equipment. The racks adhere to their own standard but still allow for innovation within the compute space. For example, the wider equipment bay allows for implementations with three motherboards or five 3.5″ disk drives side by side in one chassis. And the wider rack is much more space efficient than the 19″ rack, which, once you factor in the sidewalls and rails, results in just 17.5″ for equipment for about 73% space efficiency, versus the Open Rack delivering all 21″ out of the 24″ available for 87.5% space efficiency.

The Open Rack lowers total cost of ownership because it maximizes the product life cycle for each compute component. Rather than replacing the whole server on a regular, short (2.5 year) cycle, each component gets replaced according to its own life cycle, which can be up to 10 years in some cases. This disaggregation of compute components (CPU, hard drives, NICs) improves efficiency and reduces the amount of industrial waste.

The Open Rack implements an innovative, cable-less power distribution system, the first of its kind. Servers no longer have their own power supplies; they simply plug into bus bars at the back of the rack. The bus bars connect to the power shelves within each rack. The power shelves use the same highly efficient power supply used with the OCP Intel v2 motherboard. Two PDUs supply AC and DC power; in the event of loss of AC power, DC power can be provided by the Open Compute Project Battery Cabinet or by battery backup units (BBUs) within each power shelf in the rack.

The Open Rack is a new direction even for the Open Compute Project. The initial OCP motherboard specifications still adhered to the 19″ rack standard, and tradeoffs were made with cable routing and the necessity of putting the PSU in the back of the chassis. Having the disk drives in front caused the drives to traverse the chassis. Which made us realize a few truths:

• Simplicity is hard.
• When dealing with scale compute ecosystems, you have to design for the data center; the motherboard is just a module in a server.
• When you engineer for density, you can do the wrong thing. The Open Rack aims for practical density.

Download and read the specification and charter. And join the conversation with the community.

Today, as more than 500 people converge on San Antonio for the third OCP Summit, I think we can safely say that we’ve already achieved much more than that. The momentum that has gathered behind the project – especially in the last six months – has been nothing short of amazing.

Here are some of the major developments we’ve seen, just in the six months since our last summit:

• Dozens of new companies have joined as official members of the project, including HP, AMD, Fidelity, Quanta, Tencent, Salesforce.com, VMware, Canonical, DDN, Vantage, ZT Systems, Avnet, Alibaba, Supermicro, and Cloudscaling. HP, Quanta, and Tencent have taken the additional step of joining the OCP Incubation Committee, which reviews proposed projects to determine whether they should receive official OCP support.
• Exciting new projects have been proposed to the Incubation Committee, including a Facebook design for a vanity-free storage server (code-named “Knox”) and highly efficient motherboard designs aimed at the specific needs of financial services companies from AMD and Intel (code-named “Roadrunner” and “Decathlete,” respectively).
• We’ve begun mapping out a convergence between Open Rack, the OCP’s specification for an open-standard server rack design, and Project Scorpio, a similar spec under development by Tencent and Baidu. We expect to merge the two specs in 2013.
• HP and Dell have announced new, clean-sheet server and storage designs (code-named “Project Coyote” and “Zeus,” respectively) that will be compatible with OCP’s Open Rack specification.
• VMware has announced that it will certify its vSphere virtualization platform to run on OCP gear, and DDN has announced that it will do the same with its WOS storage system. Canonical has also announced that they will offer “zero day” certification on OCP servers, meaning that they will work with the OCP to certify new designs before those designs are released.
• We’ve launched an official OCP Solutions Provider program to help enable new opportunities for companies to sell and consume technology based on Open Compute Project designs. Companies currently pursuing Solutions Provider status include Hyve, ZT Systems, and Avnet, as well as new business units from Quanta and Wistron (called QCT and Wiwynn, respectively) that have been launched to sell directly to consumers.

This is tremendous progress for such a short period of time. But even more gratifying is the way the Open Compute Project is inspiring both consumers and suppliers to think differently about this industry. In recent months, we’ve seen a variety of new choices emerge for consumers of scale computing technology, with the rise of new solutions providers like Hyve, ZT Systems, Avnet, and Quanta’s new OCT business unit. These new options will enable consumers of all kinds to consume open, efficient hardware in the ways that suit them best.

On the other side of the equation, we’ve started to see a convergence of voices among the consumers of this technology around where we think the industry would benefit from standardization and where we think the opportunities for innovation are. Open Rack and Project Scorpio are perfect examples of these consumer voices coming together and communicating their needs more clearly – and the new Open Rack designs debuting today from HP and Dell are perfect examples of the supplier community’s response to that emerging clarity.

Perhaps most importantly, though, is the industry’s decreasing focus on what OCP founding board member Andy Bechtolsheim calls “gratuitous differentiation” and its increasing focus on driving innovation where it matters. This is the work we have ahead of us in the coming months, as we pursue even greater advances and efficiencies in scale computing technology. It is no small task that we’ve set for ourselves – but as the last year has proven, we can accomplish anything if we work together in the open.

Frank is founding board member of the Open Compute Project.

To answer those questions, we formed the Open Compute Project a year ago today. The Open Compute Project (OCP) coincided with the launch of the Facebook data center in Prineville, Ore., where OCP hardware and designs made their production debut. In our commitment to openness and sharing with the industry, we released the initial specifications and CAD models for all the OCP technologies: the data center mechanical and electrical designs, motherboards, power supplies, battery backup solution, and server chassis and “triplet” cabinet.

We’re seeing many indications that the open hardware concept is working. Our peers and technology suppliers are thinking differently about how to innovate in the scale compute space. Some are building open hardware based on OCP designs. Others are procuring and deploying OCP hardware. Resellers are beginning to build businesses around open hardware.

A community has formed around the project, attending two OCP summits last year to collaborate, share and learn.

And the pace of innovation is accelerating: great ideas are flowing through the incubation committee, ideas shaped by industry’s best minds. The committee decides on the best projects for the OCP.

OCP is evolving into a project based on open innovation, where we publish specifications before we build prototypes. This way, we can get feedback from the community and improve upon the designs before they launch as products. This is unlike anything we’ve seen in the scale compute industry so far.

We had no idea we could have made this much impact this quickly. And to keep up the momentum, we’re gathering again in San Antonio for our third OCP summit to talk about the OCP’s focus and values, the progress we’ve made and what we plan to do next.

See you in San Antonio!

LGE has helped help the foundation structure its financials, working on budgets and setting up the non-profit status of the foundation. They are bringing their expertise to the upcoming Open Compute Summit May 2-3 in San Antonio, TX, where they will track expenses and sponsorship funds.

Managing partner Rocky Bullock said they are looking forward to helping out and being part of the cloud computing environment, adding, “it’s a great honor to be part of the foundation. It’s doing the right thing for the industry. ”

Nick Bullock, LGE’s operations manager, brings non-profit experience to the table. He has served on the board of the Austin chapter of the Coastal Conservation Association for the past year, and he notes that the foundation is structured similarly. Nick will use his experience to help ensure that the foundation maintains excellent financial health for the long term.

You can download the spec now from GitHub.

If you’re interested in keeping up to date on the open rack, join the mailing list.

We look forward to your feedback and contributions to this and other Open Compute Project vectors.

The Open Compute Project Foundation and its board were announced. Modeled after the Apache Software Foundation, the OCP Foundation will design and deliver tangible goods and source files to let people deploy OCP hardware in their environments. The five members of the OCP Foundation board are:

• Andreas (“Andy”) Bechtolsheim, Founder of Arista Networks and former Chief Hardware Designer at Sun Microsystems
• Don Duet, Vice President of Information Technology at Goldman Sachs
• Frank Frankovsky, Director of Technical Operations at Facebook
• Mark Roenigk, COO at Rackspace
• Jason Waxman, General Manager of the Data Center Group at Intel

The board reflects a diversity of industries, from supplier to consumer. You might be asking yourself why Goldman Sachs sits on the board. Facebook’s Frank Frankovsky says that financial services companies “are IT companies more than they actually know,” running large-scale compute environments.

He also said Intel has “one of the richest portfolios of thought leadership of tech in the industry,” and that it’s very important for Intel to be part of the OCP, so others will follow their lead.

Rapid Innovation Requires Open Standards 

It almost goes without saying that standards in computing are vitally important. The advent of open source software and standards ushered in an era of great advancements in innovation. Andy Bechtolsheim of Arista said the same needs to be done on the hardware side, because there is a lack of standards at the system level. In order for there to be innovation in the scale compute ecosystem, the “gratuitous differentiation” where one vendor’s hardware is different than another’s must be eliminated. Calling gratuitous differentiation “the enemy” and innovation “our friend,” Bechtolsheim said there is a need to “create a mutual benefit for customers and vendors by creating a new market for open-standard system-level designs.”

This view was echoed by Jimmy Pike, Senior Distinguished Engineer, Chief Architect and Technologist at Dell Data Center Solutions. “Standards can set you free,” he said. Open hardware should have standard form and fit, physical interfaces, management interfaces, and technology elements. He mentioned Dell’s Nucleon server, its entry level OCP platform.

Brian Stevens, CTO and Vice President, Worldwide Engineering at Red Hat, likens the mission of the OCP to that of Red Hat’s, in that both are a “catalyst in communities of customers, contributors and partners building better technology the open source way.” He noted that Red Hat certified the first two OCP systems for Red Hat Enterprise Linux, and that Red Hat certification brings hardware compatibility.

Success at Scale

Innovation in data center technologies has been flourishing in recent years, and the pace is only quickening. James Hamilton, Vice President and Distinguished Engineer for Amazon Web Services, said there has been more innovation in the past five years than in the previous 15. Innovations like evaporative cooling, full building ductless cooling, and using outside air — all technologies used at Facebook’s Prineville, OR, data center — have brought down costs, increased reliability, and reduced the environmental footprint of data centers.

To give an idea about how much Amazon has had to scale in recent years, every day “Amazon Web Services adds enough capacity to support all of Amazon.com’s global infrastructure through the company’s first 5 years.”

The OCP will “democratize and bring together much more choice in the industry for people to get efficient platforms,” Jason Waxman of Intel said, adding that Intel has a long history of supporting open standards, like PCI and wi-fi. The trend is the same. “If you present people with an open spec, everyone can innovate.”

From Workshops to Working Groups

One of the primary results of the summit came from the five technical workshops that were held. The workshops covered data center design, hardware management, open rack, storage, and virtual I/O.

The output from each workshop is being turned into a charter and specification that we’ll share with the community. We also set up complementary working groups along these same themes. Join the discussions by subscribing to any or all of the lists that you are passionate about:

• For general information: lists.opencompute.org/mailman/listinfo/opencompute-all
• For data center design: lists.opencompute.com/mailman/listinfo/opencompute-datacenterdesign
• For hardware management: lists.opencompute.org/mailman/listinfo/opencompute-hardwaremngt
• For open rack: lists.opencompute.com/mailman/listinfo/opencompute-openrack
• For storage: lists.opencompute.org/mailman/listinfo/opencompute-storage
• For virtual I/O: lists.opencompute.com/mailman/listinfo/opencompute-virtualio

We look forward to your involvement in the community. See you online!

These features have enabled Facebook to reduce the energy consumption of the data center significantly, which is reflected in power usage effectiveness (PUE) of the facility. The PUE of the Prineville data center was 1.07 at full load, which was verified during commissioning. Since then, during normal operation of the facility, the PUE has varied between 1.06 and 1.1. The histogram of the available PUE trend data for the period of April 14, 2011, to September 30, 2011, is presented in figure 1 below.

Figure 1 PUE of Prineville Data Center

Challenges in Operations

Although these features have resulted in high efficiency, we have learned some lessons along the way. And as a part of our commitment to openness via the Open Compute project, we are sharing our experiences and lessons learned with the community, so that everyone might benefit from them.

One challenge we encountered was keeping our air handler lineups from “fighting” with each other as they dealt with the rapid changes in the temperature and humidity of the outside air between day and night. For example, if outside air dampers of one lineup are at 70%, the adjacent lineups would have their outside air dampers at 20-30%. This alternate modulation, or fighting, often led to stratification of air streams.

Another, more significant, issue was an error in the sequence of operation controls that led to complete closure of the outside air dampers, causing the one-pass airflow system to function like a recirculatory system. The problem began to manifest in late June as outside air conditions started changing rapidly. The economizer demand signal began responding to the changes; that’s when the erroneous control sequence drove economizer demand to 0, leading to complete closure of the outside air dampers. Thus the data center was recirculating the hot exhaust air at high temperature and low humidity. The evaporative cooling system reacted to this high temperature and low humidity, spraying at 100% to maintain the maximum allowed supply temperature and dew point temperature. This resulted in cold aisle supply temperature exceeding 80°F and relative humidity exceeding 95%. The Open Compute servers that are deployed within the data center reacted to these extreme changes. Numerous servers were rebooted and few were automatically shut down due to power supply unit failure.

Figure 2 Failed component in power supply unit

The high temperature and high humidity supply air caused condensation on the concrete slab floor (because concrete has high thermal mass and was in contact with much cooler supply air for a long time). Similarly, upon investigation of the failed power supply units (figure 2), we observed that the failure was condensation-related.

Issue Analysis

We began investigating this failure by subjecting the server to rapidly changing temperature and humidity conditions in a controlled test chamber. The relative humidity level was raised to 97% and the temperature was ramped up from 15°C to 30°C (59°F to 86°F) in the span of 10 minutes. Under these conditions, the condensation was observed on the non-heated components. The server chassis was dripping wet, as you can see in figure 3. The motherboard, however, showed no signs of condensation due to the fact that it always ran above the dew-point temperature.

Figure 3 Condensation on the server chassis

Condensation was also evident on the surfaces of power supply components such as capacitors and inductors, as shown in figure 4.

Figure 4 Condensation on the power supply components

Figure 5 below shows the surfaces of inductors in front of capacitor 1 and the forward vertical surface of capacitor 1. We can see the water droplets formed on the surfaces of these non-heated components.

Figure 5 Inductors and capacitor surfaces viewed from a borescope video

Figure 6 shows the variation in different temperatures monitored during the test interval. These are both targeted and actual values of ambient as well as dew-point temperature. The surface temperature of capacitor 1 (CAP1) is also plotted.

Figure 6 Temperature variation

The plot shows that the surface of CAP1 falls below the dew point at about 6 minutes into the temperature ramp. This is exactly the same time the borescope video starts showing a slight change in the reflectivity of the component surfaces. The condensation then continues for another 9 minutes until the surface temperature of CAP1 rises above dew point. During the entire test interval, the PCB in the power supply always ran above the dew point temperature and showed no signs of condensation.

All these findings suggest the possibility that the failures were caused by water droplets being blown onto the PCB of the power supply, rather than condensation occurring on the PCB itself. As shown in figure 7, the water droplets were observed on the AC/DC cables and connectors. It is highly likely that these droplets were blown into the power supply units when the facilities’ maintenance staff increased the airflow in efforts to mitigate the problem.

Figure 7 Condensation on cables and connectors

Corrective Actions

The erroneous control sequence was promptly corrected and additional safeguards were added to eliminate the possibility of repeated occurrence of such an event. These safeguards include reevaluation of the minimum economizer demand setting, which will avoid the complete closure of the outside air dampers. Several monitoring points and alarm settings were modified to monitor and notify ahead of time should outside air conditions begin to change rapidly. Even though the supply air humidity, which was more than 95% at times, was out of the operational range of the power supply units (10-90% RH, non-condensing), conformal coating has been applied locally in selective areas of the PCB to avoid condensation and to strengthen the power supply units against such corner cases.

As this is a community effort, we would love to hear what you’re doing with the Open Compute Project. How do you plan on implementing OCP technologies in your data centers? What do you hope to contribute back to the project?

If you have something you want to share with the community, please tell us and we will post it on this blog. And do let us know if you would like to contribute content to the blog directly.

Of course, this is a huge undertaking — one that no single organization could possibly tackle alone. And with that in mind, we’re thrilled to announce a collaboration with the Open Data Center Alliance that will enable the OCP to get its hardware and designs in front of the 300-plus members of the ODCA so they can evaluate, adopt, and innovate on them.

To begin, the two groups will work together on projects focused on rack-scale infrastructure; efficient server and storage designs; and scalable, open systems management. The collaboration will deepen over time, and we should have more details to share at the upcoming Open Compute Project Summit in New York on October 27 (which you can register for here).

We have a long way to go in figuring out what the next generations of web infrastructure should look like. But the kinds of openness and collaboration exemplified by today’s announcement are exactly what we need to get there.