Building the Phone Rack for Facebook’s Mobile Lab

July 14, 2016
To scale up its Mobile Device Lab, Facebook built custom racks to house 2,000 smartphones for application testing in its Prineville data center. Here’s a look at the design process for this offbeat challenge.

PRINEVILLE, Ore. – The call may have gone something like this:

HQ: “Hey, we need you guys to build a custom cabinet for us.”

Data Center: “OK. What kind of hardware will run inside this cabinet?”

HQ: “It’s a few dozen mobile phones.”

Data Center: “Wait. What?”

That conversation led to the creation of Facebook’s Mobile Device Lab at its huge data center complex in central Oregon. Facebook uses the lab to test drive revisions to its applications across up to 2,000 different types of smartphones, representing the diverse universe of technology found among Facebook’s 1.6 billion users.

This testing is crucial to maintaining a good user experience. But why does a rack full of phones need to live in one of the world’s most advanced data centers?

Facebook tried doing this testing at its headquarters campus in Menlo Park, Calif., but the task proved unwieldy, as engineers couldn’t create a hardware device testbed that could give it the scale and consistency it was seeking. (See Facebook’s blog post for a description of their initial attempts to create a lab in their Silicon Valley offices).

A Home in the Cold Storage Aisles

Still, it didn’t make sense to house cabinets full of phones in Facebook’s production data centers in Prineville, which run the latest Open Compute in racks that can support up to 12kW of power capacity.

One of the custom racks for Facebook’s Mobile Device Lab, which houses up tom32 smartphones. At right you can see the insulation and other adaptations to the door of the rack. (Photo: Rich Miller)

Fortunately, there was a better option: the Facebook cold storage facility in Prineville, which houses exabytes of rarely accessed photos uploaded by Facebook users. The cold storage facilities are simpler data centers with no generators or UPS (uninterruptible power supply). Emergency backup power isn’t needed because the facility doesn’t serve live production data.

The primary challenge in building racks for the Mobile Device Lab was managing wifi access to simulate how apps will run for end users – who typically don’t hold their phone next to hundreds of other phones when using Facebook.

“If we were to use the same racks for mobile device testing that we use for our servers in our data center, our tests would be unsuccessful because the Wi-Fi signal from one rack would interfere with the Wi-Fi signal from the next rack,” said Antoine Reversat, who demonstrated the lab for media during a tour of the Prineville facility is week. “We needed racks to be well-isolated from their neighbors — ensuring a consistent environment — in order for our tests to be reliable and to have reproducible results.”

To Each His Own Wireless

To solve this problem, the Facebook data center team built custom racks that function as an electromagnetic isolation (EMI) chamber. Each rack contains a wireless access point and an overhead camera. The phones lie on a slightly tilted pegboard so the cameras can record their screens.

There were several considerations in determining how many phones could be housesdin each cabinet. A key factor was the network infrastructure and hardware to monitor the tests.

Each rack holds eight Mac Minis (or four OCP Leopard servers for Android testing) that drive the phones to install, test, and uninstall the applications for Facebook, Instagram and WhatsApp. Each Mac Mini is connected to four iPhones, and each OCP Leopard server is connected to eight Android devices, for a total of 32 phones per rack.

Insulation, and More Insulation

The early prototypes needed some work.

“When we tested our EMI enclosure, we found that it was about as efficient at blocking radio signal as a sieve is at blocking water,” said Reversat.

To turn that sieve into a bucket, Facebook took the following steps:

  • Grounded the EMI enclosure by connecting it to one of the metallic surfaces of the rack itself.
  • Covered the inside seams with copper tape.
  • Added insulation foam inside the rack.
  • Added a 100 dBm power filter (i.e., a filter for power, USB, and ethernet that’s rated to filter out electromagnetic radiation and to ensure that it doesn’t interfere with the Wi-Fi signal inside the enclosure).
  • Added even more insulation foam.

Facebook now has around 60 racks for mobile device testing in one of its cold storage facilities in Prineville. The data center team continues to refine the design, and hopes to densify the current rack design to hold up to 64 phones per rack (versus the current 32).

“There are multiple challenges we’re working through here,” Reversat wrote in his blog post. “For all devices, we have to maintain a 4-foot air gap between the phones and the antenna for the signal to be sufficiently attenuated once it reaches the phones. We don’t have a lot of space to stack the phones if we want to have two shelves instead of one. Additionally, we intend to get larger form-factor phones in the rack next year, which we will need to consider in the redesign.”

Some of the 60 custom racks for Facebook’s Mobile Device Lab inside the cold storage facility at the Prineville, Oregon data center campus. (Photo: Facebook)

About the Author

Rich Miller

I write about the places where the Internet lives, telling the story of data centers and the people who build them. I founded Data Center Knowledge, the data center industry's leading news site. Now I'm exploring the future of cloud computing at Data Center Frontier.

Sponsored Recommendations

NECA Manual of Labor Rates Chart

See how Champion Fiberglass compares to PVC, GRC and PVC-coated steel in installation.

Electrical Conduit Cost Savings: A Must-Have Guide for Engineers & Contractors

To help identify cost savings that don’t cut corners on quality, Champion Fiberglass developed a free resource for engineers and contractors.

Conduit Sweeps and Elbows for Data Centers and Utilities

Data Centers and Utilities projects require a large number of electrical conduit sweeps and elbows. Learn why Champion Fiberglass is the best supplier for these projects.

Prefabricated Conduit Duct Banks Enable Smooth and Safe Electrical Installation for a Data Center

Prefabricated conduit duct banks encourage a smooth, safe electrical conduit installation for a data center.

Courtesy of ZincFive
Image courtesy of ZincFive

The Emergence of Immediate Power Solutions (IPS) for Data Centers

Tod Higinbotham, COO of ZincFive, explains why it’s essential data center operators choose the best battery for each application instead of taking a one-size-fits-all approach...

White Papers

Get the full report

Ethernet in Data Center Networks

Aug. 1, 2022
This white paper from Anritsu discusses Ethernet usage trends in data center networks, as well as the technologies helping operators meet growing bandwidth demands and verify ...