The European Organization for Nuclear Research (CERN) reported last month that researchers may have discovered a new fundamental particle of matter. Despite the cautious enthusiasm of CERN physicists, it could take months for computers to comb through the ocean of data produced by the Large Hadron Collider and confirm that the particle actually exists.
As the physicists wait for more information, CERN’s IT department is engaged in a different kind of research. Under the organization’s Openlab program, CERN has partnered with educational institutions and private companies to ensure that, in the future, its computers can sift through increasingly complex data on particle collisions.
The latest project involves using new high-speed interconnects to link thousands of processors in CERN's data center. Updating the interconnects will help to optimize its existing processing power—an alternative to adding more processors to increase its blunt force. The processors are designed to run algorithms that filter out uninteresting particle “collision events” that happen in the Large Hadron Collider. The goal is to find unexpected interactions or anomalies that could suggest the presence of new particles.
CERN’s demand for faster processors stems mainly from the enormous amount of data produced by the Large Hadron Collider. The collider smashes particles together at near the speed of light and monitors the particle shards—for instance, those of the Higgs boson particle that was discovered in 2012—spraying in different directions. The shards leave behind traces in space that are recorded by “detectors” that take roughly 14 million snapshots per second of the particle interactions. These frames of data translate into about 3 gigabytes of data per second (about 25 petabytes or 25,000 terabytes per year).
At the core of the new interconnects is the high-speed communications standard RapidIO, which has been widely used to connect processors in cellular base stations and network servers. RapidIO can support up to 20 Gbits/s interconnects directly on processors without network interface controllers, adaptors, and software intervention.
According to Integrated Device Technology Inc. (IDT), which designed the interconnects for CERN, the latency of RapidIO can be as low as 100 ns between switches and less than one microsecond between processing nodes. The standard can be used to establish a connection between chips, boards, and chassis.
Although latency was the central concern, the new interconnect also help to reduce the data center’s massive power demands. Communication between processors on the same chip takes little energy (on the order of microwatts). But according to Barry Wood, principal product applications engineer at IDT, the communication between processors is significantly higher, from hundreds of milliwatts to watts. In supercomputers and servers, that can add up to hundreds of megawatts (MW).
The interconnect project underlines the creativity that has defined CERN’s IT program over the years. In 1989, for instance, the organization created the World Wide Web as a way to distribute its research data. Building on its World Wide Web Technology, CERN invented grid computing in 2002 to process particle collision data using computer systems from around the world. Today, the Worldwide LHC Computing Grid sends information to 170 data centers in 42 countries.
CERN physicists will keep raising the energy at which the Large Hadron Collider fires protons in search of new particles that could reveal deeper physical laws. During its first two years running, the it fired protons to energies of about four trillion electron volts. But since a two-year shutdown ended last June, CERN physicists have been firing protons with 6.5 trillion electron volts of energy. To find more esoteric particles, the collider will have to operate at even higher energies, creating more violent particle collisions. In turn, these collisions produce more data.
Before the shutdown, the data center was storing 1 gigabyte-per-second, with the occasional peak of 6 gigabytes-per-second, according to Alberto Pace, head of the Data and Storage Services in CERN's IT department. But now, “what was once our ‘peak’ will now be considered average, and we believe we could go up to 10 gigabytes-per-second,” he says.
The initial phase of the interconnect project will focus mainly on connecting a small number of processor nodes. But during the three-year research collaboration, IDT and CERN engineers have plans to build large scale computer systems and start using them to analyze data.