The Large Hadron Collider (LHC) at CERN, the European Organization for Nuclear Research, is the largest and most complex scientific experiment ever constructed and a collection of superlatives. A 27-km long accelerator ring constructed underground near Geneva accelerates particles to 99.9999999996% the speed of light.
The particle beams in the LHC travel in a vacuum as empty as interplanetary space and with a pressure of 10 x 10-13 Torr, 10 times less than that on the Moon. The colliding beams generate temperatures more than 100,000 times hotter than the Sun, but circulating helium keeps the LHC at -271.3°C (1.9 K)—colder even than outer space.
The ATLAS experiment, one of four major experiments conducted with subatomic particles accelerated in the Large Hadron Collider at CERN, is housed in the largest man-made cavern. Eight toroid magnets surround the calorimeter, one of the detectors that examines particles produced when protons collide in the center of the detector.
Courtesy of CERN,
photo by Maximillen Brice
The magnets that guide the proton beams use enough superconducting filament to stretch to the Sun and back more than five times. There is enough energy in each of the two particle beams to melt 500 kg of copper. This high energy can produce short-lived heavy particles, making it possible to explore states of matter that were present just 10-25 s after the Big Bang.
The beams collide at four locations along the ring corresponding to the four main experiments with which researchers look for, among other things, evidence of particles that have never been seen, such as the Higgs boson, and for information about dark matter. Together with dark energy, dark matter makes up the rest of the universe beyond the 4% of matter that is directly observable to us.
Likewise, the job of creating sensors, instrumentation, and triggers to detect these events is a job in the superlative category. These engineers, for instance, have to deal with a whole new class of noise, here defined as any physical effect that has nothing to do with the desired physics.
An obvious concern is ambient cosmic radiation, so the experiments are placed between 45 meters and 170 meters below the surface. Another is gravity from the Moon. In the Geneva area, the Earth’s crust rises by some 0.1 mm, causing variations in the LHC’s configuration. Another effect is the weather, which can change the weight of the nearby Jura mountains and has an effect on the ring’s configuration.
Petabytes of Data
Next, consider the massive amount of data that comes from these experiments. Every second, 600 million particle collisions or events are measured, and with special triggers, scientists sort out the hundred or so per second that are interesting.
The limiting factor on the number of events that can be accepted for analysis is storage. The electronic photo of each event requires roughly 1.5 MB, and the dataflow from all four experiments is roughly 700 MB/s—some 20 petabytes or 20 million gigabytes per year. This data is backed up locally and through the LHC Computing Grid1 and distributed to researchers around the world who will analyze this data for at least the next decade.
How is this data generated? It takes very special sensors to measure the energy, trajectory, and momentum of subatomic particles traveling at such incredible speeds and whose lifetimes are as short as 10-12 s, corresponding to a flight distance of just a few hundred microns. Some particles are even shorter lived, and the scientists can measure only their decay products. Designing the sensors and data acquisition systems, filters, and triggers is just as massive an undertaking as the accelerator itself.
Even so, explained research fellow Dr. Bilge Demirköz at CERN, the four major experiments tied to the LHC all work with the same basic principles. A Toroidal LHC ApparatuS (ATLAS) is the physically largest of the experiments.
To understand the task of analyzing particle collisions, recall that particle traces from the first cloud and bubble chambers were captured on photographic film. For analysis, technicians made hand measurements literally using rulers, protractors, and spirals. From this data, physicists could determine the particles’ charge, momentum, and lifetime and identify them.
ATLAS does much the same thing but creates a sophisticated 3-D electronic image that can be analyzed in software. It does so with a series of three types of detectors arranged in concentric cylinders, each looking for a specific type of particle: the inner detector, calorimeters, and a muon detector (Figure 1).
Closest to the beam is the inner detector, which consists of three sections: the pixel detector, the semiconductor tracker (SCT), and the transitional radiation tracker (TRT). There is one large magnet surrounding all three sections of the inner detector and another at the outer layer surrounding the muon spectrometer (Figure 2a and 2b).
Inner Detector 1: Pixel Detector
Although the pixel detector located closest to the beam is the smallest detector in terms of volume, it has the highest number of readout channels for the highest position resolution: 14 µm. As its name implies, it acts much like a digital camera except that it is configured as three concentric cylinders.
It is implemented with 1,744 modules measuring 2 x 6 cm and arranged in three cylinders (Figure 3a and 3b). Each module holds a plate of 250-µm thick silicon divided into 46,080 pixels each 50 x 400 µm, leading to a total of more than 80 million pixels. By examining the points where a particle passes through the pixels, the researchers can extrapolate a particle’s path.
Figure 3b. A Barrel Pixel Module
Courtesy of CERN
When not excited, very little current flows through such a pixel diode, explained Dr. Jens Weingarten at CERN. A charged particle passing through creates electron-hole pairs that induce a current pulse of several nanoamps. A dedicated readout channel monitors each pixel and reads 8 bits of brightness.
To connect each pixel to a separate instrumentation channel, the developers use bump bonding with 0.02-mm solder droplets deposited on the pixel matrix and on the electronics, leading to a density of 5,000 connections per cm2. Each channel has a front-end (FE) chip that processes, digitizes, and compresses the signal. The signal is sent out through two fast serial links on the module controller chip (MCC), which emits a signal only when hits above a certain threshold are detected.
Inner Detector 2: Semiconductor Tracker
As you move farther out from the collision, the position resolution of the detectors decreases, which is necessary because of the tracker’s increasing volume and the associated sensor costs. Accordingly, the second level of the inner tracker is a four-layer semiconductor tracker (SCT). The sensors are silicon plates measuring 64 x 63.5 mm; each has 768 microstrips 80 µm wide x 12.6 cm long.
The SCT consists of 4,088 modules configured in four concentric layers and has 6.2 million readout channels. When added together, the silicon in the SCT, 285-µ thick, covers a surface area of approximately 60 m2.
A charged particle ionizes the n-type silicon, and the movement of 20,000 electrons in 2 ns to 3 ns induces a nanoamp signal on an aluminum base. Because one plate can identify only which microstrip a particle has hit, two plates are placed on top of each other at a slight skew. When a particle goes through two lines, the point of impact can be determined within 20 µm, and a particle traveling straight through interacts with all four layers to trace its path.
The pixel detector must be able to work through intense radiation, 10 MRad and 2 x 1014 particles/cm2, the equivalent of 1010 chest X-rays per year. As a result, radiation hardness is a top priority for detector and read- out electronics.
The silicon is treated to allow operation when partially depleted. Other measures include AC coupling and polysilicon bias resistors as well as low-temperature operation (-8°C) to minimize the effects of reverse annealing.
Further, each module in the pixel detector and SCT contains a radiation-hardened custom chip. The electronics must be immune to a single-event upset (SEU), a change of state caused by a low-energy ions or electromagnetic or nuclear radiation interferences strike to a sensitive node in a microelectronic device. Consequently, the readout and triggering electronics are routinely built up in triplicate with a voting mechanism.
Inner Detector 3: Transition Radiation Tracker
The third and outermost stage of the inner detector is the TRT, which registers points along a particle’s path and helps distinguish between electrons and pions. It consists of almost half a million gas-filled straws 4 mm in diameter and 144 cm long made of Kapton with a 0.03-mm diameter gold-plated tungsten wire in the center. A high voltage is applied between the straw and the wire, and a charged particle passing through ionizes the gas molecules. The TRT specs a position resolution of 0.17 mm.
Don’t Overlook the Obvious
“In designing the inner tracker, we spent considerable time thinking about known problems, but we also ran into challenges on what we thought was trivial,” explained Dr. Andreas Korn, who worked on the pixel detector at CERN. “We found that we also had to push the state of the art in cooling and cabling.”
The silicon detectors must be kept at a temperature of -7°C. Dissipated power in the detector, up to 15 kW, is removed by an evaporative cooling system, but the pipes must be as small as possible to not add mass. The 6-mm cooling pipes are subject to condensation from the C3F8 in the cooling loop that goes from 13 bar at the input to 0.8 bar at the exit. In certain spots, the cooling pipes themselves must be heated so they don’t subject sensitive electronics to moisture.
Another challenge was cabling, according to Dr. Daniel Dobos at CERN. For example, some cables in the pixel detector bundle 25 wires, each 100 µm to 300 µm in diameter. They are made of aluminum, which is far lighter than copper and disturbs particles less as they travel out to other parts of the detector.
The team had to develop a special process to reliably attach the fine wires to connectors. The polyurethane insulation also is very special for voltage stability, and they had to find an institute that could handle the special chemicals needed to strip off the insulation.
Calorimeters
One major design goal in the inner detector was to keep the amount of mass to an absolute minimum so the particle trajectories would not be disturbed. In contrast, the next stage, the calorimeters, is designed to absorb particle energy. Particles with short lifetimes of <10-12 s never get past the pixel or SCT detector. But to measure the path and momentum of more stable particles, the next stage consists of two different instruments: an electromagnetic calorimeter followed by a hadronic calorimeter.
The electromagnetic (EM) calorimeter absorbs energy from particles that interact electromagnetically, including all charged particles such as electrons and photons. The energy-absorbing materials are lead and stainless steel with liquid argon as the sampling material. The absorber plates have an accordion geometry that provides a uniform response over the entire angle of coverage with more than 170,000 readout channels.
Particle showers in the liquid argon produce ions, which are read out as electric pulses by Kapton electrodes. The EM calorimeter has high precision both in the amount of energy absorbed and in the location of the deposited energy. The angle that covers the particle’s trajectory is measured to within roughly 0.025 radians.
The outer hadron calorimeter absorbs energy from the particles that pass through the EM calorimeter: protons, pions, kaons, and other hadrons, which are particles composed of quarks. The energy-absorbing material is steel, and the sensors are tiles of scintillating plastic that emit light when charged particles pass through.
The light pulses are transmitted by optical fibers and converted to electrical signals by photomultiplier tubes. Each of the 256 modules in the hadron calorimeter is controlled and read out by only three optical fibers. There are 10,000 readouts in total. It is less precise in localization, here within about 0.1 radians.
Muon Spectrometer
Muons, particles very much like electrons but 200 times more massive, pass through the calorimeter without being absorbed. However, measuring their passage through the final instrument, the muon spectrometer, is vital for two reasons:
• A muon indicates that something very interesting has happened in a proton collision.
• The total energy of particles in an event cannot be measured accurately if you ignore one of them. Without a full energy balance, you can’t determine if there are other invisible or nondetected particles also resulting from a collision.
The muon trajectories are bent by a second set of magnets that has enough power to power the city of Geneva for 10 seconds. This tracking system is large, extending from a radius of 4.5 meters around the calorimeters to 11 meters. It has 1,000,000 readout channels, and its layers of detectors have a total area of 12,000 m2.
The major method of measuring the muons is with drift tubes, an array of gas-filled 3-cm tubes with anode wires along their axes. By measuring the time for electrons produced by ionization to drift to the wires, muon position can be determined to 80 µm.
Triggering
For each proton collision, 90 million channels of data in the full ATLAS detector are captured and stored in pipeline memories. Even though the LHC produces collisions in ATLAS every 25 ns, meaning 40 million events per second, only a very small fraction of these collisions produce interesting events. Therefore, ATLAS includes three-level triggering (Figure 4) to capture data on only 200 events/s. The number is limited by the speed at which event data can be sent to disk storage (200 events/s x 1.5 MB/event = 300 MB/s output data bandwidth) and the resources available for subsequent storage and analysis. The first decision from the three-stage trigger must arrive within 2.5 µs before the data buffers are overwritten during the next event.
The Level 1 trigger is performed exclusively in hardware and based on data from the calorimeters and the muon detector. It looks for obvious high-energy signatures such as the existence of a muon.
Figure 2a shows that most of the detector volume contains little of interest. So another job for this trigger is to identify regions of interest for study by the following triggers, which use full granularity information from all the detectors. Fewer than 75,000 events/s pass the Level 1 trigger test.
Dr. Simon George, who helped build the trigger system at CERN, explained that a decision for the Level 1 trigger must take place in 2.5 µs. But consider that the time just for the electrical signals to move back and forth to the trigger electronics located 60 meters away takes up more than 1.3 µs. The actual time available to the trigger electronics is roughly 1 µs.
As a result, the Level 1 trigger is simple, fast, and implemented mainly in FPGAs and ASICs. All 1.5 MB of the event data are stored in pipeline memories until the Level 1 decision is available.
If the decision is positive, data is read out to readout drivers (RODs) in an underground counting room. The RODs perform simple tasks such as digitization and data formatting and then pass the data onto readout buffers (ROBs), where it is held pending the decision of the Level 2 trigger.
A readout buffer is equipped with optical links and sophisticated on-board memory. Roughly 600 of these boards reside in the underground counting room. The board provides temporary buffering of event fragments for the duration of the Level 2 decision and for some events during the event-building process. All functions related to the receiving and buffering of event fragments on a board are implemented in an FPGA.
The Level 2 trigger checks the regions of interest (ROIs) from the first level but at full granularity, also adding information from the tracking detectors. It is implemented in custom fast algorithms running on a PC farm to meet a 10-ms target time. The event rate is now dropped to roughly 3,000/s.
The Level 3 trigger, known as the event filter, performs its analysis in approximately 1 s. Again, it looks at the ROIs from Level 2 but in more detail, using some of the same algorithms as will be used later for off-line analysis.
Only roughly 200 events/s pass through this final triggering stage, and this final data is sent on to the central computer center for storage and distribution to researchers around the world. In addition, 10% of the events selected by the triggers are done with minimum bias, meaning there are actually few selection criteria. This is to make sure that the researchers haven’t missed anything beyond what they are looking for.
Level 2 and the event filter combine to form the high-level trigger (HLT). The hardware for the HLT amounts to 850 PCs that can be assigned between the two triggers as needed. The PCs are 1U boxes with two quad-core CPUs and 16 GB of memory. These computers are located in a surface building connected to the underground counting room by a high-bandwidth network.
References
1. Schreier, P., “A Universe of Data,” Scientific Computing World, August/September 2008, pps. 41-44.
2. More than a thousand pages describing all the LHC experiments have been published in the Journal of Instrumentation. Free online access is available at http://www.iop.org/EJ/journal/-page=extra.lhc/jinst
About the Author
Paul G. Schreier is a technical journalist and marketing consultant working in Zurich, Switzerland. He was the founding editor of Personal Engineering & Instrumentation News, served as chief editor of EDN Magazine, and has written articles for countless technical magazines. Currently, he is the editor for LXI ConneXion at EE-Evaluation Engineering. Mr. Schreier earned a B.S.E.E. and a B.A. in humanities from the University of Notre Dame and an M.S. in engineering management from Northeastern University. e-mail: [email protected]
November 2008