Ever alert and always on the job, the NCCS’ control room operators keep its computer systems running all day and all night
With more than 1,700 researchers around the world using the Oak Ridge Leadership Computing Facility’s Frontier to pursue some of the biggest mysteries in science, the exascale-class supercomputer gets little downtime. At any given hour, it’s typically running complex codes to investigate everything from molecular dynamics to explosive stellar astrophysics. And that means somebody must make sure Frontier is running every minute of every day and every night, without fail.
That duty falls upon the shoulders of the control room operators in the HPC Infrastructure Operations Group at the National Center for Computational Sciences, which is located (along with the OLCF) at the Department of Energy’s Oak Ridge National Laboratory. But keeping OLCF supercomputers running is only one task among a myriad of responsibilities entrusted to ORNL’s six operators: Jack Breazeale, Connor Cunningham, Cindy Leach, George Phipps, Teresa Wilson and Curtis Wyatt.
“Operators work 24 hours a day, seven days a week, 365 days a year, so they’re a continual asset that we have on site,” said Paul Abston, HPC Infrastructure Operations Group leader. “They’re my extra 12 arms and my extra 12 eyes. Many of them have had a presence in the OLCF since its conception.”
As their job title indicates, operators spend a lot of time in the Operations Control Room (OCR). It’s a small space packed with large monitors, each one reporting the status of different systems: supercomputers, cluster machines, networks, and the High-Performance Storage System, among others. Operators staff the OCR two per day shift and one on night shift, watching for any sign of the dreaded color red, which indicates an issue. Work-slowing problems can involve anything from a login node going down to a file system hitting capacity. Once alerted, no matter the hour, operators will first make the call to whichever manager is responsible for the machine in question.
“We’re the guys that call the systems administrators at two in the morning, wake them up, and say, ‘Hey, you’ve got a problem on one of your systems,’” said Breazeale, who’s worked at ORNL for the last 35 years.
Operators do get some exercise by walking rounds each night through all the different facilities, looking for anomalies that might indicate a problem. Monitoring the electrical and cooling systems that support Frontier and the other supercomputers is also on their to-do list, so they watch out for water leaks, smoke or excess heat.
“In the event of an upset condition, whether it’s a power-quality event, a power outage, or the unexpected, they’re our first resource to bring things back to a stable state,” Abston said.
Some of the operators’ other duties are less dramatic: double-checking inventory to ensure no changes have been made inside the computer cabinets, taking in shipments, even delivering office mail.
Then there are the phone calls. During the day shift, operators typically receive calls from Frontier users who are experiencing a problem logging in and need help to resolve it. But they also get the occasional wrong number, too. Phipps, who’s worked at ORNL for 41 years, has answered calls intended for the ORNL Federal Credit Union.
“Their phone number is similar to ours, so people will call and ask for their account numbers or try to transfer some money into their account,” Phipps said. “We just take these wrong-number calls in stride.”
But it’s the night shift that gets the more unusual calls.
“On off-hours you’ll get people who used to work here on a project years ago—the atomic bomb or something—and they’ll call wanting to know if that research is still going on,” Breazeale said. “But we get some wild phone calls, too. I had one guy call who wanted to talk about some type of experience he had with aliens.”
Being an expert in providing assistance no matter the situation, Breazeale did not hesitate to offer the caller a helpful resource: “I told him I wasn’t sure if I had the information that he was requesting, but that he might be better off to call back on the day shift − perhaps he could talk with someone who had more knowledge than I did on that situation.”
UT-Battelle manages ORNL for DOE’s Office of Science, the single largest supporter of basic research in the physical sciences in the United States. The Office of Science is working to address some of the most pressing challenges of our time. For more information, visit energy.gov/science. − Coury Turczyn