Welcome to the new war games, fought not on land or sea but on the cyber frontier.
Computer scientists at Oak Ridge National Laboratory are helping the US Navy test its digital defenses and try out new ways to counter hackers and other online outlaws. The partnership offers an operational evaluation opportunity for contractors and a proving ground for researchers’ theories on how to keep the nation’s computer networks safe.
As part of this effort, ORNL’s Cybersecurity Research Group in the National Security Sciences Directorate is conducting multiple test events that will inform future US Navy acquisition decisions in computer network defense.
“The tools of future cyberdefense are going to be built around these results,” said Justin Beaver, who leads the Cybersecurity Research Group.
Today’s military procurement officers get no end of sales spin from vendors promising the next big thing in cybersecurity. Filtering out the features from the fluff sometimes calls for help from specialists.
“The sales pitch is always sparkly but actually evaluating technologies that use machine learning for cyber defense is very complicated and math-intensive. There are conventions but no industry standards on detection efficacy,” Beaver said. “The role we’re playing is the honest broker. We have the machine-learning expertise and the cybersecurity experts, so we have the ability to ask the deeper questions that help determine if and how a technology might be viable.”
The ORNL team put some of those technologies through their paces this summer at the Lockheed Martin Corp’s National Cyber Range in Orlando, Florida. Researchers spent four weeks from July 22-August 16, 2019, pitting new approaches against older methods, including a test drive with analysts from the US Department of Defense. “It’s essentially a playground for cyberdefense,” Beaver said. “We can run attacks and malware and evaluate a technology’s true capabilities, unbiased by the marketing materials. This event was about nine months in the making and was orchestrated by research engineer Kelly Huffer.”
The ORNL team wanted to find out how machine learning, which studies patterns in a computer network over time and looks for potential intrusions, held up against legacy approaches like signature-based solutions, which operate on a simpler, narrower model built around specific warning signs. In other words, a signature defense guards against only the threats it’s programmed to recognize – and only if the threat fits the definition on the checklist.
“With machine learning, the system looks for similarities,” Beaver said. Such a system, for example, might flag traffic that is behaviorally consistent with previously observed malicious traffic, or may highlight abnormal activity inside the network.
ORNL researchers pushed more than 4,000 files and several dozen user actions – some friend, some foe – at the networks during the exercise, watching for alerts and false alarms. Some of the results held a few surprises. “All the tools showed very few false positives,” Beaver said. “We probed and looked for weak spots. One finding was that machine learning did better at detecting new malware and more obscure file types. But the signature-based solutions did well with some of the more familiar, common software.”
Rather than the new approach making the old obsolete, “they complemented each other,” he said.
The military analysts found machine learning a valuable tool in watching out for cyberattacks. But software solutions still can’t replace a pair of eyes to verify the results.
“It’s not magic,” Beaver said. “There’s no flashing red lights when the system’s attacked. The operators have to be engaged and understand what they’re doing. Just because the traffic’s anomalous doesn’t necessarily mean it’s malicious. You have to drill a little deeper.”
The cyberwar games aren’t over yet. The Navy has issued a standing public challenge called Artificial Intelligence Applications to Autonomous Cybersecurity (AI ATAC) for the best solutions to combat malware, with ORNL researchers led by Jared Smith as impartial evaluators.
“The entrants can be anybody,” Beaver said. “It can be a university or somebody sitting in a garage. We develop a rigorous evaluation process with a scoring framework, execute the process for each technology, and let the Navy know who scores best.” In the case of AI ATAC, the scoring approach was developed at ORNL by mathematician Robert Bridges. It predicts the operational cost of the cyber defense technology, which includes detection accuracy but also accounts for additional factors such as resource utilization, timeliness of detection, and cost of an attack.
The results of AI ATAC will be presented at the AFSEA West Conference on March 3, 2020. At that conference, the Navy will also announce its second cybersecurity challenge, to be conducted by ORNL in 2020.
The hackers’ playground has now arrived at ORNL. Beaver’s team has built their own cyber range with Navy funding. The Air Force has shown interest in a similar partnership.
“This range is going to be a capability with great potential for the lab and for the country,” he said. “In order to establish a capability like this, you have to be strong in cybersecurity, math and data architecture. That kind of breadth of expertise is hard to find, but a national lab having that kind of diversity is expected and therefore an advantage. ” ~ ORNL science writer Matt Lakin