SOC environmental emulation & framework for testing SOAR tools

“ATAC 3” Modern security operation centers (SOCs) rely on human operators and a tapestry of diverse logging and alerting tools with large-scale collection and query abilities. Yet, SOC investigations are tedious as they rely on manual efforts to query diverse data sources, overlay related logs, and correlate the data into information, then document results in a ticketing system. Security orchestration, automation, and response (SOAR) tools are a relatively new technology that promise, with appropriate configuration, to collect, filter, and display needed diverse information, to automate many of the common tasks that unnecessarily require SOC analysts’ time, to facilitate SOC collaboration, and in doing so to improve both efficiency and consistency of SOCs. SOAR tools have never been tested in practice to evaluate their effect and to understand them in use. In this paper we design and administer the first hands-on user study of SOAR tools, involving 24 participants and six commercial SOAR tools. Our contributions include the experimental design, itemizing six characteristics of SOAR tools–install/configuration, usability, efficiency gains, context switching reduction, investigation quality, and resilience to degraded internet–and how to test them. We describe configuration of a cyber range to the test environment including network, user, and threat emulation, a full SOC tool suite, and creating artifacts allowing multiple representative investigation scenarios to permit testing.

---

---