Laser Powder Bed Fusion (L-PBF) Additive Manufacturing (AM) is among the metal 3D printing technologies most broadly adopted by the manufacturing industry. However, the current industry qualification paradigm for critical-application L-PBF parts relies heavily on expensive non-destructive inspection techniques, which significantly limits the use-cases of L-PBF. In situ monitoring of the process promises a less expensive alternative to ex situ testing, but existing sensor technologies and data analysis techniques struggle to detect sub-surface flaws (e.g., porosity and cracking) on production-scale L-PBF printers. In this work, an in situ NDE (INDE) system was engineered to detect subsurface flaws detected in X-Ray Computed Tomography (XCT) directly from process monitoring data. A multilayer, multimodal data input allowed the INDE system to detect numerous subsurface flaws in the size range of 200–1000 µm using a novel human-in-the-loop annotation procedure. Furthermore, a framework was established for generating probability-of-detection (POD) and probability-of-false-alarm (PFA) curves compliant with NDE standards by systematically comparing instances of detected subsurface flaws to post-build XCT data. We also introduce for the first time in the AM in situ sensing literature the a_(90/95) – the flaw size corresponding to a 90% detection rate on the lower 95% confidence interval of the POD curve. The INDE system successfully demonstrated POD capabilities commensurate with traditional NDE methods. Traditional ML performance metrics were also shown to be inadequate for assessing the ability of the INDE system’s flaw detection performance. It is the hope of the authors that future studies will adopt the POD and PFA approach outlined here to provide better insight into the utility of process monitoring for AM.