Phonon lifetimes are a key component of quasiparticle theories of transport; yet first-principles lifetimes are rarely directly compared with inelastic neutron scattering (INS) results. Existing comparisons show discrepancies even at temperatures where perturbation theory is expected to be reliable. In this paper, we demonstrate that the reciprocal space voxel (q voxel), which is the finite region in reciprocal space required in INS data analysis, must be explicitly accounted for within theory in order to draw a meaningful comparison. We demonstrate accurate predictions of peak widths of the scattering function when accounting for the q voxel in CaF2 and ThO2. Passing this test implies high fidelity of the phonon interactions and the approximations used to compute the Green's function, serving as a critical benchmark of theory and indicating that other material properties should be accurately predicted, which we demonstrate for thermal conductivity.