The origin of limited plasticity in metallic glasses is elusive, with no apparent link to their atomic structure. We propose that the response of the glassy structure to applied stress, not the original structure itself, provides a gauge to predict the degree of plasticity. We carried out high-energy x-ray diffraction on various bulk metallic glasses (BMGs) under uniaxial compression within the elastic limit and evaluated the anisotropic pair distribution function. We show that the extent of local deviation from the affine (uniform) deformation in the elastic regime is strongly correlated with the plastic behavior of BMGs beyond yield, across chemical compositions and sample history. The results suggest that the propensity for collective local atomic rearrangements under stress promotes plasticity.