The addition of hard fillers to a polymer matrix is a well-known process for achieving mechanical reinforcement. With a decrease in the size of the fillers, the contribution from polymer–particle nanometer-sized interfaces become significant, and these interfaces affect the mechanical performance of polymer nanocomposites (PNCs) beyond the limits established for conventional composites. However, the molecular mechanisms underlying the improvement in the mechanical performance of glassy PNCs remain unresolved, necessitating a deeper understanding of the structure–property relationships in these intrinsically heterogeneous systems. In this effort, by using Brillouin light scattering (BLS) and dynamic mechanical analysis (DMA), we demonstrated that adding shorter chains to a PNC prepared with high molecular weight polymers significantly improved the mechanical properties of the PNC in the glassy state. The strongest enhancement of mechanical properties occurred at an optimum concentration of short chains. This is in contrast to behavior of the glass transition temperature of PNCs which shows a monotonic decrease with an increase in the concentration of shorter chains. Using experimental data and coarse-grained molecular dynamics (MD) simulations, we have identified the molecular mechanism leading to the observed non-monotonic changes in mechanical reinforcement. This mechanism includes changes in the nanoscale organization at the interface, combined with chain stretching amplified by the addition of the short chains. Overall, our approach paves a simple, cost-effective pathway to fabricating glassy PNCs with significantly improved mechanical properties that will fill various practical needs.