Here, we provide an in-depth structural characterization of the amorphous ionic glasses LiPON and LiSiPON with high Li content. Based on ab initio molecular dynamics simulations, the structure of these materials is an inverted structure with either isolated polyanion tetrahedra or polyanion dimers in a Li+ matrix. Based on neutron scattering data, this type of inverted structure leads to a significant amount of medium-range ordering in the structure, as demonstrated by two sharp diffraction peaks and a periodic structural oscillation in the density function G(r). While this medium-range ordering is commonly observed in liquids and metallic glasses, it has not previously been observed in oxides. On a local scale, adding N and Si increases the number of anion bridges and polyanion dimer structures, leading to higher ionic conductivity. In the medium-range ordering, the addition of Si leads to more disorder in the polyanion substructure but a significant increase in the ordering of the O substructure. Finally, we demonstrate that this inverted structure with medium-range ordering results in a glassy material that is both mechanically stiff and ductile on the nanoscale.