Understanding elementary excitations and their couplings in condensed matter systems is critical for developing better energy-conversion devices. In thermoelectric materials, the heat-to-electricity conversion efficiency is directly improved by suppressing the propagation of phonon quasiparticles responsible for macroscopic thermal transport. The current record material for thermoelectric conversion efficiency, SnSe, has an ultralow thermal conductivity, but the mechanism behind the strong phonon scattering remains largely unknown. From inelastic neutron scattering measurements and first-principles simulations, we mapped the four-dimensional phonon dispersion surfaces of SnSe, and found the origin of the ionic-potential anharmonicity responsible for the unique properties of SnSe. We show that the giant phonon scattering arises from an unstable electronic structure, with orbital interactions leading to a ferroelectric-like lattice instability. The present results provide a microscopic picture connecting electronic structure and phonon anharmonicity in SnSe, and offers new insights on how electron–phonon and phonon–phonon interactions may lead to the realization of ultralow thermal conductivity.