Crystalline solids exhibiting glass-like thermal conductivity have attracted substantial attention both for fundamental interest and applications such as thermoelectrics. In most crystals, the competition of phonon scattering by anharmonic interactions and crystalline imperfections leads to a non-monotonic trend of thermal conductivity with temperature. Defect-free crystals that exhibit the glassy trend of low thermal conductivity with a monotonic increase with temperature are desirable because they are intrinsically thermally insulating while retaining useful properties of perfect crystals. However, this behavior is rare, and its microscopic origin remains unclear. Here, we report the observation of ultralow and glass-like thermal conductivity in a hexagonal perovskite chalcogenide single crystal, BaTiS3, despite its highly symmetric and simple primitive cell. Elastic and inelastic scattering measurements reveal the quantum mechanical origin of this unusual trend. A two-level atomic tunneling system exists in a shallow double-well potential of the Ti atom and is of sufficiently high frequency to scatter heat-carrying phonons up to room temperature. While atomic tunneling has been invoked to explain the low-temperature thermal conductivity of solids for decades, our study establishes the presence of sub-THz frequency tunneling systems even in high-quality, electrically insulating single crystals, leading to anomalous transport properties well above cryogenic temperatures.