Ba3Yb2Zn5O11 is exceptional among breathing pyrochlore compounds for being in the nearly-decoupled limit where inter-tetrahedron interactions are weak, hosting isolated clusters or molecular magnet-like tetrahedra of magnetic ytterbium (Yb3+) ions. In this work, we present the study carried out on single-crystal samples of the breathing pyrochlore Ba3Yb2Zn5O11, using a variety of magnetometry and neutron scattering techniques along with theoretical modeling. We employ inelastic neutron scattering to investigate the magnetic dynamics as a function of applied field (with respect to both magnitude and direction) down to a temperature of 70 mK, where inelastic scattering reveals dispersionless bands of excitations as found in earlier powder sample studies, in good agreement with a single-tetrahedron model. However, diffuse neutron scattering at zero field and dc-susceptibility at finite field exhibit features suggesting the presence of excitations at low-energy that are not captured by the single tetrahedron model. Analysis of the local structure down to 2 K via pair distribution function analysis finds no evidence of structural disorder. We conclude that effects beyond the single tetrahedron model are important in describing the low-energy, low-temperature physics of Ba3Yb2Zn5O11, but their nature remains undetermined.