Polymers with dynamic (transient) bonds, often called associating polymers, have been attracting significant attention in recent years due to their unique viscoelastic properties, self-healing ability, and recyclability. Nevertheless, understanding the mechanisms and the factors controlling their macroscopic properties remains limited due to the higher complexity introduced by the dynamic bonds. In this study, small-angle X-ray scattering (SAXS), broadband dielectric spectroscopy (BDS), and rheology were applied to unravel the structure and dynamics of telechelic associating polymers with different molecular weights. SAXS measurements revealed phase separation of the functional end groups with an average cluster size of ∼2–3 nm and the distance between clusters controlled by the chain length. Borrowing the interfacial layer model analysis of BDS data from the polymer nanocomposite field, we demonstrated the presence of an interfacial polymer layer with a thickness of ∼0.7–0.9 nm surrounding these clusters. Rheological measurements showed quantitatively that the presence of the interfacial layer significantly alters the viscoelastic behavior of these materials, indicating the crucial role of the interfacial layer in defining the macroscopic mechanical properties of the studied telechelic materials. The presented results emphasize that phase separation of the functional groups in associating polymers leads to very significant changes of the viscoelastic properties, opening a promising avenue in the design of novel functional materials.