Calcium aluminate cement enhances the thermomechanical properties of refractory castables through the formation of acicular calcium hexaluminate (CaO·6Al2O3), Ca2Mg2Al28O46 (CAM-I), and CaMg2Al16O27 (CAM-II) phases in MgO- or MgAl2O4-containing castables. The compatibility of CA6 with gahnite (ZnAl2O4), and acicular Ca2Zn2Al28O46 (CAZ-I) and CaZn2Al16O27 (CAZ-II) phases formation have been previously reported. In this work, the interaction between a CAC binder containing ZnAl2O4-Al2O3 matrix with commonly used refractory aggregates such as tabular alumina (TA), alumina-rich (AR90, AR78) and stoichiometric (SM72) MgAl2O4 spinels, and fused and dead-burned magnesia (FM, DBM, respectively) were investigated at 1650°C for 5 h. Microstructural analysis, using digital microscopy, scanning electron microscopy, and energy dispersive spectroscopy, revealed the formation of acicular CaZn0.18Al11.82O18.91, CAZ-I and CAZ-II grains, and strong interfacial bonding between the matrix and TA and spinel aggregates. FM and DBM were found to debond from the matrix. Thick interface layers were observed between the matrix and all the aggregates but TA. Null hypothesis significance testing (NHST) shows that the difference in the number of acicular grains between the interface zone and the bulk matrix (Z) is statistically significant for AR90/Z, SM72/Z, FM/Z, and DBM/Z interfaces, but not significant for TA/Z and AR78/Z. The role of the aggregates’ chemistry on the interfacial bonding and microstructure evolution is discussed.