The hexagonal rare-earth ferrite RFeO3 family represents a unique class of multiferroics exhibiting weak ferromagnetism, and a strong coupling between magnetism and structural trimerization is predicted. However, the hexagonal structure for RFeO3 remains metastable in conventional conditions. We have succeeded in stabilizing the hexagonal structure of polycrystalline YbFeO3 by partial Sc substitution of Yb. Using bulk magnetometry and neutron diffraction, we find that Yb0.42Sc0.58FeO3 orders into a canted antiferromagnetic state with the Néel temperature TN∼165K, below which the Fe3+ moments form the triangular configuration in the ab plane and their in-plane projections are parallel to the  axis, consistent with magnetic space group P63c′m′. It is determined that the spin canting is aligned along the c axis, giving rise to the weak ferromagnetism. Furthermore, the Fe3+ moments reorient toward a new direction below reorientation temperature TR∼40K, satisfying magnetic subgroup P63, while the Yb3+ moments order independently and ferrimagnetically along the c axis at the characteristic temperature TYb∼15K. Interestingly, reproducible modulation of electric polarization induced by magnetic field at low temperature is achieved, suggesting that the delicate structural distortion associated with two-up/one-down buckling of the Yb/Sc planes and tilting of the FeO5 bipyramids may mediate the coupling between ferroelectric and magnetic orders under magnetic field. The present work represents substantial progress to search for high-temperature multiferroics in hexagonal ferrites and related materials.