A multiscale simulation approach was developed and employed to optimize the sheet surface conditions for higher interfacial temperature and joint strength in ultrasonic welding of magnesium alloy AZ31 and dual-phase steel DP590. First, a mesoscale model was used to study the relationship between friction coefficient and surface roughness, which can be modified by various engineering methods. Then a macroscopic process model was employed to study the effects of surface roughness on heat generation, indicating that a temperature increase can be achieved with rougher surfaces on two sides of both DP590 and AZ31 sheets. Samples prepared by sanding and filing, as well as grinding, were first characterized for surface roughness and then welded under ultrasonic vibration. An infrared camera was used to measure temperatures in situ for model validation. Lap shear test results for the welded joint showed that the joint strength can be improved by 10~25% using filing and round grinding methods as a result of the enhanced heat generation and mechanical interlocking on the interface.