Buildings consume 73% of electricity produced in the United States and, currently, they are largely passive participants in the electric grid. However, the flexibility in building loads can be exploited to provide ancillary services to enhance the grid reliability. In this paper, we investigate two control strategies that allow Heating, Ventilation and Air-Conditioning (HVAC) systems in commercial and residential buildings to provide frequency regulation services to the grid while maintaining occupants comfort. The first optimal control strategy is based on model predictive control acting on a variable air volume HVAC system (continuously variable HVAC load), which is available in large commercial buildings. The second strategy is rule-based control acting on an aggregate of on/off HVAC systems, which are available in residential buildings in addition to many small to medium size commercial buildings. Hardware constraints that include limiting the switching between the different states for on/off HVAC units to maintain their lifetimes are considered. Simulations illustrate that the proposed control strategies provide frequency regulation to the grid, without affecting the indoor climate significantly.