Residential and commercial buildings in the United States accounted for 40% of total energy in 2020. Building energy modeling (BEM) is a useful tool that allows individuals, researchers, companies, or utilities to save energy by optimizing buildings through estimation of building technology savings and performance projection of building energy under various environmental conditions. Urban building energy modeling (UBEM) expands the scope beyond individual buildings to the buildings in a neighborhood, city, utility and more. Yet there is a knowledge gap in the literature as to how these models compare to measured data on an individual and aggregated basis. As UBEM data and methods continue to develop, it is important to consider the accuracy, bias, and limitations of the models. Nation-scale data and UBEM software suite named Automatic Building Energy Modeling (AutoBEM) was used to model 50,843 buildings in Chattanooga, Tennessee. The uncalibrated simulation results were compared to aggregated 15-minute electricity data for the year 2019 with visualizations highlighting sources of bias in building data and the AutoBEM framework while considering how they relate to other UBEM methods. Estimation of building type and year of constructions are found to be the major sources of bias. Accounting for the amount of conditioned area per building significantly improves the overall fit of the simulated energy use intensity. it was found that inherent variation in building energy use contributes to R2 values between 0.008 and 0.095 across building types but slope values near 1 for the total number of buildings. This indicates the need for building aggregation for representative building energy modeling with data sources available at an urban scale while illustrating the need for additional individual building data and model improvement beyond the originally produced UBEM models for individual building analysis.