Water resources, including groundwater and prominent rivers worldwide, are under duress because of excessive contaminant and nutrient loads. To help mitigate this problem, the United States Department of Energy (DOE) has supported research since the late 1980s to improve our fundamental knowledge of processes that could be used to help clean up challenging subsurface problems. Problems of interest have included subsurface radioactive waste, heavy metals, and metalloids (e.g. uranium, mercury, arsenic). Research efforts have provided insights into detailed groundwater biogeochemical process coupling and the resulting geochemical exports of metals and nutrients to surrounding environments. Recently, an increased focus has been placed on constraining the exchanges and fates of carbon and nitrogen within and across bedrock to canopy compartments of a watershed and in river–floodplain settings, because of their important role in driving biogeochemical interactions with contaminants and the potential of increased fluxes under changing precipitation regimes, including extreme events. While reviewing the extensive research that has been conducted at DOE's representative sites and testbeds (such as the Oyster Site in Virginia, Savannah River Site in South Carolina, Oak Ridge Reservation in Tennessee, Hanford in Washington, Nevada National Security Site in Nevada, Riverton in Wyoming, and Rifle and East River in Colorado), this review paper explores the nature and distribution of contaminants in the surface and shallow subsurface (i.e. the critical zone) and their interactions with carbon and nitrogen dynamics. We also describe state-of-the-art, scale-aware characterization approaches and models developed to predict contaminant fate and transport. The models take advantage of DOE leadership-class high-performance computers and are beginning to incorporate artificial intelligence approaches to tackle the extreme diversity of hydro-biogeochemical processes and measurements. Recognizing that the insights and capability developments are potentially transferable to many other sites, we also explore the scientific implications of these advances and recommend future research directions.