August 2026

Conference Paper

SetGo: Metadata Readiness for Scientific AI Datasets

By:
Wilkinson, Sean R; Shpilker, Polina H; Brewer, Wesley H
Page Number:
9
Book Title:
SSDBM '26: Proceedings of the 38th International Conference on Scalable Scientific Data Management
Publication Date:
August 2026
Publisher Location:
Association for Computing Machinery, New York, New York, United States of America
Conference Name:
The International Conference on Scalable Scientific Data Management (SSDBM 2026)
Conference Location:
San Diego, California, United States of America
Conference Sponsor:
University of California, San Diego / San Diego Supercomputer Center (SDSC)
View DOI Listing:
https://doi.org/10.1145/3828820.3828827

Abstract

Scientific datasets intended for AI use require both computational readiness for model training and metadata readiness for discovery, sharing, and reuse. The Readiness Engine for Data Integration (REDI) addresses computational readiness, but no corresponding tool evaluates whether a dataset’s metadata are sufficiently complete, governed, and standards-compliant for publication and agent-based consumption. Existing FAIR assessors operate only on published repository records, and no single system covers FAIR compliance, licensing, provenance, governance, reproducibility, and catalog readiness together. We present SetGo, an open-source Python toolkit that assesses and repairs metadata readiness across these six dimensions before a dataset is published or archived. Applied to four scientific corpora, SetGo surfaces deficiencies that general-purpose tools do not detect: ERA5 climate metadata scores 4% on ACDD 1.3 compliance; materials datasets fail OPTIMADE species-definition requirements; and PDB-derived proteomics data carries licensing terms incompatible with standard SPDX identifiers. Guided enrichment raises overall FAIR scores from 52–57% to 81–91%, and a single setgo publish command pushes to Hugging Face Hub, CKAN, or OpenMetadata with ML Commons Croissant 1.0 metadata sidecars. To support interactive and automated workflows, SetGo integrates with coding agents powered by large language models (LLMs) through a /setgo skill that enables natural-language execution of the full assess–enrich–publish loop, with user involvement limited to supplying missing metadata values.