PAPA: Online Personalized Active Preference Alignment

Abstract

Diffusion models are highly effective at modeling complex data distributions, including images and text. However, in applications like personalized recommender systems, the objective often shifts to modeling specific regions of the distribution that maximize user preferences—initially unknown but gradually uncovered through interactive feedback. This can naturally be framed as a reinforcement learning problem, where the goal is to fine-tune a diffusion model to maximize a reward function based on preferences. However, the main challenge lies in learning a parameterized reward model, which typically requires large-scale preference data—something that is often not feasible in practice. In this work, we introduce PAPA, a method that bypasses the requirement for a reward model by directly optimizing the diffusion model using real-time user feedback. PAPA enables feedback-efficient preference alignment, drawing inspiration from the variational inference framework. We demonstrate PAPA’s effectiveness through extensive experiments and ablation studies across diverse domains. Additionally, based on theoretical insights, we propose an enhanced fine-tuning strategy that requires less computational budget and accelerates the fine-tuning process, further boosting PAPA’s suitability for real-world deployment. Code is available at https://github.com/NasikNafi/papa.