Generalizable Flow-based Policy Learning in Reinforcement Learning

Brief: Oak Ridge National Laboratory (ORNL) researchers have developed a novel, generalizable policy learning approach in deep reinforcement learning (RL) for future applications in additive manufacturing.

Accomplishment: We developed a multimodal policy distribution via normalizing flows, which offers a richer flexibility to capture arbitrarily complex policy distributions.  The propped flow-based Soft Actor Critic (F-SAC) significantly improves the policy exploration and generalization when compared with unimodal Gaussian policy in vanilla SAC.  FSAC is benchmarked on several examples (ex: soft-body manipulation tasks from PlasticineLab) and shows improvements over existing methods.

Background: Generalization is critical when learning policies to interact with real physical systems in RL.  Many real-world applications involving generalizable policy learning, such as, object manipulation and autonomous driving, motivate us to develop intelligent agents to learning generalizable and reliable policies.  SAC provides a solid off-policy algorithm within the maximum entropy RL framework. Although SAC offers greater stability, improved exploration and efficiency, the policy generalization and expression capability are severally limited by its Gaussian re-parametrization.  On the other hand, normalizing flows allow for effective learning of non-gaussian policies.