When Policies Cannot Be Retrained: A Unified Closed-Form View of Post-Training Steering in Offline Reinforcement Learning
In shortA robot's control policy is often frozen once trained. We show how to steer it toward a new goal without retraining. When the new guidance is poor, the policy falls back on its original behaviour instead of collapsing; when the guidance is good, how much it helps depends on how capable the original policy was.
Technical summary
Deployment objectives change after training, but the actor often cannot be retrained. Composing a frozen offline RL actor with a goal-conditioned prior as a product of experts yields the same policy as KL-regularized adaptation, with β = α/(1−α) for diagonal-Gaussian actors. The result is graceful degradation rather than universal gain: under a degraded prior the composition stays anchored to the frozen actor while additive and prior-only adaptation collapse, and across D4RL MuJoCo, AntMaze, and Adroit its effect is bounded by actor competence (HELP/FROZEN/HURT 4/4/4 over ten seeds).
@inproceedings{hossain2026policies,
title={When Policies Cannot Be Retrained: A Unified Closed-Form View of Post-Training Steering in Offline Reinforcement Learning},
author={Hossain, Elias and Basher, Mohammad Jahid Ibna and Garibay, Ivan and Garibay, Ozlem and Yousefi, Niloofar},
booktitle={Advances in Neural Information Processing Systems},
year={2026}
}










