Off-policy PPO?

[slerman12]

Doesn't PPO, at least the vanilla variant, only work on-policy? That is, from recent data, not an experience replay?

[lucidrains]

@slerman12 i'm far from an expert on reinforcement learning, but that's in line with my thoughts!