When the next change feels easy. Not when the code is beautiful — that's a moving target and the goal is shipping.
I keep a 'refactor budget' for any feature: if I blow through it without making the next step obviously easier, I revert and try a smaller cut.