I'm the same! I don't know the "science" behind it but I think it's to do with the way your body grabs onto the "bad" food and holds it. Also, when we get back on track, maybe we are not as strict as we could be?
The only way i can lose quickly is to do Success Express, in which case I can lose 3lb ish, if I stick to plan normally it will be 1 or 2 lb.
We always expect the weight to come off quickly and sadly, it just doesn't happen :-(