A lot of doctors aren't to sure of this way of losing weight. Did he give you a specific reason for telling you to go back to slimming world? medical reason? or is it his personal opinion.
If it was a medical opinion - then do as he says, if it was a personal opinion, I don't feel he should be giving you that kind of advice, that's your own decision.
Best of luck!!
xx