I'm the same, it probably is just as cold on my UPs but somehow doesn't seem so bad with a big mug of proper hot chocolate and a few biscuits on the go lol.
I can recommend a hot bath then going to bed in thermal leggings, bed socks and a hoodie haha! X
I think there's some truth in that, I know when I was at my heaviest I was hot all of the time. I could have gone out in winter without a coat, now I'm very aware of the cold and always trying to warm up!
I think I read somewhere that when IF on down days your body's metabolism will try to slow down and will not put much energy into keeping you warm. I too get freezing on DDs.