It really depends on the individual.
I can be a size 12 or 14 and know that given my height I'm not too big and that I don't 'look' too big, but for me, personally, I know that I am more comfortable, more fit and more able if I am smaller.
At the same time I know people shorter than me that may be 'supposed' to wear a size 8, but have a different body type and shape to wear a size 12 better than I ever could because it suits the dimensions of their body.