I don't take any notice of what the size in an item of clothing says anymore!!
Last week I bought a denim shirt. I picked up a size 14, the size I usually am and could tell just from looking at it that it was small...even the 16 didn't look very big. I ended up with an 18, a size that I have never bought for my top half!! (bottom half, yes....but not at the top!!! ) It fitted perfectly!!
I don't even think it just varys from shop to shop anymore, it varys even within a shop as I think they buy items from all over the place these days (not like years ago when they manufactured their own)
And of course it depends on the style too. something tight fitted may need the next size up, whereas something loose and flowing will maybe fit you in the 'right' size...whatever the right size is!!
I certainly don't worry about what the label says, if it fits it fits!!!