I wasn't suggesting that you say you are allergic. Any responsible medical advice would probably say that cutting down on animal fats is a good thing.
But I suppose what I was really saying is that none of us is obliged to eat anything just because someone else expects us to, wants us to, will be upset if we don't, etc., etc. We all have a choice.
My grandfather had an interesting trick - he used to go to meetings where everyone drank whisky, which he didn't like. No amount of saying so made any difference, so in the end he would accept a glass of whisky, and it would sit in front of him all night. He never drank it, and no-one seemed to notice!