The HEs aren't include to promote weight loss so not having them shouldn't negatively impact on your losses. But they are there to ensure you have a proper nutritionally balanced diet so you should make an effort to have them. Try choosing to have your HEs before you choose a free or synned food. Or split them if that is easier - half the milk and half a cheese allowance = 1 HEA and so on.