Hi and welcome! It won't affect your weight loss but they are there to make sure you have a balanced diet. So I would recommend you make an effort to have your HEs. Make a conscious effort to chose your HEB before you use your free foods. There is a large list of HEb so it doesn't have to be bread or cereal ... even just using a tbsp of extra virgin olive oil when cooking your meal would count as a HEB.
Good luck!