Was going to say what Teresa just said!
SW is all about having a balanced diet, and fats and sugars are just as important in a balanced diet as carbohydrate and protein. Without it, you risk your body going into 'starvation' mode, and you will see considerably smaller losses.
You don't have to have things like a bar of chocolate - you could have an extra slice of bread, some tinned fruit, extra meat or fish if you're on green, the list goes on...
Hope that helps a bit.