It's personal choice really, I know that there are carbs in milk... Everyone has a different limit to the carbs that will effect their ketosis, some are more sensitive than others so whilst others can have milk and not have a problem others could be knocked out my even a teaspoon!
If your having some and you don't think it's effecting your ketosis or your weight loss and it helps you through the diet then crack on =)