It's individual, some can get away with a splash. If you really can't go without it have some and see if it makes a difference. Might be an idea to wait until ketosis kicks in then add it back. Hope this helps.
I wouldnt risk it at the moment, if youre struggling just now getting in ketosis will help so much with that, having milk could stop you getting into ketosis and the hunger will last even longer. Hang in there
Once you start drinking black coffee and tea you'll never go back to milk again and it'll save you loads of calories in the future. In the morning I have a big black decaf coffee with my bar and I love it I sit and drink it for about half an hour.