If you are still losing weight at the moment then I wouldn't worry about it too much but there will come a point when it will affect your weight loss, theres been a few people at group that have seen a difference when they started measuring their milk out
Its surprising easy for all those cuppas to add up and take you way over your daily allowance. Measure out your 350mls into a bottle and just use that over the course of the day. I do green so I can have 700mls of skimmed milk a day but I still easily get though that in coffee in a day x