I think you can be calm, Kind and firm without it escalating. You could do it a couple of ways. When you get there
" It's so nice we can do this, I'm always so busy * cheeky eyeroll* but its so nice to get the kids together when we can. I can only stay an hour today, I'd love it to be more but sadly I've got lots on. Oh well, They can have a nice catch up in that hour I suppose. So Have you got any plans for later on?"
Boom. Its to the point but almost manipulative to the point you're not giving him a chance.
If he says " Oh, It's a shame you can't stay longer "
your reply is " I know, I'm so glad I managed to find this hour or they wouldn't of got together at all! There really isnt enough hours in the day!! "
If he says " Well what other stuff do you have, Can they stay and you go? "
your reply is " Thats a really sweet offer, I'll bare that in mind for next time I'm a bit short on time, Don't worry about it for today though. I've got a plan! haha. * insert subject changer in here*
THEN when it comes to leaving " Ahh where does the time go!! I have to get going, Thanks so much for having us over, even if it was only for an hour, It's been lovely catching up, Hope you have fun doing * whatever he said his plans were * and don't work too hard and close the conversation, Don't leave it open. Boom. Drive home as fast as you can
lol xx