I swim twice a week and do lots of fast pace walking with pram and dog,once i start moving up the plans i will then introduce cardio and weights at the gym.
Think it depends on each individual really though,but i personally dont think the amount of calories we consume on ss or ss + is enough to be exercising too much.