if you're struggling with your weight loss or not getting the results you want, it's more likely to be something else rather than the bananas. As a free food they alone shouldn't cause a gain, check through everything else in your food diary just to be certain cos it's a shame to exclude anything that's free. Although having said that variety is everything and your body does get bored of routine after a while so why not shake things up with a whole new range of fruits. You'll def see and feel the benefits.
I think the thing to remember is that although fruits like bananas and grapes are free, they aren't speed foods so if I were you I'd stick with the bananas if you enjoy them (and let's face it they're convenient) but boost your weight loss with some superspeed berries while you're at it x