I seriously doubt that anything short of removing the plants that are close to the fence will help with your issue. Banana corms are very strong and will move wood or aluminum fences out of their way. You could dig down a make a thick concrete barrier.
I have to say that's pretty nice of you to bring him his fruit while this is going on. I know a lot of people here on this board would love to have a neighbor like you.
Good luck!