Patty! Looks great... I think you shouldn't worry so much, because it's really more your location. Even here, I only get fruit set every other year, just because they demand so many months of warmth, and I know your summers are shorter there.
Also, looks like you have some pretty nice Brugmansia growing there! Good job
