Found a video on how to do it, straight from The Man himself.
If you'll notice, the flower in the video has the stigma (female part) much taller than the surrounding anthers (male part), therefore the pollen cannot get from the male to female easily, thus the need for hand pollination.
In my flower, on the other hand, the stigma is closer to the anthers, so maybe I don't need to do it after all.