The process seems to have slowed down significantly now. I'm not watering it for 5 days now. The soil is still damp to the touch, and some mold has developed on it.
I've been removing dead leaves and leaves parts, and found some unhealthy colored tissue on a "stem" which I've also removed. Now there are only two leaves, one of them is very slowly getting yellow before drying out (previously, leaves were turning dark brown without any yellow and dried out much quicker).
I'm not sure when I should start watering it, but I plan to wait for a couple more days.
I've had yellowing leaves problem before, but that was solved by re-planting banana into a larger pot and giving some nitrogen fertilizer.
