AaronTT

Depending upon what kind of gravel is there, it might actually be beneficial to the plants. Usually it mostly quartz, which when broken down has beneficial silica. That is very important to producing stronger fibers within the leaves and stems, and its roots. It helps to blunt the effect of stress on plants. Only issue is that some have heavy metals in them.
Sand which is primarily silica, is so tightly bound, that bacteria in the soil cannot "skuff" off ions, rendering sand useless as a source. So another rich source is beneficial.
I have noticed in the past that plants that have some small amounts of rocks in their soil grow better. Just a personal observation.