filling in a fence hole, that next to a concrete patio.?

My back yard, I have a concrete patio then grass, between the 2 was a fence, I removed the fence and the pole holes, now when I try to fill the holes my dirt disappears after a good rain, I think its somehow seeping down under the cement. How can I fill this hole so I can get rid of this unsafe hole?