Wrongway Feldman was the only one who was on the island when the castaways came, left, came back again, and left again for good. (Note: Dubov the painter, the jungle boy, and the Japanese sailor were supposedly living there also before the Castaways arrived, but somehow never ran into one another).
Wrongway Feldman must have brought some provisions when he came back to the island, because Gilligan and the Skipper are seen eating celery and an apple (carrots are also mentioned), none of which are available on the island.