When Matt and his father arrive in the village, its clearly stated there are four "normal" families left amongst the happy ones and, in later episodes, Hendrick invites them one by one to his house to brainwash them. Yet, on the first night, Matthew goes outside to find one of Hendrick's ceremonies in progress, presumably to brainwash someone into the circle of Happy Ones. This is never explained about who this as or if there was fifth family also normal when Adam and Matthew first arrived.
At the end of the series, it is revealed that the village existed within a "psychic bubble" within which, once someone entered the circle of stones surrounding the village, they could not leave. However, in the very first episode, Dr. Brake arrives in the village to find that a moving company has delivered several boxes of scientific equipment and, following the delivery, departed the village without any problem.
It could of course only be people who actually live in the village as there is a road that runs through the circle meaning anyone who drove through would also be stuck.