His nephew (brother's son) is television personality John Oliver.
He was the son of Charlotte Hester (Girdlestone), a religious education adviser, and Osborne George Oliver, an electricity board official.
On his father's side, he was the great-grandson of The Rev. William Boyd Carpenter (1841-1918), who was a court chaplain to Queen Victoria and the Bishop of Ripon. He was from a prominent family that produced many Church of England clergy.