The novel Mr Smedley reads when at the Meldrews' is The Moonstone by Wilkie Collins.
This episode is one of only a few in the run of the show in which the usual opening credit sequence (with the tortoises) is not seen.The titles and names of the primary actors are superimposed on the first scene of the story.
The book Margaret Meldrew is reading on the couch is The Judas Window by Carter Dickson (aka John Dickson Carr.) Renwick's other long-running tv show is Jonathan Creek, a series inspired and influenced by locked room mysteries, specifically those like Dickson Carr's.