The building Steven goes to meet Yvonne in is the one used for Reynholm Industries, from 'The IT Crowd (2006)", which Matt Berry (III) starred in.
The song that Matt Berry sings at the karaoke is titled 'Ghost Town'. However the rights were only given to the BBC for local release. Therefore, when the series was released in foreign markets the song was either removed or scrambled unless the markets got permission from the composer. With a fee that added nothing to the plot, most markets just passed on getting permission.
During the recording session, Danny and Clem mock Steven by having him botch the names of musical artists in an ad: "Bon Jovee, Madam Gaga, and The Coldplay." On their first demo tape, the latter band's name was in fact given as "The Coldplay".