Puffins are a theme that come up repeatedly between Chris O'Dowd and Gabrielle Dennis's characters throughout the series. Chris O'Dowd served as the narrator on the Netflix series "Puffin Rock" a children's show about the adventures of Oona the puffin and her little brother.
Chris 'O Down and wife Dawn Porter did a sketch for comic relief in England. To John Prine's 'in spite of ourselves' which contains the lyrics, 'against all odds, honey we're the big door prize:
The book on which the show is based contains numerous references to the music of John Prine. The title is just one of those references.