Part of Sabrina's incantation to open the portal to Hell "Abandon all hope, ye who enter here." is taken from Dante's Inferno, specifically the inscription he finds at the door upon entering ( "Lasciate ogne speranza, voi ch'intrate").
"Pandemonium" is from "Paradise Lost". Milton called the center of Hell "Pandemonium", meaning "place of all demons".
Around the 10-minute mark, on the blackboard behind Mrs. Wardwell's back, there's a structural representation of Dante's Inferno.
When Sabrina confronts Lucifer inside Nick, he tells her, "You betrayed me. For that your suffering shall be legendary, even for Hell." This is a reference to Hellbound: Hellraiser II. In that movie, Pinhead tells Kirsty, "Your suffering will be legendary even in Hell!"
The title "The Hellbound Heart", although not directly related to the original work, is most likely an homage to the famous novel of the same name by Clive Barker.