Plans more than a decade in the works to turn Elizabeth Street Garden into affordable housing for seniors have been scrapped.