A battle is brewing between the city and a New Jersey family that owns a historic farm over land for hundreds of affordable housing units.