Councillors have refused plans for a housing development after a housebuilder claimed its site would not be viable to include affordable housing.