There are growing fears history is being “demolished” at a long-abandoned railway works as construction progresses on 303 flats.