Residents say the closure of a popular market site which ran for almost 150 years “ripped the heart out” of a town centre.