Collins' resistance to holding town halls stirs opposition