'It’s a tough thing,' Chris Collins says of ending re-election race