In all fairness, it doesn't have to be 'mercilessly hounded'. Just calmly pointed out that Morrissey supports and campaigns for far-right politics which is total anathema to most of the indie world that has always been Morrissey's natural home. If anyone disputes this, a link to For Britain's Wikipedia or Morrissey's own website (where he gave his support) ends the argument. Sadly, it's that simple. We know of course that this isn't the full picture - that Morrissey is a deeply muddled man when it comes to politics, and has probably expressed a liking for as many left wing as right-wing figures. But, tragically, the only party he has ever given a full endorsement to is of the far-right - a party that even Farage described as too extreme, and with a Conservative prime minister (Cameron) saying that Morrissey had gone beyond him, in terms of right-wing politics.
Morrissey saying he has never voted makes no difference at all, unfortunately. He needs to say that he regrets supporting For Britain and that he no longer endorses them (he could and should have done this in the recent interview with Sam Rayner). Until he says this clearly, the wider world will assume those are his politics. I remember him hitting the main national news outlets (e.g. BBC News website) three times in 2017 for supposed right-wing (or similarly ugly) views. Once or twice and people might forget. But three times and the perception of him is pretty much fixed. Since then, it's happened another two or three times in the national media.
Outside the UK, although a lot of music fans will know about this, these stories never made national news, so most people's perception of him is probably still anti-monarchy, vegetarian oddball from the Smiths, and the loss of the audience has therefore been much smaller.