The 'antisemitism' slurs, completely fabricated by the British Establishment, aided and abetted by right wing Jews and other Political opponents of the Labour Party, were just one of the reasons Labour lost the 2019 GE. The main reason being that the British working classes were fed a constant diet over the course of several years of half-truths, distortions, and outright lies designed to demonise Jeremy Corbyn and make the Labour Party unelectable.
No one knows why Morrissey endorsed For Britain, I'm sure there were several reasons, not least among them their progressive policies regarding protections for animals, but you're right, as far as Morrissey was concerned For Britain were/are neither 'far right' nor 'racist'.
I doubt there's more than 0.05% of voters who took the time to read their respective Party's manifesto, and I'm sure had they done so they would have found many things they were not in agreement with. Morrissey appears to have been attracted by a handful of policies, and his belief that AMW was a honest and sincere person.
I doubt very much that anyone who's actually studied their manifesto could legitimately argue that FB are a 'far right organisation', in fact the current Tory Government are much more deserving of the tag 'far right', but no one is cancelling or ostracising Tory voters.
None of this alters the fact that the tragic case currently using the name 'Morrissey Isn't Very Good', among many others I suspect, is an absolute c***.