I'm very much the same. I dislike a lot of what Morrissey has said and done in recent years, and I'm Not A Dog On A Chain is easily the worst album he's released, in my opinion. Just an absolute dirge.
On the other hand, his music and his words mean the world to me, and he's done a lot for me, as daft as that may sound to others. As a lad born into a working class, relatively poor family in Rochdale, The Smiths and Morrissey were a revelation when I was 15 years old (now 34, 35 in August), just like punk was when I was 12/13. His music led me to rockabilly, which I'll always be grateful for. I now just see him like a family member you love, but has views and does things that you disagree with.
I've got his face and name and lyrics tattooed on my arm anyway, so I'm stuck with him whether I like it or not (I do like it).