If you use X to insult someone, you aren't just insulting that person — you are insulting X. You are saying that X is something worthy of being insulted. It doesn't matter if you think they are a horrible person. If you don't think that X is something horrible about them, then insult the reasons you do think they are horrible.

I've seen far too many homophobic, racist, misogynistic, misandristic, transphobic, sizeist, anti-disabled and anti-semitic slurs "justified" by assertions that person being insulted doesn't personally deserve respect or that they fit some bad stereotype. Use something as an insult and you are saying that it is something worth of admonishment. Your words can say a lot more about what you think than you intend them to.