I was thinking about just that earlier, and I think I'd be hard-pressed to find fault with the practice. I actually think "nigger" would be the more interesting case, since the reaction in that case would be more confusion than outrage (or rather, confusion then outrage). People would be like "wait, you just called a white guy a . . . ?" And then they're struggle for awhile to determine just what the fuck you're on about before they decide they should be offended and start criticizing you.
I think the "faggot" case would be a lot murkier, since onlookers would obviously recognize it as a slur and assume you're claiming that, because of trait X, your target is a homosexual. It might be different if the term was applied to a woman, though.
But anyway, it's a thorny issue. I mean, no one has trouble with anyone tossing around words like "idiot", "moron", "imbecile", "cretin", and the like, so why should "retard" be any different? But OTOH "retard" does have a history associated with it that those words don't (at least in recent memory), so there's that to consider as well. Context is key, and consistency is not necessarily a virtue.