If someone calls you a nazi sympathizer, they're not saying you're a nazi. They're literally saying you sympathize with nazis, that you have sympathy for them.