> Much of the worst asshole behavior I've encountered has been done by people who are outwardly polite and high functioning and have often never had a cross word for anyone.

Indeed, there are many ways to hurt, belittle and generally make someone feel bad or unwelcome without being blamable for anything.