Actually, I'm pretty sure I do know why, sort of, now that I think about it. The connotation of a word or phrase is determined by the people who use it. So, if a phrase is used almost exclusively by a group of people who have a specific meaning in mind, then people will associate the phrase with that meaning.