But those who merely unconsciously benefit without acting to further or lessen the oppression - I don't know what you say about that. Is there a word for "apathetically allowing oppression to go on without saying a word because you are too comfortable to be bothered to say or do the slightest thing about it since it isn't hurting you?"

I think that is the sense in which the word "privileged" is used in discourse on oppression.

But, yeah... is it overstating the case to call them "oppressors" -- or under-stating the case to avoid calling them "oppressors"?

It gets even more complicated because even people with every good intention to avoid participating in oppression can find themselves doing so. I am continually amazed the more I realize my participation in racial oppression, for example.