"It's far more likely that he didn't see anything wrong with slavery per se, just the mistreatment of slaves."

Paul was a Roman, and from the rest of his writings, it is apparent he had no desire to change the Roman culture, which included slavery and oppression of women. So I think you are correct in your guess that he saw nothing wrong with the ownership of slaves.