We decided it was the fact that men can drape them around the hips, not the boobs.
So not fair, right? Pretty, though. Still, it's a law of physics. That area is far more narrow and tight than a woman's breast area.
I do like a man in a towel, though, so I guess I can put up with the injustice...