Exactly "WHY" is adultery wrong?

Lately I have been thinking about this issue and I am wondering why I feel adultery is wrong. I don't mean from just one religious standpoint, like what the Bible says; but as it seems everyone, from Christianity to Buddhists and even most atheists agree that Adultery is wrong, why does the condemnation of adultery seem so universal? So what essentially makes adultery wrong, and is there a difference between emotional adultery and physical adultery?