Is adultery a crime?

In Brazil there was a time when that adultery was a crime. I am not familiar enough with American laws to know whether it is or it was ever a crime here in the US. However I always found this criminalization kind of absurd. Adultery is a sin, yes. But a crime? How can it be a crime that a person is sexually involved with another? Crime is stealing, killing... but having sex?
Whether I have sex or not, and with whom is a personal matter. ... Right? And sex only with a spouse is a religious issue. After all, it is the religion that dictates that people should only have sex with their legitimate spouse. Why then is the civil law, which is secular, worried about this matter?
It was then that I realized that civil law also cared about the legalization of marriage. So it all made sense. What made no sense was to want adultery not to be considered a crime.
Think with me:
Marriage in our society is a social contract, determined by the law of such society. So much so that we refer to this contract in Brazil as "civil marriage". You can even get married in a church - which we call "religious wedding" - but today, if you do not sign the legal marriage papers, your marriage has no legal value (although there was a time when the religious wedding itself had already Marital validity - do not know if that was changed to make the state more secular, but that's beside the point).
According to the civil law, when signing the marriage contract - made ​​before a judge and witnesses - the two contracting parties undertake the establishing, with one another, of certain relationships in an exclusive manner. This contract also entitles both parties with certain financial privileges, but only if the contractual obligations of exclusivity are maintained. Sexual intercourse is one of the exclusive relationships that the contracting parties have committed to establish.
The sex outside marriage therefore constitutes a breach of this contract. Thus, a party who entered into the contract out sexual relationship, has broken a civil law - which is a crime .
Well, maybe my legal vocabulary is a little muddled. I know there are different words for different degrees of breaking the law, and I do not know if crime is the correct word for a breach of contract. But the central idea is roughly this: that adultery is illegal because it constitutes a breach of a contract with legal value.
The current 6 o'clock novela in Brazil depicts the decades of 30 and 40, when Brazilian society, especially the feminist movement, questioned the law of adultery as being too hard on women. Women went to jail, men walked. So instead of asking for the same punishment for men, they fought to end the punishment all together. Why? Why not punish the breach of a contract by either party?
My interpretation is that people do not understand that by signing a civil contract before witnesses and a judge, they can not break it at will. Marriage is no longer a personal or religious thing when it is recognized by the law as a civil contract.