You have to do the counseling. I know a few couples here that were in your shoes more or less, did some long counseling, and are as strong as they were in the start of their marriage.
One of them was my best high school friend.
But when it turns out the other way after making the effort, then you have to know to just move forward