Confidential to Debbie:
You have two options from where I’m sitting: confront him or just find a different church. The truth will hurt, I guarantee that, but as Jesus said, the truth will make you free. Granted, changing churches is a whole lot easier than facing a clergyman that may not be willing to be honest (been there, done that.) If you don’t talk to him and tell him why you’re leaving, though, you take away the opportunity for him to learn and you allow the same situation to develop again with another woman.
Regardless, it does sound like it’s time to find a new church. I don’t think he’s in a position to shepherd you right now. Keep praying. You do need to put space between you, but again, you really should tell him why. I understand how painful it is, but what he does with the truth is his responsiblity, not yours. As your pastor, he is in a position of authority and bares some responsiblity for what happens within his flock. You need to give him a chance to change any behavior patterns on his part that may have caused your problems.
If you do talk to him, don’t accuse him of anything, just speak the truth in love letting him know how his behavior has effected you. Unless he’s an accomplished actor, his response will answer the question even if it’s true and he tries to deny it.
If there is any risk of harm to yourself, don’t confront him alone. Consider having both spouses present. If you only take yours, he will feel ganged up on. If your spouse is out of the question, take a trusted friend with you.