When we make decisions about relationships we need to take into account all the implications of our actions and check inside with our conscience to see that our actions are correct and honorable. Your conscience is troubling you. Your peace of mind is disturbed. Do you know why?
Look wider than your immediate feelings for this man. What impact is your relationship with him having on his wife and children? You may try and rationalize this away by saying that they do not know. However you and he both know and his family will surely find out in due course. When they do what will be the implications. Are you going to fight to take him away from his wife and children? If you succeed and marry him what are the prospects for a marriage founded on such an act? By entertaining a relationship with this man you are bringing heartache, sadness and betrayal into all your lives.
This is completely unnecessary. There are many men out there who would be delighted to know you and to possibly have a relationship with you without all these complications. Focus on them. Believe in your own attractiveness and in your ability to attract the right man into your life.
My advice to you is to go to your man friend and break it off with him immediately - apologizing for any hurt you have caused him and his family. Then resolve to never spend time with him again and to never again go out with a married man.
If you do this you will know you are doing the right thing and in the course of time your guilty feelings will subside, your peace of mind will return and you will have saved a lot of people from heartache. If you do not do this and you marry this man your conscience will trouble you for the rest of your life.