I ended a bad relationship ( he was married) and I feel bad?

OK so here is the situation in short. I started dating this man right after my last realtionship ended. When we met I did not know he was married. I found out he was married, and thought this would be the perfect chance to have a no strings attached relationship. NO He fell in love with me and his wife found out about me. I guess they have an open marriage, she told me to my face that she didnt mind when me and her husband were together ( frankly I think they are both a little slow) I know I should have ended things then ( i was wrong). He treated me like gold, did any and everything for me. He wanted me to have a child with him but I told him no for obvious reasons he got all butt hurt. I wasn't the nicest to him either and last week I told him its over, and all he could say is that I treated him like dirt this whole time and I was wrong for not falling in love with him. But look here I am 23 attractive, in school working towards my Masters in Nursing, with no kids... so I felt like this relationship would no go anywhere, But I feel like I hurt this man. Should I feel bad? How do I move on from feeling guilty. No I don't love him, but I do care about him as a person. Afterall it has been 3 years. HELP NO JUDGEMENT