They say your selfish for leaving your marriage when theres children involved.....right?

We have 2 children... 5 and 6 years old. He's cheated some years ago and has since been faithful. I recently just had my first affair (and got caught...my family and my husbands family both know and are involved and the whole 9). We were married at 19 and are now 27. I got pregnant and we were kind of pushed into gettin married because it was the right thing to do. I just don't want to be here anymore ...I don't want the other person, they have nothing to do with it. I'm not happy period. I haven't been. When my husband was cheating years ago, I wanted to leave but didn't because we have kids, I was in school and financially dependant at the time. Otherwise, I would have most definitely left. But I stayed. I forgave and moved on. But I'm still not happy here. I don't want to destroy my childrens lives so Im asking myself, do I stay and suck it up like I've been doing until my kids are grown? Or should I leave?