Of course you're not wrong. It sounds like your SO is mad that he's not able to use you to the extent he'd like. What's appealing about a guy who won't hold a job or respect your boundaries?

You said, "I feel the child is HIS primary responsibility and I'm here as his support system (my emphasis)..  Your other comments show you are seeing the situation clearly.  Of course you should have some date nights and adult-only vacations.

There is a saying on Steptalk: Has he done the work?   That means has dad ironed out his residence, custody order, the child's hygiene, behavior, bedtime, childcare.  If he hasnt done that, it's terribly tough on a woman coming into the situation, as you know.  I'm getting the feeling he is using your kindness.  That spotty work record is a red flag, too.

Another saying is:  Love is not enough. Good luck, you sound like a caring, intelligent person, Shymom.

He's using you.

No you are not wrong for putting boundaries that don't suit his agenda. If you care or do more for the child than his biological parents do, in my opinion you are being used. It also seems to create a lot of resentment for other posters. My now husband never expected me to and my personal disengagement is due more to a HCBM. 
If anything put more boundaries up and make sure to keep your finances seprate .     

Easiest way to do a check test if you’re wondering whether or not you’re being unreasonable is to ask yourself what would he do if you weren’t there. Like for example if a man asks the new gf to take over all lunch preps, school pick ups, drop offs and homework help because she’s a motherly figure and they argue when she complains it’s too much. What would he do if she wasn’t there? He’d have to step up and do his actual job as the parent. And if he’s not doing that then he doesn’t have his shit together, he’s just looking for someone to palm the responsibility off to. Ultimately you’re looking for a partner, someone who you can respect. Someone who’s got his life sorted and just wants mutual company. Not another man-child, extra child and crazy ex lurking in the background. 

Nope, the child never trumps the partners or relationship.  The partners and relationship are unequivocally each other's top priority and must always come first.

Kids are the top relationship responsibility. Not the priority.

Now for your choice of mates.  Find one that is worth a shit.  This guy is a user and a loser.  You have to value yourself and your life and happiness more than this.

What is it that a man can't do for a child that a woman can? Sorry to be so blunt but I think you are being played and used and your partner sounds lazy, disrespectful and overall awful. 

What do you get from this relationship as you definitely deserve better!

This guy is a millstone round your neck. Get rid of him and take your life back. Love is not enough. 

You cook, clean, tend his child, work, support the three of you... what exactly is his role in all of this? 

Where do tyou see this in 5 years?

I always flip things around and I cannot believe the expecations the biological parents put on us.  And for what.  My motto is I didn't create it and I am sure not going to fix it.  You are NOT wrong for setting boundries at all.  I would think at this point if it's worth it.  If I could go back I would have never married my husband.  I didn't expect to be raising his kid.  I had to have a lot of meltdowns and frankly there's no reason for it.  As you know as a single mother you would have never expected this from someone else.  Why as women then do men expect it from us?  I will NEVER understand that. 

I always flip things around and I cannot believe the expecations the biological parents put on us.  And for what.  My motto is I didn't create it and I am sure not going to fix it.  You are NOT wrong for setting boundries at all.  I would think at this point if it's worth it.  If I could go back I would have never married my husband.  I didn't expect to be raising his kid.  I had to have a lot of meltdowns and frankly there's no reason for it.  As you know as a single mother you would have never expected this from someone else.  Why as women then do men expect it from us?  I will NEVER understand that. 

YOU seem to be the only adult in the house. Why is that ok with you?

You love him, sure, NOW. But love is not enough to sustain this situation. Sooner or later resentment will take over and you'll wonder how you've ended up where you are.

You don't need his permission to take a trip. If he can't pay his way, he doesn't go. If he can't find child care, he doesn't go. Not your responsibility or your problem.

"my partner is really putting a lot of parental responsibility and expectations on me"

Translation: He wants you to do the donkey work.

"dating a man with a young child means that there are just some things that he can't do"

I'm going to need a few examples here. IMO, men are quite as capable of raising their children as women are (well, those worth their salt, anyway). Women haven't cornered the market on multi-tasking.

"I get that the child comes first"

The child's needs come first, otherwise it should be your relationship that comes first.

"I would like to take a trip for my birthday and he thinks we should all go, which would be prohibitively expensive."

As he can't keep a job, I suggest you take your trip alone. Why reward bad behaviour?

"He's also not consistently working because he has to take care of his kid"

This one makes me laugh. You were a single mother, right? And I bet you fought tooth and nail to ensure you kept your job because you had children who were counting on you. So why can't he?

"always makes an excuse to quit his jobs"

You know why? Because you're there to pick up the slack. Not only does he want you to take care of his kid but he wants your money to enjoy too. He is using you on so many levels, OP ... How can you find this kind of stuff attractive. I find him pitiful.

 "Whenever he is working he's constantly complaining and makes it seem like it's my fault that he's taking a job he hates"

Tell him to shut it, he has bills to pay. You are already carrying him and his kid, do you really want to listen to him whining too?

"I'm the only one consistently working, taking care of the bills, come home, cook, clean and take care of his kid"

Now, this I take HUGE exception to - he has you raising his kid, paying his bills AND being his maid, cook, launderer, and goodness knows whatever else. How can you let yourself be USED like this? 

"I don't feel he should expect me to not take a job that conflicts with his son's and his schedule"

Well, of course you effing shouldn't! How dare he even so much as suggest it? He can't keep a job and sits on his fat arse all day, not even doing the cleaning or cooking to lighten your load (from what I understand, any help you get comes from your own children) and he wants you to get a job so you can do whatever little he actually manages to drag himself out of bed/away from the tv to do? Not on your effing nelly!

"he told me we have to figure out what to do about his son after school"

No. WRONG! HE has to figure out what to do about his son after school. His son is in no way your responsibility.

"Am I wrong for putting boundaries? "

No, but you are wrong for staying. I wonder what your kids think about living in this situation. You showed them what a strong, independent woman you are while they were growing up and now you have turned into a doormat for a person who isn't fit to tie your laces. This person is not a partner, soooo not a partner, he is a used and a leech. How can you  find him even remotely attractive? 

My intention with this post is not to hurt but maybe to give you a vision of reality. You are killing yourself for someone who does not deserve you. You are worth so much more and deserve so much better. 

Where is the child's mother in all this? Does he have his son 100% of the time?

So many red flags with this man, the biggest one being he can't/won't hold a job and yet expects you to care for his child.  He sounds like a real loser who finds women who are willing to support his lazy a$$ until they have enough and boot him.  Then he finds some other tender hearted victim who will take him in....lather, rinse, repeat.  This is a dead end guy, dump him.

If you weren't around?

He'd either have to hold down a job and make arrangements for his son, or find another poor lady to carry him

"The child comes first."

IMHO, that is what is wrong with society these days. Adults have child-centric relationships and families. Making kids into entitled, selfish brats that don't understand when they don't get their way. They also get frustrated and are unable to cope when told "no." If that ever even happens. Sounds like your SO doesn't put his foot down when it comes to affection and having some kid-free time. Let me guess, the kid rules the roost?

Relationships/marriages need to come first. The best thing for parents/adults to give their children, is to model healthy communication, respect, and partnership. Your SO isn't doing this. I would most definitely NOT let them move back in.