I think most of my friends and family know not to badger for a visit that early. I usually waited a month before asking to visit. But I also know that I will want my sister and/ or parents staying with us that first week. I just imagine DH and I wanting to focus solely on baby and my recovery rather than making dinner or cleaning. In that area, I don't mind close family coming to see us.
I was actually the opposite and didn't really want anyone except immediate family visiting in the hospital - I felt gross and I wasn't in my own space and I just wanted to be home. Once we got home I was good with people coming over - we had
lots of visitors, but I was only annoyed when people stayed too long or did not leave when it was meal time so I felt obligated to offer to feed them (or wait to eat myself which is not so good with a new mama.) I would say just be relaxed about it and
put your husband on kicking people out duty - have him prepare a couple of reasons why people need to leave and just have him ready to tell visitors that you aren't feeling well, that the baby has trouble sleeping around people, that he is putting
you to bed now that the baby is sleeping, etc.