We didn't worry either. My oldest was easy and sleeping in her room by age 3 or 4. She has never come in our bed after transitioning. My current four year old would love to sleep with us still and comes over sometimes but DH has started taking her back to her bed - too crowded. We are transitioning the 2 yo to a crib so that it's just DH, the 7 week old and I in our king.
When the 2 yo cries, DH goes to get him and brings him in.