Never downsized but never upsized. The current house we have was built in the 60s and the fridge opening drastically limits the fridge size that will fit. Basically only the basic older style models. I dream of a big fancy fridge but it's been fine without.
Honestly you will just acclimate to what you have and the change may be noticeable at first but pretty soon you'll not even notice anymore.
We have a family of 6.