The other angle to look at it is...
The purpose of immigrant visa is to reunite US citizen with their foreign spouse.
If you already live together elsewhere, you're already reunited, and why should a visa be given? This is why domicile is important.
Living together as a proof is more critical for adjustment of status cases.