No need to stress about this. You only really need 3 years worth of continuous residency as a spouse of a US citizen. But I get it, the instructions are confusing. I had exactly the same questions you have when I did my N-400. What I did was I submitted 5 years of travel history anyway (yes, even for those years I was not yet a US resident). Then I submitted an accompanying letter stating why I submitted 5 years (because the instructions said so lol) and that 2ish of those years I was not yet a US resident. The officer at the interview re-confirmed my travel dates but other than that, no other issues encountered. You will be fine.