Hello,
I went through a similar situation except I did not have an affair, given the time frame of your relationship I think you'll be ok. If your partner does decide to make a complaint you'll need an attorney for sure, provide a bunch of photos as well as affidavits from friends/families/work colleagues etc to help prove your marriage was legitimate, which in my eyes it sounds like It was genuine. Being from the UK I have some sympathy as it can be quite scary being by yourself and giving everything up.