The issues, as I see them, are that there were never any written USCIS instructions for the spouse to attend the N-400 interview, and there was never any notice that an I-751 interview was going to be conducted. It can be argued that this is an error on the part of USCIS. Whether USCIS will admit that is another story.
Note: I checked wife's N-400 interview letter. It instructs the applicant what to bring if filing as a spouse of a US citizens, but it doesn't instruct the USC spouse to be there....at least I couldn't find it anywhere on the official notice. In our case, the local office notified us 2 days prior to the N-400 interview (via an email) that I had to attend, and that we would have a combo interview.
It appears, to me, that USCIS expected the OP to read between the lines.....to somehow know that an I-751 interview would be conducted. I think that is grounds for a re-schedule. But, as you said, it might not fly.