What you need to confirm:
Did your husband check the box "Married filing jointly" in his tax returns and include your name (see example below)?
Did your husband include your children in his tax returns (also see example below, Dependents)?
If he included you and/or your children, in the end he would pay less taxes assuming the same total household income (if he got a refund, the refund would be larger), because you pay taxes on income and that % of tax reduces the larger the family size. For example if you make $50K as a single person you will pay more taxes than if your total household is $50K as a family of 4.
If he did not include you in his taxes, you should file deliquent tax return for every year there isn't a tax return with your name on it (regardless if you have worked or not). Now if you truly have not had any income, then you won't have to pay anything, but filing the taxes is the formal way you are declaring to the IRS that you did not earn income. Even if you did earn income, there is a good chance that Foreign Income Tax Credits or Exclusions would wipe out your "owed" taxes, so you still wouldn't have to pay anything.
As others have mentioned, evading taxes is a pretty serious deal (and in the extreme cases can result in punishments), but if you haven't been earning income (or very little income) there is a pretty high chance you won't be penalized or owe any taxes. You should now make sure there has been a tax returned filed for you for every year of your adult life, and going forward to include your children on those tax returns (especially if they start earning money). This is the lifelong deal you have as a US citizen (and your children), does not matter where you live you always have to file taxes.
EDIT: also, just contact an agency/accountant that works with US taxed filed from other countries. Show them what has been filed and what not. Myself being a US citizen living in another country have had very good experience with these agencies, and they should be able to clear up exactly what you need to file or not now.