Thank you everybody for your answers.
Thank you Caryh, you are correct, what we were asking for was for the I-539 to change nonimigrant status for niece+nephew from visitor to student, and not adjustment of status. I saw the instructions and forms on USCIS and they request the I-94 number which is why we called CBP because my brother in law's i-94 is not showing in the system.
They did not have any intention to stay with a tourist visa and go to school. My wife, once the kids were here, offered her brother to put the kids in our daughter's catholic school for only one year.
What i want to clarify is that what we asked was only for the kids, not for brother and wife, they will go back to their country as scheduled. If the change of status cannot be done, then the kids will return as scheduled as well. There is no intent to go against the law; as many of you said, we are just not well informed and do not know what the correct procedures are.