Hi
I had similar issue when I applied for AOS few years ago and they returned by application. It was tough to know the reason they returned it. Did you apply for employment authorization?
In my case I had put a wrong eligibility category on the I-765 form. EAD is free if you apply with AOS and that is eligibility category C 9.
If you pick any other code , you have to submit payment for it and if they do not find the payment, they might return your application.
Good luck