Taxes should be filed to avoid headache because the USCIS officers are not tax experts. It is true that there are cases when people are not required to file and same IRS rules should apply to both the U.S. citizens and permanent residents. This is what the IRS says. No income or not enough income in theory means that one is not required to file (IN THEORY). I remember one specific case on this forum where the applicant has never filed taxes but got approved (her name is antda and her situation was specific). HOWEVER, you MUST have an explanation, some kind of legal evidence, proof from the IRS, some type of correspondence saying that you are good to go, that you do not owe anything, or even that you were not required to file. And again, the question is if the officer is going to be happy with your proof, and if not, then what? So, file your taxes. Another issue here is your absence. You should work on your continuous residence first before you apply.
I used to feel very unsafe and have been here for over 2 years. I am getting better now, feeling more optimistic, but man, did that take time. And today, for the first time in awhile, I felt good about the life. It is important to be surrounded by people who understand you, support you and encourage you, and sometimes it is hard to find such people. That was my greatest problem. Also, focus on the things you love about this place, think of the positive aspects, and try not to let negativity discourage you.