As I live in San Diego County, I always carried mine with me at all times. I crossed the border regularly, and travelled as well to AZ (before the whole issue came up with AZ). I was asked for it many times at border patrol checkpoints, and by border patrol agents while waiting to get on public transportation at stops near the border.
It feels wierd opening my wallet and not seeing it anymore now that I am a US citizen, but my recomendation would be to keep it in your wallet. You just never know when you will need it.