I just shrugged.
Honestly, why should I understand what you're saying? If you had assumed I could speak Portuguese and spoke Portuguese with me I'd be fine.
Then another day I was at Walmart, wearing a BRAZILIAN flag shirt. On my way out this black guy looks at my shirt and goes - Buenos Dias!
Oh, please, no! Look at my flag! Look at my flag! Please! No Spanish!
I looked at him as if I had not the slightest clue of what he said, and kept walking. From a distance I could hear him yell, " Buenos Dias!"