I hear that.
Unfounded stereotypes are bad enough. Some folks are walking stereotypes, though, and they really ruin it for the rest of us. (Or, at least, some of us feel like they do.)
Instead of just being ourselves, we don't want to be a stereotype and we certainly don't want to be too strongly associated with those who embrace and embody the worst stereotypes.
Some say I shouldn't, but I know that I look at random Black folks as representative of us, as a whole, and me, as an individual. I'm embarrassed by certain behavior... even when someone else is exhibiting that behavior.
On top of all of this is the fact that so many non-Blacks (and Blacks, too, for that matter) are just LOOKING for something to confirm their prejudices about us. The oft-lamented "argumentative" label is a great example.
As much negativity as I've seen floating and flying around the workplace, I know that whenever *I* speak up to share a different opinion and support it, in the face of opposition, I'm slapped with the "argumentative" label.