When Did Pointing Out Racism Become "The Real Racism"?

I want to discuss something that's been on my mind for a bit. On Sunday night, right after the VMA's, (and before it-which-must-not-be-named), I had a long discussion with my 20 year old brother about why I was offended by Miley's performance. He's not totally up on social justice issues, and doesn't have the same context for this stuff as I do, but he's insightful, and willing to engage and learn. The discussion eventually segued into the nature of the word racist and how it is used in conversation of this nature.

He thinks that the word racist is "aggressive" and should only used to describe acts and people who do obviously and maliciously racist things (like... lynching I suppose?) because using it to refer to smaller, unintentional acts of racism immediately puts people on the defensive, and makes them unwilling to listen to you and your line of reasoning, even when you're right.

Now, while I see his point, and agree with his reasoning, I disagree with that sentiment. I think that it's even more important to explicitly label the smaller, unintentional acts of racism as racism, in order to help people recognize that no, you're not in the KuKluxKlan, but yes, pawing at a black woman's hair or calling a black man "boy", or dressing up as an "Indian Chief" for Halloween, or pulling your eyes back to make them slant is still, racist as fuck.

And I got this POV a lot in the comments of the article both here and over at GT over the last few days as well. Including this last one that I got just today:
That being said I don't believe that what is acceptable is one culture should be considered racist unless it actual causes harm to someone. And no one will convince me that dance moves are harmful to anyone other than the dancer.

To me, the unwillingness of people to call those smaller acts exactly what they are, feeds into the idea that racism isn't a big problem anymore. I think it contributes to the rhetoric that PoC are "pulling the race card" when they rightfully call people on their naked prejudices. 
I personally think it's incredibly important to start calling those "smaller" acts of racism out as racism, so that it becomes easier for the population at large to see why things like Miley's twerk obsession/appropriation is so problematic. 
Not calling out micro-aggressions as racism ignores the fact that Impact trumps Intent every time, and perpetuates the idea that a racist comment or joke is somehow purged of its racism because "I didn't mean it that way."
People take the label racist as a personal insult, when in fact it's just a fact of life. According to Jane Elliot:

"If you're not a racist when you leave high school, then you should have failed Social Studies because you weren't paying attention."

And I wholeheartedly agree with that. Being racist and/or prejudiced based on race is something that is ingrained in us. It leaches into our consciousness from television, movies, books, everything. It doesn't make you a bad person if you hold those prejudices. It makes you a bad person if you recognize that you have those prejudices and don't do the work to unlearn them. Because that's a huge part of intersectional feminism for me: consciously unlearning the prejudices that were fed to me as truth.

People don't get defensive when they're accused of racism because in their minds, they aren't racist, and accepting that they may have done/said something racist means acknowledging that they hold the "wrong" beliefs. It means facing the fact that they accept harmful ideas as part of their worldview. But "racism" isn't genetic. It IS something that you can change about yourself if you do the work. 
You can't fix a problem you don't know you have. How are you going to know that what you're saying is racist if people keep dancing around the word out of fear they'll offend? I'm sorry, but fuck your feelings. I'm already offended, and I have good reason to be. So I think I have the right to alert you to your racism and the fact that it's harmful to me.

I actually think that calling out racism might be the only education that PoC "owe" to white people. 
What about you guys? Is it better to tiptoe around the problem in order to avoid offending someone? Or is it more important to be clear and precise about calling our racist actions?