If people want to redefine the word "racist" so that only actual slaveholders count, let them. I'm more interested in the "critical reflection" [Ta-Nehasi] Coates rightly says that the "I'm not a racist" move is designed to shut down; in asking: does race play a role in someone's thought and action that it ought not to play? rather than in asking: does that role reach whatever bar of horrificness s/he wants to say it would have to meet to qualify as "racist"?
I've tried to make this point in my own work with my distinction between "finding the evildoer" and "solving the problem" in racial discourse. When engaging in dialogue on issues of race, I am not particularly interested in figuring out whether my interlocutros are Bad People, even if I disagree with them or think their views/policy positions are part of the structure which reinforces racial hierarchy in America. I'd much rather, as Hilzoy and Coates say, critically reflect on the manner in which those views intersect with race and racial inequality in the hopes that we can come to just solutions to the issue. I am more than willing to "sacrifice" calling these people racist in exchange for having that conversation.
But the corallary to this is that a lot of people tend assume that any liberal critique of their views on race is akin to calling them a racist. Over the past few years of attempting to avoid this pitfall, I'm beginning to think its inescapable: no matter how I phrase my inquiry, there will be a class of people who will huff "are you calling me a racist?!?!?" and there ends the discussion. It is, I suspect, a psychological defense mechanism to avoid grappling with the intricacies of the subject matter. Making the question "am I a racist" (which, in this frame, is synonymous with ultimate evildoer) allows the interlocutor to deflect attention from the particular moral or policy dimensions of the object of discussion, and instead change the inquiry into "am I an awful person?" Since most people are relatively persuaded of their status as not-awful-people, they can then react with offense that "I am implying" that they are. Having established that they are not awful, it is assumed that the original critique has been "answered", and nothing more needs to be said. So long as any racial discussion that involves more than the most perfunctory critique can be derailed in this manner, there is simply no hope of discussing these questions in any substantive, meaningful fashion.
So, new rule. Unless I specifically say the words "you are a racist", I am not calling you a racist. That can be our safe phrase: unless I actually say them, you can be assured the point of my conversation is not to label you a racist.
All clear? Excellent.