I'm not that up to date with race politics, I still have a lot of reading to do, but the way it was explained that made sense to me is that racism isn't simply prejudice against someone's race, but prejudice along with the power to act on that prejudice in a way that is supported by society in some way or built into the way things are done. You can be prejudiced but not racist, if you don't have the power to oppress those who you are prejudiced against.

You also have to look at why people say "white people are scum" vs "black people are scum." When people say the first, it's probably because of that disappointment/resentment/whatever that society has been built around these people and they themselves aren't doing anything to correct that. When people say "black people are scum," however, that comes from a very different place. It's the belief that black people are inherently inferior somehow, that they don't deserve the respect we give others because they're inherently lesser. They might sound the same but their meanings are different.