I think the real dangers of overusing the accusation of "racism" to the point of hyperbole are twofold:

One, some conservatives will internalize it and start to sympathize with racists, consciously or subconsciously, or else they will start failing to take racism seriously themselves. Most conservatives don't think of themselves as racist even though they may support some policies that affect minorities negatively. If you actually want to change their minds, be polite, approach the subject gingerly, and coddle their egos. Learn their buzzwords, thought patterns, and emotional triggers. Being confrontational gets you nowhere with people like that - they can't handle it. Most conservatives are trying to be "good people" as best they can from where they stand. If you just dismiss them collectively as "bad people", they're simply going to shut you out. As a group, they might be bad, but as individuals, they can be approachable if you know how to approach them.

You may not think that changing conservatives' minds about some of these issues is important, but it's part of the strategy of the actual Nazis. They even want to encourage a racist mindset even among portions of the population that they hate, because they feel that it helps racism set in across the board. They want LGBT people and people of color to have a racist mentality, and they target them with propaganda. They want racism to pervade everything. A lot of anti-racists don't seem to want anti-racism to pervade everything, instead believing that certain groups are simply not worth engaging with on the subject, and this is a strategic mistake. It's giving the Nazis a free win among the portions of the population MOST susceptible to their propaganda. It's gotten many groups and individuals that don't actually want to support real fascism to hate and fear anti-fascists.

Secondly, it can desensitize us and them to real acts of racist violence and fascism that take place in the background. I feel like a lot of people use things like "fascist" as an offhand accusation and compare everyone they don't like to the Nazis partially because there is, on some level, a belief that those threats actually went away in the 1940s and 50s and that some asshole politician who wants to deny transgender bathroom rights is the closest modern moral equivalent we'll ever have to the man who gassed millions on Jews in concentration camps. People think that it couldn't happen again, especially not "here" ("here" being wherever you currently live), or they never fully realized the horrors than can be a part of the rise of actual fascism.

Actual fascists are stealthy. We didn't know everything they were up to until the war was already over. They rise quietly, and crying wolf every time you see a stray dog can make it more difficult to recognize which threats are real and which ones are just vanilla conservative assholedom (which I do realize is also itself not a good thing), especially in an era where everyone is crying wolf and a lot of those people believe, on some level, that wolves are extinct and the hyperbole doesn't matter.

The wolves are real, they are hungry, and they are here. It is important to learn the difference.