TL;DR: Why are people not advocating for the removal of statues of ancient oppressors as much as modern ones?
Augustus (the first emperor of Ancient Rome) was an imperialist and probably did a lot of bad shit that wasn't recorded. There's a statue of him on my college campus (and many other places).
Pyramids (in Egypt, Nubia, Mesoamerica, etc.) and other monumental architecture (like mounds built by Mississippian cultures in the U.S.) is only possible through state power and oppression.
If some U.S. slave owner had built a massive monument with a bunch of slaves, a large number of people would want it to be taken down. However, I don't really see many people advocating the destruction of ancient monumental architecture or statues of ancient oppressors.
I am seriously conflicted about this because I study ancient history and I wish there was more evidence.
I understand that it's a bit difficult to compare the two situations because the atrocities of the civil war and slavery were recorded in other ways, whereas the pyramids are the only testament to the slaves that built them.
If I were to come up with a solution, I think that monuments to the civil war should be removed from the public, as it certainly is insulting, but I think destroying them might be a bit misguided.
As a side note, where do you draw the line? The Korean War Memorial is a reminder of genocide (against North Koreans) but veterans would get pissed if you got angry about that.