Ulfric isn't a "horrible" racist. The conditions of the Dunmer, while not ideal, aren't extremely oppressive. They're free to roam about the city and run market stalls in the Stone Quarter. The Argonians aren't allowed in the city because of concern over the potential for violence between the Argonians and the Dunmer being bad for everyone. It's to maintain order and not out of any particular benevolent disposition to the Dunmer. It also does nothing to vindicate their segregation to the Gray Quarter.
The Accession War was payback for the Arnesian War in which Morrowind seized the northern regions of Argonia. There wasn't anything particularly oppressive about the Argonian invasion in comparison to any other war in Tamriel. Historically the only enslavement that occurred between the Dunmer and Argonians was enslavement of the latter by the former.