I have more respect for the Imperials, personally. Just look at the starting quest:
Ulfric, Leader of the Stormcloaks, hides behind a door while the dragon attacks, and his soldiers cower uselessly in a tower.
Tullius, General of the Legion, is outside directing his men to fight the dragon, trying to get the townspeople to safety, etc.
I believe that Tullius is a much more honorable man than Ulfric, therefore, I would follow him. Also, I hate the Thalmor, and the only way to stand up to them is to stay united.