@DunkenMage
The point was he retreated, he ran away. It doesn't really matter why he left or where he went. He left.
If this is important to you, then you can believe whatever you want.
However, if you consider how the action of the Emperor running away from, oh, excuse me (clears throat) retreating from IC must have looked to the casual observer, I'm sure there were people saying things like, "Oh it's a corrupt and dying Empire..." what with the Emperor abandoning of all things the IC.
Very similar I would say to the ban on Talos. Which is why Ulfric was so important to the Thalmor, what with letting him go and all. I still disagree with our involvement in that, however Ulfric would have found a way to make trouble regardless.
Far as we know, in the before time there was no Ulfric around to play the accuser and hate on the Empire, TMII had everything he needed to win. And I would say the Empire walking out of that had some positive results.
Though again, I have no special love for the Empire, esp next to Elven Perfection, heh could you imagine?
It would be more beneficial to the Dominion and for the people who live in the Empire's Provinces were it to survive.
There are other things to be taken into consideration. We can rule all of Tamriel with what we have. That White Gold Tomb has always been nothing but a problem for us.
Even so, there's a ton of history and still some mystery about the White Gold Tower. And for someone to suggest 'burning it down' is unthinkable. I would also like to take this opportunity to suggest the Stormcloaks could rip down the statue of Azura in Skyrim or perhaps even the Mage's College should they have their way. Then there's eliminating East Empire CO in Skyrim, harassment of Imperial Provinces, murdering Elven civilians which even if by some twist of fate Ulfric wasn't behind, he would do nothing to stop it by the other Nords. Not to mention the Nords enslaving anyone who stands up to them, like they plan on doing in Markarth and Ysgramor did back in the day.