And yet, Hammerfell was able to repel the invaders despite the Empire turning their back on them.
Hammerfell fought a weakened Aldmeri Dominion. Largely due to success from the Legions, that weakened the Aldmeri in Hammerfell. During the war the Aldmeri made a gamble for all of Cyrodiil.
In 4E 174, the Thalmor leadership committed all available forces to the campaign in Cyrodiil, gambling on a decisive victory to end the war once and for all.
As for Hammerfell.
4E 173
The Imperial Legions under General Decianus met them outside Skaven in a bloody and indecisive clash. Decianus withdrew and left Arannelya in possession of Skaven, but the Aldmeri were too weakened to continue their advance.
4E 174
In Hammerfell, General Decianus was preparing to drive the Aldmeri back from Skaven when he was ordered to march for Cyrodiil. Unwilling to abandon Hammerfell completely, he allowed a great number of "invalids" to be discharged from the Legions before they marched east. These veterans formed the core of the army that eventually drove Lady Arannelya's forces back across the Alik'r late in 174, taking heavy losses on their retreat from harassing attacks by the Alik'r warriors.
Hammerfell only fought the Aldmeri Dominion to a standstill, and formed a treaty with the Thalmor allowing Dominion forces to withdraw from Southern Hammerfell. It came at great cost, Hammerfell was largely devastated.
Don't underestimate a small number of folks protecting their homeland.
Stormcloaks aren't protecting their homeland. They're going to invade the Aldmeri Dominion.
Tullius says it well.
"But you're willing to sacrifice thousands for your own selfish ambitions."
Near as I can tell, Galmar is more the warmonger than Ulfric.
Ulfric does things for image, willing to toy with a man's life.
"Just kill him and let's be done with it already."
"Come, Galmar. Where's your sense of the dramatic moment?"
Ulfric is a manipulative politician. If you listen to his end speech, about returning to traditions and the moot and what not. You catch him talking to Galmar after.
"To return to our glory and traditions, to determine our own future! And it is for these reasons that I cannot accept the mantle of "High King." Not until the Moot declares that title should adorn my shoulders will I accept it."
Ulfric:
"How'd I do?"
Galmar:
"Eh, not so bad. Nice touch about the High King."
Ulfric:
"Thank you, I thought so, too."
Galmar:
"It's a foregone conclusion, you know."
Ulfric:
"Oh, I know."