The Empire only banned the worship of Talos because not doing so would result in a war against Aldmeri dominion, which they would have lost. Not to mention the fact that Talos isn't actually a divine, just a deified human. The Thalmor don't ban the acknowledgement of Talos, they just don't want people worshipping him as Divine. In fact, the Empire has, on multiple occasions, turned a blind eye to Talos worshippers, it is only the Thalmor than en
force the ban.
'We allowed them to worship Talos, in full violation of the White-Gold Concordat with the Aldmeri Dominion (which recognizes the elven belief that Talos, as a human, cannot be one of the Divines). In jeopardizing the treaty that so many sacrificed for during the Great War, the Empire was wrong. But what choice did they have, I ask you? Against the Bear of Markarth, Ulfric Stormcloak, "no" is not an answer.'
Source:
The Bear Of Markarth.
If the Empire had not signed the White-Gold Concordant then the Aldmeri would have won the Great War and enforced far stricter restraints on Nordic and Imperial customs, traditions and culture. The only thing Skyrim had to forfeit in return for its safety was the worship of Talos as a Divine, something that Torygg was equally as bitter about as Ulfric was; Torygg was simply intelligent enough to realise that abandoning Talos worship temporarily to allow the Empire to reform and rebuild after the Great War would have the greatest chance of defeating the Aldmeri Dominion in the end. Torygg, in spite of his young age, had something Ulfric doesn't: foresight. If Ulfric wins freedom from the Empire then the Nords will be doomed in the face of the Aldmeri - as they will cut off any allies they had, massively reduce trading and leave politics in the region in a state of turmoil. And for this ability to see beyond the moment Ulfric killed Torygg, a young king who looked up to and trusted Stormcloak. Regardless of whether or not Ulfric challenged the high king to a duel, he killed him with help of a Thu'um, hardly an honourable method of combat.
Rule under the Empire isn't perfect, but independence will simply weaken Skyrim in the face of the Aldmeri, who would have little difficulty taking the region. Under the Empire there is ultimately the chance for freedom from the Aldmeri in the end, something that would be impossible under Stormcloak rule. Mankind either needs to stand together or fall divided.