And saying the Empire doesn't care about it's people is fairly incorrect. The first thing General Tullius said when Alduin attacked was, "Get the townspeople to safety!"
And Hadvar saves a little child from being a baby BBQ.
So the Empire actually does care.
Small instances, but largely? No. Many examples of their uncaring attitude towards citizens, the Empire is more concerned with making profits and keeping nobles happy.
This is the Empire that exploited slave labor in Morrowind.
The Empire had no choice and HAD to accept the White-Gold Concordat
So the Empire being unable to properly protect the citizenry, due to them being unprepared for war... even after being warned by the Penitus Oculatus and the Blades. People should give the Empire a pass? Why should people fight for something, that is unable to protect them?
We have no idea if the Empire had to accept the White-Gold Concordat, the Redguards showed another treaty was possible. I personally believe the Elder Council pressured Titus Mede II to surrender, and take peace. Their daggers at his back, figuratively... or not given he is killed later on.
The Mede Dynasty was politically weak, and knowing that Medes came into power was orchestrated by the Elder Council after they were reassured they could control them. It seems fairly logical; treaties, laws and legislation are the Elder Council's area, so the White-Gold Concordat is their doing.
In any event, the Empire is either unwilling or unable to properly defend the people. The Empire takes resources from Skyrim in return for protection they're not offering. Wayrest was lost in High Rock, Bravil and many other cities around Cyrodiil erupted into violence. Even Cheydinhal was lost to violence, and seems to have been that way up until 4E 200, only less than a year.
If the Empire can barely their house together in the Heartlands, their continued occupation of Skyrim is merely stretching themselves too far. The Legion is already stretched thin, Tullius isn't getting reinforcements (Even though Maven Black-Briar is given Imperial soldiers to personally escort mead shipments).
Mercenaries under the employ of the Empire are raiding trade shipments for supplies, citizens from Cyrodiil are escaping into Skyrim to get away from violence. The Empire needs to focus on Cyrodiil, and get their act together.
So far this shadow of an Empire has been one of inaction and incompetence, especially after many warnings about the Thalmor for the last hundred and fifty years.