Mount & Blade was a game that fit on a floppy disc and yet it had horseback fighting. Why doesn't Skyrim? In fact, the entire horse mechanic could be changed. Why can't I have saddlebags with permanent storage? This way, I don't have to have a follower be a pack mule ... I can have an actual pack mule. And why can't I shoot my bow from horseback? Or swing a mace? Or just run someone over?
And different horse personalities...warhorses that will attack dragons to nellies that buck at an attacking skeever and flees like the wind. A whistling command to call them. Magic spells to call them from longer distances. Tie them to things.