Horses allow you to fast travel even if you're over-encumbered, the best feature about them, they also help you fight, if it weren't for my Khajiit's horse in helping me fight off conjurors and mages at Fort Kastav, I surely would've died. He sacrificed himself and I was able to take out the last big threat there (especially at a low level with extremely limited supplies, to see a Frost Atronach charging at you is certainly a shock). The worst is when my horse fell to his death at a drop that would've only hurt you but not too badly. I was annoyed but also surprised to see the horse collapsing underneath me as we made a simple jump, must've been beaten around too much from previous encounters and unable to rest.
I like horses, other than the fact that as soon as I reach a gallop a bear or wolves appear. Now, the Oblivion horses... those were worthless.
When I first played Oblivion, I wanted to buy a horse, I went inside the stables and talked to the guy about buying one, I didn't actually purchase one (I may have spoken to the wrong guy) and thought he wanted me to go outside and pick my horse. So I did so only to be branded a thief, I hopped off yelling out "NO NO! I didn't want to STEAL it, look, I'm off, it's okay!" but the guards came and I went ahead and took the horse anyway (why chance dying and why not just go ahead and take it since they want to kill me for 'stealing it'), I eventually got away but couldn't return to the city without dying.