It really is a very different game on the higher difficulties. You have conjuration, illusion, and followers completely unaffected and therefore much more effective compared to 1h, 2h, archery, and destruction.
With a need for more health, you have less magica for destruction and less stamina...