It all depends on your play style, I guess. I started on the default difficulty (adept?), then moved up to expert at around level 15 as the game continued to get progressively easier, and finally to master at level 25 or thereabouts. Eventually even master started to offer less challenges than it should, so I downloaded a difficulty increasing mod so I now do 0.4 x damage and my enemies do triple damage. It's fun and moderately challenging but not yet frustrating. I can still kill most melee enemies in 1-3 swings apart from bosses, absorb lots of punishment before going down and generally survive most fights with my dignity intact (i.e. not having to run from enemies etc). Some magic users give me grief, chiefly shock mages as I have no such resistances, but that just adds nicely to the challenge and keeps me on my toes! I would definitely recommend expert or master difficulty if you feel the game has become a cake walk.
I'm playing a heavy armored 2-handed axe wielding Nord btw, a pure warrior with all 3 crafting skills (haven't abused the alchemy-smithing-enchanting loop, though) and some sneaking thrown in the mix. No magic, apart from shouts.