I also agree with this. There are peaks and valleys like in any game, but generally I think the game is too easy. Dragonborn did up the level of difficulty, kind of like Point Lookout and the added enemies of Broken Steel in Fallout 3, but the overall challenge of surviving combat is not there. Depending on the type of character you play, you have have a fairly difficult start; it can be particularly challenging for mages early one. However, once you reach the early 20's you are well on your path to godhood.
Arena, the first game in the series, was very much the opposite. It became so incredibly challenging that out of 10 plays, you might get out of the tutorial dungeon 3 times. It seems that Bethesda slowly became shy to that sort of challenge, and each game in the series has become easier. While I think that the difficulty of Arena was a bit too much, bordering on 'NES hard' at points, I would like to see some increased challenge in future titles.