Master Vampire - they are always difficult to fight alone, and there is a chance of infecting you (if you have no form of disease immunity, i.e racial bonuses, werewolf or amulets etc.) - if you have unwanted vampirism, it can be a pain to sort out. And also, the fact that in Dawnguard DLC, they can randomly attack towns and cities in Skyrim, and if you like having a town full of people (like me), this can be very annoying.
When I first got Skyrim, I did the Companions quest 'Trouble in Skyrim', which is a radiant quest and is usually just a bandit lair near Whiterun or something - I got the misfortune of having to go to Shimmermist Cave, which was for me, the most frustrating place in the game at a low level - Falmer (and a Dwemer Centurion) are not kind to a level 5-6 player, especially when I was a total noob to Skyrim itself.