The hired thugs encounter is supposed to be triggered by your character wronging an NPC in some way. So for example if you steal something from someone, assault or murder someone (or someone they are close to), or tresspass in their homes, there's a chance they will send hired thugs after you to teach you a lesson/completely straight-up murder you to death.
I used the words 'supposed to' there very deliberately, though, because some NPCs are notoriously touchy and will seemingly send hired thugs after you to murder you to death for utterly trivial offences. Belethor once sent hired thugs after me for no reason, for example. I'd never stolen from him, always traded fairly with him, never assaulted him or that weird, clingy assistant of his, and I'd even manipulated the fabric of time and space (read: reloaded a savegame) once to save him from death at the hands of vampires! I couldn't have been any nicer to him! I couldn't figure out what I might have done that deserved me being killed to death! So I was forced to look it up, and apparently he will send hired thugs after you for something as trivial as knocking a vase slightly as you walk past it in his shop. So while I like the mechanic Bethesda have employed here (actual consequences for being mean to people, at last!) I think they could have tweaked it a bit.
My absolute favourite 'hired thugs' encounter was when a Black-Briar Mercenary that I'd killed sent them after me. Because I'd killed him. Ghost vengeance is alive and well in Skyrim, it seems. But there are all sorts of examples of the mechanic going a bit haywire. Frickin' bandits will send hired thugs after you if you pickpocket them, even if you subsequently kill them! And I once had a Forsworn Briarheart send hired thugs after me because I'd pickpocketed his frickin' heart.
The Dark Brotherhood assassins, on the other hand, don't seem to have a trigger. They just come after you after a certain level until you either join up or murder them all in turn. I could be wrong on this, but all of my characters have universally faced Dark Brotherhood assassins at roughly the same time, no matter what they've done, so I don't think there's a particular trigger for this.