I guess you decided to "meet them at Karthspire," then forgot about it until you ended up at Karthspire, arguably the biggest Forsworn camp in the game. Yeah, they're there. If you come in from the south, you meet them first... but if you come in from the north, the fight starts with you on one...