I agree there. But like you said, it is very possible they went on and created their own culture later on but we probably won't know for sure until TES Online or a new Elder Scrolls game with that information. The Forsworn may be one group but as far as i know they are the only group completely made up of Bretons so it's basically the only window we have to really look through at Bretons. Though i think i read something saying they were Bretons mixed with Nord blood, but i don't know how true that is. I'm interested into finding out more about Bretons in general.
The Forsworn might be the splinter group descending from runaway Bretons or something like that. In Tamrielic history humanity's been at each others necks for quite a bit, having their own little tribes and territories while elves for example had entire civilizations going at that point. Even when the Ayleid started kidnapping, enslaving, murdering and brutally torturing humans just for fun they didn't step up, I guess they just didn't consider themselves to belong together for whatever reasons (just like humanity sees differences in one another because of irrelevant characteristics such as skin color, facial structures etc despite being genetically the same).
Maybe the Forsworn were one of those tribes, only larger than others and being able to hold themselves up against the Nords for quite a while. Well, now that there's an Empire, I doubt they'll live for much longer as long as they behave aggressively and with hostility towards everyone but themselves.
(Admittedly, while being in Cidhna Mine I felt some sympathy for the Forsworn, but when I went to their dungeons and saw innocent people, kidnapped Nord women and many more laying about... no, that's just not a decent thing to do.)