I don't want to speak to soon, but since i fired my bard, removed every workbench from the house, and always tell to wait to my follower, no freezing for me. In fact i spent hours in the house, with the kids and stewart running around the house, the pets too.
I hope it stays this way. :P...