I returned the key because it was the right thing to do, and as a side note, my lockpick skill is at 100 (no perk points) so I have no problem opening locks. After returning the Skeleton key, the reward is well worth giving it up.
I know... I know... It has been bugging me, and I will return it eventually. I have been hoarding lock picks preparing myself to go ahead with it.