Nope, sorry, not impossible, my build: Name: Raoh (from Fist of the North Star!) Khajiit, Heavy armor, enchanting, smithing. I'm not really that into playing this character because it's very hard to level up since there's no unarmed combat skill to increase, but other than that it works just fine. At lower levels it's actually quite overpowered, so it kinda' got boring real quick. As long as the heavy armor perk for punching is obtained before you get too far along in the game, then the character should stay overpowered...it becomes godly once you get some serious enchantments and your smithing is real high.
I ran through Bleak Falls Barrow in one of my fastest times, after that I felt there was no need to play this anymore since I knew where it was going (as stated above). Now, if I wasn't Khajiit...that would be a different story, but would also eventually become very powerful with enchanting, smithing, and that heavy armor perk. So no, not officially a challenging build, but certainly interesting, and I might revisit it some day and introduce some magic into the mix...that would allow for faster leveling, 'cause right now it's freakin' slow, I barely got to level 3 in a couple hours of total play time. Of course, I should have let the enemies hit me more to increase the heavy armor skill, I was floating like a butterfly the whole time, even in that heavy gear, they couldn't hardly hit me...then I'd sting like a bee...que the Rocky music! (now I have three references related to unarmed combat in one post...epic)