You just need to remember that as you level, so do your adversaries. So power leveling something like armor is not really detrimental as it means that particular disproportionately high skill will be usable against your opponents. However if you power level skills that doesn't improve your combat, something like pickpocketing, then you may find combat more difficult.