That's a loaded question.
Mentally, yes, I certainly think it it's plausible for man to lose such a substantial amount of IQ due to laziness and complacency. The data is present in standardized tests. Children are scoring lower and lower while a rises in social media activity, video-game popularity, and shocking mental disorders can be noted alongside that trend.
However, physically, I'd argue against it. A caveman's lifestyle revolves around their ability to gather materials, hunt, and build with little to no assistance. It's a highly active lifestyle that we continually move further away from.
Can we devolve into cavemen? Partially.
However, this is my opinion, and not a solidly researched and proven fact. Take it as you will.