As far as I know, the Headless Horseman shows up between 10pm to 5am, and his only importance is leading you to a certain area. I prefer you chase him with a horse if you wish to find the destination.
It's actually not that important, there's no quest (that I'm aware of), but there are rumors of talking to him by going spectral or using the ethereal shout. I tried becoming ethereal and nothing really happened, but if you want to look further into it you can just check YouTube.