Where from Horker Island did you see it? Had my char swim beyond that, plus to an island north of it (2-3 Spriggan Matrons on it), and saw no ship (just a boat on the above mentioned island).
Never noticed a ship out there, even before finding the Stahlrim.