If this question is answered in the game, I must have missed it. You learn a couple of things about Lermitage later in the game, but Prince James' men were looking for the baby not the treasure so I don't see how those things are relevant.

Curiously, you don't get points for matching the tire tracks from the car to the ones at Lermitage, although you do get points for matching the tire tracks at Chateau de Serras. I'm guessing it's because Chateau de Serras is where they were captured and killed but that doesn't explain why they went to Lermitage