If you don't do the mission as soon as he tells you to, later down the line it gets bugged. And you can't do it. I've seen some other threads on it but damn, talk about gamebreaking. Where the hell is the next patch...

yeah it is horrible. Waiting for the patch and cannot continue with the main story. Its horrible, I have already done all of the quests in velen and novigrad and now doing quests in skellige. I hope they fix it, otherwise i will ahve to use save that is 30hours gameplay before :(