I found a way to workaround to this bug (which is even in version 1.31).
If someone struggles with this same problem, here is one way :

You must download and install witcher 3 console and after getting the black pearl:
- open console (§) and enter
xy(-800,100)
- then open console again and enter same command (up key helps) so you don't die when you fall.
- Then you can kill the enemies and complete the quest