I am assuming you don't know Polish. Therefore the content of Polish voices should not concern you because even if it was uber-brilliant you would not be able to appreciate - I don't say it in a mean way, I am just stating facts.
So if you like Polish voices, leave them on - you will be reading the English script anyway so it's the quality of that script is most important for you.

As long as English subtitles cover 100% of what the characters are actually saying in English, you are not 'missing' anything by switching to Polish voices, and i would even argue that you are gaining better immersion, as the world of Witcher seems more exotic and foreign when you are listening to an unknown language - kind of like watching Almodovar movie in Spanish, with English subs ;)

Polish is in my opinion the best language in which to experience the Witcher (assuming you know Polish) because the game was written in Polish and then translated.

Anyways, from what I heard the first non-Enhanced edition of Witcher had pretty poor non-polish translation/voicing which was much "cut-down" compared to Polish version (there were some famous examples from Zoltan's monologues where in Polish he would say 3 sentences which were compacted to one sentence in English subs - when he was replying to Geralt's question "why humans hate non humans" or some such).
This was supposedly for the most part fixed in EE, they retranslated and rerecorded hundreds of lines.

Personally, I would recommend to everyone playing with Polish voices and English (or their native language) subtitles - would you watch Japanese anime/Almodovar movie with English dubbing? I would not :) Immersion FTW! :)