There is a series of books following the plot of the Myst series. The books are:
• The Book of Atrus
• The Book of Ti'ana
• The Book of D'ni
They are available on Amazon both together and separately: http://www.amazon.com/Rand-Miller/e/B000AP9Q0A/ref=ntt_athr_dp_pel_1
I just finished playing Myst and am contemplating playing Riven. My question is if I should read any of these books before doing so. If so, when should they be read in relation to the games?

If you like the series, you should really get all 3 Books. Probably the Myst Reader which is all 3 Books in one volume and it's usually significantly cheaper than buying them separate.
The Book of Atrus ties directly with Riven, as it deals with Atrus' liffe from birth to the time he settled on Myst with Catherine (and Anna) and it somewhat explains the back story events that led to the situation you'll be walking into in Riven.
The Book of Ti'ana and Book of D'ni don't tie as directly with any specific game of the series as BoA does with Riven. They focus more on providing a better insight about the D'ni story as a whole, which if you put all Myst/URU games togheter plus the Books it's huge.
The Book of Ti'ana adresses the time when the D'ni reached the surface (Earth's surface) and the subsequent events that lead to the plague being released into the Ages while the Book of D'ni mostly adresses the events in Terahnee (D'ni's 'sister' society result of the Gartenay exodus/split) after Atrus and Catherine find the Terahnee book in D'ni and visit the Age. These two are invaluable in giving a more in depth look in regards to many subjects, from Atrus' grandparents lives (Anna - or Ti'ana, and Aitrus) to many of the subjects briefly adressed in the DRC documents in URU, etc, etc.
In terms of timeline, if you want 'to do' things in order:
-Book of Ti'Ana
-Book of Atrus
-Myst
-Riven
-Book of D'ni
Having said that, i don't think you have to worry too much about the order in which you play/read the games/books as long as you play/read them all. The Myst Reader it's an absolute must if you like the series, and considering its price...well...just get it ;)

Thank you.
Will reading any of the books before playing Riven ruin the exploration of the game; the plot as it develops?

You're welcome.
No. Reading the BoA before playing Riven will give you a better understanding of the characters and their relations in the game as you play, but BoA doesn't have any 'spoilers' in regards to Riven's plot. Same for the other two Books, no spoilers.