ymgve: If some files still break, you can change the number of slashes from 2 to 3, 4 etc to get a "new" URL. Managed to get all my XML files now without any proxying tricks.
Yeah, I also thought that might work. So I tried that right now.
But it is still a crutch since there is no guarantee you will get the XML after X retries and you have to give up at some point. I settled for 5 retries (after the first failure, so a total of 6 tries) and according to the log some files took up to 5 tries.
I did get everything without resorting to CDN server hopping by adding one more slash on every retry though.
Geralt_of_Rivia: Sadly it doesn't work as perfect as it seemed. I deleted my database again and tried getting everything again from scratch with the double slash method to check if this method doesn't make any problems with other XMLs. And it does make problems.
It's still a big improvement though. There were only 6 unreadable XMLs which I had to get using the usual CDN server hopping shenanigans instead of 89.
Magnitus: Right now, it takes me ~12 hours to generate a manifest and someone else reported ~34 hours (with the workaround method of computing the hash twice when you can' get the metadata).
I'll take any performance improvement that I can. I'll give it a try this weekend.
Twelve hours???!!!!?!?! And I thought 2:15 was slow...
How big is your library?