It seems that you're using an outdated browser. Some things may not work as they should (or don't work at all).
We suggest you upgrade newer and better browser like: Chrome, Firefox, Internet Explorer or Opera

×
Community
General discussiongogrepo.py - Python script for regularly backing up your purchased GOG collection for full offline enjoyment(3861 posts)(3861 posts)
(3861 posts)
Pages:
1
50
100
150
200
204
205
206
207
208
258
This is my favourite topic
Posted April 04, 2022
avatar
mrkgnao: I'm not sure that's the entire picture. I may be wrong, but I think that if a file fails to obtain the XML during the manifest stage, it is not verified during the verify stage.
avatar
Magnitus: At that point, it is tool specific, but there is a path forward: You download the file twice, each download being completely independent from the other, the first time being only to get a checksum.

That way, you safeguard against corruption. The likelihood that two separate instances of corruption will result in the same checksum is close to impossible. At that point, we'd be talking about a consistent flaw in the hardware that is present in the design and would always result in the computation being bad in the exact same way.
Yes. That's the method I had used prior to gogrepoc, except that I didn't use any checksums. I simply compared the two files (binary comparison). In that respect, I was quite fortunate in having a very stable internet. In what had surely been several tens of thousands of downloads over the years, I had only ever had one (1) miscomparison.
Posted April 04, 2022
So, the xml error is still around eh? Did Gog say anything about it? :\

Also, are the gogrepoc fixes only in the dev version or also in the stable one?
Posted April 04, 2022
avatar
Tarhiel: Hello Kalanyr,

Have you made any progress about updating gogrepoc script to omit checking md5 data?
Every now and then script spews xml parsing errors in regard to md5 data and it is very slow then, and even if it manages to successfully update the manifest (at least I hope does, as it just finishes), when I try to download some game, it won´t downloads them successfully (download will fail).

Even when I use skipmd5 -skipsize -skipzip flags, it won´t start downloading's:
14:07:58 | download setup_banners_of_ruin_1.0.0_(64bit)_(48829).exe
14:07:58 | ------------------------------------------------------------
14:07:58 | preallocating '617406560' bytes for 'c:\Downloads\GOG\!downloading\banners_of_ruin\setup_banners_of_ruin_1.0.0_(64bit)_(48829).exe'
14:09:49 | xml parsing error occurred trying to get md5 data for [url=https://gog-cdn-lumen.secure2.footprint.net/token=nva=1648815779~dirs=6~token=0d4788c32cfef4d3d9885/secure/offline/1590989280/1590989280/55142836439868358/4576/setup_banners_of_ruin_1.1.30_%2864bit%29_%2853271%29.exe]https://gog-cdn-lumen.secure2.footprint.net/token=nva=1648815779~dirs=6~token=0d4788c32cfef4d3d9885/secure/offline/1590989280/1590989280/55142836439868358/4576/setup_banners_of_ruin_1.1.30_%2864bit%29_%2853271%29.exe[/url]
14:09:50 | chunk request has unexpected Content-Range. expected '0-617406559/617406560' received '0-617406559/632833656'. skipping.
14:09:50 | not moving uncompleted download 'd:\Downloads\GOG\!downloading\banners_of_ruin\setup_banners_of_ruin_1.0.0_(64bit)_(48829).exe', success: False remaining bytes: 617406560 / 617406560
avatar
Kalanyr: Sorry, it's not my intention to stop checking md5 data all together, way too much of the functionality depends on it (my comment was about not trusting old md5 data if new md5 data can't be retrieved.
Okay, is there something I can do with that error?
The script is useless to me if I can´t download the games.
Posted April 04, 2022
avatar
phaolo: So, the xml error is still around eh? Did Gog say anything about it? :\

Also, are the gogrepoc fixes only in the dev version or also in the stable one?
Also in master (what you call "the stable one").
Posted April 04, 2022
avatar
Kalanyr: Sorry, it's not my intention to stop checking md5 data all together, way too much of the functionality depends on it (my comment was about not trusting old md5 data if new md5 data can't be retrieved.
avatar
Tarhiel: Okay, is there something I can do with that error?
The script is useless to me if I can´t download the games.
I haven't tried it myself (because for me downloads still work despite the XML error), but perhaps you could attempt to modify line #200 in your own copy.

The line reads:
SKIP_MD5_FILE_EXT = ['.txt', '.zip','.tar.gz','']

I believe it lists the file extensions for which gogrepoc will not attempt to obtain MD5 XML info. I expect that if you add the extensions of installers to the list, it might work. For example,
SKIP_MD5_FILE_EXT = ['.txt', '.zip','.tar.gz','.exe','.bin','']
Posted April 04, 2022
avatar
Kalanyr: Sorry, it's not my intention to stop checking md5 data all together, way too much of the functionality depends on it (my comment was about not trusting old md5 data if new md5 data can't be retrieved.
avatar
Tarhiel: Okay, is there something I can do with that error?
The script is useless to me if I can´t download the games.
That seems unrelated to the recent issue. That just indicates the metadata and data from the server disagree on the size. It probably means GOG silently updated banners of ruin and you should manually update it using -ids and then try to download again.
Posted April 04, 2022
avatar
Tarhiel: Okay, is there something I can do with that error?
The script is useless to me if I can´t download the games.
avatar
Kalanyr: That seems unrelated to the recent issue. That just indicates the metadata and data from the server disagree on the size. It probably means GOG silently updated banners of ruin and you should manually update it using -ids and then try to download again.
Thank you I tried that, it ended with mxl error:
xml parsing error occurred trying to get md5 data for setup_banners_of_ruin_1.1.30_(64bit)_(53271).exe

avatar
mrkgnao: I expect that if you add the extensions of installers to the list, it might work. For example,
SKIP_MD5_FILE_EXT = ['.txt', '.zip','.tar.gz','.exe','.bin','']
Thank you, that did the trick: when I did that, game updated, and now it downloads as I write this.
I will let the whole manifest be updated like this (a have a backup copy), maybe there was something botched from the previous updates.

Then I will return that line to the default state.
Post edited April 05, 2022 by Tarhiel
Posted April 05, 2022
avatar
Kalanyr: That seems unrelated to the recent issue. That just indicates the metadata and data from the server disagree on the size. It probably means GOG silently updated banners of ruin and you should manually update it using -ids and then try to download again.
avatar
Tarhiel: Thank you I tried that, it ended with mxl error:
xml parsing error occurred trying to get md5 data for setup_banners_of_ruin_1.1.30_(64bit)_(53271).exe

avatar
mrkgnao: I expect that if you add the extensions of installers to the list, it might work. For example,
SKIP_MD5_FILE_EXT = ['.txt', '.zip','.tar.gz','.exe','.bin','']
avatar
Tarhiel: Thank you, that did the trick: when I did that, game updated, and now it downloads as I write this.
I will let the whole manifest be updated like this (a have a backup copy), maybe there was something botched from the previous updates.

Then I will return that line to the default state.
XML parsing errors shouldn't prevent downloads of the actual file, by themselves. It's just there to let you know that the XML data couldn't be fetched (and so eg resuming a partial download and verifying the file won't work as well).
Posted April 05, 2022
avatar
phaolo: So, the xml error is still around eh? Did Gog say anything about it? :\

Also, are the gogrepoc fixes only in the dev version or also in the stable one?
avatar
mrkgnao: Also in master (what you call "the stable one").
Thanks, but did anyone manage to report this error to Gog?
Posted April 05, 2022
avatar
mrkgnao: Also in master (what you call "the stable one").
avatar
phaolo: Thanks, but did anyone manage to report this error to Gog?
I didn't try. Doesn't seems worth the trouble of facing the blood-curdling chat bot and then waiting weeks for the ticket to be closed, given that we are talking here about an unofficial tool using an unofficial GOG feature.
Posted April 05, 2022
avatar
phaolo: Thanks, but did anyone manage to report this error to Gog?
avatar
mrkgnao: I didn't try. Doesn't seems worth the trouble of facing the blood-curdling chat bot and then waiting weeks for the ticket to be closed, given that we are talking here about an unofficial tool using an unofficial GOG feature.
Ugh.. that's true. But maybe we could talk with a blue?
Or maybe I'm too naive to believe it could work XD
Posted April 05, 2022
avatar
phaolo: Ugh.. that's true. But maybe we could talk with a blue?
Or maybe I'm too naive to believe it could work XD
GOG is truly in a fortunate situation with community tooling.

Essentially, they get the community doing their work for them with absolutely zero support commitment on their end (outside of what is needed for the website to function properly) for a feature they really ought to be providing themselves given that a core functionality they are offering is ownership and the desirable outcome for them is that their users buy hundreds or even thousands of games (which become unmanageable in the browser).

And make no mistake, that community tooling has business value for them. People who are geared toward ownership would feel a lot less inclined to buy a lot without a convenient way to manage their game backups.
Post edited April 05, 2022 by Magnitus
Posted April 05, 2022
avatar
Magnitus: People who are geared toward ownership would feel a lot less inclined to buy a lot without a convenient way to manage their game backups.
Me included. Managing standalones is unfeasible without an automatic process like gogrepo.
Post edited April 05, 2022 by phaolo
Posted April 05, 2022
avatar
Tarhiel: Thank you I tried that, it ended with mxl error:
xml parsing error occurred trying to get md5 data for setup_banners_of_ruin_1.1.30_(64bit)_(53271).exe

Thank you, that did the trick: when I did that, game updated, and now it downloads as I write this.
I will let the whole manifest be updated like this (a have a backup copy), maybe there was something botched from the previous updates.

Then I will return that line to the default state.
avatar
Kalanyr: XML parsing errors shouldn't prevent downloads of the actual file, by themselves. It's just there to let you know that the XML data couldn't be fetched (and so eg resuming a partial download and verifying the file won't work as well).
Well, for some mysterious reason, it did prevent me from downloading the games themselves.

But I managed modify my own file for creating .bat files for individual games to first do the update of manifest info of said game (with -ids) and THEN start to download it - and now it works.
Posted April 06, 2022
avatar
Kalanyr: XML parsing errors shouldn't prevent downloads of the actual file, by themselves. It's just there to let you know that the XML data couldn't be fetched (and so eg resuming a partial download and verifying the file won't work as well).
avatar
Tarhiel: Well, for some mysterious reason, it did prevent me from downloading the games themselves.

But I managed modify my own file for creating .bat files for individual games to first do the update of manifest info of said game (with -ids) and THEN start to download it - and now it works.
It's this: chunk request has unexpected Content-Range. expected '0-617406559/617406560' received '0-617406559/632833656'. skipping.
That was the error that stopped the download.

If you see errors of that form, it's best to do a quick chest for updates to the game and if there isn't to manually force a refresh with -ids.

That happens when GOG does a stealth update (they change a file, without marking the game as updated or changing the file name), it's a result of the data in the manifest being for a different file (so it's expecting a different file size). It's actually unrelated to the XML error.


(There are very rare occassions when the update won't help though, because GOG actually has the wrong metadata for the file. )
Pages:
1
50
100
150
200
204
205
206
207
208
258
This is my favourite topic
Community
General discussiongogrepo.py - Python script for regularly backing up your purchased GOG collection for full offline enjoyment(3861 posts)(3861 posts)
(3861 posts)