Brand new error I saw seems like its a python issue, hope it helps!

I thought of (or rather experienced) one difference.

If you create a new manifest, when you later come to verify your library, gogrepo will verify every single file.

If you use the -full flag, when you later come to verify your library, gogrepo will verify only those files that have not been verified yet or have changed since the last verify (since previously-verified files are remembered in the manifest).

Considering that a complete verify of my entire library takes more than 12 hours, I prefer the -full flag.

I've been travelling, so I only got around to testing your new script these last couple of days. Thank you for your efforts.

Here is the procedure I used:

1) Before trying your new script, I ran a full update using your old gogrepoc script (with the two xml-related manual patches mentioned earlier in this thread).
I got failures for 18 files (17 games):
09:15:05 | xml parsing error occurred trying to get md5 data for setup_avernum5_2.1.0.4.exe
09:17:29 | failed to fetch https://www.gog.com/downloads/batman_the_enemy_within_the_telltale_series/en1patch0 because of non-HTTP Error
09:17:56 | xml parsing error occurred trying to get md5 data for patch_battle_brothers_1.5.0.9_(54278)_to_1.5.0.10_(54352).exe
09:18:15 | xml parsing error occurred trying to get md5 data for setup_battle_brothers_-_blazing_deserts_1.5.0.10_(54352).exe
09:53:32 | failed to fetch https://www.gog.com/downloads/ethan_meteor_hunter/en1installer0 because of non-HTTP Error
09:54:44 | failed to fetch https://www.gog.com/downloads/evil_islands/en1installer1 because of non-HTTP Error
10:03:00 | xml parsing error occurred trying to get md5 data for setup_ghostrunner_pre-order_katana_dlc_42441_444_(53890).exe
10:33:38 | failed to fetch https://www.gog.com/downloads/magic_carpet/en1installer0 because of non-HTTP Error
10:37:54 | failed to fetch https://www.gog.com/downloads/master_of_orion_terran_khanate/en1installer0 because of non-HTTP Error
10:44:56 | xml parsing error occurred trying to get md5 data for setup_moto_racer_2.1.0.11.exe
10:56:09 | xml parsing error occurred trying to get md5 data for setup_pixeljunk_monsters_hd_r3132_(21326).exe
11:23:52 | xml parsing error occurred trying to get md5 data for setup_star_renegades_1.5.1.5_(53756)-1.bin
11:41:05 | xml parsing error occurred trying to get md5 data for setup_the_hand_of_merlin_677512_(64bit)_(54062)-1.bin
11:44:42 | xml parsing error occurred trying to get md5 data for setup_the_riftbreaker_prologue_1.5_(64bit)_(53557)-1.bin
11:49:25 | xml parsing error occurred trying to get md5 data for setup_this_war_of_mine_stories_-_the_last_broadcast_6.0.7.5_(51540).exe
11:52:07 | failed to fetch https://www.gog.com/downloads/thronebreaker_the_witcher_tales/en1installer1 because of non-HTTP Error
12:05:04 | failed to fetch https://www.gog.com/downloads/warlords_battlecry_2/en1installer0 because of non-HTTP Error
12:10:09 | failed to fetch https://www.gog.com/downloads/xenonauts_2_demo/en1installer1 because of non-HTTP Error

2) I waited 24 hours and retested the 17 failed games, still on the old script.
Only one game disappeared (pixeljunk_monsters_hd). The rest stayed more or less the same.

3) I installed your new script, deleted the old manifest (to have a clean slate), and ran a full update.
Things didn't change much. One game disappeared (ghostrunner) and three new games appeared (democracy_3, layers_of_fear, supreme_league_of_patriots_episode_2):
16:09:17 | xml parsing error occurred trying to get md5 data for setup_avernum5_2.1.0.4.exe
16:11:17 | xml parsing error occurred trying to get md5 data for patch_batman_the_enemy_within_-_the_telltale_series_Episode_5_(19607)_to_Noir-gog_(37813).exe
16:12:31 | xml parsing error occurred trying to get md5 data for patch_battle_brothers_1.5.0.9_(54278)_to_1.5.0.10_(54352).exe
16:15:30 | unexpected connection error fetching md5 data for setup_battle_brothers_-_blazing_deserts_1.5.0.10_(54352).exe This error may be temporary. Please retry in 24 hours.
16:36:31 | xml parsing error occurred trying to get md5 data for setup_democracy_3_electioneering_1.34_final_update_(23486).exe
16:48:22 | unexpected content decoding error fetching md5 data for setup_ethan_meteor_hunter_1.0_(20745).exe This error may be temporary. Please retry in 24 hours.
16:49:19 | xml parsing error occurred trying to get md5 data for setup_evil_islands_2.0.0.5.exe
17:17:01 | xml parsing error occurred trying to get md5 data for setup_layers_of_fear_inheritance_1.1.0_(16815).exe
17:23:13 | xml parsing error occurred trying to get md5 data for setup_magic_carpet_plus_1.0_(28186).exe
17:24:41 | unexpected content decoding error fetching md5 data for setup_master_of_orion_terran_khanate_55.1.1_2.1.41258_locfix_(11709).exe This error may be temporary. Please retry in 24 hours.
17:29:25 | xml parsing error occurred trying to get md5 data for setup_moto_racer_2.1.0.11.exe
18:09:49 | xml parsing error occurred trying to get md5 data for setup_star_renegades_1.5.1.5_(53756)-1.bin
18:17:29 | xml parsing error occurred trying to get md5 data for setup_slop_episode2_2.0.0.1.exe
18:27:05 | xml parsing error occurred trying to get md5 data for setup_the_hand_of_merlin_677512_(64bit)_(54062)-1.bin
18:30:25 | xml parsing error occurred trying to get md5 data for setup_the_riftbreaker_prologue_1.5_(64bit)_(53557)-1.bin
18:34:54 | xml parsing error occurred trying to get md5 data for setup_this_war_of_mine_stories_-_the_last_broadcast_6.0.7.5_(51540).exe
18:37:36 | unexpected connection error fetching md5 data for setup_thronebreaker_3553184_(64bit)_(38743)-1.bin This error may be temporary. Please retry in 24 hours.
18:52:33 | unexpected connection error fetching md5 data for setup_warlords_battlecry_2_1.04_(30487).exe This error may be temporary. Please retry in 24 hours.
18:57:49 | xml parsing error occurred trying to get md5 data for setup_xenonauts_2_demo_kickstarter_demo_(21327)-1.bin

4) I waited 36 hours and retested the 20 failed games (17 from first run + 3 from second) on the new script.
Three games disappeared (batman_the_enemy_within_the_telltale_series, evil_islands, star_renegades). The rest stayed more or less the same.

Summary:
- 13 files failed consistently in all 4 runs, spanning about 72 hours
- 5 files stopped failing after 24-72 hours
- 3 files failed consistently in the 2 later runs, spanning about 36 hours
- My limited experience seems to point out that the "Please retry in 24 hours" message is not very useful. The same file failing on different runs sometimes had it and sometimes not. Files that didn't have the message, still stopped failing at some point, while files that did have it, did not.

Questions:
- What is the effect of a file consistently failing the xml parsing? Does it mean that it is not verified during the verify stage? Something else?
- If the former, is there some indication in the verification summary that some files were not verified? I can't see such an indication.
- Does it matter if the manifest already had information from a successful past xml parse, or is it overwritten by a failed parse? Assuming the file hasn't changed.

A much more basic verification is done for files without valid md5 data, (just a file size check and a check that (non-spanned) zip files aren't corrupt, I need to fix this at some point to not false positive the first volume of a multi-archive zip file as corrupt but GOG uses those only extremely rarely (I've only seen it once for a Mac installer over 4GB) and they seem to have moved to unnamed data files instead more recently).

Existing information is stomped unless the game data isn't written back into the manifest, it has to be that way because GOG sometimes does stealth updates of files without changing their name, so there's no guarantee that old data is still valid, I could theoretically do some introspection about whether or not that's a reasonable assumption but I suspect the general loss of performance would be a more practical problem than the potential loss of XML md5 data that's pretty suspect to start with, if it's not reliably available (it's probably best not to make a particularly strong assumption that XML data that is no longer available is still valid).

Seems like GOG broke something with this... I tried to do a manifest update and download and got a ton of errors, and no downloads....0

Logs to follow. And more followed like this through the whole update.\

Then Download script ran: This sucks because it feels like they are actively trying to sabotage their user base here, knowing we use tools for this and having no documentation or anything for it. I don't know if there is any solution, but it reinforces that I'm done with this shitty company.

I think xml data is nice to have and you should definitely look for it and use it if it is there.

However, it has proven consistently unreliable (over a long enough time period not to be considered simply a transient issue) and at this point, a robust solution should have a fallback if it isn't there.

Prior to this new wave of incidents, I did hit a case where the xml data was malformed (there was a semblance of xml, but it was not syntactically valid, though the first part of the xml document that contained most of the info was readable so I implemented a fallback to read just that if the xml document was not syntactically valid).

So yeah, even if GOG fixes everything perfectly with this, there is enough of a precedent at this point for me that I'd keep my fallback anyways. By default, I try to read the xml data and if that if valid everywhere, my fallback will never be used, but if there are further issues, it will not break the flow of the tool anymore.

I'll do some testing later on but at a first glance that log suggests your computer ran out of memory or had a memory error of some sort. Might be worth restarting the computer and trying again when you get a chance and maybe doing a RAM test if your BIOS supports that.

Hi Everyone,

I am a novice at this, but through research, I was able to successfully create a manifest file

Is there a more user-friendly way to read it? For example, is there a way to import it into Microsoft Excel?

I didn't happen to see any commands that gave this type of summary information within the cmd window. I am using gogrepoc, on Windows 10.

I am trying to find out what total size of my library is. It would be great to be able to get a list of windows installers, and how large each one is.

Thank you in advance!

EDIT: I was able to get some basic information by using the -dryrun flag with the download command, as follows:

python gogrepoc.py download -dryrun -skipgalaxy -skipextras -os windows -lang en

I ran the command with and without the -skipextras flag, and got two different results (which makes sense).

Hello Kalanyr,

Have you made any progress about updating gogrepoc script to omit checking md5 data?
Every now and then script spews xml parsing errors in regard to md5 data and it is very slow then, and even if it manages to successfully update the manifest (at least I hope does, as it just finishes), when I try to download some game, it won´t downloads them successfully (download will fail).

Even when I use skipmd5 -skipsize -skipzip flags, it won´t start downloading's:
14:07:58 | download setup_banners_of_ruin_1.0.0_(64bit)_(48829).exe
14:07:58 | ------------------------------------------------------------
14:07:58 | preallocating '617406560' bytes for 'c:\Downloads\GOG\!downloading\banners_of_ruin\setup_banners_of_ruin_1.0.0_(64bit)_(48829).exe'
14:09:49 | xml parsing error occurred trying to get md5 data for [url=https://gog-cdn-lumen.secure2.footprint.net/token=nva=1648815779~dirs=6~token=0d4788c32cfef4d3d9885/secure/offline/1590989280/1590989280/55142836439868358/4576/setup_banners_of_ruin_1.1.30_%2864bit%29_%2853271%29.exe]https://gog-cdn-lumen.secure2.footprint.net/token=nva=1648815779~dirs=6~token=0d4788c32cfef4d3d9885/secure/offline/1590989280/1590989280/55142836439868358/4576/setup_banners_of_ruin_1.1.30_%2864bit%29_%2853271%29.exe[/url]
14:09:50 | chunk request has unexpected Content-Range. expected '0-617406559/617406560' received '0-617406559/632833656'. skipping.
14:09:50 | not moving uncompleted download 'd:\Downloads\GOG\!downloading\banners_of_ruin\setup_banners_of_ruin_1.0.0_(64bit)_(48829).exe', success: False remaining bytes: 617406560 / 617406560

Sorry, it's not my intention to stop checking md5 data all together, way too much of the functionality depends on it (my comment was about not trusting old md5 data if new md5 data can't be retrieved.

Update #1: I continued testing the games that had XML failures on a daily basis. A week or so later, of the 16 games that had failures, 9 "corrected" themselves, 7 not yet.

Depends on where you live. With my last update of 20 games or so, I had 3 bad ones:

[sdk] Bad metadata for /downloads/crystal_crisis/en1installer0: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/intravenous/en1patch0: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/hellish_quart/en1installer0: File metadata was still fetched using much longer workaround method.

That's 15% with patches and 10% with just the installers. Maybe it is worse for recently updated games.

Personally, I'm happy to just live with it as I now consider it a low priority problem. Manifest generation will just take longer and consume more bandwidth on GOG's end since the md5 checksum now needs to be computed twice separately (which to me seems fair since they are the ones providing shoddy support for their offline installers).

I'm not sure that's the entire picture. I may be wrong, but I think that if a file fails to obtain the XML during the manifest stage, it is not verified during the verify stage.

At that point, it is tool specific, but there is a path forward: You download the file twice, each download being completely independent from the other, the first time being only to get a checksum.

That way, you safeguard against corruption. The likelihood that two completely separate instances of corruption will result in the same checksum is close to impossible. At that point, we'd be talking about a consistent design flaw in the hardware (or os) that would always result in the computation being bad in the exact same way.

So I guess if it is not done already, the only thing to do would be to modify the gogrepoc scripts to do the above and you'll be in business. Manifest generation might be slower because of files with bad metadata will need to be downloaded during manifest generation to get their checksum, but that's it. Then, you can mostly ignore this issue.