It seems that you're using an outdated browser. Some things may not work as they should (or don't work at all).
We suggest you upgrade newer and better browser like: Chrome, Firefox, Internet Explorer or Opera

×
Community
General discussiongogrepo.py - Python script for regularly backing up your purchased GOG collection for full offline enjoyment(4211 posts)(4211 posts)
(4211 posts)
Pages:
1
50
100
150
200
250
273
274
275
276
277
281
This is my favourite topic
Posted March 09, 2025
avatar
mrkgnao: Feature request:

GOG seems to be adding more and more meaningless entries to some games. For example:

18:47:22 | no known filename for "blood_west (Blood West)"
18:47:33 | no known filename for "hacknet (Labyrinths DLC)"
18:47:33 | no known filename for "iratus_lord_of_the_dead (additional skins for all minions)"
18:47:33 | no known filename for "jagged_alliance_2_wildfire (Jagged Alliance 2: Wildfire)"
18:47:33 | no known filename for "omerta_city_of_gangsters (Omerta - City of Gangsters - The Bulgarian Colossus DLC)"
18:47:33 | no known filename for "omerta_city_of_gangsters (The Arms Industry DLC)"
18:47:33 | no known filename for "omerta_city_of_gangsters (Damsel In Distress DLC)"
18:47:33 | no known filename for "omerta_city_of_gangsters (The Con Artist DLC)"
18:47:33 | no known filename for "shadow_tactics_blades_of_the_shogun (Shadow Tactics: Blades of the Shogun)"
18:47:46 | no known filename for "stray_gods_the_roleplaying_musical (Stray Gods: The Roleplaying Musical)"
18:47:46 | no known filename for "warhammer_skulls_2022_digital_goodie_pack_goodies (Warhammer Chaos Gate)"

Perhaps gogrepoc can somehow learn to ignore these kind of entries.
Should be mostly ignoring them already, I assume this is from verify ? I don't think I've checked it yet.

Bit of an update:

Ignoring older timestamps on last-modified to work around the server desynching doesn't quite work, forgot to account for the possibility of Old -> New -> Old -> New, which will seesaw. I need to do something more sophisticated there and update the Timestamps fakely, so that the one in the manifest is always the most recent.

Also got a report about a download using Deflate64 Zip compression so I need to work on that too.
Post edited March 09, 2025 by Kalanyr
Posted March 09, 2025
avatar
mrkgnao: Feature request:

GOG seems to be adding more and more meaningless entries to some games. For example:

18:47:22 | no known filename for "blood_west (Blood West)"
18:47:33 | no known filename for "hacknet (Labyrinths DLC)"
18:47:33 | no known filename for "iratus_lord_of_the_dead (additional skins for all minions)"
18:47:33 | no known filename for "jagged_alliance_2_wildfire (Jagged Alliance 2: Wildfire)"
18:47:33 | no known filename for "omerta_city_of_gangsters (Omerta - City of Gangsters - The Bulgarian Colossus DLC)"
18:47:33 | no known filename for "omerta_city_of_gangsters (The Arms Industry DLC)"
18:47:33 | no known filename for "omerta_city_of_gangsters (Damsel In Distress DLC)"
18:47:33 | no known filename for "omerta_city_of_gangsters (The Con Artist DLC)"
18:47:33 | no known filename for "shadow_tactics_blades_of_the_shogun (Shadow Tactics: Blades of the Shogun)"
18:47:46 | no known filename for "stray_gods_the_roleplaying_musical (Stray Gods: The Roleplaying Musical)"
18:47:46 | no known filename for "warhammer_skulls_2022_digital_goodie_pack_goodies (Warhammer Chaos Gate)"

Perhaps gogrepoc can somehow learn to ignore these kind of entries.
avatar
Kalanyr: Should be mostly ignoring them already, I assume this is from verify ? I don't think I've checked it yet.
Yes. Verify. Thanks.
Posted March 10, 2025
I've hopefully dealt with the filename stuff on my local copy ( skipping outputting missing filenames for unreleased stuff ( or stuff with 0 MB size, which should make Galaxy only convenience listings skip too ), should still throw out the info if the filename is genuinely missing when it shouldn't be though.

Okay, think I've got the last-modified date thing solved but I need to do a couple of full updates in a row to test that.

Archive testing shouldn't be too much of a problem.

The Deflate64 thing seems like it's going to be a pain, a) I need to check if it's handled properly on new Python versions and b) the documentation and support for this format on Python is hot garbage.

I'll probably finish all of these and bundle them together for upload to Dev before the normal testing period and rolling the whole thing live. Maybe I should roll the last-modified to Dev but the current Dev state is at least as good as the previous version so doesn't seem that important.
Posted March 10, 2025
avatar
Kalanyr: The Deflate64 thing seems like it's going to be a pain, a) I need to check if it's handled properly on new Python versions and b) the documentation and support for this format on Python is hot garbage.
If you have problems with compression just add the following header to your requests:

Accept-Encoding: identity

That will instruct the server to only send uncompressed data.
Posted March 11, 2025
avatar
Kalanyr: The Deflate64 thing seems like it's going to be a pain, a) I need to check if it's handled properly on new Python versions and b) the documentation and support for this format on Python is hot garbage.
avatar
Geralt_of_Rivia: If you have problems with compression just add the following header to your requests:

Accept-Encoding: identity

That will instruct the server to only send uncompressed data.
No, this is a problem with the Zips provided as Extras. Deflate64 is a ( relatively kinda sort of new) compression level for Zip files that GOG hasn't used before but since Windows added Zip support and uses it automatically for Zip files over 2 GB it's become significantly more common (GOG doesn't seem to use it much I've only had 1 report and have no instances of it in my pretty large collection but it's weird seeing it at all ).
Posted March 11, 2025
avatar
Geralt_of_Rivia: If you have problems with compression just add the following header to your requests:

Accept-Encoding: identity

That will instruct the server to only send uncompressed data.
avatar
Kalanyr: No, this is a problem with the Zips provided as Extras. Deflate64 is a ( relatively kinda sort of new) compression level for Zip files that GOG hasn't used before but since Windows added Zip support and uses it automatically for Zip files over 2 GB it's become significantly more common (GOG doesn't seem to use it much I've only had 1 report and have no instances of it in my pretty large collection but it's weird seeing it at all ).
Ah, that's what you were talking about.

Instead of using Python's internal zip support you could always execute an external program like 7zip from within Python to test the archives.
Posted March 11, 2025
avatar
Kalanyr: No, this is a problem with the Zips provided as Extras. Deflate64 is a ( relatively kinda sort of new) compression level for Zip files that GOG hasn't used before but since Windows added Zip support and uses it automatically for Zip files over 2 GB it's become significantly more common (GOG doesn't seem to use it much I've only had 1 report and have no instances of it in my pretty large collection but it's weird seeing it at all ).
avatar
Geralt_of_Rivia: Ah, that's what you were talking about.

Instead of using Python's internal zip support you could always execute an external program like 7zip from within Python to test the archives.
Yeah, I know but crossplatform that sucks so much. May end up where I'm going though since licensing seems to be the primary problem for rar and Deflate64 which means Python support isn't likely to substantially improve at any point in the forseeable future.
Posted March 13, 2025
Paradox just nuked my plans for a bit, this is at least stable and functional unlike my CK3 mods right now, so I'll upload the fixed last-modified to Dev shortly and let it have it's usual week of testing before I roll it over to master.

I'll have to come back to the other stuff later on.
Posted March 13, 2025
Okay, pushed to dev for testing, I'll give it a week for testing then roll to master if there's no errors reported.
Posted March 16, 2025
Pushed a potential temporary fix for Deflate64 causing verification to continually orphan those files to dev so resetting the master count to 7 days again.
Posted March 24, 2025
Rolled to master
Posted March 25, 2025
I think that save_manifest function is missing dryrun as one of the arguments passed or its value shouldn't be checked in line 582.

Another bug: function save_manifest_core_worker cannot handle situation where gog-resume-manifest.dat doesn't exist yet and needs to be created, resulting in error "FileNotFoundError: [Errno 2] No such file or directory: 'gog-resume-manifest.dat'"
Post edited March 25, 2025 by KoЯni
Posted March 25, 2025
avatar
KoЯni: Another bug: function save_manifest_core_worker cannot handle situation where gog-resume-manifest.dat doesn't exist yet and needs to be created, resulting in error "FileNotFoundError: [Errno 2] No such file or directory: 'gog-resume-manifest.dat'"
Similar here. Had to delete after aborted verify with the old working version. Decided to use the new master and ran into this issue with gog-manifest.dat

used this :
sudo python3 gogrepoc.py update -resumemode noresume -strictverify -os windows linux -lang en de -skiphidden -installers both -nolog -strictdupe

EDIT: Reading helps. In my case it was gog-manifest.dat not the resume file....
Post edited March 26, 2025 by chalice
Posted March 26, 2025
avatar
KoЯni: Another bug: function save_manifest_core_worker cannot handle situation where gog-resume-manifest.dat doesn't exist yet and needs to be created, resulting in error "FileNotFoundError: [Errno 2] No such file or directory: 'gog-resume-manifest.dat'"
avatar
chalice: Same here. Had to delete after aborted verify with the old working version. Decided to use the new master and ran into this issue too.

used this :
sudo python3 gogrepoc.py update -resumemode noresume -strictverify -os windows linux -lang en de -skiphidden -installers both -nolog -strictdupe
That's one solution, but I think it's easier to just create an empty file ;) (Win7 here)
Post edited March 26, 2025 by KoЯni
Posted March 26, 2025
Huh, interesting, not sure how that hadn't been triggered before ( missing resume manifest), AFAIK the refactoring didn't actually change the logic just transferred it.

ETA - *ahh* I see, it's not the change to core_worker that's caused this, it's the change to the anti-file corruption measures. Though now I need to work out why the resume manifest would be in a different state to the actual manifest wrt that.

Dryrun is probably just an overight by me.
ETA2 - Yeah, it was, shouldn't be there, the MD5 folder can be regarded as an extension of the manifest, and it doesn't make sense to execute save_manifest at all in a real dryrun context.

I'll fix those ASAP ( probably in the next day ).

ETA3 - Think I've got it fixed locally, but I won't get a chance to test it for a few hours since I'm in the middle of an update right now.
Post edited March 26, 2025 by Kalanyr
Pages:
1
50
100
150
200
250
273
274
275
276
277
281
This is my favourite topic
Community
General discussiongogrepo.py - Python script for regularly backing up your purchased GOG collection for full offline enjoyment(4211 posts)(4211 posts)
(4211 posts)