Thank you Kalanyr for giving us this awesome tool, I was always frightened by the prospect of backing up my gog library, and with gogrepoc.py it is so easy.

I have to admit I feel a bit stupid though. I spent considerable time duplicating my archive on the same hdd so that I would have one "untouched legacy" archive and one that I would always keep up to date with the latest updates.

Of course, only after the first game update I noticed that the old files are also kept next to the new installers and gogrepoc allows basically allows you to keep a complete version history starting from the first time you backup your library. My "read only" legacy archive was pointless.

Running into an issue downloading Ion Fury with the new Aftershock DLC.

Output from gogrepo which creates an "empty" executable named "setup_ion_fury_2.0_(51180).exe" in the game folder:

14:27:40 | chunk request has unexpected Content-Range. expected '0-101928607/101928608' received '0-101343279/101343280'. skipping.

Output from gogrepoc which detects an issue and leaves the "setup_ion_fury_2.0_(51180).exe" in the !downloading folder:

14:30:38 | XML verification data size does not match manifest size for setup_ion_fury_3.0.0.9_(64bit)_(68519).exe. manifest 101928608, received 101343280, skipping.
14:30:38 | not moving uncompleted download '.\!downloading\ion_maiden_game\setup_ion_fury_2.0_(51180).exe', success: False remaining bytes: 101928608 / 101928608

Is this a situation where resetting the manifest would help? If so, how would I go about doing that?

I suggest you run update on the game again:
python gogrepoc.py update [add-here-all-the-flags-you-regularly-use-e.g.-os-lang-etc.] -ids ion_maiden_game
then clean and redownload and reverify.

I've uploaded a new version on the dev branch with various minor bugfixes and some rewrites to hopefully stop some issues if weird clashes on the GOG side happen again. Nothing that I think anyone can deliberately test but I'd appreciate it, if people can use it and let me know if they have any issues / any regressions. I'll roll it up to main next week probably if there's no issues.

I have downloaded the new version and merged it with my local copy (in which I had previously commented out many of the info prints). I am now running a full update and will report in several hours.

Meanwhile, can you please elaborate on the new "-strictdupe" flag. What happens when it is used and when it is not?

Sure, the default behavior is currently that if 2 different files have the same manifest filename and same file size and one of them has a missing MD5 and the other has an MD5 specified, the second one encountered will be treated as a duplicate and discarded (this means that eg duplicate installers where one is part of the installers and one is part of the extras, will match), strictdupe changes this so that those files would only be considered duplicates ifr they both have missing MD5s (in the case where the MD5s of both files match they are treated as duplicates in either case obviously )

Thank you.

Meanwhile, a full update, followed by a regular clean/download/verify, has finished running with the new version, without any issue.

The -strictdupe flag doesn't seem to be mentioned anywhere in the documentation (unless I turned half-blind and missed it) - I wouldn't even know about it, if it wasn't talked about here.

Which command is it used with? (I'm guessing Update or Download?)

Update.

FWIW, it is listed in the integrated help:
gogrepoc update -h

You can search the source code as well which can give you some hints (python is quite readable), just make sure to check out the dev branch not the master branch.

git clone https://github.com/Kalanyr/gogrepoc
cd gogrepoc
git switch dev

Hmmm. I think I properly need to code something up to deal with the GameData sets for the Mac in certain games (eg Baldur's Gate 3) because they always use the same file name and most of them are exactly the file size limit there's no good way to tell when they are updated, my current though is to detect a GameData set and nuke them all if any of them change (this will usually be because the last file in the set, which is below the file size cap) changes file size indicating an update has happened. Does anyone have any ideas for a better way to handle these files ? Another thought I had was doing a timestamp sanity check during verification (if the last file in the set doesn't have a last modified within 24 hours of the other files than nuke the other files )

My approach to checking if a file has changed:

If there is an XML for the file get it and parse all the information (filename, filesize, filedate, md5) in it. The file is unchanged if all the data matches the data I already had from an earlier run.

If there is no XML do a HEAD request on the file. From the reply headers you can get the filename, filesize and filedate. The file is unchanged if all the data matches the data I already had from an earlier run.

So even if for Mac games the filename and filesize typically stays unchanged for most of the files I do hope the filedate would at least change if GOG updates a file.

That sounds sensible, thanks.

We were talking (tangentially) about this type of topic over at the Level 1 Tech Forums
(https://forum.level1techs.com/t/btrfs-raid-confusion/204194/11)

While the basics of that thread were about the type of raid, this comment might apply:
------
I also have another jumble of very large drives, running mergerfs, snapraid and samba. It’s a very very surprisingly good working setup for lots of media and archival storage and secondary backups. Adding a drive and rebalancing using rsync of all things is a bizarre experience, … but it works.
------

It's something I'll be playing with to clean up my pile of old drives.

Anyone can help with this?

---
17:11:44 | loading local manifest...
17:11:45 | loading token...
17:11:45 | loading local resume manifest...
17:11:45 | fetching game product data (page 1)...
17:11:45 | refreshing token
17:11:45 | saving token...
17:11:45 | saved token
17:11:45 | refreshed token
17:11:46 | scanning found "dying_light_the_following_enhanced_edition" in product data!
17:11:46 | saving resume manifest...
17:11:46 | saved resume manifest
17:11:46 | (1 / 1) fetching game details for dying_light_the_following_enhanced_edition...
17:12:08 | request failed: 403 Client Error: Forbidden for url: https://content-system.gog.com/products/1448452156/users/46989201154138636/namespaces/website/download?path=%2Fsecure&token=***&access_token=***. will not retry.
17:12:08 | failed to fetch https://www.gog.com/downloads/dying_light_the_following_enhanced_edition/88891
17:12:08 | request failed: 403 Client Error: Forbidden for url: https://content-system.gog.com/products/1448452156/users/46989201154138636/namespaces/website/download?path=%2Fsecure&token=***&access_token=***. will not retry.
17:12:08 | failed to fetch https://www.gog.com/downloads/dying_light_the_following_enhanced_edition/88945
17:12:09 | request failed: 403 Client Error: Forbidden for url: https://content-system.gog.com/products/1448452156/users/46989201154138636/namespaces/website/download?path=%2Fsecure&token=***&access_token=***. will not retry.
17:12:09 | failed to fetch https://www.gog.com/downloads/dying_light_the_following_enhanced_edition/88948
17:12:09 | request failed: 403 Client Error: Forbidden for url: https://content-system.gog.com/products/1448452156/users/46989201154138636/namespaces/website/download?path=%2Fsecure&token=***&access_token=***. will not retry.
17:12:09 | failed to fetch https://www.gog.com/downloads/dying_light_the_following_enhanced_edition/88951
17:12:10 | request failed: 403 Client Error: Forbidden for url: https://content-system.gog.com/products/1448452156/users/46989201154138636/namespaces/website/download?path=%2Fsecure&token=***&access_token=***. will not retry.
17:12:10 | failed to fetch https://www.gog.com/downloads/dying_light_the_following_enhanced_edition/88954
17:12:22 | saving manifest...
17:12:23 | saved manifest
17:12:23 | saving resume manifest...
17:12:23 | saved resume manifest
17:12:23 | --
17:12:23 | total time: 0:00:38.393612
17:12:23 | exiting...