Yes, it is worth running an update on a game, prior to installing it.

GOGDownloader is obsolete and there might not be updates in the future for its download links.
New games should not be a problem.

`gogrepo.py clean` command is almost ready. I'll push it to github tomorrow once I'm finished testing.

"clean" will move any unknown folders and files from your games root directory to an "orphaned/" directory keeping the same directory structure.
Anything not known by the manifest will be moved.

eg.
-> a_golden_wake\gog_a_golden_wake_1.0.0.3.deb
would be moved to:
-> orphaned\a_golden_wake\gog_a_golden_wake_1.0.0.3.deb

Sample output:

./gogrepo.py clean <cleandir>
00:06:22 | loading local manifest...
00:06:23 | scanning local directories within 'T:\gog'...
00:06:23 | orphaning file 'a_golden_wake\gog_a_golden_wake_1.0.0.3.deb'
00:06:23 | orphaning file 'a_golden_wake\gog_a_golden_wake_1.0.0.3.tar.gz'
00:06:23 | orphaning file 'afterlife\gog_afterlife_1.0.0.5.deb'
00:06:23 | orphaning dir 'alone_in_the_dark'
00:06:23 | orphaning file 'alone_in_the_dark_1\setup_alone_in_the_dark_2.1.0.8.exe'
00:06:23 | orphaning file 'alone_in_the_dark_2\aitd_2_newspaper.zip'
00:06:23 | orphaning dir 'american_conquest'
00:06:23 | orphaning file 'amnesia_the_dark_descent\gog_amnesia_the_dark_descent_1.0.0.2.tar.gz'
00:06:23 | orphaning file 'anomaly_2\gog_anomaly_2_1.0.0.1.tar.gz'
00:06:23 | orphaning file 'anomaly_korea\gog_anomaly_korea_1.0.0.1.tar.gz'
00:06:23 | orphaning file 'anomaly_korea\gog_anomaly_korea_1.0.0.1.deb'
00:06:23 | orphaning dir 'the_penumbra_collection'
00:06:23 | orphaning file 'wasteland_2_kickstarter\gog_wasteland_2_1.9.0.13.tar.gz'
00:06:23 | orphaning file 'wasteland_the_classic_original\gog_wasteland_1_the_original_classic_1.0.0.1.deb'
00:06:23 | orphaning file 'wasteland_the_classic_original\gog_wasteland_1_the_original_classic_1.0.0.1.tar.gz'
00:06:23 | orphaning file 'witcher_adventure_game\setup_the_witcher_adventure_game_2.4.0.22.exe'
00:06:23 | orphaning file 'witcher_adventure_game\setup_the_witcher_adventure_game_2.2.0.20.exe'
00:06:23 | orphaning file 'witcher_adventure_game\setup_the_witcher_adventure_game_2.3.0.21.exe'
00:06:23 | orphaning file 'witcher_adventure_game\setup_the_witcher_adventure_game_2.1.0.17.exe'
00:06:23 | orphaning file 'witcher_adventure_game\setup_the_witcher_adventure_game_2.1.0.19.exe'
00:06:23 | orphaning file 'wizardry_7_gold\setup_wizardry7dos_german_2.2.0.1.exe'
00:06:23 | orphaning file 'worlds_of_ultima_the_savage_empire\setup_ultima_the_savage_empire_2.0.0.26.exe'
00:06:23 | orphaning file 'xenonauts\setup_xenonauts_2.1.0.10-1.bin'
00:06:23 | orphaning file 'xenonauts\setup_xenonauts_2.1.0.10.exe'
00:06:23 | orphaning file 'zak_mckracken_and_the_alien_mindbenders\gog_zak_mckracken_and_the_alien_mindbenders_1.0.0.4.deb'
00:06:23 | orphaning file 'zombie_shooter\setup_zombie_shooter_2.0.0.2.exe'
00:06:23 | orphaning file 'zombie_shooter_2\setup_zombie_shooter2_2.0.0.3.exe'

Then you can review the orphaned/ directory and delete it should you wish.

Thank you, does it do anything to custom folders or just files?
I keep my extra files like mods, cheats, walkthroughts etc. in a subfolder of the setup.exe files.

Iron Storm seems to have an interesting issue. Not sure if this is a problem with gogrepo script, or the GOG database.

I noticed the !info.txt file for Iron Storm doesn't contain any game items (installers or patches), only extras. On the GOG account page, I notice that for Iron Storm there is an installer, but for some reason the only selectable language is "POLSKI". No English. According to the Iron Storm gamecard though, it should support various languages:

Languages: Audio and text: English. Text only: Deutsch, español, français, italiano, polski, português
(maybe they are not separate downloads, but selectable within the game)

Doesn't gogrepo find any game items because the manifest file was updated with "lang en"?

(I have a feeling this issue was discussed already before, either in this or some other context? Deja vu, veni vidi vici and all that. I wonder if there are other games with that issue?)

Right now there's a filter for matching dir's. Also it only looks at files within a matching directory and currently doesn't look at sub dirs (ie. witcher_2/mods would be left untouched.. but witcher_4 would be unknown and be moved).

Let me know if you have suggestions.
99% this is GOG side problem.

don't have that game to check. Indeed it depends on the language from the drop down. If POLSKI is the only language, then that's a bug on GOG side. This observation of yours tells me that GOG json data is not including any data for english langauge.

I suggest you open a ticket with them to fix it.

Sent. I'll try to check if I detect anything similar on other games (ie. games where the '"game items" is empty, if you try to download English versions specifically). Of course I don't have absolutely every GOG game either, but most of them anyway.

Do you think it is feasible for the script to e.g. override the language selection during the download (or updating the manifest file), if there is only one language selection to choose from? So that it would download at least one version of the game in similar cases?

I submitted the new clean command to github (https://github.com/eddie3/gogrepo). You can run clean with -dryrun to see what would be moved and total size of the files. Without the -dryrun option, it will orphan files into a "!orphaned" directory relative to the cleandir.

Usage:

python gogrepo.py clean -h
usage: gogrepo.py clean [-h] [-dryrun] cleandir

positional arguments:
cleandir root directory containing gog games

optional arguments:
-h, --help show this help message and exit
-dryrun do not move/orphan files, only display what would be cleaned

Example:

./gogrepo.py clean T:\gog

It works like this:

* Any unknown folders found in the root cleandir will be moved to "!orphaned" folder (eg. unknown_dir -> !orphaned\unknown_dir)
* Files in the root cleandir are ignored (since gogrepo.py would have only ever created game folders from root eg. the_witcher_2\)
* Any files in a game directory that are unknown to manifest will be moved to "!orphaned" folder (eg. the_witcher_2\old_patch.exe -> !orphaned\the_witcher_2\old_patch.exe)
* Any sub directories found within a game folder will be ignored (eg. the_witcher_2\mods would be left unscanned and untouched). Since gogrepo.py doesn't create subdirectories within game folders, it assumes this is created by the user manually.

So basically this is a pretty safe clean where you can review the stuff moved into !orphaned and delete as you wish.

You can add root cleandir subdir exclusions if you want by modifying ORPHAN_DIR_EXCLUDE_LIST in the source.
You can change the orphan foldername by modifying ORPHAN_DIR_NAME. I could add an command line option later, but I assume most people will just delete the folder after reviewing.

Example of clean with -dryrun (you can see what would be moved)

python gogrepo.py clean t:\gog -dryrun
22:40:20 | loading local manifest...
22:40:21 | scanning local directories within 't:\gog'...
22:40:23 | orphaning file 'tex_murphy_1_mean_streets\gog_tex_murphy_mean_streets_1.0.0.1.tar.gz'
22:40:23 | orphaning file 'tex_murphy_1_mean_streets\gog_tex_murphy_mean_streets_1.0.0.1.deb'
22:40:23 | orphaning file 'tex_murphy_2_martian_memorandum\gog_tex_murphy_martian_memorandum_1.0.0.1.deb'
22:40:23 | orphaning file 'tex_murphy_2_martian_memorandum\gog_tex_murphy_martian_memorandum_1.0.0.1.tar.gz'
22:40:23 | orphaning file 'tex_murphy_the_pandora_directive\gog_tex_murphy_the_pandora_directive_1.0.0.5.tar.gz'
22:40:23 | orphaning file 'tex_murphy_under_a_killing_moon\gog_tex_murphy_under_a_killing_moon_1.0.0.5.tar.gz'
22:40:23 |
22:40:23 | total size of newly orphaned files: 4.52GB
22:40:23 | --
22:40:23 | total time: 0:00:02.642685
22:40:23 | exiting...


Do a real clean where unknown files are moved to !orphaned folder

python gogrepo.py clean t:\gog

22:52:39 | loading local manifest...
22:52:39 | scanning local directories within 't:\gog'...
22:52:41 | orphaning file 'tex_murphy_1_mean_streets\gog_tex_murphy_mean_streets_1.0.0.1.tar.gz'
22:52:41 | orphaning file 'tex_murphy_1_mean_streets\gog_tex_murphy_mean_streets_1.0.0.1.deb'
22:52:41 | orphaning file 'tex_murphy_2_martian_memorandum\gog_tex_murphy_martian_memorandum_1.0.0.1.deb'
22:52:41 | orphaning file 'tex_murphy_2_martian_memorandum\gog_tex_murphy_martian_memorandum_1.0.0.1.tar.gz'
22:52:41 | orphaning file 'tex_murphy_the_pandora_directive\gog_tex_murphy_the_pandora_directive_1.0.0.5.tar.gz'
22:52:41 | orphaning file 'tex_murphy_under_a_killing_moon\gog_tex_murphy_under_a_killing_moon_1.0.0.5.tar.gz'
22:52:41 |
22:52:41 | total size of newly orphaned files: 4.52GB
22:52:41 | orphaned items moved to: t:\gog\!orphaned
22:52:41 | --
22:52:41 | total time: 0:00:02.551700
22:52:41 | exiting...

Doing a clean on an already clean collection:

python gogrepo.py clean t:\gog -dryrun

22:38:50 | loading local manifest...
22:38:50 | scanning local directories within 't:\gog'...
22:38:52 | nothing to clean. nice and tidy!
22:38:52 | --
22:38:52 | total time: 0:00:02.519361
22:38:52 | exiting...

There seems to be a minor inconsistency with regards to case:

clean orphans simon_the_sorceror_3D (due to the official GOG file name being simon_the_sorceror_3d) .

That would be odd on Windows but understandable since *nix is case sensitive. But update and download have been using the directory without problem.

I'm happy with whichever way it gets resolved, just thought it was worth noting.

ETA - Would it be possible to add an option to the verify command to orphan things that fail md5/ size checks (as an option along with the current delete) ? Seems useful for verifying if there's actually a problem with the file or if there's a problem with GOG.

Unfortunately I don't have that game. I should add a debug option to output the raw json data from GOG to help with bug reports.

Hmm how did your game folder name end up with a capital "3D"? Did you manually rename it as some point?

The only explanation I have for your case is that you ran an update, and GOG changed the game name on their server from "3D" to "3d". So now the older folder isn't recognized anymore. Now when you run clean, the "3D" version doesn't match anything known in the manifest and so it gets orphaned.

I'll soon be adding auto detection of moved/renamed game folders and files. If my explanation is correct, then the 'update' would detect the folder rename (by game id # being the same) and just rename the folder (in this case "3D" to "3d").

Yeah I can add orphan support to the verify command as well.

I run it on Windows and FreeBSD (case sensitive ZFS) and no issues with the ~850+ games I have. The clean command did save me 500GB though :)

They did a similar thing with Witcher3, that was first something like "The Witcher3" and then "The Witcher3 game", which made the script obviously download the whole content again.

Yeah I noticed that. I'll add a fix where on update if a "new" game is detected but has the same game id as an already known one, assume it's a game folder rename.

I made it so user has to explicit state directory as a safety thing (ie. if someone ran it from c:\ by accident)

yup GOG seems to be renaming lots of game titles lately. Detecting that is next on my list.

I came to have it called Simon_the_Sorceror_3D because thats what the GOG Downloader called it, I think.

I don't think they changed the name of Simon the Sorceror 3D to 3d (I did an Update, Download , Verify) last Friday and it was detected and passed fine but its possible. It's pretty minor as I said though. Even if it did get accidentally cleaned once its not even a gig in size and the next update would fix the folder name. I'll check the Manifest in a couple of hours and let you know (my GOG archival collection is on an external drive).

Yeah, clean saved me 134 GB , so I'm really happy. I moved the extra stuff I had (missing manuals, mods, etc) into a folder called "extras" in the relevant game folder so I can just add cleaning to my Update, Download, Verify batch file and then just do a quick check for any problems before nuking !orphaned.

Cool, glad it's working for you.