Geralt_of_Rivia: And so do lists. That's the beauty of it.
In C# for example lists have the methods Add, Insert, Remove, Find, FindAll, Exists, Sort, Reverse, just to name some of the most used ones.

Geralt_of_Rivia: The table of files, obviously. Only the files table knows which files have not been downloaded and what type the file is (Installer, DLC, Patch, Extra) and what language (for games with separate installers for every language) while only the games table knows the name of the game. So without a join it is impossible to have a result table that is sorted by game names.

timppu: Either way, splitting the manifest file before download is good enough solution for me, especially since I made a bash script that does that splitting automatically and copies the partial manifest files to different paths, I just run that script after gogrepoc update and before gogrepoc download (if needed). I don't need to touch the gogrepoc.py script myself.

UrsaCorvalis: Very nice. You wouldn't fee like sharing the splitting script would you? Might come in handy...

Another really dumb question...

What is the role of ".gitignore"? Is that a github thing (I rarely use github) or should that be with the script?

Thanks

Geralt_of_Rivia: The .gitignore file is a simple text file which tells Git which files or subdirectories to ignore. Files and directories listed in .gitignore will neither have their changes tracked nor will they be included in online repositories.

If you don't use Git then you do not really need this file. But it is still a good idea to leave it there should you ever change your mind and use Git in the future.

KoЯni: BTW !images folder is empty for every single game I have, is there any reason for its existence?

Cheers!

neumi5694: The image directory is always created, even if you don't use -covers and -backgrounds.
I guess that's just an oversight.
I downloaded the images for all games by excluding all files and using thse two options.

KoЯni: Hi Kalanyr,

There seems to be an issue in handling The Witcher 3 game (both IDs actually: the_witcher_3_wild_hunt_game and the_witcher_3_wild_hunt_game_of_the_year_edition_game) resulting in duplicating some files - be advised that it affects only these two games out of my 1000+ games collection and I think it might be related to dual language download process I use gogrepoc for.

For the clarity, I will focus on the first game.
To reproduce, I deleted whole game folder and then:

1) Updated the game info info with:
gogrepoc.py update -ids the_witcher_3_wild_hunt_game -lang en pl -os windows -full

It results in these messages:
17:07:14 | -> setup_the_witcher_3_wild_hunt_1.31_(a)_(9709).exe already exists in this game entry with a different size and/or md5, this file renamed to setup_the_w
itcher_3_wild_hunt_1.31_(a)_(9709)(1).exe
17:07:14 | -> setup_the_witcher_3_wild_hunt_1.31_(a)_(9709)-1.bin already exists in this game entry with a different size and/or md5, this file renamed to setup_the
_witcher_3_wild_hunt_1.31_(a)_(9709)(1)-1.bin
17:07:14 | -> setup_the_witcher_3_wild_hunt_1.31_(a)_(9709)-2.bin already exists in this game entry with a different size and/or md5, this file renamed to setup_the
_witcher_3_wild_hunt_1.31_(a)_(9709)(1)-2.bin
17:07:14 | -> setup_the_witcher_3_wild_hunt_1.31_(a)_(9709)-3.bin already exists in this game entry with a different size and/or md5, this file renamed to setup_the
_witcher_3_wild_hunt_1.31_(a)_(9709)(1)-3.bin
17:07:14 | -> setup_the_witcher_3_wild_hunt_1.31_(a)_(9709)-4.bin already exists in this game entry with a different size and/or md5, this file renamed to setup_the
_witcher_3_wild_hunt_1.31_(a)_(9709)(1)-4.bin
17:07:14 | -> setup_the_witcher_3_wild_hunt_1.31_(a)_(9709)-5.bin already exists in this game entry with a different size and/or md5, this file renamed to setup_the
_witcher_3_wild_hunt_1.31_(a)_(9709)(1)-5.bin
17:07:14 | -> setup_the_witcher_3_wild_hunt_1.31_(a)_(9709)-6.bin already exists in this game entry with a different size and/or md5, this file renamed to setup_the
_witcher_3_wild_hunt_1.31_(a)_(9709)(1)-6.bin
17:07:14 | -> setup_the_witcher_3_wild_hunt_1.31_(a)_(9709)-7.bin already exists in this game entry with a different size and/or md5, this file renamed to setup_the
_witcher_3_wild_hunt_1.31_(a)_(9709)(1)-7.bin
17:07:14 | -> witcher3_lang_pl_2.0.0.52.exe already exists in this game entry with same size/md5, skipping adding this file to the manifest
17:07:14 | -> witcher3_lang_pl_2.0.0.52-1.bin already exists in this game entry with a different size and/or md5, this file renamed to witcher3_lang_pl_2.0.0.52(1)-
1.bin
17:07:14 | saving manifest...
17:07:16 | saved manifest

2) Downloaded game files with:
gogrepoc.py download -ids the_witcher_3_wild_hunt_game

Among other files, I get these:
setup_the_witcher_3_wild_hunt_1.31_(a)_(9709).exe
setup_the_witcher_3_wild_hunt_1.31_(a)_(9709)(1).exe
setup_the_witcher_3_wild_hunt_1.31_(a)_(9709)(1)-1.bin
setup_the_witcher_3_wild_hunt_1.31_(a)_(9709)(1)-2.bin
setup_the_witcher_3_wild_hunt_1.31_(a)_(9709)(1)-3.bin
setup_the_witcher_3_wild_hunt_1.31_(a)_(9709)(1)-4.bin
setup_the_witcher_3_wild_hunt_1.31_(a)_(9709)(1)-5.bin
setup_the_witcher_3_wild_hunt_1.31_(a)_(9709)(1)-6.bin
setup_the_witcher_3_wild_hunt_1.31_(a)_(9709)(1)-7.bin
setup_the_witcher_3_wild_hunt_1.31_(a)_(9709)-1.bin
setup_the_witcher_3_wild_hunt_1.31_(a)_(9709)-2.bin
setup_the_witcher_3_wild_hunt_1.31_(a)_(9709)-3.bin
setup_the_witcher_3_wild_hunt_1.31_(a)_(9709)-4.bin
setup_the_witcher_3_wild_hunt_1.31_(a)_(9709)-5.bin
setup_the_witcher_3_wild_hunt_1.31_(a)_(9709)-6.bin
setup_the_witcher_3_wild_hunt_1.31_(a)_(9709)-7.bin
setup_the_witcher_3_wild_hunt_4.03_(64430).exe
setup_the_witcher_3_wild_hunt_4.03_(64430)-1.bin
setup_the_witcher_3_wild_hunt_4.03_(64430)-2.bin
setup_the_witcher_3_wild_hunt_4.03_(64430)-3.bin
setup_the_witcher_3_wild_hunt_4.03_(64430)-4.bin
setup_the_witcher_3_wild_hunt_4.03_(64430)-5.bin
setup_the_witcher_3_wild_hunt_4.03_(64430)-6.bin
setup_the_witcher_3_wild_hunt_4.03_(64430)-7.bin
setup_the_witcher_3_wild_hunt_4.03_(64430)-8.bin
setup_the_witcher_3_wild_hunt_4.03_(64430)-9.bin
setup_the_witcher_3_wild_hunt_4.03_(64430)-10.bin
setup_the_witcher_3_wild_hunt_4.03_(64430)-11.bin
setup_the_witcher_3_wild_hunt_4.03_(64430)-12.bin
setup_the_witcher_3_wild_hunt_4.03_(64430)-13.bin
setup_the_witcher_3_wild_hunt_4.03_(64430)-14.bin
setup_the_witcher_3_wild_hunt_4.03_(64430)-15.bin

The 1.31 files seems to be Classic version of the game, and are downloaded twice.
In the update step 1), the console output says the files have different size and/or md5, but I have compared them and their counterparts with (1) in their name after the download step 2) and they have the same content (so both size and md5 match). The files are working fine, but it's a disk space waste (and data transfer waste if I delete them manually and then update / download again).
I'm pretty sure this is a GOG's issue and not gogrepoc's one, but maybe you can build some workaround?
BTW !images folder is empty for every single game I have, is there any reason for its existence?

Cheers!

phaolo: - what are the backgrounds, those seen on the top of a game page?

phaolo: Questions:

- how big are the cover images? (resolution and avg size)

- what are the backgrounds, those seen on the top of a game page?

- Kalanyr, is it possible to avoid creating the folders, if the previous commands are skipped?

- do I still need to apply Mrkgnao's workaround (if I'll find it again) due to Gog's CDN errors? If so, Kalanyr could you please add it directly?

Kalanyr: Hmmmm. Not sure what's happening there, it doesn't happen for me so I suspect it's a consequence of downloading both Polish and English, and the size / md5 check are against the GOG provided data not the actual files but that should still block the downloads if size or md5 mismatched the real file size. I'll have to look into it further.

Could you run a verify and see if any of those files fail it please ?

KoЯni: [snip - large post about Witcher 3 download duplicates]

KoЯni: The "clean" command doesn't move any of these files to "!orphaned" subdir (sometimes one or few files are moved there, probably because they're missing from manifest because of "update -full" timeouts on them) and the verify command happily verifies them - i.e. no errors.

But I think I know what the root cause is - e.g. here's main executable snip from the manifest file DOWNLOAD section:

//English version part
{'desc': 'The Witcher 3: Wild Hunt (część 1 z 16)',
'href': 'https://www.gog.com/downloads/the_witcher_3_wild_hunt_game/en1installer0',
'lang': 'English',
'md5': '797a6a2e58b2b252d8b7082dc92e86d8',
'name': 'setup_the_witcher_3_wild_hunt_4.03_(64430).exe',
'old_name': None,
'os_type': 'windows',
'prev_verified': True,
'size': 1859808,
'version': '4.03'},

//Polish version part
{'desc': 'The Witcher 3: Wild Hunt (część 1 z 8)',
'href': 'https://www.gog.com/downloads/the_witcher_3_wild_hunt_game/pl1installer16',
'lang': 'polski',
'md5': '4c99511bc7b66633eb556fd5ff0113be',
'name': 'setup_the_witcher_3_wild_hunt_1.31_(a)_(9709).exe',
'old_name': None,
'os_type': 'windows',
'prev_verified': True,
'size': 948152,
'version': '1.31'},

and here's one from the EXTRAS section:

{'desc': 'Witcher 3 Classic Part1',
'href': 'https://www.gog.com/downloads/the_witcher_3_wild_hunt_game/92908',
'lang': '',
'md5': None,
'name': 'setup_the_witcher_3_wild_hunt_1.31_(a)_(9709)(1).exe',
'old_name': None,
'os_type': 'extra',
'prev_verified': True,
'size': 948152,
'version': None},

So apparently the script sees these two files with the same name, but mismatching md5 (the one from extras has md5 of "None"), so it renames the one from the extras, adding "(1)" to the name and downloading both.

EDIT: Please also take a look at the differences from GOG library in the attachments. For English version, you get 4.03 offline installers, for Polish (or any version other than English, for that matter) you get 1.31 plus a language pack - duplicating the files from EXTRAS section. This is reflected in the manifest snippets above.

Let me know if you need anything else to eliminate this issue. Thanks for your effort!

KoЯni: The easiest thing that comes to my mind is comparing size - and if it matches, then calculate MD5 in the script and compare. There's a very limited risk of same-size files, and for these few calculating MD5 should be quick.