It seems that you're using an outdated browser. Some things may not work as they should (or don't work at all).
We suggest you upgrade newer and better browser like: Chrome, Firefox, Internet Explorer or Opera

×
avatar
Geralt_of_Rivia: And so do lists. That's the beauty of it.
In C# for example lists have the methods Add, Insert, Remove, Find, FindAll, Exists, Sort, Reverse, just to name some of the most used ones.
Was talking about SQL and queries. Of course lists have basic functionality. They wouldn't work without. That a list needs a method to add a element, should go without saying.

The example what you made later is what I meant. If you are thinking in rows and columns instead of objects and properties, SQL can be a lot more elegant to handle these things.


avatar
Geralt_of_Rivia: The table of files, obviously. Only the files table knows which files have not been downloaded and what type the file is (Installer, DLC, Patch, Extra) and what language (for games with separate installers for every language) while only the games table knows the name of the game. So without a join it is impossible to have a result table that is sorted by game names.
Not so obviously, since that would require to have a table with files to begin with.
I used relational databases for ages, by now I prefer object oriented models, where each game instance would contain all of it's informations, including categorized downloadable files.

If you are strictly thinking in relational data tables, then SQL is of course a good readable syntax to achieve what you described.
But if you think object oriented with the files being a property of the game and not stored in a separated table, you can do the same by first filtering and sorting the list of games, then either handling each game on it's own or mapping the games to the list of their files and use a Collector to collecte these into one long list.
It would of course also be possible with separated lists, but not as elegant.

The right way to do it really depends on if you see the data as properties of an object or as a collection of columns.
Post edited June 11, 2023 by neumi5694
avatar
timppu: Either way, splitting the manifest file before download is good enough solution for me, especially since I made a bash script that does that splitting automatically and copies the partial manifest files to different paths, I just run that script after gogrepoc update and before gogrepoc download (if needed). I don't need to touch the gogrepoc.py script myself.
Very nice. You wouldn't fee like sharing the splitting script would you? Might come in handy...

Another really dumb question...

What is the role of ".gitignore"? Is that a github thing (I rarely use github) or should that be with the script?

Thanks
Post edited June 17, 2023 by UrsaCorvalis
avatar
timppu: Either way, splitting the manifest file before download is good enough solution for me, especially since I made a bash script that does that splitting automatically and copies the partial manifest files to different paths, I just run that script after gogrepoc update and before gogrepoc download (if needed). I don't need to touch the gogrepoc.py script myself.
avatar
UrsaCorvalis: Very nice. You wouldn't fee like sharing the splitting script would you? Might come in handy...

Another really dumb question...

What is the role of ".gitignore"? Is that a github thing (I rarely use github) or should that be with the script?

Thanks
Git is a version control system. Using Git you can turn a directory into a repository which tracks all changes to files within it. You can create commits which are basically fixed versions which allow you to return to any older commit should you ever mess up the program.

You can also synchronize your files with an online repository (like for example GitHub) which is not only useful to distribute your software but also vastly simplifies shared development if you work together with several people because your changes get automatically synchronized with the changes others made to the software.

The .gitignore file is a simple text file which tells Git which files or subdirectories to ignore. Files and directories listed in .gitignore will neither have their changes tracked nor will they be included in online repositories.

If you don't use Git then you do not really need this file. But it is still a good idea to leave it there should you ever change your mind and use Git in the future.
avatar
Geralt_of_Rivia: The .gitignore file is a simple text file which tells Git which files or subdirectories to ignore. Files and directories listed in .gitignore will neither have their changes tracked nor will they be included in online repositories.

If you don't use Git then you do not really need this file. But it is still a good idea to leave it there should you ever change your mind and use Git in the future.
Awesome. Thank you sir
Hi Kalanyr,

There seems to be an issue in handling The Witcher 3 game (both IDs actually: the_witcher_3_wild_hunt_game and the_witcher_3_wild_hunt_game_of_the_year_edition_game) resulting in duplicating some files - be advised that it affects only these two games out of my 1000+ games collection and I think it might be related to dual language download process I use gogrepoc for.

For the clarity, I will focus on the first game.
To reproduce, I deleted whole game folder and then:

1) Updated the game info info with:
gogrepoc.py update -ids the_witcher_3_wild_hunt_game -lang en pl -os windows -full

It results in these messages:
17:07:14 | -> setup_the_witcher_3_wild_hunt_1.31_(a)_(9709).exe already exists in this game entry with a different size and/or md5, this file renamed to setup_the_w
itcher_3_wild_hunt_1.31_(a)_(9709)(1).exe
17:07:14 | -> setup_the_witcher_3_wild_hunt_1.31_(a)_(9709)-1.bin already exists in this game entry with a different size and/or md5, this file renamed to setup_the
_witcher_3_wild_hunt_1.31_(a)_(9709)(1)-1.bin
17:07:14 | -> setup_the_witcher_3_wild_hunt_1.31_(a)_(9709)-2.bin already exists in this game entry with a different size and/or md5, this file renamed to setup_the
_witcher_3_wild_hunt_1.31_(a)_(9709)(1)-2.bin
17:07:14 | -> setup_the_witcher_3_wild_hunt_1.31_(a)_(9709)-3.bin already exists in this game entry with a different size and/or md5, this file renamed to setup_the
_witcher_3_wild_hunt_1.31_(a)_(9709)(1)-3.bin
17:07:14 | -> setup_the_witcher_3_wild_hunt_1.31_(a)_(9709)-4.bin already exists in this game entry with a different size and/or md5, this file renamed to setup_the
_witcher_3_wild_hunt_1.31_(a)_(9709)(1)-4.bin
17:07:14 | -> setup_the_witcher_3_wild_hunt_1.31_(a)_(9709)-5.bin already exists in this game entry with a different size and/or md5, this file renamed to setup_the
_witcher_3_wild_hunt_1.31_(a)_(9709)(1)-5.bin
17:07:14 | -> setup_the_witcher_3_wild_hunt_1.31_(a)_(9709)-6.bin already exists in this game entry with a different size and/or md5, this file renamed to setup_the
_witcher_3_wild_hunt_1.31_(a)_(9709)(1)-6.bin
17:07:14 | -> setup_the_witcher_3_wild_hunt_1.31_(a)_(9709)-7.bin already exists in this game entry with a different size and/or md5, this file renamed to setup_the
_witcher_3_wild_hunt_1.31_(a)_(9709)(1)-7.bin
17:07:14 | -> witcher3_lang_pl_2.0.0.52.exe already exists in this game entry with same size/md5, skipping adding this file to the manifest
17:07:14 | -> witcher3_lang_pl_2.0.0.52-1.bin already exists in this game entry with a different size and/or md5, this file renamed to witcher3_lang_pl_2.0.0.52(1)-
1.bin
17:07:14 | saving manifest...
17:07:16 | saved manifest

2) Downloaded game files with:
gogrepoc.py download -ids the_witcher_3_wild_hunt_game

Among other files, I get these:
setup_the_witcher_3_wild_hunt_1.31_(a)_(9709).exe
setup_the_witcher_3_wild_hunt_1.31_(a)_(9709)(1).exe
setup_the_witcher_3_wild_hunt_1.31_(a)_(9709)(1)-1.bin
setup_the_witcher_3_wild_hunt_1.31_(a)_(9709)(1)-2.bin
setup_the_witcher_3_wild_hunt_1.31_(a)_(9709)(1)-3.bin
setup_the_witcher_3_wild_hunt_1.31_(a)_(9709)(1)-4.bin
setup_the_witcher_3_wild_hunt_1.31_(a)_(9709)(1)-5.bin
setup_the_witcher_3_wild_hunt_1.31_(a)_(9709)(1)-6.bin
setup_the_witcher_3_wild_hunt_1.31_(a)_(9709)(1)-7.bin
setup_the_witcher_3_wild_hunt_1.31_(a)_(9709)-1.bin
setup_the_witcher_3_wild_hunt_1.31_(a)_(9709)-2.bin
setup_the_witcher_3_wild_hunt_1.31_(a)_(9709)-3.bin
setup_the_witcher_3_wild_hunt_1.31_(a)_(9709)-4.bin
setup_the_witcher_3_wild_hunt_1.31_(a)_(9709)-5.bin
setup_the_witcher_3_wild_hunt_1.31_(a)_(9709)-6.bin
setup_the_witcher_3_wild_hunt_1.31_(a)_(9709)-7.bin
setup_the_witcher_3_wild_hunt_4.03_(64430).exe
setup_the_witcher_3_wild_hunt_4.03_(64430)-1.bin
setup_the_witcher_3_wild_hunt_4.03_(64430)-2.bin
setup_the_witcher_3_wild_hunt_4.03_(64430)-3.bin
setup_the_witcher_3_wild_hunt_4.03_(64430)-4.bin
setup_the_witcher_3_wild_hunt_4.03_(64430)-5.bin
setup_the_witcher_3_wild_hunt_4.03_(64430)-6.bin
setup_the_witcher_3_wild_hunt_4.03_(64430)-7.bin
setup_the_witcher_3_wild_hunt_4.03_(64430)-8.bin
setup_the_witcher_3_wild_hunt_4.03_(64430)-9.bin
setup_the_witcher_3_wild_hunt_4.03_(64430)-10.bin
setup_the_witcher_3_wild_hunt_4.03_(64430)-11.bin
setup_the_witcher_3_wild_hunt_4.03_(64430)-12.bin
setup_the_witcher_3_wild_hunt_4.03_(64430)-13.bin
setup_the_witcher_3_wild_hunt_4.03_(64430)-14.bin
setup_the_witcher_3_wild_hunt_4.03_(64430)-15.bin

The 1.31 files seems to be Classic version of the game, and are downloaded twice.
In the update step 1), the console output says the files have different size and/or md5, but I have compared them and their counterparts with (1) in their name after the download step 2) and they have the same content (so both size and md5 match). The files are working fine, but it's a disk space waste (and data transfer waste if I delete them manually and then update / download again).
I'm pretty sure this is a GOG's issue and not gogrepoc's one, but maybe you can build some workaround?
BTW !images folder is empty for every single game I have, is there any reason for its existence?

Cheers!
Post edited July 04, 2023 by KoЯni
avatar
KoЯni: BTW !images folder is empty for every single game I have, is there any reason for its existence?

Cheers!
The image directory is always created, even if you don't use -covers and -backgrounds.
I guess that's just an oversight.
I downloaded the images for all games by excluding all files and using thse two options.
avatar
neumi5694: The image directory is always created, even if you don't use -covers and -backgrounds.
I guess that's just an oversight.
I downloaded the images for all games by excluding all files and using thse two options.
Thanks, didn't even know they exist - they're missing from readme.md, might be worth updating...
Post edited July 04, 2023 by KoЯni
avatar
KoЯni: Hi Kalanyr,

There seems to be an issue in handling The Witcher 3 game (both IDs actually: the_witcher_3_wild_hunt_game and the_witcher_3_wild_hunt_game_of_the_year_edition_game) resulting in duplicating some files - be advised that it affects only these two games out of my 1000+ games collection and I think it might be related to dual language download process I use gogrepoc for.

For the clarity, I will focus on the first game.
To reproduce, I deleted whole game folder and then:

1) Updated the game info info with:
gogrepoc.py update -ids the_witcher_3_wild_hunt_game -lang en pl -os windows -full

It results in these messages:
17:07:14 | -> setup_the_witcher_3_wild_hunt_1.31_(a)_(9709).exe already exists in this game entry with a different size and/or md5, this file renamed to setup_the_w
itcher_3_wild_hunt_1.31_(a)_(9709)(1).exe
17:07:14 | -> setup_the_witcher_3_wild_hunt_1.31_(a)_(9709)-1.bin already exists in this game entry with a different size and/or md5, this file renamed to setup_the
_witcher_3_wild_hunt_1.31_(a)_(9709)(1)-1.bin
17:07:14 | -> setup_the_witcher_3_wild_hunt_1.31_(a)_(9709)-2.bin already exists in this game entry with a different size and/or md5, this file renamed to setup_the
_witcher_3_wild_hunt_1.31_(a)_(9709)(1)-2.bin
17:07:14 | -> setup_the_witcher_3_wild_hunt_1.31_(a)_(9709)-3.bin already exists in this game entry with a different size and/or md5, this file renamed to setup_the
_witcher_3_wild_hunt_1.31_(a)_(9709)(1)-3.bin
17:07:14 | -> setup_the_witcher_3_wild_hunt_1.31_(a)_(9709)-4.bin already exists in this game entry with a different size and/or md5, this file renamed to setup_the
_witcher_3_wild_hunt_1.31_(a)_(9709)(1)-4.bin
17:07:14 | -> setup_the_witcher_3_wild_hunt_1.31_(a)_(9709)-5.bin already exists in this game entry with a different size and/or md5, this file renamed to setup_the
_witcher_3_wild_hunt_1.31_(a)_(9709)(1)-5.bin
17:07:14 | -> setup_the_witcher_3_wild_hunt_1.31_(a)_(9709)-6.bin already exists in this game entry with a different size and/or md5, this file renamed to setup_the
_witcher_3_wild_hunt_1.31_(a)_(9709)(1)-6.bin
17:07:14 | -> setup_the_witcher_3_wild_hunt_1.31_(a)_(9709)-7.bin already exists in this game entry with a different size and/or md5, this file renamed to setup_the
_witcher_3_wild_hunt_1.31_(a)_(9709)(1)-7.bin
17:07:14 | -> witcher3_lang_pl_2.0.0.52.exe already exists in this game entry with same size/md5, skipping adding this file to the manifest
17:07:14 | -> witcher3_lang_pl_2.0.0.52-1.bin already exists in this game entry with a different size and/or md5, this file renamed to witcher3_lang_pl_2.0.0.52(1)-
1.bin
17:07:14 | saving manifest...
17:07:16 | saved manifest

2) Downloaded game files with:
gogrepoc.py download -ids the_witcher_3_wild_hunt_game

Among other files, I get these:
setup_the_witcher_3_wild_hunt_1.31_(a)_(9709).exe
setup_the_witcher_3_wild_hunt_1.31_(a)_(9709)(1).exe
setup_the_witcher_3_wild_hunt_1.31_(a)_(9709)(1)-1.bin
setup_the_witcher_3_wild_hunt_1.31_(a)_(9709)(1)-2.bin
setup_the_witcher_3_wild_hunt_1.31_(a)_(9709)(1)-3.bin
setup_the_witcher_3_wild_hunt_1.31_(a)_(9709)(1)-4.bin
setup_the_witcher_3_wild_hunt_1.31_(a)_(9709)(1)-5.bin
setup_the_witcher_3_wild_hunt_1.31_(a)_(9709)(1)-6.bin
setup_the_witcher_3_wild_hunt_1.31_(a)_(9709)(1)-7.bin
setup_the_witcher_3_wild_hunt_1.31_(a)_(9709)-1.bin
setup_the_witcher_3_wild_hunt_1.31_(a)_(9709)-2.bin
setup_the_witcher_3_wild_hunt_1.31_(a)_(9709)-3.bin
setup_the_witcher_3_wild_hunt_1.31_(a)_(9709)-4.bin
setup_the_witcher_3_wild_hunt_1.31_(a)_(9709)-5.bin
setup_the_witcher_3_wild_hunt_1.31_(a)_(9709)-6.bin
setup_the_witcher_3_wild_hunt_1.31_(a)_(9709)-7.bin
setup_the_witcher_3_wild_hunt_4.03_(64430).exe
setup_the_witcher_3_wild_hunt_4.03_(64430)-1.bin
setup_the_witcher_3_wild_hunt_4.03_(64430)-2.bin
setup_the_witcher_3_wild_hunt_4.03_(64430)-3.bin
setup_the_witcher_3_wild_hunt_4.03_(64430)-4.bin
setup_the_witcher_3_wild_hunt_4.03_(64430)-5.bin
setup_the_witcher_3_wild_hunt_4.03_(64430)-6.bin
setup_the_witcher_3_wild_hunt_4.03_(64430)-7.bin
setup_the_witcher_3_wild_hunt_4.03_(64430)-8.bin
setup_the_witcher_3_wild_hunt_4.03_(64430)-9.bin
setup_the_witcher_3_wild_hunt_4.03_(64430)-10.bin
setup_the_witcher_3_wild_hunt_4.03_(64430)-11.bin
setup_the_witcher_3_wild_hunt_4.03_(64430)-12.bin
setup_the_witcher_3_wild_hunt_4.03_(64430)-13.bin
setup_the_witcher_3_wild_hunt_4.03_(64430)-14.bin
setup_the_witcher_3_wild_hunt_4.03_(64430)-15.bin

The 1.31 files seems to be Classic version of the game, and are downloaded twice.
In the update step 1), the console output says the files have different size and/or md5, but I have compared them and their counterparts with (1) in their name after the download step 2) and they have the same content (so both size and md5 match). The files are working fine, but it's a disk space waste (and data transfer waste if I delete them manually and then update / download again).
I'm pretty sure this is a GOG's issue and not gogrepoc's one, but maybe you can build some workaround?
BTW !images folder is empty for every single game I have, is there any reason for its existence?

Cheers!
Hmmmm. Not sure what's happening there, it doesn't happen for me so I suspect it's a consequence of downloading both Polish and English, and the size / md5 check are against the GOG provided data not the actual files but that should still block the downloads if size or md5 mismatched the real file size. I'll have to look into it further.

Could you run a verify and see if any of those files fail it please ?
Questions:

- how big are the cover images? (resolution and avg size)

- what are the backgrounds, those seen on the top of a game page?

- Kalanyr, is it possible to avoid creating the folders, if the previous commands are skipped?

- do I still need to apply Mrkgnao's workaround (if I'll find it again) due to Gog's CDN errors? If so, Kalanyr could you please add it directly?
Post edited July 05, 2023 by phaolo
avatar
phaolo: - what are the backgrounds, those seen on the top of a game page?
Yes, these are the images used for the homepage.
Try downloading them for one game for the sizes.
avatar
phaolo: Questions:

- how big are the cover images? (resolution and avg size)

- what are the backgrounds, those seen on the top of a game page?

- Kalanyr, is it possible to avoid creating the folders, if the previous commands are skipped?

- do I still need to apply Mrkgnao's workaround (if I'll find it again) due to Gog's CDN errors? If so, Kalanyr could you please add it directly?
Not at my computer right now so can't answer the first but they are pretty small.

Yes

I think so, I'll check into that when I get back.

No, it was never necessary (md5s are optional) I'll see what I can do about using it to retry failed md5s dynamically though, but there's a lot fewer now so it's harder to test.
[snip - large post about Witcher 3 download duplicates]

avatar
Kalanyr: Hmmmm. Not sure what's happening there, it doesn't happen for me so I suspect it's a consequence of downloading both Polish and English, and the size / md5 check are against the GOG provided data not the actual files but that should still block the downloads if size or md5 mismatched the real file size. I'll have to look into it further.

Could you run a verify and see if any of those files fail it please ?
The "clean" command doesn't move any of these files to "!orphaned" subdir (sometimes one or few files are moved there, probably because they're missing from manifest because of "update -full" timeouts on them) and the verify command happily verifies them - i.e. no errors.

But I think I know what the root cause is - e.g. here's main executable snip from the manifest file DOWNLOAD section:

//English version part
{'desc': 'The Witcher 3: Wild Hunt (część 1 z 16)',
'href': 'https://www.gog.com/downloads/the_witcher_3_wild_hunt_game/en1installer0',
'lang': 'English',
'md5': '797a6a2e58b2b252d8b7082dc92e86d8',
'name': 'setup_the_witcher_3_wild_hunt_4.03_(64430).exe',
'old_name': None,
'os_type': 'windows',
'prev_verified': True,
'size': 1859808,
'version': '4.03'},

//Polish version part
{'desc': 'The Witcher 3: Wild Hunt (część 1 z 8)',
'href': 'https://www.gog.com/downloads/the_witcher_3_wild_hunt_game/pl1installer16',
'lang': 'polski',
'md5': '4c99511bc7b66633eb556fd5ff0113be',
'name': 'setup_the_witcher_3_wild_hunt_1.31_(a)_(9709).exe',
'old_name': None,
'os_type': 'windows',
'prev_verified': True,
'size': 948152,
'version': '1.31'},

and here's one from the EXTRAS section:

{'desc': 'Witcher 3 Classic Part1',
'href': 'https://www.gog.com/downloads/the_witcher_3_wild_hunt_game/92908',
'lang': '',
'md5': None,
'name': 'setup_the_witcher_3_wild_hunt_1.31_(a)_(9709)(1).exe',
'old_name': None,
'os_type': 'extra',
'prev_verified': True,
'size': 948152,
'version': None},

So apparently the script sees these two files with the same name, but mismatching md5 (the one from extras has md5 of "None"), so it renames the one from the extras, adding "(1)" to the name and downloading both.

EDIT: Please also take a look at the differences from GOG library in the attachments. For English version, you get 4.03 offline installers, for Polish (or any version other than English, for that matter) you get 1.31 plus a language pack - duplicating the files from EXTRAS section. This is reflected in the manifest snippets above.

Let me know if you need anything else to eliminate this issue. Thanks for your effort!
Attachments:
w3en.jpg (72 Kb)
w3pl.jpg (69 Kb)
Post edited July 06, 2023 by KoЯni
avatar
KoЯni: [snip - large post about Witcher 3 download duplicates]

avatar
Kalanyr: Hmmmm. Not sure what's happening there, it doesn't happen for me so I suspect it's a consequence of downloading both Polish and English, and the size / md5 check are against the GOG provided data not the actual files but that should still block the downloads if size or md5 mismatched the real file size. I'll have to look into it further.

Could you run a verify and see if any of those files fail it please ?
avatar
KoЯni: The "clean" command doesn't move any of these files to "!orphaned" subdir (sometimes one or few files are moved there, probably because they're missing from manifest because of "update -full" timeouts on them) and the verify command happily verifies them - i.e. no errors.

But I think I know what the root cause is - e.g. here's main executable snip from the manifest file DOWNLOAD section:

//English version part
{'desc': 'The Witcher 3: Wild Hunt (część 1 z 16)',
'href': 'https://www.gog.com/downloads/the_witcher_3_wild_hunt_game/en1installer0',
'lang': 'English',
'md5': '797a6a2e58b2b252d8b7082dc92e86d8',
'name': 'setup_the_witcher_3_wild_hunt_4.03_(64430).exe',
'old_name': None,
'os_type': 'windows',
'prev_verified': True,
'size': 1859808,
'version': '4.03'},

//Polish version part
{'desc': 'The Witcher 3: Wild Hunt (część 1 z 8)',
'href': 'https://www.gog.com/downloads/the_witcher_3_wild_hunt_game/pl1installer16',
'lang': 'polski',
'md5': '4c99511bc7b66633eb556fd5ff0113be',
'name': 'setup_the_witcher_3_wild_hunt_1.31_(a)_(9709).exe',
'old_name': None,
'os_type': 'windows',
'prev_verified': True,
'size': 948152,
'version': '1.31'},

and here's one from the EXTRAS section:

{'desc': 'Witcher 3 Classic Part1',
'href': 'https://www.gog.com/downloads/the_witcher_3_wild_hunt_game/92908',
'lang': '',
'md5': None,
'name': 'setup_the_witcher_3_wild_hunt_1.31_(a)_(9709)(1).exe',
'old_name': None,
'os_type': 'extra',
'prev_verified': True,
'size': 948152,
'version': None},

So apparently the script sees these two files with the same name, but mismatching md5 (the one from extras has md5 of "None"), so it renames the one from the extras, adding "(1)" to the name and downloading both.

EDIT: Please also take a look at the differences from GOG library in the attachments. For English version, you get 4.03 offline installers, for Polish (or any version other than English, for that matter) you get 1.31 plus a language pack - duplicating the files from EXTRAS section. This is reflected in the manifest snippets above.

Let me know if you need anything else to eliminate this issue. Thanks for your effort!
That's super annoying, I think I'd have to write a special case for that and I need one that doesn't fail if those files become different. I'll have to think about that one for a bit. Maybe I'll write a special check for extras that only uses size since extras never have MD5s.
The easiest thing that comes to my mind is comparing size - and if it matches, then calculate MD5 in the script and compare. There's a very limited risk of same-size files, and for these few calculating MD5 should be quick.
avatar
KoЯni: The easiest thing that comes to my mind is comparing size - and if it matches, then calculate MD5 in the script and compare. There's a very limited risk of same-size files, and for these few calculating MD5 should be quick.
Problem is that MD5s don't exist for extras , which is causing the issue here, it's passing the size check but failing the MD5 check because none != 4c99511bc7b66633eb556fd5ff0113b , so I'll have to change the comparison to assume it passes if the size matches but the item is an extra / it can't find an MD5