It seems that you're using an outdated browser. Some things may not work as they should (or don't work at all).
We suggest you upgrade newer and better browser like: Chrome, Firefox, Internet Explorer or Opera

×
Community
General discussiongogrepo.py - Python script for regularly backing up your purchased GOG collection for full offline enjoyment(3853 posts)(3853 posts)
(3853 posts)
Pages:
1
26
27
28
29
30
50
100
150
200
250
257
This is my favourite topic
Posted July 18, 2016
avatar
disi: In your example it would be like this:
So I would have to create a symlink manually for each subdirectory? That's not doable because it is the tool (gogrepo.py) that creates, deletes and manages the individual subdirectories under C:\GOG\, and there are something like 1286 of those subdirectories (one per GOG game).

Maybe then spanned volumes or some form of RAID (e.g. RAID 0 or RAID 5) are the only options, but as said I have no idea if they can be used with (several) external USB drives, or is the assumption that they are "fixed" drives which are always connected when the system boots up. I guess I can try it though, starting with spanned volumes...
Post edited July 18, 2016 by timppu
Posted July 18, 2016
avatar
disi: In your example it would be like this:
avatar
timppu: So I would have to create a symlink manually for each subdirectory? That's not doable because it is the tool (gogrepo.py) that creates, deletes and manages the individual subdirectories under C:\GOG\, and there are something like 1286 of those subdirectories (one per GOG game).

Maybe then spanned volumes or some form of RAID (e.g. RAID 0 or RAID 5) are the only options, but as said I have no idea if they can be used with (several) external USB drives, or is the assumption that they are "fixed" drives which are always connected when the system boots up. I guess I can try it though, starting with spanned volumes...
Hmm, yes it would be problematic keeping those updated manually.

With JBOD, the problem is that if one fails or there is a problem with consitency, you are doomed. There is also the problem with drives shutting down to save power etc. This is why raid-drives are much more expensive, because they use better materials.

Better get one trusty large drive. Let is run for a month, if there are no problems then, it is likely to last a long time.
Post edited July 18, 2016 by disi
Posted July 18, 2016
It shouldn't be that hard to add the functionality to limit the number of games put into a game folder i.e. by a range of starting letters. I've thought about changing the script two weeks ago – and forgot about it afterwards. ;-)
Posted July 19, 2016
I will be adding support for configuration file soon... there will definitely be some kind of option to be able to "split" your game folders.. Perhaps a wildcard or regex matching such that user can split by letter or whatever they want.
Posted August 24, 2016
[terminal command] python gogrepo.py update -os windows linux mac -lang en

At some point I get the following:
[snip]
fetching game details for amerzone_the_explorer_legacy...
request failed: HTTP Error 404: Not Found. will not retry.
no md5 data found for amerzone_the_explorer_s_legacy_2.0.0.9.dmg
[/snip]

Update: I contacted support, so we will see if they add md5 data for that file.
Post edited August 24, 2016 by solar_dome
Posted August 24, 2016
avatar
solar_dome: [terminal command] python gogrepo.py update -os windows linux mac -lang en

At some point I get the following:
[snip]
fetching game details for amerzone_the_explorer_legacy...
request failed: HTTP Error 404: Not Found. will not retry.
no md5 data found for amerzone_the_explorer_s_legacy_2.0.0.9.dmg
[/snip]

Update: I contacted support, so we will see if they add md5 data for that file.
You can ignore these. MD5 is nice to have, but GOG doesn't really support it. Verify will at least check the filesize.
Posted August 24, 2016
avatar
woolymethodman: MD5 is nice to have, but GOG doesn't really support it.
What does it mean GOG doesn't really support it (while still providing them for almost all of their installer files)?

I think it is very good to have because otherwise checking the integrity of your whole GOG game collection is almost impossible. in case the filesize is correct but a file has otherwise become corrupted (bitrot?). I think I had a case like this with one Dead Space bin file a couple of weeks ago, gogrepo verify noticed it. If it hadn't, I guess I would have just unknowingly kept corrupted Dead Space installers in my collection.

In case GOG didn't provide md5 sums at all, would there be some alternative method for a script like gogrepo to check the installer files? For instance, I recall GOG game installers are made with innosetup and could be extracted with innoextractor... so would that work, e.g. uncompressing the files temporarily? Maybe that would be quite slow, dunno, but better than nothing?
Post edited August 25, 2016 by timppu
Posted August 25, 2016
If md5 is not supported officially, and not supplied for extra files (zip), then an alternative way to verify our downloaded files would be a community supported list of md5 hashes/checksums/values.
The list could contain hashes from other algorithms as well. e.g. sha1, sha256, tiger, etc.

I would like to avoid bitrot on my files. I think file size and time-stamp verification is a good first check, but hash values an even more important second check.
Posted August 25, 2016
avatar
woolymethodman: MD5 is nice to have, but GOG doesn't really support it.
avatar
timppu: What does it mean GOG doesn't really support it (while still providing them for almost all of their installer files)?

I think it is very good to have because otherwise checking the integrity of your whole GOG game collection is almost impossible. in case the filesize is correct but a file has otherwise become corrupted (bitrot?). I think I had a case like this with one Dead Space bin file a couple of weeks ago, gogrepo verify noticed it. If it hadn't, I guess I would have just unknowingly kept corrupted Dead Space installers in my collection.

In case GOG didn't provide md5 sums at all, would there be some alternative method for a script like gogrepo to check the installer files? For instance, I recall GOG game installers are made with innosetup and could be extracted with innoextractor... so would that work, e.g. uncompressing the files temporarily? Maybe that would be quite slow, dunno, but better than nothing?
innoextract --test <installer.exe>
should do what you want
similar "unzip -t" for zip files (or use pythons builtin zip support)
the linux installers have a md5 check builtin as well afaik, not sure if there is a way to trigger that without starting the installation.
don't know about OSX installers
avatar
solar_dome: If md5 is not supported officially, and not supplied for extra files (zip), then an alternative way to verify our downloaded files would be a community supported list of md5 hashes/checksums/values.
The list could contain hashes from other algorithms as well. e.g. sha1, sha256, tiger, etc.

I would like to avoid bitrot on my files. I think file size and time-stamp verification is a good first check, but hash values an even more important second check.
are there any extra files that don't come as zip files?
The builtin CRC checksum of zip files should be enough to detect bitrot.

If you want additional security that the file hasn't been willfully tempered with during the download than CRC is of course insufficient. But an external, user-provided hash isn't exactly the most secure solution either ;)
Post edited August 25, 2016 by immi101
Posted August 25, 2016
avatar
solar_dome: If md5 is not supported officially, and not supplied for extra files (zip)
The extras (zip files) don't really need the md5 checksums, the zip file integrity is checked by the script I think. That's good enough confirmation of integrity for me. EDIT: Yeah what immi said above.

Anyway, 99% of GOG game installers, at least the English Windows versions, seem to have the md5 checksum, at least for the 1292 GOG games I have. And quite often the missing md5 checksum is for some update patch, not the main installer files. These are the md5 checksum fetching errors I encountered the last time I ran gogrepo (on 8th of August, fetching only English Windows versions):

21:24:21 | ( 8 / 1292) fetching game details for a_golden_wake...
21:24:28 | xml parsing error occurred trying to get md5 data for setup_a_golden_wake_2.0.0.4.exe

22:07:28 | ( 145 / 1292) fetching game details for broken_age...
22:07:44 | request failed: HTTP Error 404: Not Found. will not retry.
22:07:44 | no md5 data found for patch_broken_age_2.3.0.10.exe

22:15:53 | ( 172 / 1292) fetching game details for chaos_reborn...
22:16:10 | xml parsing error occurred trying to get md5 data for patch_chaos_reborn_2.8.0.9.exe

00:16:01 | ( 559 / 1292) fetching game details for kyn...
00:16:08 | request failed: HTTP Error 404: Not Found. will not retry.
00:16:08 | no md5 data found for setup_kyn_2.1.0.4.exe
00:16:13 | request failed: HTTP Error 404: Not Found. will not retry.
00:16:13 | no md5 data found for setup_kyn_2.1.0.4-1.bin

03:38:03 | (1220 / 1292) fetching game details for walking_dead_season_1_the...
03:38:19 | request failed: HTTP Error 404: Not Found. will not retry.
03:38:19 | no md5 data found for patch_walking_dead_season_1_2.1.0.5.exe

03:51:47 | (1263 / 1292) fetching game details for x3_terran_war_pack...
03:51:55 | xml parsing error occurred trying to get md5 data for setup_x3_terran_war_pack_2.2.0.5.exe
03:52:00 | xml parsing error occurred trying to get md5 data for setup_x3_terran_war_pack_2.2.0.5-1.bin
03:52:05 | xml parsing error occurred trying to get md5 data for setup_x3_terran_war_pack_2.2.0.5-2.bin
03:52:14 | xml parsing error occurred trying to get md5 data for patch_x3_terran_war_pack_2.2.0.5.exe
So I guess for those particular files gogrepo can currently only verify that the filesize is correct, and that's all.

If additional verification methods for installers would be added to the script, I presume checking the md5 checksum should still be the primary method, I presume it is the fastest way to verify? So maybe the priority could go something like this:

1. If a md5 checksum exists for the file, check it. If not, then:
2. if it is a zip file, test the zip file integrity (CRC for the extras). If not, then:
3. Test the installer files with innoextractor.
4. If for some reason none of the above works, then just check the filesize?
Post edited August 25, 2016 by timppu
Posted August 25, 2016
avatar
immi101: innoextract --test <installer.exe>
should do what you want
Oh, one thing popped into my mind though... is innoextract Windows-only or is it available for other platforms too? The power of gogrepo is that it can be ran on various different platform. I usually run it in Linux even though I specifically download Windows versions of games.

If it is e.g. Windows-only, dunno, maybe even I tried to make a separate script that specifically just verifies all the GOG installers with innoextract, even if it works only in Windows. Then I just need to run it separately, if needed.

Anyway, fortunately nowadays most GOG installers seem to have the md5 checksum, it is a rare occurence that it is missing. Hopefully it stays that way too. Maybe this is not a high-priority request at the moment, but would be good to have if it became common for GOG installers to be missing the md5sum.

EDIT: Ok apparently innoextract is multiplatform, luckily:

http://constexpr.org/innoextract/

It mentions in that page that it works for _most_ GOG installers. So if some installer is missing the md5sum, and for some reason doesn't work with innoextract either, then I guess it could be checked only against the filesize.
Post edited August 25, 2016 by timppu
Posted August 25, 2016
avatar
immi101: innoextract --test <installer.exe>
should do what you want
avatar
timppu: Oh, one thing popped into my mind though... is innoextract Windows-only or is it available for other platforms too?
it's multiplatform. It's available on most linux distros and BSD. Windows binaries are provided on the homepage. It works on OSX, but it seems binaries are not offered for download.

Personally I never encountered missing MD5 data for installer files. But then, I have far less games than you ;). And I usually don't use the patches.

One thing where the hashes would be useful for extras are cases like the Broken Sword, where the installers of the original game are provided as extras.
maybe we should start a wishlist entry :)
Posted August 25, 2016
avatar
immi101: And I usually don't use the patches.
Yeah I am not really fretting over the patch integrity either, as long as the main installers are up to date and integrity ok.
Posted August 25, 2016
Community MD5 database would be interesting.. sigh I wish I had more time to keep working on gogrepo. Maybe if GOG hired me and I can quit my full time job :P
Posted August 25, 2016
Thank you for the time you spent already on the script. Much easier downloading games via gogrepo than via a browser.
I see how the zip file integrity check is useful to avoid bitrot.

I started a community md5 database of sorts. https://www.gogwiki.com/wiki/Hashes
Not very useful yet.
Even if the page was populated with hashes for all the available files, would it be useful, or a waste of time?
Possibly useful for people that do not know about the gogrepo script, or unable to use it?
Post edited August 25, 2016 by solar_dome
Pages:
1
26
27
28
29
30
50
100
150
200
250
257
This is my favourite topic
Community
General discussiongogrepo.py - Python script for regularly backing up your purchased GOG collection for full offline enjoyment(3853 posts)(3853 posts)
(3853 posts)